# dbt™️ Source checks

## `check-source-columns-have-desc`

{% hint style="info" %}
**What it does**

Ensures that the source has columns with descriptions in the properties file (usually `schema.yml`).

**When to use it**

You want to make sure that all specified columns in the properties files (usually `schema.yml`) have some description. **This hook does not validate if all database columns are also present in a properties file.**
{% endhint %}

**Example**

```yaml
repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-columns-have-desc
```

**Requirements**

| Model exists in `manifest.json` *1* | Model exists in `catalog.json` *2* |
| :---------------------------------: | :--------------------------------: |
|                  ❌                  |            ❌ Not needed            |

1 It means that you need to run `dbt parse` before run this hook (dbt >= 1.5).\
2 It means that you need to run `dbt docs generate` before run this hook.

**How it works**

* Hook takes all changed `yml`.
* All sources from yml file are parsed.
* If any column in the source does not contain a description, the hook fails.

***

## `check-source-has-all-columns`

{% hint style="info" %}
**What it does**

Ensures that all columns in the database are also specified in the properties file. (usually `schema.yml`).

**When to use it**

You want to make sure that you have all the database columns listed in the properties file, or that your properties file no longer contains deleted columns.
{% endhint %}

**Arguments**

`--manifest`: location of `manifest.json` file. Usually `target/manifest.json`. This file contains a full representation of dbt project. **Default: `target/manifest.json`**\
`--catalog`: location of `catalog.json` file. Usually `target/catalog.json`. dbt uses this file to render information like column types and table statistics into the docs site. In dbt-checkpoint is used for column operations. **Default: `target/catalog.json`**

**Example**

```yaml
repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-all-columns
```

**Requirements**

| Model exists in `manifest.json` *1* | Model exists in `catalog.json` *2* |
| :---------------------------------: | :--------------------------------: |
|                 ❌ No                |                ✅ Yes               |

1 It means that you need to run `dbt parse` before run this hook (dbt >= 1.5).\
2 It means that you need to run `dbt docs generate` before run this hook.

**How it works**

* Hook takes all changed `yml`.
* All sources from yml file are parsed.
* The catalog is scanned for a model.
* If there is any discrepancy between found yml sources and catalog sources, the hook fails.

**Known limitations**

If you did not update the catalog and manifest results can be wrong.

***

## `check-source-table-has-description`

{% hint style="info" %}
**What it does**

Ensures that the source table has a description in the properties file (usually `schema.yml`).

**When to use it**

You want to make sure that all sources have a description.
{% endhint %}

**Example**

```yaml
repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-table-has-description
```

**Requirements**

| Model exists in `manifest.json` *1* | Model exists in `catalog.json` *2* |
| :---------------------------------: | :--------------------------------: |
|             ❌ Not needed            |            ❌ Not needed            |

1 It means that you need to run `dbt parse` before run this hook (dbt >= 1.5).\
2 It means that you need to run `dbt docs generate` before run this hook.

**How it works**

* Hook takes all changed `yml`.
* All sources from yml file are parsed.
* If the source table does not have a description, the hook fails.

***

## `check-source-has-freshness`

{% hint style="info" %}
**What it does**

Ensures that the source has freshness options in the properties file (usually `schema.yml`).

**When to use it**

You want to make sure that all freshness is correctly set.
{% endhint %}

**Example**

```yaml
repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-freshness
        args: ["--freshness", "error_after", "warn_after", "--"]
```

⚠️ do not forget to include `--` as the last argument. Otherwise `pre-commit` would not be able to separate a list of files with args.

**Requirements**

| Model exists in `manifest.json` *1* | Model exists in `catalog.json` *2* |
| :---------------------------------: | :--------------------------------: |
|             ❌ Not needed            |            ❌ Not needed            |

1 It means that you need to run `dbt parse` before run this hook (dbt >= 1.5).\
2 It means that you need to run `dbt docs generate` before run this hook.

**How it works**

* Hook takes all changed `yml`.
* All sources from yml file are parsed.
* If the source does not have freshness correctly set, the hook fails.

***

## `check-source-has-loader`

{% hint style="info" %}
**What it does**

Ensures that the source has a loader option in the properties file (usually `schema.yml`).

**When to use it**

You want to make sure that the source has loader specified.
{% endhint %}

**Example**

```yaml
repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-loader
```

**Requirements**

| Model exists in `manifest.json` *1* | Model exists in `catalog.json` *2* |
| :---------------------------------: | :--------------------------------: |
|             ❌ Not needed            |            ❌ Not needed            |

1 It means that you need to run `dbt parse` before run this hook (dbt >= 1.5).\
2 It means that you need to run `dbt docs generate` before run this hook.

**How it works**

* Hook takes all changed `yml`.
* All sources from yml file are parsed.
* If the source does not have a loader set, the hook fails.

***

## `check-source-has-meta-keys`

{% hint style="info" %}
**What it does**

Ensures that the source has a list of valid meta keys. (usually `schema.yml`).

**When to use it**

If every source needs to have certain meta keys.
{% endhint %}

**Arguments**

`--meta-keys`: list of the required keys in the meta part of the model.

**Example**

```yaml
repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-meta-keys
        args: ['--meta-keys', 'foo', 'bar', "--"]
```

⚠️ do not forget to include `--` as the last argument. Otherwise `pre-commit` would not be able to separate a list of files with args.

**Requirements**

|          Model exists in `manifest.json` *1*          | Model exists in `catalog.json` *2* |
| :---------------------------------------------------: | :--------------------------------: |
| ❌ Not needed since it also validates properties files |            ❌ Not needed            |

1 It means that you need to run `dbt parse` before run this hook (dbt >= 1.5).\
2 It means that you need to run `dbt docs generate` before run this hook.

**How it works**

* Hook takes all changed `yml`.
* All sources from yml file are parsed.
* If the source does not have the required meta keys set, the hook fails.

***

## `check-source-has-labels-keys`

{% hint style="info" %}
**What it does**

Ensures that the source has a list of valid labels keys. (usually `schema.yml`).

**When to use it**

If every source needs to have certain labels keys.
{% endhint %}

**Arguments**

`--labels-keys`: list of the required keys in the labels part of the model.

**Example**

```yaml
repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-labels-keys
        args: ['--labels-keys', 'foo', 'bar', "--"]
```

⚠️ do not forget to include `--` as the last argument. Otherwise `pre-commit` would not be able to separate a list of files with args.

**Requirements**

|          Model exists in `manifest.json` *1*          | Model exists in `catalog.json` *2* |
| :---------------------------------------------------: | :--------------------------------: |
| ❌ Not needed since it also validates properties files |            ❌ Not needed            |

1 It means that you need to run `dbt parse` before run this hook (dbt >= 1.5).\
2 It means that you need to run `dbt docs generate` before run this hook.

**How it works**

* Hook takes all changed `yml`.
* All sources from yml file are parsed.
* If the source does not have the required labels keys set, the hook fails.

***

## `check-source-has-tests-by-name`

{% hint style="info" %}
**What it does**

Ensures that the source has a number of tests of a certain name (e.g. data, unique).

**When to use it**

You want to make sure that every source has certain tests.
{% endhint %}

**Arguments**

`--tests`: key-value pairs of test names. Key is the name of test and value is required minimal number of tests eg. --test unique=1 not\_null=2 (do not put spaces before or after the = sign).

**Example**

```yaml
repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-tests-by-name
        args: ["--tests", "unique=1", "data=1", "--"]
```

⚠️ do not forget to include `--` as the last argument. Otherwise `pre-commit` would not be able to separate a list of files with args.

**Requirements**

| Model exists in `manifest.json` *1* | Model exists in `catalog.json` *2* |
| :---------------------------------: | :--------------------------------: |
|                ❌ Yes                |            ❌ Not needed            |

1 It means that you need to run `dbt parse` before run this hook (dbt >= 1.5).\
2 It means that you need to run `dbt docs generate` before run this hook.

**How it works**

* Hook takes all changed `yml`.
* All sources from yml file are parsed.
* If the source does not have the required test names, the hook fails.

***

## `check-source-has-tests-by-type`

{% hint style="info" %}
**What it does**

Ensures that the source has a number of tests of a certain type (data, schema).

**When to use it**

You want to make sure that every source has certain tests.
{% endhint %}

**Arguments**

`--tests`: key-value pairs of test types. Key is a type of test (data or schema) and value is required eg. --test data=1 schema=2 (do not put spaces before or after the = sign).

**Example**

```yaml
repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-tests-by-type
        args: ["--tests", "schema=1", "data=1", "--"]
```

⚠️ do not forget to include `--` as the last argument. Otherwise `pre-commit` would not be able to separate a list of files with args.

**Requirements**

| Model exists in `manifest.json` *1* | Model exists in `catalog.json` *2* |
| :---------------------------------: | :--------------------------------: |
|                ❌ Yes                |            ❌ Not needed            |

1 It means that you need to run `dbt parse` before run this hook (dbt >= 1.5).\
2 It means that you need to run `dbt docs generate` before run this hook.

**How it works**

* Hook takes all changed `yml`.
* All sources from yml file are parsed.
* If the source does not have the required test types, the hook fails.

***

## `check-source-has-tests`

{% hint style="info" %}
**What it does**

Ensures that the source has a number of tests.

**When to use it**

You want to make sure that every source was tested.
{% endhint %}

**Arguments**

`--test-cnt`: Minimum number of tests required.

**Example**

```yaml
repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-tests
        args: ["--test-cnt", "2", "--"]
```

⚠️ do not forget to include `--` as the last argument. Otherwise `pre-commit` would not be able to separate a list of files with args.

**Requirements**

| Model exists in `manifest.json` *1* | Model exists in `catalog.json` *2* |
| :---------------------------------: | :--------------------------------: |
|                ❌ Yes                |            ❌ Not needed            |

1 It means that you need to run `dbt parse` before run this hook (dbt >= 1.5).\
2 It means that you need to run `dbt docs generate` before run this hook.

**How it works**

* Hook takes all changed `yml`.
* All sources from yml file are parsed.
* If the source does not have the required test count, the hook fails.

***

## `check-source-has-tests-by-group`

{% hint style="info" %}
**What it does**

Ensures that the source has a number of tests from a group of tests.

**When to use it**

You want to make sure that every source has one (or more) of a group of eligible tests (e.g. a set of unique tests).
{% endhint %}

**Arguments**

`--tests`: list of test names. `--test_cnt`: number of tests required across test group.

**Example**

```yaml
repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-tests-by-group
        args: ["--tests", "unique", "unique_where", "--test-cnt", "1", "--"]
```

⚠️ do not forget to include `--` as the last argument. Otherwise `pre-commit` would not be able to separate a list of files with args.

**Requirements**

| Model exists in `manifest.json` *1* | Model exists in `catalog.json` *2* |
| :---------------------------------: | :--------------------------------: |
|                ✅ Yes                |            ❌ Not needed            |

1 It means that you need to run `dbt parse` before run this hook (dbt >= 1.5).\
2 It means that you need to run `dbt docs generate` before run this hook.

**How it works**

* Hook takes all changed `SQL` files.
* The source name is obtained from the `SQL` file name.
* If any source does not have the number of required tests, the hook fails.

***

## `check-source-tags`

{% hint style="info" %}
**What it does**

Ensures that the source has only valid tags from the provided list.

**When to use it**

Make sure you did not typo in tags.
{% endhint %}

**Arguments**

`--tags`: A list of tags that sources can have.

**Example**

```yaml
repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-tags
        args: ["--tags", "foo", "bar", "--"]
```

⚠️ do not forget to include `--` as the last argument. Otherwise `pre-commit` would not be able to separate a list of files with args.

**Requirements**

| Model exists in `manifest.json` *1* | Model exists in `catalog.json` *2* |
| :---------------------------------: | :--------------------------------: |
|             ❌ Not needed            |            ❌ Not needed            |

1 It means that you need to run `dbt parse` before run this hook (dbt >= 1.5).\
2 It means that you need to run `dbt docs generate` before run this hook.

**How it works**

* Hook takes all changed `yml`.
* All sources from yml file are parsed.
* If the source has different tags than specified, the hook fails.

***

## `check-source-childs`

{% hint style="info" %}
**What it does**

Ensures the source has a specific number (max/min) of childs.

**When to use it**

You want to find orphaned sources without any dependencies. Or you want to make sure that every source is used somewhere.
{% endhint %}

**Arguments**

`--manifest`: location of `manifest.json` file. Usually `target/manifest.json`. This file contains a full representation of dbt project. **Default: `target/manifest.json`**\
`--min-child-cnt`: Minimal number of child models. `--max-child-cnt`: Maximal number of child models.

**Example**

```yaml
repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-childs
        args: ["--min-child-cnt", "2", "--"]
```

⚠️ do not forget to include `--` as the last argument. Otherwise `pre-commit` would not be able to separate a list of files with args.

**Requirements**

| Model exists in `manifest.json` *1* | Model exists in `catalog.json` *2* |
| :---------------------------------: | :--------------------------------: |
|                ✅ Yes                |            ❌ Not needed            |

*1* It means that you need to run `dbt parse` before run this hook (dbt >= 1.5).\
2 It means that you need to run `dbt docs generate` before run this hook.

**How it works**

* Hook takes all changed `yml`.
* All sources from yml file are parsed.
* The manifest is scanned for child models.
* If any source does not have a number of required childs, the hook fails.<br>