dbt™️ Source checks

`check-source-columns-have-desc`

What it does

Ensures that the source has columns with descriptions in the properties file (usually schema.yml).

When to use it

You want to make sure that all specified columns in the properties files (usually schema.yml) have some description. This hook does not validate if all database columns are also present in a properties file.

Example

repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-columns-have-desc

Requirements

Model exists in manifest.json 1

Model exists in catalog.json 2

❌

❌ Not needed

1 It means that you need to run dbt parse before run this hook (dbt >= 1.5). 2 It means that you need to run dbt docs generate before run this hook.

How it works

Hook takes all changed yml.
All sources from yml file are parsed.
If any column in the source does not contain a description, the hook fails.

`check-source-has-all-columns`

What it does

Ensures that all columns in the database are also specified in the properties file. (usually schema.yml).

When to use it

You want to make sure that you have all the database columns listed in the properties file, or that your properties file no longer contains deleted columns.

Arguments

--manifest: location of manifest.json file. Usually target/manifest.json. This file contains a full representation of dbt project. Default: target/manifest.json --catalog: location of catalog.json file. Usually target/catalog.json. dbt uses this file to render information like column types and table statistics into the docs site. In dbt-checkpoint is used for column operations. Default: target/catalog.json

Example

repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-all-columns

Requirements

Model exists in manifest.json 1

Model exists in catalog.json 2

❌ No

✅ Yes

1 It means that you need to run dbt parse before run this hook (dbt >= 1.5). 2 It means that you need to run dbt docs generate before run this hook.

How it works

Hook takes all changed yml.
All sources from yml file are parsed.
The catalog is scanned for a model.
If there is any discrepancy between found yml sources and catalog sources, the hook fails.

Known limitations

If you did not update the catalog and manifest results can be wrong.

`check-source-table-has-description`

What it does

Ensures that the source table has a description in the properties file (usually schema.yml).

When to use it

You want to make sure that all sources have a description.

Example

repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-table-has-description

Requirements

Model exists in manifest.json 1

Model exists in catalog.json 2

❌ Not needed

1 It means that you need to run dbt parse before run this hook (dbt >= 1.5). 2 It means that you need to run dbt docs generate before run this hook.

How it works

Hook takes all changed yml.
All sources from yml file are parsed.
If the source table does not have a description, the hook fails.

`check-source-has-freshness`

What it does

Ensures that the source has freshness options in the properties file (usually schema.yml).

When to use it

You want to make sure that all freshness is correctly set.

Example

repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-freshness
        args: ["--freshness", "error_after", "warn_after", "--"]

⚠️ do not forget to include -- as the last argument. Otherwise pre-commit would not be able to separate a list of files with args.

Requirements

Model exists in manifest.json 1

Model exists in catalog.json 2

❌ Not needed

1 It means that you need to run dbt parse before run this hook (dbt >= 1.5). 2 It means that you need to run dbt docs generate before run this hook.

How it works

Hook takes all changed yml.
All sources from yml file are parsed.
If the source does not have freshness correctly set, the hook fails.

`check-source-has-loader`

What it does

Ensures that the source has a loader option in the properties file (usually schema.yml).

When to use it

You want to make sure that the source has loader specified.

Example

repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-loader

Requirements

Model exists in manifest.json 1

Model exists in catalog.json 2

❌ Not needed

1 It means that you need to run dbt parse before run this hook (dbt >= 1.5). 2 It means that you need to run dbt docs generate before run this hook.

How it works

Hook takes all changed yml.
All sources from yml file are parsed.
If the source does not have a loader set, the hook fails.

`check-source-has-meta-keys`

What it does

Ensures that the source has a list of valid meta keys. (usually schema.yml).

When to use it

If every source needs to have certain meta keys.

Arguments

--meta-keys: list of the required keys in the meta part of the model.

Example

repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-meta-keys
        args: ['--meta-keys', 'foo', 'bar', "--"]

⚠️ do not forget to include -- as the last argument. Otherwise pre-commit would not be able to separate a list of files with args.

Requirements

Model exists in manifest.json 1

Model exists in catalog.json 2

❌ Not needed since it also validates properties files

❌ Not needed

1 It means that you need to run dbt parse before run this hook (dbt >= 1.5). 2 It means that you need to run dbt docs generate before run this hook.

How it works

Hook takes all changed yml.
All sources from yml file are parsed.
If the source does not have the required meta keys set, the hook fails.

`check-source-has-labels-keys`

What it does

Ensures that the source has a list of valid labels keys. (usually schema.yml).

When to use it

If every source needs to have certain labels keys.

Arguments

--labels-keys: list of the required keys in the labels part of the model.

Example

repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-labels-keys
        args: ['--labels-keys', 'foo', 'bar', "--"]

⚠️ do not forget to include -- as the last argument. Otherwise pre-commit would not be able to separate a list of files with args.

Requirements

Model exists in manifest.json 1

Model exists in catalog.json 2

❌ Not needed since it also validates properties files

❌ Not needed

1 It means that you need to run dbt parse before run this hook (dbt >= 1.5). 2 It means that you need to run dbt docs generate before run this hook.

How it works

Hook takes all changed yml.
All sources from yml file are parsed.
If the source does not have the required labels keys set, the hook fails.

`check-source-has-tests-by-name`

What it does

Ensures that the source has a number of tests of a certain name (e.g. data, unique).

When to use it

You want to make sure that every source has certain tests.

Arguments

--tests: key-value pairs of test names. Key is the name of test and value is required minimal number of tests eg. --test unique=1 not_null=2 (do not put spaces before or after the = sign).

Example

repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-tests-by-name
        args: ["--tests", "unique=1", "data=1", "--"]

⚠️ do not forget to include -- as the last argument. Otherwise pre-commit would not be able to separate a list of files with args.

Requirements

Model exists in manifest.json 1

Model exists in catalog.json 2

❌ Yes

❌ Not needed

1 It means that you need to run dbt parse before run this hook (dbt >= 1.5). 2 It means that you need to run dbt docs generate before run this hook.

How it works

Hook takes all changed yml.
All sources from yml file are parsed.
If the source does not have the required test names, the hook fails.

`check-source-has-tests-by-type`

What it does

Ensures that the source has a number of tests of a certain type (data, schema).

When to use it

You want to make sure that every source has certain tests.

Arguments

--tests: key-value pairs of test types. Key is a type of test (data or schema) and value is required eg. --test data=1 schema=2 (do not put spaces before or after the = sign).

Example

repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-tests-by-type
        args: ["--tests", "schema=1", "data=1", "--"]

⚠️ do not forget to include -- as the last argument. Otherwise pre-commit would not be able to separate a list of files with args.

Requirements

Model exists in manifest.json 1

Model exists in catalog.json 2

❌ Yes

❌ Not needed

1 It means that you need to run dbt parse before run this hook (dbt >= 1.5). 2 It means that you need to run dbt docs generate before run this hook.

How it works

Hook takes all changed yml.
All sources from yml file are parsed.
If the source does not have the required test types, the hook fails.

`check-source-has-tests`

What it does

Ensures that the source has a number of tests.

When to use it

You want to make sure that every source was tested.

Arguments

--test-cnt: Minimum number of tests required.

Example

repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-tests
        args: ["--test-cnt", "2", "--"]

⚠️ do not forget to include -- as the last argument. Otherwise pre-commit would not be able to separate a list of files with args.

Requirements

Model exists in manifest.json 1

Model exists in catalog.json 2

❌ Yes

❌ Not needed

1 It means that you need to run dbt parse before run this hook (dbt >= 1.5). 2 It means that you need to run dbt docs generate before run this hook.

How it works

Hook takes all changed yml.
All sources from yml file are parsed.
If the source does not have the required test count, the hook fails.

`check-source-has-tests-by-group`

What it does

Ensures that the source has a number of tests from a group of tests.

When to use it

You want to make sure that every source has one (or more) of a group of eligible tests (e.g. a set of unique tests).

Arguments

--tests: list of test names. --test_cnt: number of tests required across test group.

Example

repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-has-tests-by-group
        args: ["--tests", "unique", "unique_where", "--test-cnt", "1", "--"]

⚠️ do not forget to include -- as the last argument. Otherwise pre-commit would not be able to separate a list of files with args.

Requirements

Model exists in manifest.json 1

Model exists in catalog.json 2

✅ Yes

❌ Not needed

1 It means that you need to run dbt parse before run this hook (dbt >= 1.5). 2 It means that you need to run dbt docs generate before run this hook.

How it works

Hook takes all changed SQL files.
The source name is obtained from the SQL file name.
If any source does not have the number of required tests, the hook fails.

`check-source-tags`

What it does

Ensures that the source has only valid tags from the provided list.

When to use it

Make sure you did not typo in tags.

Arguments

--tags: A list of tags that sources can have.

Example

repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-tags
        args: ["--tags", "foo", "bar", "--"]

⚠️ do not forget to include -- as the last argument. Otherwise pre-commit would not be able to separate a list of files with args.

Requirements

Model exists in manifest.json 1

Model exists in catalog.json 2

❌ Not needed

1 It means that you need to run dbt parse before run this hook (dbt >= 1.5). 2 It means that you need to run dbt docs generate before run this hook.

How it works

Hook takes all changed yml.
All sources from yml file are parsed.
If the source has different tags than specified, the hook fails.

`check-source-childs`

What it does

Ensures the source has a specific number (max/min) of childs.

When to use it

You want to find orphaned sources without any dependencies. Or you want to make sure that every source is used somewhere.

Arguments

Example

repos:
  - repo: https://github.com/dbt-checkpoint/dbt-checkpoint
    rev: v1.0.0
    hooks:
      - id: check-source-childs
        args: ["--min-child-cnt", "2", "--"]

⚠️ do not forget to include -- as the last argument. Otherwise pre-commit would not be able to separate a list of files with args.

Requirements

Model exists in manifest.json 1

Model exists in catalog.json 2

✅ Yes

❌ Not needed

1 It means that you need to run dbt parse before run this hook (dbt >= 1.5). 2 It means that you need to run dbt docs generate before run this hook.

How it works

Hook takes all changed yml.
All sources from yml file are parsed.
The manifest is scanned for child models.
If any source does not have a number of required childs, the hook fails.

Previousdbt™️ Script checks Nextdbt™️ Macro checks

Last updated 9 months ago

Was this helpful?