dbt™️ generator
Last updated
Was this helpful?
Last updated
Was this helpful?
To use this feature, make sure to first install the dbt™️-codegen package in your dbt™️ project.
The dbt™️ generator integration allows you to generate based models from your sources and transform model at scale. This will help you save time generating base models from the Paradime Integrated Terminal.
To generate base models, use the dbt-generator generate
command. This is a wrapper around the codegen
command that will generate the base models. This is especially useful when you have a lot of models, and you want to generate them all at once.
Usage: dbt-generator generate [OPTIONS]
Generate base models based on a .yml source
Options:
-s, --source-yml PATH Source .yml file to be used
-o, --output-path PATH Path to write generated models
-m, --model STRING Model name
-c, --custom_prefix. Enter a Custom String Prefix for Model Filename
--model-prefix BOOLEAN optional prefix of source_name + "_" to the resulting modelname.sql to avoid model name collisions across sources
--source-index INTEGER Index of the source to generate base models for
--help Show this message and exit.
$ dbt-generator generate -s ./models/source.yml -o ./models/staging/source_name/
This will read in the source.yml
file and generate the base models in the staging/source_name
folder. If you have multiple sources defined in your yml
file, use the --source-index
flag to specify which source you want to generate base models for.
Usage: dbt-generator transform [OPTIONS]
Transform base models in a directory using a transforms.yml file
Options:
-m, --model-path PATH The path to models
-t, --transforms-path PATH Path to a .yml file containing transformations
-o, --output-path PATH Path to write transformed models to
--drop-metadata BOOLEAN (default=False) optionally drop source columns prefixed with "_" if that designates metadata columns not needed in target
--case-sensitive BOOLEAN (default=False) treat column names as case-sensitive - otherwise force all to lower
--help Show this message and exit.
With this package, you can write a transforms.yml
file that will be read in (the .yml
file can be named anything). This file will contain the transforms that you want to apply to all the base models. You can just rename the fields in the base models or apply a custom SQL select to the transformed fields.
For the same source, you often have consistent naming conventions between tables. For example, the created_at
and modified_at
fields are often named the same for all tables. Changing all these fields to common values across different sources is a best practice. However, doing that for all the date columns in 10+ tables is a pain.
Supported data warehouse:
BigQuery: bq_transform
Snowflake: sf_transform
Usage: dbt-generator bq-transform/sf-transform [OPTIONS]
Transform base models in a directory for BigQuery source
Options:
-m, --model-path PATH The path to models
-o, --output-path PATH Path to write transformed models to
--drop-metadata BOOLEAN (default=False) optionally drop source columns prefixed with "_" if that designates metadata columns not needed in target
--case-sensitive BOOLEAN (default=False) treat column names as case-sensitive - otherwise force all to lower
--split-columns BOOLEAN Split column names. E.g. currencycode =>
currency_code
--id-as-int BOOLEAN Convert id to int
--convert-timestamp BOOLEAN Convert timestamp to datetime
--help Show this message and exit.
ID:
name: ID
sql: CAST(ID as INT64)
CREATED_TIME:
name: CREATED_AT
UPDATED_TIME:
name: MODIFIED_AT
DATE_START:
name: START_AT
DATE_STOP:
name: STOP_AT
This .yml
file when applied to all models in the staging/source_name
folder will cast all ID
field to INT64 and rename all the date columns to a value in the name
key. For example, CREATED_TIME
will be renamed to CREATED_AT
and DATE_START
will be renamed to START_AT
. If no sql
is provided, the package will just rename the field. If a sql
is provided, the package will execute the SQL and rename the field using the name
key.
$ dbt-generator transform -m ./models/staging/source_name/ -t ./transforms.yml
This will transform all models in the staging/source_name
folder using the transforms.yml
file. You can also drop the metadata by setting the drop-metadata
flag to true
(dropping columns start with _
). The --case-sensitive
flag will determine if the transforms will use case-sensitive names or not.\
Unexpected error with integration github-files: Integration is not authenticated with GitHub