Using GitHub

With Paradime Turbo CI, you can run and test code changes to your dbt™️ project prior to merge them in your main branch, by configuring a Bolt schedule to run when a new pull request is opened against your dbt™️ repository.

Paradime will build the models affected by your changed into a temporary schema, and will run tests that you have written against the changes models to validate your test are still passing. When opening a pull request, Paradime Turbo CI run status will be showing in the pull request.

How Turbo CI works

When Turbo CI is configured, Paradime will listen for the pull request event from Github - Turbo CI will run on creation, reopening and when there is a new commit in a PR.

When Paradime receive the webhook event, Bolt will trigger a new run based on the configuration of the Turbo CI job.

When running, models will be built in a temporary schema using the prefix paradime_turbo_ci_pr_ followed by the PR number (e.g paradime_turbo_ci_pr_123). This will enabled you to see the results of your changes associated with the code in your pull request in your production data warehouse

After the pull request is merged, Paradime will delete the temporary schema created as part of the turbo CI job to keep your database in order.

Pre-requisites

To use this feature it is required to install the Paradime Github app and authorize access to the dbt™️ repository used in Paradime.

ℹ️ Check our installation guide for the GitHub integration here.

To use this feature it is required to have an additional production environment configured in Paradime, where the target is set to ci.

ℹ️ Check our setup guide here based on your data warehouse provider.

Configure a custom schema for CI runs

Customize you dbt™️ schema generation at runtime, to make sure that when a PR is opened, Paradime Turbo CI will build and test your changed dbt™️ models in a temporary schema. We will want to update the dbt™️ generate_schema_name.sql macro.

In your dbt™️ project, in your macros folder create a macro called: generate_schema_name.sql and use a similar logic as below, where when a Bolt run is executed using the ci target the schema where your models will be built will use the paradime_turbo_ci_pr prefix.

{% macro generate_schema_name(custom_schema_name, node) -%}

    {%- set default_schema = target.schema -%}
    {%- if target.name == 'prod' -%}

        {{ custom_schema_name | trim }}

    {%- elif target.name == 'ci' -%}

        {{ default_schema }}

    {%- else -%}

        {%- if custom_schema_name is none -%}

            {{ default_schema }}

        {%- else -%}

            {{ default_schema }}_{{ custom_schema_name | trim }}

        {%- endif -%}

    {%- endif -%}

{%- endmacro %}

See also:

Custom schemas | dbt Developer Hub

Deferral and State Comparison explained

When creating the Turbo CI job in Paradime, you can set your execution settings to defer to a previous run state by adding in the deferred_schedule_name configuration the name of the Bolt productions schedule you want to defer to

Paradime will look at the manifest.json from the specified schedule's most recent successful run (unless setting the optional parameter successful_run_only: false) to determined the set of new and modified dbt™️ resources. In your Turbo CI commands, you can then use the state:modified+ argument to only run the modified resources and their downstream dependencies.

A common example is:

dbt build --select state:modified+ --target ci

This will run and test all modified models with the downstream dependencies. Useful to validate that test are still passing after making changes to upstream dbt™️ models.

👉 To read more about state comparison check the dbt Core™️ documentation here.

Configuring Bolt Turbo CI job

Paradime offers two distinct methods for running Turbo CI:

• UI-Based Configuration: Create, edit, and update your Turbo CI configurations within the user interface. All the configurations available in the YAML-based schedule is available in the UI.

• YAML-Based Configuration: Create, edit, and update your Turbo CI configurations in the paradime_schedules.yml file, which is located in your Git repository.

A Turbo CI job differ from a bolt schedule as it involves:

• The Bolt schedule to defer to

• The commands to use the state:modified+ selector to build / tests only new dbt™️ models and their downstream dependencies. State comparison can only happen when there is a deferred schedule name configured to compare state to.

• Being triggered by a pull request being opened in your Github dbt™️ repository

UI-Based Configuration

UI-Based Configuration

To create a UI-Based Turbo CI schedule , go to Bolt on the home screen and click + New Schedule.

Select turbo-ci as the schedule type and fill out the various fields to set up your Turbo CI schedule.

For additional details, watch the demo, and read more about the available fields.

YAML-Based Configuration

YAML-Based Configuration

Additional schedule parameters available for turbo CI schedules

Parameters	Description	Example
enabled	[Required] Set this to TRUE to enabled deferred for the schedule	true
deferred_manifest_schedule	[Required] The name of the Bolt schedule used to to look for the most recent successful run manifest.json / sources.json / run_results.json for state comparison.	hourly
successful_run_only	[Optional] By default paradime will look for the last successful run. Set this to false to let paradime to use defer with last run of the selected schedule, irrespective of the run status.	false

Example Turbo CI configuration

paradime_schedules.yml

- name: turbo ci job # the name of your Turbo CI job
  turbo_ci:
    enabled: true # true to enabled this Turbo CI job to run on pull request
    deferred_manifest_schedule: hourly #the name of the bolt schedule where the Turbo CI job will look for the most recent successful run manifest.json for state comparison
    successful_run_only: false #[optional]by default paradime will look for the last successful run. Set this to false to let paradime to use defer with last run of the selected schedule, irrespective of the run status.
  schedule: "OFF" # set the schedule configuration to not run on a schedule (to be used for PR only)
  environment: production #the environment used to run the schedule -> this is always production
  commands:
    - dbt build --select state:modified+ --target ci #the dbt command you want to run when the pull request is opened
  owner_email: "[e-mail]" #the email of the Turbo CI job owner_email
  slack_on: # the configuration of when a notification is triggered. Here we want to send a notification when the run is completed either successfully or when failing
    - passed
    - failed
  slack_notify: # the channel/user that will be notified
    - "#data-alerts"
    - "@john"
  email_notify: # the email addresses that will be notified
    - "[e-mail]"
    - "[e-mail]"
  email_on: # the configuration of when a notification is triggered. Here we want to send a notification when the run is completed either successfully or when failing
    - passed
    - failed

View Turbo CI run Logs

As per other bolt schedules, you can inspect run logs for each of the Turbo CI jobs running when a pull request is opened.

pageView Bolt runs logs

Paradime provides a set of system environment variables that can be added to your dbt™️ models to inject metadata at run time to add more information to your materialized tables in your data warehouse.

These can be used to help identify which schedules run relates to a given row of data in your tables.

Environment variables

Paradime provides a set of system environment variables that can be used to build custom logic for macro or schema generation when a PR is opened.

Vairable name	example
PARADIME_TURBO_CI_PREFIX	paradime_turbo_ci_pr_123