Turbo CI for dbt™ pull requests

Turbo CI runs and tests only your changed dbt™ models on every pull request, before they merge. When a pull request opens or gets a new commit, a Turbo CI Bolt schedule builds and tests the changes in a temporary schema, and the run status shows on the pull request.

GitHub and GitLab: Paradime starts the runs itself, through the Paradime GitHub App or the GitLab connection. You don’t add a CI file or an API key.
Azure DevOps and Bitbucket: a pipeline in your repository starts the runs through the Bolt API.

How Turbo CI works

When a pull request is opened, Paradime (on GitHub and GitLab) or your pipeline (on Azure DevOps and Bitbucket) starts a Turbo CI schedule, which runs your configured dbt™ command against the changed models. Models build into a temporary schema using the prefix paradime_turbo_ci_pr_ followed by the pull request number (for example, paradime_turbo_ci_pr_123), so you can inspect the results of your changes in your warehouse. A model counts as changed when its file differs from the manifest of the schedule you defer to, so a branch that is behind your default branch can also build models it didn’t touch. See the Turbo CI FAQ.

Prerequisites

Turbo CI runs in its own Bolt environment, so CI builds use a separate warehouse connection and target from your production runs:

Create a CI environment. In Settings > Environments, click New environment and create one for CI (for example, CI). See Create and manage environments.
Add a connection to it. Attach a warehouse connection to the CI environment with its target name set to ci. See the Bolt environment setup guide for your data warehouse.
Point the Turbo CI schedule at it. Set the schedule’s Environment to the CI environment in schedule settings, or set environment: <ci-environment-slug> in the schedule’s YAML.

Without a dedicated CI environment, Turbo CI runs would use your default environment’s connection and target, so the ci target logic in the schema macro below would never engage.

Already running Turbo CI with --target ci?

Follow the migration guide to move to a dedicated CI environment and drop the --target ci flags from your commands.

Configure a custom schema for CI runs

Update your dbt™ generate_schema_name.sql macro so that, when a Bolt run uses the ci target, models build into a temporary schema with the paradime_turbo_ci_pr prefix. In your project’s macros folder, create generate_schema_name.sql:

{% macro generate_schema_name(custom_schema_name, node) -%}

    {%- set default_schema = target.schema -%}
    {%- if target.name == 'prod' -%}

        {{ custom_schema_name | trim }}

    {%- elif target.name == 'ci' -%}

        {{ default_schema }}

    {%- else -%}

        {%- if custom_schema_name is none -%}

            {{ default_schema }}

        {%- else -%}

            {{ default_schema }}_{{ custom_schema_name | trim }}

        {%- endif -%}

    {%- endif -%}

{%- endmacro %}

See the dbt custom schemas documentation.

Configure the Turbo CI job

A Turbo CI job is a Bolt schedule with a few additions:

Its environment set to your CI environment, so runs use the CI connection and ci target.
A production schedule to defer to, so state comparison can determine what changed.
commands that use the state:modified+ selector to build and test only new and changed models.
A trigger set to run when a pull request is opened.

Test Code Changes On Pull Requests

Set up your Git provider

Once the shared setup above is in place, connect Turbo CI to your provider:

GitHub

Native, through the Paradime GitHub App. No CI file.

GitLab

Native, through the GitLab connection on gitlab.com or Self-Managed. No CI file.

Azure DevOps

An Azure Pipeline calls the Bolt API.

BitBucket

A Bitbucket Pipeline calls the Bolt API.

Run schedules in order

On GitHub and GitLab, you can chain several Turbo CI schedules so they run one after another instead of all at once, and stop at the first failure.

Schedule Anatomy

Deploy Schedules

Managing Schedules

CI/CD

Turbo CI for dbt™ pull requests

How Turbo CI works

Prerequisites

Already running Turbo CI with --target ci?

Configure a custom schema for CI runs

Configure the Turbo CI job

Test Code Changes On Pull Requests

Set up your Git provider

GitHub

GitLab

Azure DevOps

BitBucket

Run schedules in order

Run Bolt schedules in order (chains)

​How Turbo CI works

​Prerequisites

Already running Turbo CI with --target ci?

​Configure a custom schema for CI runs

​Configure the Turbo CI job

Test Code Changes On Pull Requests

​Set up your Git provider

GitHub

GitLab

Azure DevOps

BitBucket

​Run schedules in order

Run Bolt schedules in order (chains)

How Turbo CI works

Prerequisites

Configure a custom schema for CI runs

Configure the Turbo CI job

Set up your Git provider

Run schedules in order