Paradime Turbo CI Schema Cleanup

This dbt macro automates the cleanup of temporary datasets created during Paradime Turbo CI runs.

Overview

During CI/CD processes, Paradime Turbo CI creates temporary schemas for testing. The schemas will be dropped automatically on Pull Request merged if:

• Snowflake is your connected Data Warehouse

• GitHub is your git provider

This macro identifies and drops these schemas on demand by identifying all schemas with prefix paradime_turbo_ci

Configure the drop_turbo_ci_schema macro

To clean up temporary schemas created during CI runs, you can create a custom macro in your dbt™️ project. Here's how:

1. Navigate to your dbt project's macros folder

2. Create a new file called drop_turbo_ci_schema.sql

3. Add the following code to the file:

drop_turbo_ci_schema.sql

{#   Deletes BigQuery datasets created by Paradime Turbo CI #}
{% macro drop_turbo_ci_schema(dryrun=True) %}


{# Get project ID from BigQuery connection config based on the target used to executed the macro #}
{%- set default_database = target.database -%}

{# Set schema pattern to match with schemas created by Paradime Turbo CI #}
{%- set paradime_turbo_ci_schema = 'paradime_turbo_ci%' -%}

{# Query to generate DROP commands for matching schemas #}
{% set cleanup_query %}

      WITH TURBO_CI_DROP_SCHEMA AS (
                SELECT 
                    catalog_name, 
                    schema_name 
                FROM {{default_database}}.INFORMATION_SCHEMA.SCHEMATA
                WHERE schema_name LIKE "{{ paradime_turbo_ci_schema }}"
              ) 
      SELECT 
        'DROP SCHEMA ' || '`' || catalog_name || '.' || schema_name || '`' || ' CASCADE' || ';' as DROP_COMMANDS
      FROM 
        TURBO_CI_DROP_SCHEMA
  {% endset %}

{# Get list of DROP commands to execute #}
{% set drop_commands = run_query(cleanup_query).columns[0].values() %}

{# Execute or print DROP commands based on dryrun parameter #}
{% if drop_commands %}
  {% if dryrun | as_bool == False %}
    {% do log('Executing DROP commands...', True) %}
  {% else %}
    {% do log('Printing DROP commands...', True) %}
  {% endif %}
  {% for drop_command in drop_commands %}
    {% do log(drop_command, True) %}
    {% if dryrun | as_bool == False %}
      {% do run_query(drop_command) %}
    {% endif %}
  {% endfor %}
{% else %}
  {% do log('No relations to clean.', True) %}
{% endif %}

{%- endmacro -%

The above example is based on BigQuery, use DinoAI to adapt it to other Data Warehouse providers.

Usage

The drop_turbo_ci_schema macro helps clean up temporary schemas created during CI runs, with an optional dry-run mode for safe testing.

Arguments

• dryrun (bool, optional)

  • True: Preview DROP commands without executing them

  • False: Execute DROP commands and remove schemas

  • Default: True (Safe mode)

# Preview DROP commands
dbt run-operation drop_turbo_ci_schema

# Execute DROP commands
dbt run-operation drop_turbo_ci_schema --args '{dryrun: false}'

Configure a Bolt Schedule

Set up a scheduled job in Paradime Bolt with the Below:

Schedule Settings

Setting	Value	Explanation
Schedule Type	Standard	Ensures consistent execution for production workloads in a single environment. Best for regular data pipeline runs
Schedule Name	drop turbo ci schemas	Descriptive name that indicates purpose
Git Branch	main	Uses your default production branch to ensure you're always running the latest approved code

Command Settings

The template uses one commands that execute our macro and passes the argument dryrun set to false

• dbt run-operation drop_turbo_ci_schema --args '{dryrun: false}' --target ci

Make sure to use the same --target used when running Paradime Turbo CI

Trigger Type

• Type: Scheduled Run (Cron)

• Cron Schedule: 0 8 * * 0 (Every Sunday at 8AM UTC)

For custom command configurations, see Command Settings documentation.

Notification Settings

• Email Alerts:

  • Success: Confirms all schemas were dropped successfully, letting you know your data pipeline is healthy

  • Failure: Immediately alerts you when the macro fails to drops your Paradime Turbo Ci schema