Using GitLab Pipelines

You can build your custom Continuous Deployment jobs using GitLab Pipelines and Bolt APIs.

To use this feature it is required to have a production environment configured in Paradime.

ℹ️ Check our setup guide here based on your data warehouse provider.

Setting up Continuous Deployment Bolt Schdeule

1. Create or Edit a Bolt Schedule: In the Bolt UI, create a new schedule or modify an existing one.

2. Set Schedule Type to deferred: This allows your job to leverage results from previous runs, improving efficiency.

3. Name your continuous deployment schedule: Choose a descriptive name for easy identification.

4. Choose a Bolt Schedule to defer to: Select a previous job run to use as a reference, either from the same schedule or a different one.

5. Set the comparison method: Determine how the current run compares with previous runs (e.g., Last Run, Last Successful Run).

6. Enter dbt Commands: Specify the dbt tasks to execute as part of the continuous deployment. For example:

dbt run --select state:modified

1. Specify the Git Branch: Choose which branch of your dbt project will be used when the continuous deployment is triggered (e.g., main).

2. Set Trigger Type to On Merge: This will allow the API to trigger multiple runs

3. Publish Bolt Schedule: Save and activate your continuous deployment schedule.

Implement using Paradime's Bolt Template - Test Code Changes on Pull Request.

Tutorial

Generate API keys

API keys are generate at a workspace level.

To be able to trigger Bolt using the API, you will first need to generate API keys for your workspace. Got to account settings and generate your API keys, make sure to save in your password manager:

• API key

• API secret

• API Endpoint

You will need this later when setting up the secret in GitLab pipelines.

API Keys

Create a GitLab Pipeline

GitLab merged results pipelines is a feature available only on Tier: Premium, Ultimate. Make sure to enable merge results pipelines in GitLab before proceeding to next step.

Now you will need to create a new .gitlab-ci.yml file at the root of your project your dbt™️ repository. Copy the code block below and enter the values required.

Example GitLab pipelines configuration file

.gitlab-ci.yml

# Define the stages of the pipeline. In this case, we only have one stage
stages:
  - paradime_continuous_deployment

# Main job definition
paradime_continuous_deployment:
  # Specify which stage this job belongs to
  stage: paradime_continuous_deployment
  
  # Use Python 3.11 as the base Docker image for this job
  image: python:3.11
  
  # Define when this pipeline should be triggered
  rules:
    # This job will only run when all these conditions are met:
    # - CI_PIPELINE_SOURCE == "merge_request_event": Triggered by a merge request
    # - CI_MERGE_REQUEST_EVENT_TYPE == "merged": The merge request is being merged
    # - CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main": The target branch is 'main'
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_EVENT_TYPE == "merged" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"'
  
  # Define environment variables needed for the job
  variables:
    # These variables should be configured in GitLab CI/CD settings as protected variables
    PARADIME_API_KEY: ${PARADIME_API_KEY}         # API key for Paradime authentication
    PARADIME_API_SECRET: ${PARADIME_API_SECRET}   # API secret for Paradime authentication
    PARADIME_API_ENDPOINT: ${PARADIME_API_ENDPOINT} # Paradime API endpoint URL
    PARADIME_SCHEDULE_NAME: "continuous_deployment_run" # Name of the Paradime schedule to run
  
  # Commands to run before the main script
  before_script:
    # Install the Paradime Python SDK
    - pip install paradime-io==4.7.1  # Check for latest version of the Paradime Python SDK on https://github.com/paradime-io/paradime-python-sdk/releases
  
  # Main script to execute
  # Run the Paradime bolt schedule with the specified name
  # The --wait flag makes the script wait for the schedule to complete
  script: |
    paradime bolt run "$PARADIME_SCHEDULE_NAME" --wait
  
  # Set a timeout of 60 minutes for this job
  # If the job hasn't completed within this time, it will be terminated
  timeout: 60 minutes

Add the API key and Credential in the GitLab variables

Finally you need to add the API key and credentials generated in the previous step in GitLab CI/CD pipelines.

Set the corresponding values using your credentials for the variable names:

• PARADIME_API_KEY

• PARADIME_API_SECRET

• PARADIME_API_ENDPOINT