Using BitBucket Pipelines

You can build your custom Continuous Deployment jobs using BitBucket Pipelines and Bolt APIs.

To use this feature it is required to have a production environment configured in Paradime.

ℹ️ Check our setup guide here based on your data warehouse provider.

Setting up Continuous Deployment Bolt Schdeule

1. Create or Edit a Bolt Schedule: In the Bolt UI, create a new schedule or modify an existing one.

2. Set Schedule Type to deferred: This allows your job to leverage results from previous runs, improving efficiency.

3. Name your continuous deployment schedule: Choose a descriptive name for easy identification.

4. Choose a Bolt Schedule to defer to: Select a previous job run to use as a reference, either from the same schedule or a different one.

5. Set the comparison method: Determine how the current run compares with previous runs (e.g., Last Run, Last Successful Run).

6. Enter dbt Commands: Specify the dbt tasks to execute as part of the continuous deployment. For example:

dbt run --select state:modified

1. Specify the Git Branch: Choose which branch of your dbt project will be used when the continuous deployment is triggered (e.g., main).

2. Set Trigger Type to On Merge: This will allow the API to trigger multiple runs

3. Publish Bolt Schedule: Save and activate your continuous deployment schedule.

Implement using Paradime's Bolt Template - Test Code Changes on Pull Request.

Tutorial

Generate API keys

API keys are generate at a workspace level.

To be able to trigger Bolt using the API, you will first need to generate API keys for your workspace. Got to account settings and generate your API keys, make sure to save in your password manager:

• API key

• API secret

• API Endpoint

You will need this later when setting up the secrete in BitBucket pipelines.

API Keys

Create an BitBucket Pipeline

Now you will need to create a new bitbucket-pipelines.yml file in your dbt™️ repository. Copy the code block below and enter the values required.

Example BitBucket pipelines configuration file

bitbucket-pipelines.yml

# Specify the base Docker image to use for all pipeline steps
image: python:3.11

# Define reusable pipeline steps
definitions:
 steps:
   # Create a reusable step template using YAML anchor (&)
   - step: &paradime-continuous-deployment
       # Name of this pipeline step for identification
       name: Paradime Continuous Deployment
       
       # Commands to execute in this step
       script:
         # Export required Paradime environment variables
         - export PARADIME_API_KEY=$PARADIME_API_KEY
         - export PARADIME_API_SECRET=$PARADIME_API_SECRET
         - export PARADIME_API_ENDPOINT=$PARADIME_API_ENDPOINT
         
         # Install Paradime Python SDK
         - pip install paradime-io==4.7.1  # Check for latest version of the Paradime Python SDK on https://github.com/paradime-io/paradime-python-sdk/releases
         
         # Run the Paradime bolt schedule and wait for completion
         - paradime bolt run "continuous_deployment_run" --wait
       
       # Specify compute resources for this step
       size: 2x  # Double the default resources
       
       # Set maximum execution time in minutes
       max-time: 60

# Define when and how pipelines should run
pipelines:
 branches:
   # This pipeline will only run when changes are pushed to the main branch
   main:  
     - step:
         # Reference the template step defined above using YAML alias (*)
         <<: *paradime-continuous-deployment
         
         # Define environment variables needed for this step
         # These should be configured in Bitbucket repository settings
         variables:
           API_KEY: PARADIME_API_KEY      # API key for Paradime authentication
           API_SECRET: PARADIME_API_SECRET # API secret for Paradime authentication
           API_ENDPOINT: PARADIME_API_ENDPOINT # Paradime API endpoint URL

Add the API keys and Credential in the BitBucket Pipeline variables

Finally you need to add the API key and credentials generated in the previous step in BitBucket Pipelines.

Set the corresponding values using your credentials for the variable names:

• PARADIME_API_KEY

• PARADIME_API_SECRET

• PARADIME_API_ENDPOINT