DataHub Sync
Overview
Paradime provides a convenient way to push dbt™ metadata from your Bolt runs directly into DataHub, keeping your DataHub catalog up to date with the latest models, lineage, columns, and descriptions from your dbt™ project.
By leveraging Paradime's DataHub Integration, you can orchestrate your data pipeline by pushing dbt™ artifacts to DataHub immediately after your dbt™ transformations complete. You can configure this by adding the push command directly in the same Bolt schedule as your dbt™ commands, or by using the on run completion trigger in a separate pipeline.
Under the hood, Paradime feeds the manifest file and the catalog file into DataHub's native dbt ingestion source, so the result matches what a dbt Cloud / dbt Core ingestion produces.

Prerequisites
1. Configure the DataHub Integration
You will need a DataHub GMS URL and a Personal Access Token to use these commands. See our docs here.
🔧 dbt™ Artifact Configuration
DataHub's dbt source needs both
target/manifest.jsonandtarget/catalog.json.The catalog is only produced by
dbt docs generate, so your schedule must rundbt docs generatebefore the push command.Whenever a dbt™ command is executed, files in the
targetdirectory will be overwritten. So you will need to run the push command after your dbt™ artifacts have been generated.
2. Set Environment Variables
Set your DataHub credentials and configuration as environment variables for Bolt Schedules
DATAHUB_GMS_URLDATAHUB_GMS_TOKENDATAHUB_TARGET_PLATFORMDATAHUB_DOMAIN(optional)
How to Configure DataHub Artifacts Push
When setting up the DataHub Artifacts Push task, add the command after the dbt™ commands that generate your artifacts (including dbt docs generate). Paradime searches the resources directory and pushes every dbt™ project it finds (any folder containing both target/manifest.json and target/catalog.json), so a multi-project setup is handled in one task.
We recommend running dbt docs generate as the step immediately before the push command. This guarantees the catalog.json DataHub needs is present and up to date for every run.
datahub-artifacts-push
Push Bolt dbt™ artifacts (manifest + catalog) to DataHub.
--datahub-server
Required, TEXT
The DataHub GMS server URL (e.g. https://<instance>.acryl.io/gms).
- Env: DATAHUB_GMS_URL
--datahub-token
Required, TEXT
The DataHub personal access token used to authenticate to GMS.
- Env: DATAHUB_GMS_TOKEN
--target-platform
Required, TEXT
The data warehouse platform the dbt™ models run on (e.g. snowflake, bigquery, redshift).
- Env: DATAHUB_TARGET_PLATFORM
--domain
Optional, TEXT
DataHub domain URN to associate the pushed assets with (e.g. urn:li:domain:sales). When set, every emitted dataset is associated with that domain.
- Env: DATAHUB_DOMAIN
--paradime-resources-directory
Optional, TEXT
The directory where the Paradime resources are stored. The command searches this directory for target/manifest.json and target/catalog.json.
- Default: current directory (.)
- Env: PARADIME_RESOURCES_DIRECTORY
--json
Optional, FLAG
Output results as JSON.
Note
Each flag can be supplied either on the command line or via its environment variable. Configuring the integration in Paradime sets these environment variables for you, so in a Bolt schedule you can usually call paradime run datahub-artifacts-push with no flags.
Examples
Standard push
When DATAHUB_GMS_URL, DATAHUB_GMS_TOKEN, and DATAHUB_TARGET_PLATFORM are set as environment variables, the command picks them up automatically, so no flags are needed.

Push with a domain
Use the --domain flag to associate every pushed asset with a specific DataHub domain URN. This overrides the DATAHUB_DOMAIN environment variable for this run.

Learn More: For details on configuring the integration and generating credentials, refer to the DataHub integration documentation.
Now, let's learn how to configure the Trigger Types of a Bolt Schedule.
Last updated
Was this helpful?