Skip to main content

​
Overview

Paradime provides a convenient way to push dbt™ metadata from your Bolt runs directly into DataHub, keeping your DataHub catalog up to date with the latest models, lineage, columns, and descriptions from your dbt™ project. By leveraging Paradime’s DataHub Integration, you can orchestrate your data pipeline by pushing dbt™ artifacts to DataHub immediately after your dbt™ transformations complete. You can configure this by adding the push command directly in the same Bolt schedule as your dbt™ commands, or by using the on run completion trigger in a separate pipeline. Under the hood, Paradime feeds the manifest file and the catalog file into DataHub’s native dbt ingestion source, so the result matches what a dbt Cloud / dbt Core ingestion produces.
Prerequisites1. Configure the DataHub IntegrationYou will need a DataHub GMS URL and a Personal Access Token to use these commands. See our docs here.🔧 dbt™ Artifact Configuration
  • DataHub’s dbt source needs both target/manifest.json and target/catalog.json.
  • The catalog is only produced by dbt docs generate, so your schedule must run dbt docs generate before the push command.
  • Whenever a dbt™ command is executed, files in the target directory will be overwritten. So you will need to run the push command after your dbt™ artifacts have been generated.
2. Set Environment VariablesSet your DataHub credentials and configuration as environment variables for Bolt Schedules
  • DATAHUB_GMS_URL
  • DATAHUB_GMS_TOKEN
  • DATAHUB_TARGET_PLATFORM
  • DATAHUB_DOMAIN (optional)

​
How to Configure DataHub Artifacts Push

When setting up the DataHub Artifacts Push task, add the command after the dbt™ commands that generate your artifacts (including dbt docs generate). Paradime searches the resources directory and pushes every dbt™ project it finds (any folder containing both target/manifest.json and target/catalog.json), so a multi-project setup is handled in one task.
We recommend running dbt docs generate as the step immediately before the push command. This guarantees the catalog.json DataHub needs is present and up to date for every run.

​
datahub-artifacts-push

Push Bolt dbt™ artifacts (manifest + catalog) to DataHub.
NoteEach flag can be supplied either on the command line or via its environment variable. Configuring the integration in Paradime sets these environment variables for you, so in a Bolt schedule you can usually call paradime run datahub-artifacts-push with no flags.

​
Examples

Standard push When DATAHUB_GMS_URL, DATAHUB_GMS_TOKEN, and DATAHUB_TARGET_PLATFORM are set as environment variables, the command picks them up automatically, so no flags are needed.
Push with a domain Use the --domain flag to associate every pushed asset with a specific DataHub domain URN. This overrides the DATAHUB_DOMAIN environment variable for this run.
Learn More: For details on configuring the integration and generating credentials, refer to the DataHub integration documentation.
Now, let’s learn how to configure the Trigger Types of a Bolt Schedule.

Related topics

DataHubFivetran Connector SyncTrigger Hightouch SyncsTrigger Census SyncsAirbyte Connector Sync