Databricks - Paradime

Databricks is a cloud data platform for storing, processing, and analyzing your data. Connect it to Paradime to run dbt™ from the Code IDE, deploy production jobs with Bolt, and let the DinoAI background agent work against your warehouse.

IP restrictionsMake sure to allow traffic from one of the Paradime IPs in your firewall depending on the data location selected.See also: Paradime IP addresses.

Available in: Code IDE, Bolt, and Agent environments.

Add a connection

Click Settings in the top menu bar of the Paradime interface to access Workspace Settings.
In the left sidebar, click Environments.
Click Add New next to the section for the environment you are configuring:
- Code IDE for the Code IDE connection
- Bolt for the Bolt connection
- Agent for the Agent connection
Select Databricks and fill out the corresponding fields.

Field details

Set Target to match the environment you are configuring (dev, prod, or dinoai). The Code IDE environment uses a Personal Access Token (PAT), while the Bolt and Agent environments use a Service Principal Token (SPT).

Field access levels apply only to the Code IDE environment: an Admin sets workspace-level fields, and each developer sets their own user-level fields. In the Bolt and Agent environments, an Admin sets all fields.

Field	Description	Example
Profile Name	The profile name set in your `dbt_project.yaml`. See more here.	`dbt-databricks`
Target	The target name used to identify the connection. See more here.	`dev`
Service Host Name	Databricks server hostname.	`dbc-e7c65a5c-0521.cloud.databricks.com`
HTTP path	Databricks compute resources URL.	`/sql/1.0/warehouses/fbc246efa4369412`
Unity Catalog Name (Optional)	The name of the Unity Catalog. See more here.	`analytics`
Personal Access Token (PAT)	The PAT created by each user, for the Code IDE environment. See more here.	`dapid7cc51fb15d5889136351110e66119b3`
Service Principal Token (SPT)	The SPT created for your service user, for the Bolt and Agent environments. See more here.	`dapid7cc51fb15d5889136351110e66119b3`
Schema	The default schema used to build dbt™ objects at runtime.	`dbt_john`
Threads	The number of threads used in this connection. See more here.	`8`

Route models to different SQL warehouses

The HTTP path field sets the default SQL warehouse for the connection. To send specific models to other warehouses, define named compute resources in the connection’s optional connection attributes and select one per model with the databricks_compute config.

Open the Databricks connection in Settings > Environments.
Keep your default warehouse in the HTTP path field.
In Optional connection attributes, add a compute block:

compute:
  large_wh:
    http_path: /sql/1.0/warehouses/<large_warehouse_id>
  xxl_wh:
    http_path: /sql/1.0/warehouses/<xxl_warehouse_id>

In a model, select a compute resource by name:

{{ config(materialized='incremental', databricks_compute='xxl_wh') }}

Models without databricks_compute run on the default HTTP path.

Add the same compute block to every Databricks connection that runs these models, in the Code IDE, Bolt, and Agent environments. If a model names a compute resource that its connection does not define, the run fails with Compute resource 'xxl_wh' does not exist or does not specify http_path.

Optional connection attributes

Set additional profiles.yml attributes for this connection.

​Add a connection

​Field details

​Route models to different SQL warehouses

​Related

Optional connection attributes

Add a connection

Field details

Route models to different SQL warehouses

Related