Elementary Data

Elementary Data - Open Source Data Observability

With Elementary, an open-source data observability solution, data and analytics engineers can achieve instant visibility into their dbt™️ projects. It streamlines the process of detecting data anomalies, issuing targeted alerts, and unraveling both impact and root causes.

Getting started

Paradime seamlessly integrates with Elementary, offering a ready-to-use installation of the Elementary CLI.

Install Elementary dbt™️ package

To start using Elementary, you need to add our the Elementary dbt™️ package to your dbt™️ project.

Add elementary to packages.yml

Add the following to your packages.yml file (if missing, create it where dbt_project.yml is):

dbt_project.yml

packages:
  - package: elementary-data/elementary
    version: 0.15.0
    # check latest package version on https://hub.getdbt.com/elementary-data/elementary/latest/

dbt - Package hub

Configure your dbt™️ project

Paradime already provides an Elementary profile configured to work with Elementary based on your default connections (for both IDE and Scheduler connnections.

The schema name is by default set as <target_schema>_elementary. You can override this by configured a system variable called PARADIME_EDR_SCHEMA.

Check how to configure environment variables here.

Update your dbt™️ project with the below code:

dbt_project.yml

models:
  elementary:
    +schema: "elementary"
    ## To disable elementary for dev, uncomment this:
    # enabled: "{{ target.name in ['prod','analytics'] }}"

This means Elementary models will have their own schema. Depending on your project custom schema macro, the schema will be named elementary or <target_schema>_elementary.

Important: Materialization config

For elementary to work, it needs to create some of the models as incremental tables. Make sure that there are no global materialization configurations that affect elementary, such as:

dbt_project.yml

materialized: "{{ 'table' if target.name == 'prod-cloud' else 'view' }}"

Make sure to place the 'elementary' configuration under the models key, and other configs under your project name, example"

models:
  my_project:
    materialized: "{{ 'table' if target.name == 'prod-cloud' else 'view' }}"

  elementary:
    +schema: "elementary"

Build Elementary models

This will mostly create empty tables, that will be updated with artifacts, metrics and test results in your future dbt executions. (This is required both environments, IDE and Scheduler.

Finally, lets run dbt™️ deps to install the package and build the Elementary models in your databse.

dbt deps
dbt run --select elementary