Elementary Data

For new Elementary data users, we recommend starting out Elementary Data's Tutorial Guide to understand the core concepts and implementation.

With Elementary, an open-source data observability solution, data and analytics engineers can achieve instant visibility into their dbt projects. It streamlines the process of detecting data anomalies, issuing targeted alerts, and unraveling both impact and root causes.

Getting started

Paradime seamlessly integrates with Elementary, offering a ready-to-use installation of the Elementary CLI.

1. Install Elementary dbt™️ package

To start using Elementary, you need to add the Elementary dbt package to your dbt project.

2. Add elementary to packages.yml

Add the following to your packages.ymlfile (if missing, create it on the same path s yourdbt_project.yml):

dbt_project.yml

packages:
  - package: elementary-data/elementary
    version: 0.15.0
    # check latest package version on https://hub.getdbt.com/elementary-data/elementary/latest/

dbt - Package hub

3. Configure Elementary Profile in your dbt_project.yml

Paradime simplifies the setup process by providing a pre-configured Elementary profile that works with your default connections for both IDE and Scheduler.

By default, the Elementary schema name is set as <target_schema>_elementary. You can override this by setting a system variable called PARADIME_EDR_SCHEMA. For instructions on configuring environment variables, click here.

3.1 - Add the following code to you dbt_project.yml:

dbt_project.yml

models:
  elementary:
    +schema: "elementary"
    ## To disable elementary for dev, uncomment this:
    # enabled: "{{ target.name in ['prod','analytics'] }}"

This configuration ensures Elementary models have their own schema.

3.2 - Add this flag to your dbt_project.yml:

dbt_project.yml

flags:
  require_explicit_package_overrides_for_builtin_materializations: False
  source_freshness_run_project_hooks: True

Starting from dbt 1.8, you must explicitly allow the Elementary packages to override default dbt™ materializations. Elementary needs this permission to collect samples and failed row counts for dbt tests.

These flags may trigger a deprecation warning. Elementary is collaborating with the dbt core team on a permanent solution. To mute the warning, to mute this warning, add the following flag:

dbt_project.yml

flags:
  mute_ensure_materialization_override: TRUE

Important: Materialization config

Elementary requires some models to be created as incremental tables. Ensure there are no global materialization configurations that might interfere with Elementary. For example, avoid settings like:

dbt_project.yml

materialized: "{{ 'table' if target.name == 'prod-cloud' else 'view' }}"

Instead, structure your dbt_project.yml like this:

dbt_project.yml

models:
  my_project:
    materialized: "{{ 'table' if target.name == 'prod-cloud' else 'view' }}"
  elementary:
    +schema: "elementary"

4. Build Initial Elementary Models

Once you've configured your Elementary in your packages.yml and dbt_project.yml, Run the following dbt commands to install Elementary and materialize the initial elementary models in your data warehouse:

Terminal

dbt deps
dbt run --select elementary

This will mostly create empty tables, that will be updated with artifacts, metrics and test results in your future dbt™ executions. (This is required for both environments, IDE and Scheduler.)

5. Generate Elementary Tests

After installing and configuring Elementary, you're ready generate a variety on test, including: Anomaly Detection Tests, Schema Tests, etc.

Additionally, you can leverage you can leverage DinoIA's one-click command, Generate Elementary Tests to automatically create comprehensive tests tailored to your dbt™ model's structure and content. This ensures a seamless setup of data observability across your project.

When complete, execute execute dbt testto verify setup.

6. (Optional) Configure Slack / Teams notifications

Paradime Alerts allow you to set up and manage notifications for your dbt™ projects, keeping you informed of important events and anomalies. With support for both Slack and Microsoft Teams, you can customize your alerts to fit your team's communication preferences. See docs for details.

Example Slack Notifications

7. (Optional) Run Elementary in Production

While this documentation demonstrates running Elementary in a development environment, you can execute Elementary command in your Production Schedules (Bolt)