Data Pipeline Configuration

This guide demonstrates how DinoAI Agent helps configure your dbt™ project settings to align with best practices and your folder structure.

Properly configuring dbt™ projects involves managing folder-specific materializations, schema naming conventions, and consistent tagging. As projects grow, maintaining these configurations becomes increasingly complex.

DinoAI Agent can analyze your project structure and automatically update your dbt_project.yml file to implement best practices for materializations, schemas, and tags.

Example Prompt

Can you update my dbt_project.yml to map all my models to my folder structure so that each SQL file in the respective folder will write to the respective schema and also make sure they are tagged?

Optional: Add context by selecting your existing dbt_project.yml file. This context is useful for configuration tasks as it allows DinoAI to understand your current settings before making changes.

How It Works

After you enter your prompt:

1. DinoAI analyzes your project's folder structure and existing configuration

2. It identifies model types based on folder organization (staging, intermediate, marts, etc.)

3. It generates appropriate configuration sections with schema naming and materializations

4. If configured, DinoAI applies your .dinorules preferences for project organization

Note: Adding your existing dbt_project.yml as context ensures DinoAI preserves your custom settings while adding new configurations

Example Output

DinoAI will generate configuration like this:

# dbt_project.yml update

models:
  formula_one:
    # Default materialization and schema settings
    +materialized: view
    +schema: "{{ target.schema }}"
    
    # Source models
    source:
      +schema: "{{ target.schema }}_source"
      +tags: ["source", "formula_one"]
    
    # Staging models
    staging:
      +schema: "{{ target.schema }}_staging"
      +tags: ["staging", "formula_one"]
      # Sub-folders within staging
      f1:
        +tags: ["f1"]
    
    # Intermediate models
    intermediate:
      +schema: "{{ target.schema }}_intermediate"
      +tags: ["intermediate", "formula_one"]
    
    # Marts models (typically materialized as tables)
    marts:
      +materialized: table
      +schema: "{{ target.schema }}_marts"
      +tags: ["marts", "formula_one", "reporting"]
      # Sub-folders within marts
      f1:
        +tags: ["f1"]

Key Benefits

• Optimized Performance: Ensures efficient materialization strategies

• Organizational Clarity: Makes project structure more intuitive

• Consistent Naming: Implements standardized schema naming

• Logical Tagging: Makes it easier to run specific model groups

• Maintainability: Creates a configuration that scales with your project

When to Use This

• When setting up a new dbt™ project

• When restructuring or reorganizing existing projects

• After adding new model categories or folders

• When implementing or updating tagging strategies

• Before optimization efforts to ensure proper materializations