Configuring Source Freshness
Monitor the freshness of your source data by configuring the dbt source freshness
command in your Bolt schedules. This helps ensure your data meets defined SLAs and maintains quality standards.
Overview
When you include the dbt source freshness
command in your schedule, Bolt tracks and reports the last update time of your source tables. This information appears in the Source Freshness section of your run details.
Configuration Steps:
1. Add Required Fields
To enable source freshness checking, add these elements to your source configuration:
A
freshness
block to define timing thresholdsA
loaded_at_field
to specify the timestamp column
2. Define Freshness Rules
In your freshness
block, specify one or both timing thresholds:
warn_after
: Triggers a warning when data exceeds this ageerror_after
: Marks data as error state when exceeding this age
Configuration Example
sources:
- name: analytics_raw
freshness: # default freshness
warn_after: {count: 12, period: hour}
error_after: {count: 24, period: hour}
loaded_at_field: fivetran_loaded_at
tables:
- name: orders
freshness: # make this a little more strict
warn_after: {count: 6, period: hour}
error_after: {count: 12, period: hour}
- name: customers # this will use the freshness defined above
- name: product_skus
freshness: null # do not check freshness for this table
Last updated
Was this helpful?