Snapshot Source Data Freshness

This template creates a schedule to monitor source data freshness using dbt source freshness, ensuring your data meets defined SLAs by regularly checking when source tables were last updated and alerting you of any staleness issues. Use this template when you need to monitor critical data sources - for example, ensuring your daily sales data is loading properly or verifying that customer data is being updated within expected timeframes.

Key Benefits

• Ensures critical data sources meet defined freshness SLAs

• Quickly identifies issues with upstream data loading jobs

• Prevents analytics teams from using stale data

• Tracks and reports on source data freshness for downstream processes

Prerequisites

• Scheduler Environment is connected to your data warehouse provider.

• Source freshness configurations in your sources.yml file

Default Configuration

Schedule Settings

Setting	Value	Explanation
Schedule Type	Standard	Ensures consistent monitoring of source freshness in production environment
Schedule Name	source freshness check	Descriptive name that indicates purpose
Git Branch	main	Uses your default production branch to ensure you're checking against the latest source configurations

Command Settings

The template uses a single command to monitor your data sources:

• dbt source freshness: Validates the freshness of all configured source tables by:

  • Checking the timestamp in each table's loaded_at_field

  • Comparing this timestamp against your defined warning and error thresholds

  • Reporting success or failure based on these comparisons

For additional source freshness configurations, see Source Freshness documentation.

Trigger Type

• Type: Scheduled Run (Cron)

• Cron Schedule: 0 */2 * * * (Every 2 hours, starting at minute 0 to balance frequent data updates and reasonable resource usage)

For custom Trigger configurations, see Trigger Types documentation.

Notification Settings

• Email Alerts:

  • Success: Confirms all sources are within freshness thresholds

  • Failure: Alerts when sources exceed error thresholds

  • SLA Breach: Alerts when the freshness check itself exceeds 120 minutes

For custom notification configurations, see Notification Settings documentation.

When to Customize

Tailor this template to your specific needs:

• Adjust trigger type based on SLA requirements:

  • Hourly checks for critical sources (0 * * * *)

  • Daily checks for stable sources (0 0 * * *)

  • Weekly checks for historical analysis (0 0 * * 0)

• Add notification destinations (Slack, MS Teams) for team collaboration