Freshness anomalies

elementary.freshness_anomalies

Monitors the freshness of your table over time, as the expected time between data updates.

Upon running the test, your data is split into time buckets (daily by default, configurable with the time bucket field), and then we compute the maximum freshness value per bucket for the last training_period (by default 14 days).

The test then compares the freshness of each bucket within the detection period (last 2 days by default, controlled by the detection_period var), and compares it to the freshness of the previous time buckets. If there were any anomalies during the detection period, the test will fail.

Models

models:
  - name: < model name >
    tests:
      - elementary.freshness_anomalies:
          timestamp_column: < timestamp column > # Mandatory
          where_expression: < sql expression >
          time_bucket: # Daily by default
            period: < time period >
            count: < number of periods >

Models example

models:
  - name: login_events
    tests:
      - elementary.freshness_anomalies:
          timestamp_column: "updated_at"
          # optional - use tags to run elementary tests on a dedicated run
          tags: ["elementary"]
          config:
            # optional - change severity
            severity: warn

Test configuration

Required configuration: timestamp_column Default configuration: anomaly_direction: spike to alert only on delays.

tests:
  — elementary.freshness_anomalies:
    timestamp_column: column name
    where_expression: sql expression
    anomaly_sensitivity: int
    detection_period:
      period: [hour | day | week | month]
      count: int
    training_period:
      period: [hour | day | week | month]
      count: int
    time_bucket:
      period: [hour | day | week | month]
      count: int
    detection_delay:
      period: [hour | day | week | month]
      count: int
    ignore_small_changes:
      spike_failure_percent_threshold: int
      drop_failure_percent_threshold: int
    anomaly_exclude_metrics: [SQL expression]