Event freshness anomalies

The elementary.event_freshness_anomalies test monitors the freshness of event data over time, measuring the expected time it takes each event to load. This is calculated as the time between when the event actually occurs (the event timestamp) and when it is loaded to the database (the update timestamp).

This test complements the freshness_anomalies test and is primarily intended for data that is updated in a continuous or streaming fashion.

How it works

The test can operate in two modes:

1. If only an event_timestamp_column is supplied:

   • The test measures the difference between the current timestamp ("now") and the most recent event timestamp.

2. If both an event_timestamp_column and an update_timestamp_column are provided:

   • The test measures the difference between these two columns.

Models

models:
  - name: < model name >
    tests:
      - elementary.event_freshness_anomalies:
          event_timestamp_column: < timestamp column > # Mandatory
          update_timestamp_column: < timestamp column > # Optional
          where_expression: < sql expression >
          time_bucket: # Daily by default
            period: < time period >
            count: < number of periods >

Models example

models:
  - name: login_events
    tests:
      - elementary.event_freshness_anomalies:
          event_timestamp_column: "occurred_at"
          update_timestamp_column: "updated_at"
          # optional - use tags to run elementary tests on a dedicated run
          tags: ["elementary"]
          config:
            # optional - change severity
            severity: warn

Test configuration

tests:
  — elementary.event_freshness_anomalies:
    event_timestamp_column: column name
    update_timestamp_column: column name
    where_expression: sql expression
    anomaly_sensitivity: int
    detection_period:
      period: [hour | day | week | month]
      count: int
    training_period:
      period: [hour | day | week | month]
      count: int
    time_bucket:
      period: [hour | day | week | month]
      count: int
    seasonality: day_of_week
    detection_delay:
      period: [hour | day | week | month]
      count: int
    ignore_small_changes:
      spike_failure_percent_threshold: int
      drop_failure_percent_threshold: int
    anomaly_exclude_metrics: [SQL expression]

Important Notes

• Required configuration: event_timestamp_column is mandatory.

• Default configuration: anomaly_direction: spike to alert only on delays.