Event freshness anomalies

elementary.event_freshness_anomalies

Monitors the freshness of event data over time, as the expected time it takes each event to load - that is, the time between when the event actually occurs (the event timestamp), and when it is loaded to the database (the update timestamp).

This test compliments the freshness_anomalies test and is primarily intended for data that is updated in a continuous / streaming fashion.

The test can work in a couple of modes:

• If only an event_timestamp_column is supplied, the test measures over time the difference between the current timestamp (“now”) and the most recent event timestamp.

• If both an event_timestamp_column and an update_timestamp_column are provided, the test will measure over time the difference between these two columns.

Models

models:
  - name: < model name >
    tests:
      - elementary.event_freshness_anomalies:
          event_timestamp_column: < timestamp column > # Mandatory
          update_timestamp_column: < timestamp column > # Optional
          where_expression: < sql expression >
          time_bucket: # Daily by default
            period: < time period >
            count: < number of periods >

Models example

models:
  - name: login_events
    tests:
      - elementary.event_freshness_anomalies:
          event_timestamp_column: "occurred_at"
          update_timestamp_column: "updated_at"
          # optional - use tags to run elementary tests on a dedicated run
          tags: ["elementary"]
          config:
            # optional - change severity
            severity: warn

Test configuration

Required configuration: event_timestamp_column Default configuration: anomaly_direction: spike to alert only on delays.

tests:
  — elementary.event_freshness_anomalies:
    event_timestamp_column: column name
    update_timestamp_column: column name
    where_expression: sql expression
    anomaly_sensitivity: int
    detection_period:
      period: [hour | day | week | month]
      count: int
    training_period:
      period: [hour | day | week | month]
      count: int
    time_bucket:
      period: [hour | day | week | month]
      count: int
    seasonality: day_of_week
    detection_delay:
      period: [hour | day | week | month]
      count: int
    ignore_small_changes:
      spike_failure_percent_threshold: int
      drop_failure_percent_threshold: int
    anomaly_exclude_metrics: [SQL expression]