Volume anomalies

elementary.volume_anomalies

Monitors the row count of your table over time per time bucket (if configured without timestamp_column, will count table total rows).

Upon running the test, your data is split into time buckets (daily by default, configurable with the time bucket field), and then we compute the row count per bucket for the last training_period days (by default 14).

The test then compares the row count of each bucket within the detection period (last 2 days by default, configured as detection_period), and compares it to the row count of the previous time buckets.

The test will only run on completed time buckets, so if you run it with daily buckets in the middle of today, the test would only count yesterday as a complete bucket. If there were any anomalies during the detection period, the test will fail.

Models

models:
  - name: < model name >
    tests:
      - elementary.volume_anomalies:
          timestamp_column: < timestamp column >
          where_expression: < sql expression >
          time_bucket: # Daily by default
            period: < time period >
            count: < number of periods >

Models example

models:
  - name: login_events
    config:
      elementary:
        timestamp_column: "loaded_at"
    tests:
      - elementary.volume_anomalies:
          where_expression: "event_type in ('event_1', 'event_2') and country_name != 'unwanted country'"
          time_bucket:
            period: day
            count: 1
          # optional - use tags to run elementary tests on a dedicated run
          tags: ["elementary"]
          config:
            # optional - change severity
            severity: warn

  - name: users
    # if no timestamp is configured, elementary will monitor without time filtering
    tests:
      - elementary.volume_anomalies:
          tags: ["elementary"]

Test configuration

No mandatory configuration, however it is highly recommended to configure a timestamp_column.

tests:
  — elementary.volume_anomalies:
    timestamp_column: column name
    where_expression: sql expression
    anomaly_sensitivity: int
    anomaly_direction: [both | spike | drop]
    detection_period:
      period: [hour | day | week | month]
      count: int
    training_period:
      period: [hour | day | week | month]
      count: int
    time_bucket:
      period: [hour | day | week | month]
      count: int
    seasonality: day_of_week
    fail_on_zero: [true | false]
    ignore_small_changes:
      spike_failure_percent_threshold: int
      drop_failure_percent_threshold: int
    detection_delay:
      period: [hour | day | week | month]
      count: int
    anomaly_exclude_metrics: [SQL expression]