Service Level Objectives (SLOs)

SLOs are promises about your data's reliability. They answer the question every stakeholder asks: "Can I trust this data?"

Think of SLOs like a delivery guarantee. When you order next-day delivery, the retailer promises your package arrives within 24 hours. If it doesn't, they notify you and fix it. SLOs do the same for data — they promise freshness, completeness, and quality, and alert your team when those promises are broken.

SLOs can be defined at three levels, with contract-level taking highest precedence:

_domain.yaml (defaults)  →  _system.yaml (overrides)  →  contract.yaml (overrides)

Contract values deep-merge with system/domain SLOs — field by field.

Freshness

Business value: Ensures your dashboards and reports are never showing stale data. If the marketing team is making campaign decisions based on yesterday's data when they think it's today's — that's a freshness SLO breach.

Analogy: Like checking the "use by" date on food. If the data's timestamp is older than the threshold, it's stale and shouldn't be served to consumers.

Example: Customer data must be refreshed daily

service_levels:
  freshness:
    threshold: "24h"              # "30m", "6h", "1d"
    field: "updated_at"           # MAX of this column vs current time
    description: "Customer data must be updated daily"

    # Source-time freshness (catches stale upstream data)
    source_field: "event_timestamp"
    source_threshold: "2h"

Two checks in one:

Check	What It Catches
`threshold` + `field`	"Our pipeline hasn't run in 24 hours"
`source_threshold` + `source_field`	"The upstream system stopped sending data 2 hours ago"

The second check is critical — without it, your pipeline could run on time but process zero new records, and you'd never know.

Availability (Field Completeness)

Business value: Guarantees that critical fields actually have values. A customer table with 30% missing email addresses is technically "fresh" but practically useless for email campaigns.

Analogy: Like checking that a form has been filled in completely. A job application without a name is valid structurally, but useless operationally.

Example: 99.9% of rows must have a customer_id

  availability:
    threshold: 99.9               # % of rows with non-null value
    field: "customer_id"

Row Count

Business value: Catches two common silent failures: (1) an upstream source that suddenly stops sending data, and (2) a misconfigured filter that accidentally drops 90% of rows. Both would pass quality rules but produce a suspiciously small dataset.

Analogy: Like a restaurant checking the morning delivery. If you usually get 500 eggs and today you got 12, something went wrong upstream — even if all 12 eggs are perfect.

Example: Row count bounds with anomaly detection

  row_count:
    min_rows: 100
    max_rows: 10000000
    check_field: "counts_source"   # counts_source | counts_good | counts_total
    skip_reprocess_days: 3         # Skip checks on large backfills
    warn_only: false               # true = log warning, false = fail pipeline

    # Anomaly detection (compares against recent history)
    anomaly:
      enabled: true
      lookback_runs: 14            # Compare against last 14 runs
      min_ratio: 0.5               # Alert if < 50% of baseline
      max_ratio: 2.0               # Alert if > 200% of baseline
      method: "seasonal_median"    # median | rolling_average | seasonal_median
      lookback_days: 35            # seasonal_median: days of history
      critical_ratio: 0.3          # below 30% of baseline (or zero rows) = critical
      min_runs_before_enforcement: 3

Anomaly detection is the smart version of min/max — instead of hard-coding thresholds, it learns your baseline and alerts on deviations. This catches gradual drift (data volume slowly declining) that fixed thresholds miss.

How the baseline is built:

Per environment. A run is compared only with earlier runs from its own environment, so a small dev run is never judged against prod volume, even when both write to one run log.
seasonal_median compares a run with the same weekday over lookback_days, because weekends are often quieter by design. With fewer same-weekday runs than min_runs_before_enforcement, it uses the median of the whole window instead. Runs that wrote 0 rows are left out of the baseline.
median / rolling_average use the last lookback_runs runs, zeros included — they also baseline ratio expressions, where 0 is a real value.
Severity. A breach is a warning. A drop below critical_ratio, or to zero rows, is critical. The result carries anomaly_severity with the expected count, floor, ceiling, method, sample count and environment it was judged against.

Quality

Business value: Sets the minimum "pass rate" for your data. If your order table has a 15% quarantine rate, something is fundamentally wrong with the source — even if each individual quality rule is working correctly.

Analogy: Like a factory's defect rate. Even if every individual inspection catches bad products, a 20% defect rate means the manufacturing process itself needs fixing.

Example: At least 99% of rows must pass all quality rules

  quality:
    min_good_ratio: 0.99          # 99% of rows must be clean
    max_quarantine_ratio: 0.01    # No more than 1% quarantined

Schedule (retired)

slo.schedule — an expected run window — is no longer supported. It was hard to capture and harder to keep true, and a window that is not yours raises false alerts or hides real lateness. Freshness answers the question that matters to a consumer (is the data recent enough?), so use it instead. An existing schedule: block still loads; it is ignored with a warning.

Run Log Capture

Every SLO result is automatically recorded in the run log table (metadata.run_log_table) — giving you a full audit trail of data reliability over time.

Field	Description
`slo_freshness_seconds`	Time since newest source record
`slo_freshness_pass`	Whether freshness threshold was met
`slo_row_count_min` / `max`	Configured thresholds
`slo_row_count_anomaly_pass`	Whether anomaly check passed
`slo_quality_pass`	Whether quality ratio met threshold

Business value: This run log data is what powers your data reliability dashboard. Instead of asking "is our data trustworthy?", your team can show stakeholders a chart of SLO compliance over the last 90 days — just like an uptime SLA for a web service.

SLO breaches emit "slo_breach" notification events that route through the Notifications system — so the right people get alerted immediately.