Agent Reliability SLOs: Measure Correctness, Safety, and Recovery
An agent can return HTTP 200, finish on time, and still fail the user. Availability and latency remain necessary, but agent reliability also includes semantic correctness, policy compliance, effect integrity, evidence completeness, and recoverability.
The practical response is not a bigger dashboard. It is a small set of user-centered service level indicators (SLIs), explicit service level objectives (SLOs), and an error-budget policy that changes engineering behavior.
Start with one eligible journey
Define the unit before defining the percentage. For an outreach agent, one journey might be:
An authorised request is accepted, the correct recipient and message are verified, exactly one message is sent, the provider receipt is reconciled, and the required evidence is retained.
The denominator is every eligible accepted journey. Eligibility must be machine-checkable: supported intent, valid authority, and required inputs present at acceptance time. Unsupported requests rejected before acceptance can be excluded; failures discovered after acceptance cannot.
Denominator policy is production code. If a team can relabel failures as “unsupported” after the fact, the SLI becomes a vanity metric.
Use an end-to-end success SLI
verified_task_success =
eligible journeys with every acceptance check passing
-------------------------------------------------------
all eligible journeysMeasure this at the user boundary, not by multiplying component availability figures. Agent-stage failures are correlated: a malformed retrieval result can poison planning, tool choice, and verification together.
Keep diagnostic component metrics—model errors, tool latency, retrieval misses—but do not mistake them for the journey SLI.
The minimum agent reliability scorecard
| SLI | Good event | Bad event |
|---|---|---|
| Verified task success | Every acceptance check passes | Wrong, incomplete, unverifiable, or timed-out outcome |
| Forbidden-effect avoidance | No mutation outside policy or delegated scope | Any unauthorised or prohibited effect |
| Duplicate-effect avoidance | Each intended mutation occurs once | Duplicate payment, message, deployment, or deletion |
| Evidence completeness | Every required decision and effect has a resolvable evidence reference | Missing, inaccessible, or mismatched evidence |
| Recovery within objective | Failure is restored or safely terminated before the recovery-time objective | Unresolved or late recovery |
| Decision latency | Verified result arrives before the journey threshold | Threshold exceeded, including verification time |
Forbidden effects deserve a separate safety policy. A 99.9% target does not make one unauthorised payment acceptable. Record the rate for observability, but treat every miss as an incident rather than “spendable budget.”
Set targets from user harm and operating cost
There is no universal “three nines” for agents. Segment by journey and risk:
- a draft summariser may tolerate abstention and human correction;
- an internal data lookup needs high provenance completeness;
- an external messaging workflow needs near-zero duplicate effects;
- a financial or deletion workflow needs strict authority and reconciliation.
For each SLO, document the user consequence of a miss, the cost of prevention, the measurement source, and the owner. Set a target below 100% where an error budget is meaningful. Google's SRE guidance similarly warns that demanding 100% can produce excessive cost and slower change; the target should support a deliberate risk decision in the service-level objective.
Error budgets turn percentages into decisions
allowed_bad_ratio = 1 - SLO
error_budget_events = eligible_events × allowed_bad_ratio
burn_rate = observed_bad_ratio ÷ allowed_bad_ratioExample: with 10,000 eligible monthly journeys and a 99% verified-success SLO, the budget is 100 bad journeys. Forty failures consume 40% of the budget. A burn rate of 2 means the current failure ratio would exhaust the budget twice as fast as planned.
An error-budget policy should say what happens:
- healthy budget: normal feature delivery;
- fast burn: page and mitigate;
- slow sustained burn: create owned reliability work;
- budget exhausted: freeze risky releases and prioritise recovery.
Alert on burn, not every bad output
For sufficiently high-volume journeys, multi-window burn-rate alerts balance detection speed and noise. The Google SRE Workbook recommends this approach and gives 14.4× over one hour and 6× over six hours as starting points for paging, with shorter confirmation windows.
| Signal | Example action | Purpose |
|---|---|---|
| 14.4× burn over 1h and 5m | Page | Catch acute budget loss |
| 6× burn over 6h and 30m | Page | Catch sustained severe degradation |
| 1× burn over 3d | Ticket | Catch slow exhaustion |
| Any forbidden effect | Safety incident | Respond to harm, independent of volume |
Low-volume agents need different treatment. One failure may appear as an enormous burn rate with little statistical confidence. Show raw counts, confidence intervals, and longer windows; page directly only for high-severity event classes.
Semantic evaluation must be versioned
A model grader is another fallible component. Store:
- verifier version and rubric digest;
- the evidence it evaluated;
- score, threshold, and reason codes;
- human-review result for an independent sample;
- disagreement rate by journey and risk class.
Run calibration sets before changing the verifier. Where possible, blind reviewers to model version and treatment. A rising SLI caused by a more permissive grader is measurement drift, not improved reliability.
Instrument the causal path
Every journey should carry a stable trace ID and immutable configuration dimensions:
workflow_version
model_provider + model_id
prompt_digest
toolset_digest
policy_digest
tenant_or_risk_class
verifier_version
accepted_at → started_at → first_effect_at
→ completed_at → verified_atAvoid raw prompts, personal data, and secrets in metric labels. Keep detailed evidence in access-controlled storage and expose low-cardinality digests in telemetry.
Review the scorecard as an operating contract
A useful monthly review answers five questions:
- Which user journey consumed the most budget?
- Which cohort was hidden by the aggregate?
- Did verifier or denominator policy change?
- Which incident added a new detection or recovery control?
- What release decision follows from the remaining budget?
If no engineering decision changes when the SLO misses, it is reporting—not an objective.