Issue Management as a Reliability Control Plane
An alert is an event. An issue is a durable control object. It connects impact, evidence, hypotheses, ownership, corrective actions, verification, and recurrence across many alerts and deployments.
This is the role of an issue-management control plane: preserve accountability and system learning after the runtime emergency ends.
Keep signals, incidents, problems, and changes separate
| Object | Purpose | Typical lifetime |
|---|---|---|
| Signal | One observation: log, metric, trace, failed check, user report | Seconds to retention period |
| Incident | Coordinate restoration for active user impact | Minutes to days |
| Issue | Explain and eliminate a recurring or material failure class | Days to months |
| Change | Implement one containment, detection, recovery, or prevention action | One delivery lifecycle |
Many signals may map to one issue. One issue may produce several changes. Closing an incident because service recovered must not close the deeper issue automatically.
Fingerprint the failure class
A useful fingerprint uses stable causal features, not volatile text:
fingerprint = hash(
affected_journey,
failure_class,
component_boundary,
normalized_error_code,
violated_invariant
)Exclude timestamps, request IDs, model prose, and full stack traces. Those belong in evidence. The fingerprint should group the same failure while keeping distinct causes apart.
When a new signal matches a closed issue, increment recurrence and reopen it under policy. Do not create an unrelated ticket that hides the repeat.
The issue lifecycle
detected → triaged → contained → diagnosing
│
▼
remediating
│
▼
verifying
┌───┴────┐
▼ ▼
resolved reopened
│
▼
closedresolved means corrective actions passed their checks. closed means the verification window elapsed without recurrence and required learning artifacts were published. A matching failure during verification moves the issue to reopened.
Evidence before explanation
Attach immutable references to relevant logs, traces, metrics, artifacts, receipts, and tests. Record digests so later reviewers know whether evidence changed.
Maintain competing hypotheses:
| Hypothesis field | Why it exists |
|---|---|
| Causal statement | Precise, falsifiable explanation |
| Evidence for | Observations predicted by the hypothesis |
| Evidence against | Contradictions that must be explained |
| Status | Open, supported, or rejected |
| Reproduction | Controlled path from cause to failure |
“The model hallucinated” and “bad deploy” are symptoms unless they identify the violated invariant and the system condition that allowed harm.
Use five whys carefully
Five Whys is a prompt for deeper inquiry, not proof that every incident has one linear root. Stop when evidence stops. Branch when independent contributing causes exist.
- Why did the user journey fail?
- Why did the system permit that local failure to escape?
- Why did detection or containment not stop it?
- Why was the vulnerable design or policy accepted?
- Why did organisational feedback not correct it earlier?
The result may be a causal graph: triggering event, latent condition, missing control, and amplification path.
Corrective actions need distinct classes
| Class | Goal | Example |
|---|---|---|
| Containment | Limit current harm | Disable one effectful tool |
| Detection | Find the condition sooner | Add stale-state invariant check |
| Recovery | Restore safely and faster | Reconcile unknown effects before retry |
| Prevention | Remove or reduce the causal path | Add optimistic concurrency control |
Each action needs an owner, due date, acceptance condition, and evidence. “Update documentation” cannot be the only prevention for a machine-enforceable invariant.
Verification is not deployment
A merged change proves only that code changed. Verification should include:
- a regression reproducer that failed before and passes now;
- integration coverage at the affected journey boundary;
- observability showing the control is active;
- a defined recurrence window appropriate to traffic volume;
- negative checks for newly introduced failure modes.
Low-volume workflows may need replay, simulation, or fault injection because waiting for organic traffic would leave the issue open indefinitely.
Agent-specific issue dimensions
In addition to conventional service fields, capture:
- workflow, model, prompt, toolset, policy, and verifier versions;
- authority and approval state;
- semantic failure label and evaluation evidence;
- external effect state: confirmed, rejected, unknown, or duplicated;
- memory or retrieval IDs that influenced the decision.
This lets the control plane join a plausible but wrong output to the exact configuration and evidence chain that produced it.
Metrics for learning, not ticket throughput
| Metric | Interpretation |
|---|---|
| Recurrence rate by fingerprint | Whether the failure class returns |
| Reopen rate | Whether verification was premature |
| Time to containment | Speed of limiting harm |
| Time to supported cause | Speed of evidence-backed diagnosis |
| Corrective-action escape rate | Changes that failed to remove the issue |
| Detection lead-time gain | How much earlier the updated control catches recurrence |
Ticket closure count rewards throughput. Recurrence elimination rewards reliability.
How this differs from self-healing
The runtime self-healing loop detects a known condition and executes a bounded remediation. The issue control plane decides whether the condition is understood, who owns systemic prevention, what evidence supports closure, and whether the same failure came back. The runtime loop is a consumer and producer of issue records—not their replacement.