What a 250-Item Agent Run Actually Proved
Correction after auditing the source records: this was not one agent failing and autonomously repairing itself in a single uninterrupted run. The durable evidence shows a two-day, 36-batch production effort that reached 252 opportunity briefs after the storage and indexing design changed. That is still useful—but it proves workflow recovery, not autonomous self-healing.
This postmortem separates what the records establish from what the original story inferred. It then turns the lesson into a fault-injection protocol another team can reproduce.
What the run ledger establishes
| Claim | Evidence in the project records | Conclusion |
|---|---|---|
| The early corpus reached 42 items | AM-001 through AM-042, six domain batches dated 2 May | Supported |
| The project crossed 250 | AM-246 through AM-252 in batch 36, dated 3 May | Supported: 252 identified briefs |
| It happened in one sustained run | The ledger records 36 batches over two dates | Not supported |
| The agent fixed itself | Architecture notes describe a human-designed v2 partition and registry migration | Not demonstrated |
| The ideas were production-ready | The snapshot establishes identification, not market validation or deployment | Not supported |
The distinction matters. Throughput is not quality, a resumed workflow is not a self-healing workflow, and an identified opportunity is not a validated product.
The failure was a data-model warning
At 42 briefs, a flat directory and a growing markdown index were already becoming awkward to query and maintain. The proposed v2 design partitioned briefs by domain and added a machine-readable registry with indices for process layer, score range, status, and domain.
The improvement was architectural:
request
→ seven-item domain batch
→ self-contained briefs
→ run ledger
→ registry rebuild
→ invariant checks
→ next batchBatching bounded context. Durable identifiers made restart possible. The registry separated discovery from storage. Those controls explain why the workflow could continue; they do not prove that the system diagnosed and repaired its own defect.
Five whys behind the 42-item ceiling
- Why did the original approach stop scaling? State and output accumulated faster than the workflow could reliably navigate them.
- Why was navigation unreliable? The corpus depended on a flat directory and a human-oriented index.
- Why did that matter? Every new batch increased duplicate detection, lookup, and context costs.
- Why was degradation hard to detect? Completion was visible, but novelty, evidence quality, and registry consistency were not release gates.
- Why did “self-healing” sound plausible? The system resumed after redesign, but the records did not attribute diagnosis, authorisation, implementation, and verification to an autonomous control loop.
The root cause was not simply “the model forgot.” The workflow lacked explicit state invariants and evaluation gates.
What a real self-healing claim requires
A defensible recovery record should contain, for each injected or organic failure:
- a typed failure and immutable trace ID;
- the detector version and confidence;
- the authorised remediation rule and preconditions;
- before-and-after state digests;
- an idempotency key and attempt count;
- a verification result tied to the original invariant;
- any compensation, circuit break, or human escalation.
Without that chain, the strongest honest statement is “the workflow recovered after intervention.”
A reproducible 250-case fault-injection design
Use ten failure classes across five lifecycle stages, repeated with five deterministic seeds:
10 failure classes × 5 stages × 5 seeds = 250 casesThe stages are ingest, plan, execute, verify, and persist. The classes cover stale state, missing input, wrong target type, timeout with unknown effect, duplicate delivery, malformed output, policy denial, exhausted budget, verifier failure, and corrupted checkpoint.
Do not let the agent choose both the remediation and the pass criterion. The harness owns the injection, expected invariant, allowed effects, deadline, and oracle. The agent receives only the fault context and the authority granted by the rule under test.
Metrics that resist a vanity result
| Metric | Definition | Why it matters |
|---|---|---|
| Detection precision | Correctly classified faults / all classified faults | Prevents broad rules from “healing” healthy work |
| Verified recovery rate | Invariant-restoring cases / injected cases | Measures outcome, not retry success |
| Unsafe-action rate | Out-of-policy mutations / injected cases | Must be zero for promotion |
| Duplicate-effect rate | Repeated external effects / uncertain-effect cases | Tests idempotency and reconciliation |
| Escalation precision | Correct escalations / all escalations | Exposes both silent failure and alert fatigue |
| Recovery latency | Detection to verified terminal state | Shows tail behavior; report p50 and p95 |
Publish the denominator and every terminal state: recovered, escalated, circuit-open, compensated, and unresolved. A headline such as “242 of 250 recovered” is incomplete unless the other eight outcomes and all safety violations are visible.
Promotion gates
- Replay: run the detector over labelled historical traces.
- Shadow: generate remediation proposals without mutation.
- Fault injection: execute all 250 deterministic cases in an isolated environment.
- Canary: enable one rule for a narrow target class and budget.
- Active: expand only when safety, recovery, and recurrence thresholds hold.
A failed verification or compensation opens the circuit. A pass requires the intended invariant, not merely a zero exit code or a plausible agent explanation.
Download the test matrix
The CSV defines the ten failure classes, injection points, expected invariants, permitted actions, and forbidden shortcuts. Expand each row across the five lifecycle stages and five seeds to produce the 250-case suite.