Self-Healing Agent Systems: A Bounded Remediation State Machine
Self-healing is not “catch an error and try something else.” It is a bounded state machine that detects a known condition, proves preconditions, executes one authorised remediation, verifies the invariant, compensates if needed, and escalates before uncertainty spreads.
The live OpenClaw workspace has three documented rules—stale edit, missing path, and directory read. Auditing them shows why rule documentation alone is not yet safe runtime automation.
The rule audit
| Rule | Documented recovery | Safety gap |
|---|---|---|
| Stale edit | Re-read, retry, then replace the whole file | Whole-file replacement expands the mutation and may overwrite concurrent work |
| Missing path | Create parents and an empty file, then retry | A failed read does not imply authority or intent to create data |
| Directory read | List files instead | A directory listing is not semantically equivalent to requested file content |
All three are marked auto-recoverable, but their verification checklists are incomplete. One stale-edit rule records 37 occurrences against a threshold of five while auto-filed remains false. The documents capture valuable patterns; they do not prove the wrapper, reporting, recovery, or learning path is wired end to end.
The remediation state machine
observed
│
▼
classified ──unknown──▶ escalated
│
known + confident
▼
preflight ──fail──────▶ escalated
│
▼
remediating ──timeout──▶ uncertain ──reconcile──▶ escalated
│
▼
verifying ──pass──────▶ recovered
│
└──fail──▶ compensating ──success──▶ escalated
└──failure──▶ circuit_openEvery transition records rule version, trace ID, attempt, actor, timestamps, and evidence. recovered is terminal only for that attempt; a recurrence may open a systemic issue.
Prefer typed errors over regex stories
Match a structured tool error code plus context:
{
"code": "STALE_PRECONDITION",
"tool": "file_edit",
"target": "repo://project/file.py",
"expectedDigest": "sha256:…",
"actualDigest": "sha256:…",
"effectState": "not_attempted"
}Free-form error text changes across versions and may contain untrusted content. Regex can support legacy adapters, but the classifier should emit a versioned internal failure type and confidence. Low confidence routes to observation or escalation, not mutation.
Preconditions carry the safety argument
Before remediation, verify:
- the original action was authorised;
- the current target and environment match the rule scope;
- the external effect state is known;
- the intended action remains valid after refreshing state;
- the remediation stays within file, line, cost, and time limits;
- compensation is defined for any mutation.
For ENOENT, preflight must distinguish “expected existing file is missing” from “caller intends to create a new file.” The safe default for a read is to report missing—not to manufacture an empty success.
Design idempotency around the intended effect
Use a stable key such as:
trace_id + original_tool_call_id + rule_versionPersist attempts and terminal effect state. If a timeout occurs after an external mutation, move to uncertain and reconcile before retrying. This prevents duplicate messages, deployments, payments, or writes.
Bound the remediation
| Boundary | Example control |
|---|---|
| Attempts | One retry after refreshed state |
| Scope | One declared file or provider operation |
| Magnitude | Maximum changed lines, bytes, records, or cost |
| Time | Per-step and total deadline |
| Authority | No new tools, secrets, destinations, or permissions |
| Fallback | Escalate; never substitute a more destructive action |
For stale edits, re-read and recompute the minimal patch. If intent no longer maps cleanly, stop. Replacing the entire file because an exact patch failed is not a fallback; it is a larger operation with a different risk profile.
Verification must test the invariant
A successful tool response is insufficient. Verify:
- the intended invariant now holds;
- the artifact parses, compiles, or passes the targeted check;
- only authorised state changed;
- the original failure no longer reproduces;
- no external effect was duplicated;
- evidence is retained for the issue control plane.
For directory reads, the invariant may be “caller receives a typed path-kind mismatch,” not “some output was returned.” Listing a directory can be offered as a separate next action.
Compensation is not always rollback
Local file bytes can often be restored. An email cannot be unsent and a payment cannot be unmade. Classify effects:
| Effect | Failure response |
|---|---|
| Reversible local state | Restore captured pre-change state and verify |
| Provider supports cancellation | Cancel with receipt, then reconcile |
| Compensatable business effect | Execute a separately authorised compensating action |
| Irreversible effect | Contain, notify, preserve evidence; never claim rollback |
Circuit breakers stop recursive harm
Open the rule's circuit when:
- verification fails twice in a short window;
- one compensation or rollback fails;
- the same target recurs three times in one run;
- failure rate or cost exceeds its budget;
- classifier confidence drifts below threshold.
An open circuit disables remediation, continues safe observation where possible, and creates an owned issue. Reset requires evidence and owner review—not merely elapsed time.
Promote rules safely
- Replay: evaluate the detector against historical failures and normal traffic.
- Shadow: propose remediations without executing them.
- Canary: enable for a narrow scope with immediate evidence.
- Active: expand only after precision, recovery, and no-harm thresholds pass.
- Retire: disable rules whose underlying failure class was eliminated.
Track detector precision, precondition rejection, verified recovery, compensation, circuit-open, recurrence, and change-induced incident rates by rule version.