Agent Memory Architecture: Admission, Retrieval, Consolidation, and Forgetting
Agent memory is not “put the chat in a vector database.” It is a governed lifecycle that decides what may be remembered, how it is represented, when it is retrieved, how contradictions are resolved, and when it must disappear.
Without those decisions, memory can make an agent worse: stale instructions outrank current policy, private facts cross tenants, summaries lose provenance, and irrelevant retrieval consumes the context it was meant to save.
Four memory classes, four jobs
| Class | Question answered | Typical lifetime |
|---|---|---|
| Working | What does this turn need right now? | One task or session |
| Episodic | What happened in this run, incident, or decision? | Days to months |
| Semantic | What reusable fact, constraint, or pattern is believed? | Until review, supersession, or deletion |
| Procedural | How should a scoped workflow be performed? | Versioned policy lifetime |
Do not search every class with one ranking function. Working memory is injected by task state. Episodes need actor, project, and time filters. Semantic memory benefits from hybrid retrieval. Procedures require exact scope and version compatibility.
The lifecycle is the architecture
observation
│
▼
candidate ──admit──▶ active ──merge──▶ consolidated
│ │ │ │
reject │ ├──replace──────▶ superseded
│ └──doubt────────▶ quarantined
│
└──expire / erase──▶ deletedEach transition should be explicit and auditable. A record is not durable merely because it was written. It becomes active only after admission. Consolidation creates a stable synthesis while preserving source links. Supersession keeps history without allowing old instructions to compete. Quarantine removes doubtful material from retrieval. Deletion produces a receipt and propagates to indexes and caches.
Admission: remember less, but remember it well
A candidate memory needs:
- a future decision or action it can improve;
- source URI and observation time;
- owner, tenant or project scope, and sensitivity;
- confidence and whether the statement is observed, inferred, or prescribed;
- review or expiry time;
- deduplication and contradiction checks.
Reject raw speculation phrased as fact, secrets, disallowed personal data, and “notes to self” with no reusable value. Store the evidence separately when it is large; keep a digest and resolvable reference in the memory record.
Retrieval is an authorization decision
Filter before ranking:
authorized scope
∩ project or tenant
∩ compatible memory class
∩ active state
∩ not expired
│
▼
hybrid candidate retrieval
│
▼
rank by relevance + source quality + recency
+ successful use − contradiction penaltyEvery returned item should carry its memory ID, provenance, confidence, observation time, and state. The model needs to know whether it is seeing a current policy, an old episode, or an uncertain inference.
Cap the memory token budget independently from the task-input budget. A retrieval system that always fills the context window turns “available memory” into mandatory noise.
Consolidation must preserve disagreement
When several episodes support one lesson, consolidation can reduce duplication. But do not flatten conflicts into a confident average. Keep:
- the consolidated statement;
- supporting and contradicting source IDs;
- scope and exceptions;
- the consolidator version;
- human approval where the memory becomes policy.
If two memories prescribe incompatible actions under the same scope, quarantine or surface the conflict. Ranking one slightly higher is not conflict resolution.
Forgetting is a feature
Decay should lower retrieval priority or trigger reverification; it should not silently rewrite historical evidence. Use a stable base score and a last_evaluated_at checkpoint.
priority_now =
base_quality
× freshness_factor(time since last verification)
× use_factor(successful uses, corrections)
× source_factor
− contradiction_penaltyThis avoids a subtle failure I found in the local ALMS implementation: its decay function derives lifetime decay from creation time, then subtracts that amount from the already-decayed current score. Repeated sweeps can therefore punish the same elapsed time more than once. The safe alternatives are to recompute from an immutable base or apply only the delta since the last checkpoint.
Pinning should be rare and reviewed. Safety policy, contractual constraints, and human-approved durable decisions may justify it. “This was useful once” does not.
Deletion needs propagation and proof
Soft deletion is helpful for audit and recovery, but it is not the end of erasure. A deletion workflow should cover:
- primary record state;
- vector and lexical indexes;
- materialized summaries and consolidated memories;
- agent-local caches and sync queues;
- backups under the retention policy;
- a receipt recording what was removed, when, and by which authority.
Measure deletion-propagation latency. “Deleted from the database” is incomplete if the record remains retrievable from an embedding index.
Evaluate memory as a treatment
Run the same task set under three conditions: no memory, retrieval only, and retrieval plus consolidation. Segment by workflow and memory class.
| Metric | What it detects |
|---|---|
| Precision@k | Retrieved items that were actually relevant |
| Helpful recall@k | Known useful memories that were retrieved |
| Task-success delta | Whether memory improves the outcome |
| Contradiction rate | Mutually incompatible items shown together |
| Stale-recall rate | Expired or superseded memories reaching context |
| Sensitive-recall rate | Unauthorized records crossing the boundary |
| Retrieval token overhead | Context spent to obtain the benefit |
| Correction latency | Time from reported error to retrieval-safe repair |
Do not optimise retrieval relevance alone. A highly relevant false memory can do more damage than an irrelevant one because the model is more likely to act on it.
The economics of “token rent”
avoided cost =
successful reuses × (reconstruction cost − retrieval cost)
+ avoided human reconstruction time
net memory value =
avoided cost
− extraction, storage, indexing, retrieval, evaluation,
correction, deletion, and wrong-recall damageUse provider bills and measured human time, not generic token prices. A memory system can reduce prompt tokens while increasing total cost through extraction models, embeddings, review, and incident cleanup.
The decisive metric is not “memories stored.” It is successful, attributable reuse without unacceptable stale, contradictory, or sensitive recall.
What the ALMS implementation contributes
The local Agent Learning Management System already implements several sound primitives: shared learning records, sync plus acknowledgement, pinning, scoring, TTL, soft deletion, protocol distribution, and an architecture that stays out of the agent's hot path. The next maturity step is to make admission state, provenance-aware retrieval, contradiction handling, and decay checkpoints equally explicit.