Part I · Understand the problemChapter 2
Anatomy of a coding agent's context
The question Where do the tokens in a coding agent's context actually go, and which parts can you control?
Research contents
- Understand the problem
- The methods
- Measure it
- Decide and avoid
- Act on it
- Reference
Reference
In 30 seconds
- A context has nine segments: four common prefix segments and five growing body segments.
- In this audit model, prefix content is sent on every call; body content grows with the session, and cache billing depends on hits. Compaction or memory policy can rewrite or reposition body context D.
- The constructed audit finds its largest controllable segments in tool definitions and tool output.
You will be able to
- Split any context into prefix and body and explain why that split governs cost
- Estimate the size, owner and failure mode of each of the nine segments
- Rank clean-up actions by size, cost and reversibility
- Track the four numbers that belong on a dashboard
Why break the context down at all#
Chapter 1 gave the constraints; this chapter locates them in the nine segments you can measure and change.
"My context is full" is not something you can act on. A segment-by-segment audit is.
In this constructed example, the instruction file is only 4% of the window; tool definitions occupy 25% for tools the agent rarely calls C.
This chapter splits a coding agent's context into nine segments. For each one it gives the typical size, who controls it, how it fails, how to measure it, and how it behaves in the cache.
Prefix and body#
The nine segments sit in a common prompt layout. Exact ordering, cache behaviour and position effects depend on the harness and provider.
The nine segments, in a common prompt layout
Scroll sideways to see all nine segments.
Use the prefix/body split as an audit model; exact ordering and cache semantics depend on the harness and provider.
The nine segments at a glance#
| # | Segment | Typical size D | Controlled by | Cache behavior D | Typical failure |
|---|---|---|---|---|---|
| 1 | System prompt and harness | 2K–12K | Mostly the vendor | Usually stable; provider-cacheable if unchanged | Invisible; you cannot trim it and often cannot see it |
| 2 | Tool definitions | 1K–60K+ | You (which servers and tools) | Stable when the tool set is stable | Bloat; selection degrades as the set grows |
| 3 | Project instruction files | 0.3K–15K | Entirely you | Stable when unchanged | Grows forever; buried by position; contradicts itself |
| 4 | Skill descriptions | 0.5K–5K | You | Stable when unchanged | Too vague to trigger, or so many they become segment 2 |
| 5 | Retrieved code and files | 2K–80K | Agent plus your tooling | Growing body; often churns | Whole files read for one function; stale after its own edits |
| 6 | Tool results | 5K–150K | Agent plus output shaping | Growing body; often uncached while changing | Flooding, truncation and long retention |
| 7 | Agent reasoning and messages | 3K–60K | Model verbosity, your prompting | Growing body; churns each turn | Self-written distractors; loops |
| 8 | User turns | 0.2K–10K | You | Growing body; unchanged turns may be reused | Vague early, contradictory later |
| 9 | Summaries and memory | 0.5K–8K | Your policy | Compaction may rewrite; memory placement varies | Launders errors; loses the decisive detail |
The ranges are order-of-magnitude bands from observed sessions and reported figures. Treat them as a place to start measuring, not as targets D.
How to act on each segment#
For each segment, the useful questions are the same: how it fails, how to measure it and what to change first. The row number maps to the inventory above.
| # | Decisive risk | Measure | Default action |
|---|---|---|---|
| 1 · System | Fixed prefix tax and the largest harness confound. A scaffold change can look like a model regression. | In a fresh session, send one token and record input tokens; repeat after harness upgrades. | Treat it as a versioned dependency and keep it stable. |
| 2 · Tools | Displacement plus selection confusion. Reported examples reach 42K tokens P; selection degrades as tool count grows S. | Schema tokens, active tool count and defined-to-called ratio across 20 sessions. | Treat zero-call tools as deletion candidates; verify task coverage and keep rollback; use about 20 active tools as a soft ceiling D. |
| 3 · Instructions | Growth, contradiction, staleness and position effects. | Lines, tokens and violations per rule. | Keep under about 150 lines P; retain only what a competent engineer cannot infer in two minutes; scope specialist rules beside their code. |
| 4 · Skills | Vague triggers never fire; too many descriptions recreate tool bloat. | Trigger precision, trigger recall and total always-loaded tokens. | Keep the trigger short; load the procedure on demand. |
| 5 · Retrieved code | Whole-file waste and stale copies after the agent edits the file. | Read-utilisation and read-coverage on failures. | Read symbols or bounded ranges; re-read before re-editing. |
| 6 · Tool results | Can be the largest, low-density body segment; flooding, truncation and long retention. | Result tokens, top commands and retention integral: tokens × turns retained. | One line on success, full trace on failure, full log offloaded to a file. |
| 7 · Agent messages | Narration becomes a plausible self-authored distractor; repeated actions become a loop. | Assistant-token share and identical tool-call repeats. | Persist plans, invariants and ruled-out hypotheses; cut narration. |
| 8 · User turns | Under-specification causes exploration; later changes create clash. | Turns to first productive edit and requirement reversals. | State scope early; after a change, restate the complete current requirement. |
| 9 · Summaries / memory | Lossy re-encoding can launder errors; compaction may rewrite earlier context. | Compactions, compression ratio, post-compaction re-fetch rate and Pass². | Offload first; compact once at a semantic boundary with an explicit schema. |
Four evidence results change the default action and are worth keeping here:
| Evidence | Consequence |
|---|---|
| Tool selection fell from 43% to under 14% as tool count grew; another test went from 19/20 correct at 20 tools to failure at 107 S. | Fewer tools solve both displacement and confusion; deferred schemas solve only displacement D. |
| Input tokens were 99.75–99.87% of usage in tool-heavy agents, and full-context runs used 2.68× the tokens while completing fewer tasks S. | Shape tool output before tuning model output D. |
| Changing only the summariser moved SWE-bench from 49.0% to 55.5% S. | A compactor is a quality component, not plumbing D. |
| On AppWorld, no compression reached 85.7% accuracy / 77.4% Pass²; prompt compaction reached 71.4% / 59.5%; FIFO reached 63.7% / 53.0% S. | Compression damage appears in repeatability before averages D. |
Reading a whole budget: a worked example C#
Here is a constructed 152K-token mid-session snapshot from a large monorepo. It is illustrative C, not a measured incident.
A 152K-token context, by segment
- 1 System prompt9,000 · 6%
- 2 Tool definitions38,000 · 25%
- 3 Instruction files6,500 · 4%
- 4 Skill descriptions1,800 · 1%
- 5 Retrieved code22,000 · 14%
- 6 Tool results58,000 · 38%
- 7 Agent messages12,000 · 8%
- 8 User turns1,700 · 1%
- 9 Summaries3,000 · 2%
npm ci log alone is 14,000 tokens). The instruction file is 4% C.View data
| Segment | Tokens |
|---|---|
| 1 System prompt | 9,000 |
| 2 Tool definitions | 38,000 |
| 3 Instruction files | 6,500 |
| 4 Skill descriptions | 1,800 |
| 5 Retrieved code | 22,000 |
| 6 Tool results | 58,000 |
| 7 Agent messages | 12,000 |
| 8 User turns | 1,700 |
| 9 Summaries | 3,000 |
What a practitioner does with this, in priority order:
- Start with unused tool definitions. In this example, treat zero-call or duplicated capabilities as removal candidates; verify task coverage and keep rollback before deleting, or defer their schemas. This is the largest reversible prefix win in this example.
- Shape the loudest command output. Filter
npm cion success and offload the full log so it does not remain in the body. - Prune the instruction file. Apply the inference test; this changes the rules, so verify the resulting behaviour.
- Change retrieval last. Symbol reads instead of whole files alter how the agent searches, so measure the trade-off rather than treating the saved tokens as free.
Tokens recovered by each action
View data
| Action | Tokens recovered |
|---|---|
| Defer or delete unused tools | 30,000 |
| Filter or offload the npm ci log | 13,000 |
| Inference test on instruction file | 4,000 |
| Symbol reads instead of whole files | 2,000 |
The example shows why the optimisation plan opens with measurement instead of technique: the highest-leverage segment is workload-specific.
The four numbers for your dashboard#
If you track nothing else, track these.
| Metric | Formula | What it tells you | Healthy |
|---|---|---|---|
| Prefix tax | Tokens in segments 1–4 | Fixed input present on each call; cached billing varies | Start below 15K; calibrate to provider |
| Relevance density | Plausibly citable tokens ÷ total | Dilution risk | Start above 15% mid-session; calibrate to workload |
| Retention integral | Σ (segment tokens × turns retained) | The true cost of hoarding | Trending down |
| Cache hit rate | Cached prefix tokens ÷ total input | Whether your optimisations defeat themselves | Start above 70%; calibrate to provider |
These are derived starting targets, not universal health values; calibrate them to your provider, harness and workload. The full catalogue of 26 metrics is in chapter 10.
Your next action#
Run the context budget audit. Record the nine segments, identify the two largest retention contributors, make one reversible change, and re-measure prefix tax, relevance density, retention integral and cache hit rate.
Key takeaways
- "My context is full" is not actionable. A segment-by-segment breakdown is.
- In the constructed audit, tool definitions and tool results are the largest controllable segments.
- Optimise the retention integral (tokens × turns kept), not the peak.
- Compaction may rewrite earlier context, so treat it as cache-hostile until measured.
- The worked audit ranks recoverable segments before behaviour-changing fixes; validate every change against task coverage C.
Terms used in this chapter
- Prefix — Segments 1–4 of the context: system prompt, tool definitions, instruction files and skill descriptions. Usually stable and sent on each call; it may be reusable when unchanged, while billing depends on cached versus uncached rates.
- Body — Segments 5–9 of the context: retrieved code, tool results, agent messages, user turns and summaries. It grows during a session.
- Prefix tax — The tokens present in segments 1–4 on each call before any work. Measure them by sending a one-token message in a fresh session; billing depends on cache hits.
- Harness — The software wrapping the model: system prompt, tools, loop, compaction and retrieval behaviour.
- Read-utilisation — The share of retrieved tokens that appear in the final change or explanation. Low values mean retrieval is wasteful.
- Read-coverage — Whether the decisive file was ever opened during a failed task. It separates starvation from dilution, which need opposite fixes.
- Retention integral — The sum of each segment's tokens multiplied by the number of turns it stays in context. The true cost of hoarding.
- Clash — A failure mode where the context contains contradictions from different phases and the model resolves them by recency or by blending.
- Pass^k — Pass@k counts a task solved if any of k runs succeed ("can it ever?"). Pass^k, written Pass² for k = 2, requires all k runs to succeed ("can it reliably?").