Evaluating a context change

Why this is harder than ordinary agent evaluation#

Context changes have four properties that break the standard playbook. Ignore them and you will run experiments that cannot answer your question, which is worse than running none.

Twelve results that shape the design#

These come first because the design choices follow from them.

Eight design choices#

1. What you measure#

Recommendation: primary metric Pass^k with k ≥ 2, reported with tokens per solved task.

2. The task set#

Recommendation: 30–60 replayed tickets, stratified by size and subsystem, with the repository pinned to the commit before each fix. Use public benchmarks for sanity checks, never for choosing your configuration.

3. How many runs per task#

Recommendation: k = 2 minimum, always. Not for the interval, but because Pass² is the only way to see intermittency. k = 1 is not a cheaper version of the experiment. It is a different, less valid one.

4. Comparison design#

Recommendation: paired, always, analysed with McNemar's test on the tasks where the arms disagree. As shown below, it needs roughly a third of the tasks.

5. What to hold constant#

The most violated row is the compaction prompt. A team comparing retrieval strategies while the summariser differs between arms is measuring the summariser.

6. Cost accounting#

Report cache-adjusted cost. Cached and uncached tokens differ in price by about an order of magnitude, and context changes often move the ratio rather than the total. A change that cuts tokens 20% while halving the cache hit rate is a cost increase, and a raw token count reports it as a win. Chapter 10 works the numbers.

7. Where you measure#

Always instrument the 10 turns after each compaction, reset or sub-agent return (E11). Aggregate metrics average this window away. The re-fetch rate inside it is the cheapest high-signal metric in this research.

8. The baseline#

Recommendation: three arms: masking, your proposal, and full context.

Pass^k, worked#

Pass@k and Pass^k answer opposite questions and diverge sharply. Assume each run succeeds independently with probability p = 0.80.

A 32-point spread from one underlying number. Now suppose aggressive compaction drops p to 0.70.

Sample size, honestly#

The interval on one arm#

50 tasks, 34 solved. The Wilson interval at 95% (z = 1.96):

p̂        = 34/50                        = 0.680
z²/n     = 3.8416/50                    = 0.076832
z²/2n    = 0.038416
z²/4n²   = 3.8416/10000                 = 0.00038416
p̂q̂/n     = (0.680)(0.320)/50            = 0.0043520

√(0.0043520 + 0.00038416) = √0.0047362  = 0.068819
1.96 × 0.068819                         = 0.134885

centre = 0.680 + 0.038416               = 0.718416
lower  = (0.718416 − 0.134885)/1.076832 = 0.5419
upper  = (0.718416 + 0.134885)/1.076832 = 0.7924

68%, with a 95% interval of 54.2% to 79.2%: 25 points wide. The treatment arm, 37 of 50, gives 74%, interval 60.4% to 84.1%.

How many tasks you would need#

For an unpaired comparison of 60% versus 65% (a 5-point effect), at α = 0.05 two-sided and 80% power:

p̄ = 0.625, q̄ = 0.375
z_α/2 · √(2p̄q̄)      = 1.9600 × 0.684653 = 1.34192
z_β · √(p₁q₁ + p₂q₂) = 0.8416 × 0.683740 = 0.57544
(1.34192 + 0.57544)² = 3.67627
n = 3.67627 / 0.05²  = 1470.5   →  about 1,471 tasks per arm

At k = 2 that is 5,884 agent runs, for a 5-point effect. For a 10-point effect (60% versus 70%) the same method gives about 356 per arm. The detectable effect shrinks only with the square root of your budget: halving it costs four times the runs.

The paired design rescues this#

McNemar's test uses only the discordant pairs, tasks where the two arms disagree. With ψ the share of discordant pairs that favour the treatment:

n_disc ≈ (z_α/2 + z_β)² / (2ψ − 1)²
       = (1.9600 + 0.8416)² / (2 × 0.70 − 1)²
       = 7.84896 / 0.16
       = 49.06   →  about 49 discordant pairs

If 20% of tasks are discordant, you need about 245 tasks in total, against 712 unpaired for the same 10-point effect.

When you cannot afford even that#

Three honest fallbacks, most valid first:

1. Measure a bigger effect. Do not A/B a 2-point tweak; A/B the whole stack against no management.
2. Measure a tighter proxy. Tokens per solved task varies far less than solve rate; cache hit rate is almost deterministic. Weaker evidence, but measurable.
3. Adopt on mechanism, then monitor. Adopt because the mechanism is sound, and watch a small canary set for regressions. This is what most teams should do for most changes, and it is defensible only if you say so instead of presenting it as measured.

The minimum viable experiment#

tasks:        40 replayed tickets from your repository, stratified by size
arms:         2   # control = current config, treatment = ONE change
runs:         k = 2 per task per arm   # 160 runs in total
design:       paired; identical tasks; identical seeds where possible
held_fixed:   [model, harness, tool_surface, compaction_prompt, temperature]
primary:      Pass² per arm; McNemar on discordant pairs
secondary:    cache-adjusted input tokens per solved task
diagnostic:   re-fetch rate in the 10 turns after each boundary
stop_rule:    decided before running; no peeking and extending

The experiment spec template turns this into a form.

Attributing a stack: ablation#

Once several methods are adopted, "which one is doing the work?" needs leave-one-out ablation on a fixed task set.

Expect two things. Effects are not additive: semantic compaction's value depends on offload being present. And some ablations will come out neutral or positive. A method that does not help on your workload shows up as a wash when removed. That is the point: removal is a legitimate result. Six arms × 40 tasks × k = 2 is 480 runs, so run it quarterly, not weekly.

Evaluating a memory system#

Memory is the hardest thing here to evaluate and the easiest to fool yourself about, because writing works and reading is what matters.

The gate: if retrievals that changed an action are under one per week, the system is decoration. Delete it.

Evaluation mistakes to avoid#