How I Built a 3 ms Workspace Index for AI Agents
The original headline compared “30 seconds” with “3 ms” without publishing a protocol. I could reproduce the index side, not the 30-second baseline. On the current main-machine workspace, loading the complete semantic registry takes a median 2.142 ms over 500 runs; p95 is 2.545 ms and max is 5.8 ms.
| Benchmark input | Value |
|---|---|
| Machine | Intel Core i5-2410M @ 2.30 GHz, Linux 6.12 |
| Workspace files observed by indexer | 2,408 |
| Registry size | 302,261 bytes |
| JSON parse, median / p95 / max | 2.142 / 2.545 / 5.8 ms (500 runs) |
| Warm linear ID scan, median / p95 / max | 0.0153 / 0.0263 / 0.0717 ms (5,000 runs) |
This is a microbenchmark of local bytes and in-memory lookup—not end-to-end agent startup, filesystem discovery, or semantic answer quality.
The data model
A standard file tree tells an agent where bytes live. The registry says which files represent skills, ideas, issues, articles, products, research, automation opportunities, and projects. Each compact entry retains a stable ID, title, path, lifecycle state, and type-specific fields.
{
"v": 2,
"s": {"files": 2408, "sk": 22, "id": 40, "is": 19, "am": 386},
"d": [
{"n": "Articles", "x": "ART", "c": 30, "e": [...]},
{"n": "Automation Mining", "x": "AM", "c": 386, "e": [...]}
],
"x": {"idea_to_product": {...}, "idea_to_article": {...}},
"t": {"agent": "ART-...,IS-..."}
}Short keys reduce bytes, but they also reduce readability. Version the schema and keep the generator as the source of truth.
Build path versus query path
cold rebuild: walk files → parse frontmatter → classify → deduplicate → write JSON
warm query: read JSON → parse once → query in memoryThe 3 ms result measures the second path. Rebuild time was not captured in this audit and should not be implied by the headline.
Update strategy
The current implementation performs a full rebuild and overwrites Registry/index.json. At 2,408 files, simplicity is a reasonable tradeoff. Run it after workspace-changing workflows and before long agent sessions. Write to a temporary file, validate counts and references, then atomically replace the last good index; the current script writes the destination directly, so interruption can leave a partial file.
Bugs the schema had to absorb
- idea sidecars shared an ID-shaped filename but lacked primary frontmatter;
- article image prompts and briefs appeared as duplicate articles;
- issues moved between legacy and current directory layouts;
- nested project directories were invisible to a top-level glob;
- the same logical article existed in draft, ready, and published states.
The indexer encodes precedence and filtering once so every agent does not rediscover those rules.
When to use each retrieval tool
| Tool | Use it for | Do not use it for |
|---|---|---|
| Registry | Known IDs, types, lifecycle, relationships | Searching arbitrary file contents |
rg | Exact strings, symbols, filenames, source text | Conceptual similarity without matching terms |
| Vector search | Fuzzy semantic recall over content | Authoritative identity or lifecycle state |
These approaches complement one another. A compact registry narrows the search space; exact or semantic retrieval opens the relevant content.