Benchmarking a Markdown Knowledge Graph as AI Agent Memory

The IWE project set out to test whether a local directory of linked markdown files — no embeddings, no hosted pipeline — could serve as viable memory for AI agents, competing with commercial vector-based 'memory layer' products. To get comparable numbers, the team adopted LOCOMO, the dataset behind most published memory-system evaluations: ten long fictional text conversations, each split into dozens of timestamped sessions, followed by hundreds of context-free questions graded strictly correct/wrong by an LLM judge.

The benchmark repeatedly disproved the team's own assumptions, each failure pointing at a concrete fix. It reshaped IWE's search engine, motivated a new block-level editing language, killed several design rules, and forced the retraction of an earlier, overstated result. The whole harness runs through claude -p as a Rust cargo xtask, with isolated agent workspaces, a curator that never sees the benchmark's questions, and a mostly-sealed test set — mechanical safeguards against overfitting rather than trust in instructions.

The eventual result: a cheap curation model, run for about $4.50 and constrained by mechanical guards, produced a markdown memory store that a single retrieval call reads back at 96 percent of a hand-built ceiling. Yet plain grep over raw transcripts still holds its own against most published memory products on raw accuracy — a reminder that simple, tool-native baselines remain a serious bar for agent-memory systems to clear.