Building an LLM Wiki: Persistent Memory for AI Agents Over Live Sources
How one team built a self-updating LLM Wiki memory layer for AI agents, tackling live source drift and comparing grep, BM25, vector, and LLM retrieval.
A team building an internal question-answering agent hit a familiar RAG problem: every query re-fetched raw documents and re-derived the same facts, burning time and money while answer quality depended on whatever chunks happened to get retrieved. Their fix was to implement Andrej Karpathy's "LLM Wiki" idea — compile what you learn once into a persistent, AI-maintained knowledge layer of markdown files, then answer from that instead of starting from scratch each time.
The hard part wasn't covered by existing research: their sources aren't static personal files but shared, constantly-changing systems (Notion, GDrive, Slack, the web) edited by people with no idea a wiki depends on them. There's no signal when something changes. Their workaround is a weekly lint job that catches contradictions and stale claims, plus re-validating every wiki fact against its source at query time before using it.
Architecturally, the wiki splits into category-based sub-wikis, each navigated via index.md summaries at every folder level so the agent can walk the tree instead of scanning everything. The wiki fills itself as a byproduct of answering questions: queries trigger both retrieval and opportunistic ingestion of new facts, and answers that get positive feedback get distilled into reusable entries.
The team benchmarked four retrieval strategies — grep, BM25 with FTS5, vector search, and LLM-based judgment — plus a hybrid using Reciprocal Rank Fusion, across 24 queries over an 88-entry production wiki. Grep proved unusable, LLM judgment gave the cleanest, most accurate results but cost 69 seconds per query, while BM25 landed as the practical middle ground, nearly matching LLM quality at sub-millisecond latency.