A deterministic memory layer for local LLMs at near-zero cost
An open-source deterministic memory layer bolts onto frozen LLMs, enabling zero-forgetting recall beyond context at near-zero marginal cost.
A developer built a small memory layer that grafts onto any frozen model — GPT-2, Llama, Qwen, from 124M to 7B parameters. It doesn't compete with the base model; it adds something the model structurally lacks: the ability to remember and revise facts beyond its context window. Tested on Qwen2.5-7B (4-bit), the system stores 30,000 real Wikipedia facts in just 14.7 MB with 100% recall and zero forgetting, taking recall of out-of-window facts from 0/10 with the base model alone to 10/10 with the memory attached.
The scaling story is the real point. Standard transformer attention is O(N²), and it exhausts a 24 GB GPU at roughly 79,000 tokens. The same model with a fixed window plus this memory processed a 20-billion-token stream with flat, zero memory growth — process RSS stayed at one constant value for the entire run, a meaningful difference for anything meant to run continuously.
The modeled cost comparison is striking too: for 1 million users, re-sending each user's facts via long-context runs about $3M/month and eventually OOMs past the window; a RAG-style vector index costs roughly $45k–165k/month in ongoing infrastructure; this memory layer needs no extra context tokens and no vector database, pushing marginal cost to near-zero — literally zero server cost when run on-device. Known facts are answered in about 0.1 microseconds with zero model forward passes.
The honest caveat: what collapses to near-zero is the cost of holding and recalling memory, not the model's own compute — running the model for actual generation or reasoning still costs what it always did. This is a fact memory, not a reasoning engine, tested so far only on public Wikipedia data and on a 7B model at 4-bit precision. The code is open-source under AGPL-3.0 and reproducible with a single command.