Stanford researchers have open-sourced Memorilla, a memory system that lets an unmodified (“frozen”) large language model draw on an entire document collection through just 16 memory tokens. The code was released on November 14, 2026, a month after the paper, “Memorilla: Latent Semantic Memory for LLMs”, appeared on arXiv. On one fact-checking benchmark, a model using those 16 tokens beat the same model reading every document in full.
Key takeaways
- Memorilla turns a whole document collection into a fixed set of 16 query-specific “memory tokens” for a frozen LLM. The base model is never fine-tuned.
- On FactKG, a frozen Qwen3-8B scored 84.33% with Memorilla, against 80.43% with every document in its context. Memorilla used 108x fewer document tokens.
- It led on five of eight benchmarks against memory systems such as Mem0 and MemoRAG. It lost on three, most clearly on PubMedQA.
- Because the model always gets 16 tokens however large the collection is, the authors report answers in 0.74 seconds on one H100 with 512 documents (about 128K tokens).
- The work comes from Jure Leskovec’s SNAP group at Stanford, with co-authors from other universities including Toronto and Penn.
What happened
Most production LLM apps give a model memory in one of two ways. They paste as much source text into the context window as they can afford, or they use retrieval-augmented generation (RAG) to pick a few relevant chunks and paste those in. Both put raw text in front of the model. The more text there is, the more each request costs and the longer it takes. And as contexts grow, models get worse at finding the detail that matters.
Memorilla works differently. According to the paper’s abstract, documents such as past chats or book chapters are encoded once into reusable embeddings that act as a persistent memory store. When a question arrives, a small trainable memory module looks across every embedding in that store, not just the top few results as RAG would. It then writes what the question needs into 16 “soft” tokens placed directly in the LLM’s input embedding space. The frozen model reads those 16 vectors together with the question text and answers. The source documents never enter the context as text.
The team, led by Adibvafa Fallahpour and Jure Leskovec, has published the project at memory.stanford.edu. The release was covered by MarkTechPost and other outlets.
The numbers, including where it falls short
The headline result is on FactKG, a benchmark that asks whether facts from a knowledge graph support or refute a claim. With a frozen Qwen3-8B as the reader, Memorilla reached 84.33% accuracy. Feeding the model every source document directly scored 80.43%. The full-context run averaged about 1,727 document tokens per query; Memorilla always uses 16, a 108x reduction. Doing better with less text suggests that a focused, query-specific summary can help a model more than raw text it has to sort through itself.
The compression is largest on long documents. On NarrativeQA, which asks questions about whole books and scripts, the 16 memory tokens stand in for 67,385 document tokens, a reduction of more than 4,000x.
It is not a clean sweep. Across eight benchmarks covering personalization, long-document QA and knowledge-heavy QA, Memorilla came first on five. The clearest miss is PubMedQA, where each question comes with only a few short abstracts (324 tokens in total). There, Memorilla scored 59.00% while the prompt-compression method LLMLingua-2 reached 69.00%. When the source text is already that small, squeezing it further gains little and can lose detail.
The authors also tested Memorilla as memory for an agent. In the text-adventure environment TextWorld, an agent whose past turns kept piling up wandered less between rooms it had already explored. Its average success rate rose from 18.6% with its full history in context to 24.2% with Memorilla.
Why it matters
Long context is expensive. Every token in a prompt costs money and adds delay, and teams building assistants with chat history, support bots over large knowledge bases, or agents with long trajectories pay that cost on every request. Memorilla is a research case for a different trade-off: do the heavy reading once at encoding time, then send the model a small, fixed input on every call.
The “frozen” part is just as important. You don’t retrain or fine-tune the base model; only the memory module learns. In principle that means a team could add this kind of memory to an open-weight model it already runs without touching the model’s weights. That fits the wider move toward open models that teams host themselves, such as Thinking Machines’ Inkling 2 Small.
What it means for developers
For now, Memorilla is a research release, not a drop-in replacement for your RAG pipeline. A few practical points:
- You need access to embeddings. The memory tokens go straight into the model’s input embedding space, so this works with models whose weights you run yourself. It does not work with closed APIs that accept only text.
- Expect a training step. The memory module is trained for a given reader model. Check the repository for which models and checkpoints it supports before planning around it.
- Fit the tool to the data. The results favour large or growing collections (long books, chat histories, agent logs). For short inputs, plain prompting or lighter compression may still win, as PubMedQA shows.
- Interpretability changes. With RAG you can log exactly which chunks the model saw. Soft memory tokens are vectors, not text, so debugging and auditing what the model “remembered” gets harder.
For teams building agents, the TextWorld result is the one to watch. Memory is one of the hardest parts of running agents in production, and agent tooling is drawing serious investment (see our coverage of the LangChain Series C). A fixed-size memory that doesn’t grow with the history would make long-running agents cheaper and more predictable.
What to watch next
Three things will decide whether Memorilla moves from paper to practice. First, independent reproductions: these benchmark results come from the authors, and outside tests on other models and datasets will show how well they hold up. Second, support for more models: how easily the memory module can be trained for other open-weight models, beyond the Qwen3 models used in the paper. Third, integration: whether vector databases and agent frameworks add “latent memory” as an option next to classic RAG.
The broader point is clear already. Bigger context windows are not the only way to give a model more knowledge. Memorilla shows that a small learned memory can sometimes beat simply giving the model all the text, at a fraction of the token cost.