Building a Memory Layer Into an AI Agent Product: What Actually Persists Between Sessions

Because model weights are frozen at inference time and nothing in an ordinary request writes to them. Correcting an agent changes the current conversation and nothing beyond it. Aparna, another buildDay registrant, described the symptom exactly: "I have to keep on training it all the time. self learning doesnt work as I expect." Fine-tuning is the obvious next thought, and the measurements do not support it. FineTuneBench, by Eric Wu, Kevin Wu and James Zou at Stanford University (arXiv, November 2024), tested five frontier models through commercial fine-tuning APIs and found an average generalisation accuracy of 37% for learning new information, falling to 19% for updating knowledge the model already held. Updating a fact the model already believes is precisely what "learn from my correction" means, and it worked fewer than one time in five. Retraining also damages what already works. Google Research, announcing its Nested Learning paradigm on 7 November 2025, wrote that "the simple approach, continually updating a model's parameters with new data, often leads to catastrophic forgetting" (Ali Behrouz and Vahab Mirrokni, Google Research). Andrej Karpathy, a founding member of OpenAI and former Director of AI at Tesla, drew the architectural line on the Dwarkesh Podcast on 17 October 2025: "Anything that happens during the training of the neural network, the knowledge is only a hazy recollection of what happened in training time. Whereas anything that happens in the context window of the neural network is very directly accessible to the neural net." Harrison Chase, co-founder and CEO of LangChain, made the same point about what actually ships. Writing on 19 October 2024, he said he does not "see many (any?) agentic systems that update the weights of their LLM automatically or rewrite their code". So memory is retrieval, not learning. You store the correction as a record and put it back in front of the model next time.

Three separate mechanisms get the same name, and conflating them is where most agent memory designs go wrong. | Tier | Where it lives | How it is read | Best for | |---|---|---|---| | Always in context | Injected into every prompt | No retrieval, always present | User profile, account state, tone rules | | Retrieved on demand | External store the agent queries | Key lookup or search | Facts, past decisions, documents | | Conversation history | Message log for one thread | Replayed or summarised | Continuity inside a single session | Letta, the product built from the MemGPT paper (Packer, Wooders, Lin, Fang, Patil, Stoica and Gonzalez, UC Berkeley, arXiv, October 2023), calls the first tier memory blocks and describes them as sections of the context window that are always visible with no retrieval needed, each with a character limit (Letta documentation, 2026). LangChain splits the same ground differently, calling thread-scoped state short-term memory and cross-session state long-term memory, and borrowing the semantic, episodic and procedural labels from the CoALA paper by Sumers, Yao, Narasimhan and Griffiths at Princeton (arXiv, published in TMLR, March 2024). Pick the tier before you pick the technology. A user's timezone belongs in tier one and should never become a similarity search you can get wrong.

Start with a database row, not a vector store. Structured facts about a user (their name, plan, timezone, stated preferences, last five orders) are lookups by key, and turning a lookup into a similarity search adds a failure mode you did not need. The shipped implementations back this up. Anthropic's memory tool uses no embeddings at all: it exposes view, create, str_replace, insert, delete and rename over plain files, and the documentation says it "operates client-side: Claude requests file operations, and your application executes them" (Anthropic, 2026). LangChain's long-term store is a namespaced key-value store where semantic search is optional rather than the default access path. Reach for a vector store when the query is genuinely fuzzy, such as "what has this user said about their renovation" across hundreds of past messages. Reach for a graph when facts change and the change itself matters. Zep's temporal knowledge graph (Rasmussen, Paliychuk, Beauvais, Ryan and Chalef, arXiv, January 2025) stamps every edge with four timestamps, so a superseded fact is invalidated rather than deleted and the system can still answer what it believed and when. Check what the platform keeps for you before building your own. OpenAI's Assistants API shuts down on 26 August 2026 and conversation state moves to the Responses API, where Conversation objects are explicitly not subject to the 30 day retention default that applies to stored responses (OpenAI developer documentation, 2026). That is session continuity, though, not a memory layer: it replays what was said, it does not decide what mattered. Cost rarely decides this at small scale. Pinecone's Standard tier carries a minimum spend of USD $50 per month (roughly AUD $77) plus USD $16 to $18 per million reads, while pgvector inside a Postgres database you already run adds nothing to your monthly bill (Pinecone pricing, 2026; converted at approximately 1.54 AUD per USD, August 2026).

Write memories in a background job, not in the middle of the user's turn. Deciding what is worth saving needs its own model call, and running it on the hot path adds latency to every single response. LangChain's LangMem documentation names the two options directly: active or hot path formation has a higher latency impact with immediate updates, background formation has none with delayed updates. Their recommended production delay for background processing is 30 to 60 minutes, paired with a debounce that cancels queued extraction when new messages arrive. Without that debounce, a 40 message conversation fires 40 extraction calls to produce roughly one useful fact. The payoff for extracting rather than replaying everything is real, though the headline numbers are vendor published. The Mem0 paper (Chhikara, Khant, Aryan, Singh and Yadav, arXiv, 28 April 2025) reports 91% lower p95 latency and over 90% token cost savings against feeding the full conversation back in, with a 26% relative improvement in LLM-as-a-judge score over the OpenAI memory baseline. Read their own results table before you get excited. In the same paper, the plain full-context baseline scored 72.90 overall on LOCOMO against Mem0's 66.88. The memory layer bought speed and cost, not accuracy. That is the honest trade, and it is the right one at scale, but do not tell yourself extraction is making the agent smarter. Background writes can do more than save latency. The sleep-time compute paper (Lin, Snell, Wang, Packer, Wooders, Stoica and Gonzalez, arXiv, 17 April 2025) had an offline agent pre-process context between sessions and cut the test-time compute needed for the same accuracy by roughly 5x, with accuracy gains up to 13% on one stateful benchmark and 18% on another. Two rules that survive contact with production. Never write a memory the user cannot see. And store the source turn alongside every extracted fact, so when a wrong memory shows up you can trace where it came from instead of guessing.

Timestamp every memory and make superseding an explicit operation, because no current memory system handles it well on its own. MemoryAgentBench (Hu, Wang and McAuley, arXiv, July 2025) includes a FactConsolidation task built for exactly this case: a stored fact is later contradicted and the system has to prefer the newer one. The results are unflattering for the purpose-built products. On the single-hop version, plain GPT-4o with a long context scored 60.0%, MemGPT 28.0%, Mem0 18.0% and MIRIX 14.0%. On the multi-hop version every system scored 7% or below. The authors conclude that selective forgetting "still poses a significant challenge to all memory mechanisms". Hallucination during the write step compounds it. HaluMem (Chen and colleagues, arXiv, November 2025) scored memory systems at the extraction, updating and question-answering stages separately, and found every system tested hallucinated on 15% to 30% of memory questions and omitted answers on 17% to 35%. Its finding is that systems "generate and accumulate hallucinations during the extraction and updating stages, which subsequently propagate errors to the question answering stage". The practical design: store each fact with a valid-from date and an optional valid-to, resolve conflicts at read time by recency plus source confidence, and show the user both values rather than silently picking one.

Retrieval quality falls as the store grows, which is the opposite of what most builders assume when they design one. HaluMem ran the same systems over conversation histories of roughly 1,500 turns and roughly 2,600 turns. Mem0's memory extraction recall fell from 42.91% to 3.23% between the two, and its answer omission rate rose from 27.81% to 54.60%. LongMemEval (Wu, Wang, Yu, Zhang, Chang and Yu, arXiv, October 2024, ICLR 2025) measured the same shape from the other direction, reporting that commercial chat assistants and long-context models show "a 30% accuracy drop on memorizing information across sustained interactions". Treat published memory benchmarks with care. An independent audit of the widely cited LoCoMo benchmark by Penfield Labs (9 April 2026) found 6.4% of its answer key wrong and that the LLM judge accepted 63% of deliberately wrong answers. Two leading memory vendors have publicly accused each other of running invalid evaluations on it. There is no benchmark you can currently point at to prove your memory layer works, which means you need your own eval set built from your own product's failures. "Everyone designs the write path and nobody designs the delete path. Six months in, the agent is confidently working from something the user said once, in passing, in March." - Callum Holt, Founder, 13Labs Design for forgetting on day one: a per-user cap, a decay or archive policy, and a read step that returns 5 to 10 memories rather than everything that matched.

Scope every memory record explicitly, and store the scope as a column rather than as a naming convention. The cheapest catastrophic bug in an agent product is one user's memory surfacing in another user's conversation. Four scopes cover most products: - Session: true only for this conversation, discarded when it ends. - User: preferences, corrections and facts about the person. This is the default and where most records belong. - Account or organisation: shared context for a team, where one member's correction should apply to their colleagues. Decide this deliberately, because it is often wrong. - Global: rules that apply to everyone. That is your system prompt, not your memory store. Getting the boundary wrong is now a legal event in Australia, not only a support ticket. The statutory tort of serious invasion of privacy commenced on 10 June 2025 as Schedule 2 to the Privacy Act 1988, is actionable without proof of damage, and is not limited to entities covered by the Act (OAIC, 2025). A startup under the $3 million small business turnover threshold can still be sued directly by an individual whose stored disclosures were exposed to someone else.

Yes, and the attacks are documented and cheap. Memory is a write surface, and anything the agent reads can carry instructions aimed at it. OWASP's Top 10 for Agentic Applications (OWASP GenAI Security Project, 9 December 2025) ranks ASI06, Memory and Context Poisoning, sixth of ten, noting that memory poisoning "reshaped behaviour long after the initial interaction". Two papers give the numbers. MINJA (Dong and colleagues, arXiv, March 2025) plants malicious records using nothing but ordinary queries to the agent, with no access to the store, reporting a 98.2% average injection success rate and a 76.8% attack success rate at eliciting the malicious reasoning. AgentPoison (NeurIPS, 2024) reached over 80% attack success with a poison rate below 0.1% of the store, sometimes with as few as two poisoned entries, while degrading benign performance by less than 1%. It has already happened in a shipped consumer product. Security researcher Johann Rehberger published a Gemini long-term memory attack on 10 February 2025 using delayed tool invocation, planting permanent false facts including a user's age recorded as 102. Google classified it as low likelihood and low impact. Four defences that carry their weight: never write to memory directly from content the agent merely read, require a schema-constrained tool call rather than free text, log every write with its source, and give the user a memory viewer so wrong entries surface fast.

Stored memories are personal information under the Privacy Act 1988, including the ones your agent inferred rather than the user typing them. The OAIC's guidance on privacy and the use of commercially available AI products (21 October 2024, updated 17 January 2025) states that "if AI systems are used to generate or infer personal information, including images, this is a collection of personal information and must comply with APP 3". Four obligations shape the build: - APP 5 requires telling users at collection that memories are created, inferred and kept across sessions. A line about improving our services does not describe a memory feature. - APP 6 limits secondary use. The OAIC says that "in many cases it will be difficult to establish that a secondary use for AI-related purposes (such as training an AI system) was within reasonable expectations". Training on your memory store needs separate opt-in consent with a working opt-out. - APP 11.2 requires destroying or de-identifying personal information once it is no longer needed. Indefinite retention is hard to defend, and hard delete has to reach the primary store, the backups and any derived embeddings. Deleting a row while its embedding stays retrievable is not destruction. - APP 12 and APP 13 give users access and correction rights with a 30 day response window, which in practice means shipping a memory viewer with edit and delete. The exposure is not hypothetical. The OAIC recorded 1,205 data breach notifications in the 2025 calendar year, up 8% on 2024 and the highest since the scheme began in 2018, with 59.4% attributed to malicious or criminal attack (OAIC, 6 July 2026). A memory store is a per-user file of inferred facts written in plain language, which is a considerably worse thing to lose than a mailing list.

Does an AI agent learn from my corrections automatically? No. Model weights are frozen at inference, so a correction only affects the current conversation. FineTuneBench (Stanford, November 2024) found commercial fine-tuning APIs updated knowledge the model already held with 19% generalisation accuracy, so retraining does not fix it either. Store the correction and re-inject it. Should agent memories go in a vector database? Only the fuzzy ones. Structured user facts such as plan, timezone and stated preferences are key lookups and belong in a normal database row. Anthropic's memory tool uses plain files with no embeddings, and LangChain's long-term store makes semantic search optional rather than the default path. How do you handle two memories that contradict each other? Timestamp every record with a valid-from date and resolve by recency at read time. Do not rely on the memory system to do it: MemoryAgentBench (July 2025) measured Mem0 at 18.0% and MemGPT at 28.0% on single-hop fact consolidation, with every system under 7% on multi-hop. Can someone poison an agent's memory? Yes. The MINJA attack (arXiv, March 2025) reported a 98.2% injection success rate using only ordinary queries to the agent, and AgentPoison (NeurIPS 2024) achieved over 80% attack success with under 0.1% of the store poisoned. Never write memory straight from content the agent read. Do Australian privacy rules apply to AI memories? Yes, once you are an APP entity. Inferred facts count as collection under APP 3, secondary use for training is restricted under APP 6, APP 11.2 requires destruction when no longer needed, and APP 12 and 13 require access and correction within 30 days. The statutory privacy tort, in force since 10 June 2025, applies even below the $3 million threshold.