Skip to content

Welcome

How an AI model remembers

Suppose you have worked with an AI agent all afternoon. At the start it followed every instruction you gave it. By the evening it suggests an approach you ruled out an hour ago, asks for a file name you already gave it, and breaks a rule you set in your first message.

The agent has not changed, and nothing is broken. The problem is that it is reading less of the conversation than it was at the start. The reasons for that come from how the language model inside it works.

An agent is a program built around a language model, and the model is the part that reads text and writes a reply. When you ask an agent a question, the agent does not send only your question. It builds a request that holds its instructions, the conversation so far and your new message, and sends all of it to the model. For a coding agent, a short request might hold this:

What the agent sends to the model
Instructions: You are a coding assistant. Run the tests before you
say a change is finished. Never edit files in the vendor folder.

Conversation so far:
  User: The login page shows an error after I sign in.
  Agent: I will look at the sign-in handler first.

New message:
  User: It only happens in Safari.

The model's reply can be an answer, or a request to use a tool: to search the web, read a file, run a command or query a database. When it asks for a tool, the agent runs the tool, adds the result to the request, and sends the whole request to the model again. This repeats until the model writes an answer. One question can take a dozen of these rounds, and each round adds text to the conversation: the tool's request, its result, and the model's next step.

A language model is a program trained on a very large amount of text, such as books, websites and code, to predict what text comes next. Companies such as OpenAI, Anthropic and Google train their models over weeks or months, on thousands of specialised computers, and then release the finished model for people and programs to use.

Training a model costs these companies a great deal, and they earn it back by charging for its use. An agent sends each request over the internet to the company that runs the model, the model writes a reply, the company charges for the amount of text read and written, and the agent acts on the reply. A better model draws more customers, which is why new models keep being released, and why an agent can switch to a newer model without changing anything else. Some companies also publish models that anyone can download and run on their own computers.

Training is closer to education than to memory. A person who has studied for years can think through a new problem quickly, but still has to be told what was said in a meeting they did not attend. A model is in the same position: training made it able to reason, write and use tools, and it knows many general facts from the text it was trained on, but it knows nothing about you or your work. Using a model does not train it further, so nothing you tell it becomes part of the model.

A model keeps no record of a request after it replies. A chat app or an agent makes a conversation look continuous by sending the whole conversation again with every message. On the tenth message, the model reads the instructions it was given, the nine earlier messages and their replies, and then the new message, all as one request. To the model, each request is the first one.

This sounds strange, but people live in a similar way. A person only ever experiences the present moment, one moment after another, and never lives through yesterday again. What makes a life feel continuous is what memory brings into each moment: you do not relive yesterday's conversation, you remember the parts of it that matter now. A model has the same present moment, which is the request in front of it. What it has in place of memory is whatever text the request holds.

The example below shows the requests that a chat app sends for the first three messages of a conversation. The exact format differs between providers, but each request holds the whole conversation so far. Some providers can store the conversation on their side, so that the app sends only the new message, and the model still reads the whole conversation on every request.

The request grows with every message
Request for message 1
{
	"system": "You help people plan dinner parties.",
	"messages": [
		{
			"role": "user",
			"content": "Dinner party for 8 on Saturday. Can you help?"
		}
	]
}
Request for message 2
{
	"system": "You help people plan dinner parties.",
	"messages": [
		{
			"role": "user",
			"content": "Dinner party for 8 on Saturday. Can you help?"
		},
		{
			"role": "assistant",
			"content": "Of course. A set menu or a buffet?"
		},
		{
			"role": "user",
			"content": "A set menu, three courses."
		}
	]
}
Request for message 3
{
	"system": "You help people plan dinner parties.",
	"messages": [
		{
			"role": "user",
			"content": "Dinner party for 8 on Saturday. Can you help?"
		},
		{
			"role": "assistant",
			"content": "Of course. A set menu or a buffet?"
		},
		{
			"role": "user",
			"content": "A set menu, three courses."
		},
		{
			"role": "assistant",
			"content": "A first idea: soup, a roast, then a fruit tart."
		},
		{
			"role": "user",
			"content": "Cold starter, please. I'm allergic to peanuts."
		}
	]
}

By message 40, the same request holds 78 earlier messages and replies before the new one, and an agent's request also holds every tool request and tool result from those rounds.

This has three consequences:

  • Each request costs more than the last. The model reads the whole conversation every time, so a long conversation is slower and more expensive per message than a short one.

  • A session that ends is gone. When the app stops sending the old messages, the model has no way to recover them.

  • The text has a size limit. Each model accepts only so much text in one request, and a long conversation reaches that limit.

A model reads text as tokens: short pieces of text, often a whole common word or part of a longer one. In English, one token is about four characters on average, so 1,000 words is roughly 1,300 tokens.

The context window is the most tokens a model can read in one request. Everything the model works with has to fit in it:

  • the instructions the agent was given, often called the system prompt

  • the conversation so far

  • any documents, search results or tool outputs added for this request

  • the reply the model writes

Context windows range from tens of thousands of tokens to around a million, depending on the model. That sounds large, but an agent fills it quickly: a few long documents, the output of some tool calls, and an afternoon of conversation can use all of it.

A large window also has a cost beyond the price of the tokens. Research on long contexts, such as Lost in the middle, has found that models use information at the start and end of a long context more reliably than information in the middle. A detail can be inside the window and still be missed.

Even worse, eventually the details of a conversation reach a size that it can't be expanded any longer.

When a conversation no longer fits, the agent has to remove something before it can send the next request. There are two common ways to do it.

Truncation drops the oldest messages. It is simple, and the most recent messages stay exactly as they were. Everything from the start of the conversation is lost, including anything the user said once and expected the agent to remember.

Compaction asks the model to write a summary of the earlier conversation, and sends the summary in place of the messages it replaces. The conversation then fits again, and its main points survive. Many coding agents and chat apps compact a long session automatically when it reaches the limit.

A context window drawn as a box of fixed size at four points in a conversation about planning a dinner party. At turn 3 the window holds the agent's instructions and three turns, one of which says the user is allergic to peanuts. At turn 20 the window is half full. At turn 40 it is full. After compaction, turns 1 to 30 are replaced by one summary that reads: dinner party for 8 on Saturday, menu chosen, wine ordered. The recent turns stay as they were, and the turn about the peanut allergy is no longer in the window.A context window drawn as a box of fixed size at four points in a conversation about planning a dinner party. At turn 3 the window holds the agent's instructions and three turns, one of which says the user is allergic to peanuts. At turn 20 the window is half full. At turn 40 it is full. After compaction, turns 1 to 30 are replaced by one summary that reads: dinner party for 8 on Saturday, menu chosen, wine ordered. The recent turns stay as they were, and the turn about the peanut allergy is no longer in the window.

A summary is shorter than what it replaces, so it leaves things out. The model writing the summary decides what matters at that moment, and it cannot know which details a later question will need.

In the conversation in the diagram, the user mentions a peanut allergy early on, while talking about something else. Thirty turns later, the summary keeps the plan for the dinner party and drops the allergy, which looked like a passing remark. When the user asks for a dessert recipe at turn 45, the agent suggests one with peanuts. It did not forget in the way a person forgets: the fact was simply no longer in the text it was given.

Compaction also repeats. A session that fills the window a second time is compacted again, and the new summary is made from text that already includes the old summary. Each round keeps the gist and loses more detail, in the same way as a photocopy of a photocopy.

Other details that a summary tends to lose are:

  • When something was said, and by whom. "The budget is 500" and "the budget was 500 until Tuesday" can both become "the budget is 500" in the summary, which loses the fact that it changed.

  • Corrections. If the user gave one value and later corrected it, a summary can keep either one, or both without saying which is current. Newer models tend to write better summaries, but nothing guarantees that a correction survives, and the summary keeps no record of what it left out, so there is no way to check afterwards what was lost.

  • Exact wording. Names, numbers and quotations are often rounded or paraphrased.

Truncation and compaction both work inside the context window, so whatever they drop is gone for good. The alternative is to keep what matters somewhere else, in storage that lasts after the session ends, and to put the relevant part of it back into the context window for each request. That storage is what agent memory means.

The context window stays the only thing the model reads. Memory changes how its contents are chosen: instead of "the most recent messages" or "a summary of everything", each request gets the facts that answer the question being asked, however long ago they were said.

From context to memory describes the common ways to build that storage, and what each one can and cannot answer.

Was this page helpful?