Context and memory, why AI "forgets"

The model doesn’t remember: it rereads

This may be the most counter-intuitive thing about AI assistants. When you chat, it seems the model remembers what you said. Actually, every time you send a message, the app sends the model the whole conversation from the beginning, plus hidden system instructions, plus any files. The model reads it all, writes the answer and “forgets” everything. At the next message it starts again, with one more piece.

Two immediate consequences:

• Long conversations cost more and are slower, because each turn has more text to reread.
• There is a limit: when the conversation exceeds the context window, the oldest parts are cut or summarised by the app. At that point the model really no longer “knows” what you agreed at the start.

How big the window is

The context window is measured in tokens (see the guide on how an LLM works). As of October 2026, according to Artificial Analysis, almost all top models have a window of about 1 million tokens: Claude Opus and Sonnet 5.5, GPT-6.1 Sol, Gemini 4 Argon. Llama 4 Scout reaches 10 million.

But beware: in apps the window is often smaller than the model’s via API. For example, ChatGPT’s pricing page lists for Instant mode a total context of 27,000 tokens on Free (about 12 pages of text), 54,000 on Go and Plus and 128,000 on Pro; for reasoning it rises to 256,000 on Go and Plus and 400,000 on Pro. Claude states “up to 1M, varies by model” on every plan.

Use the calculator to see how much a document “weighs”:

Why long conversations degrade

Even when everything fits, not everything gets equal weight. A 2023 study (Lost in the Middle) showed models use information at the start and end of the context better than in the middle. Today’s models are much improved, but the effect hasn’t vanished: with long, meandering conversations, quality drops. Developers call it context rot.

Signs you’ve passed the useful point:

• the model repeats mistakes you already corrected;
• it forgets a constraint given at the start (“max 200 words”, “no bullet lists”);
• it mixes details from two versions of the same document.

Where to put important things

Since the start and end weigh more:

• put instructions and constraints at the start of the message;
• when pasting a long document, repeat the question at the end, after the document;
• for very long documents, first ask for a map (“list the sections with page numbers”) and then work section by section.

Memory, projects and custom instructions

Apps have added features that look like “memory”, but they all work the same way: they automatically insert text into the context.

• Custom instructions: text you write once (“I’m an accountant, be concise”) added at the start of every chat.
• Memory: the assistant saves notes about you during conversations (“works with freelancers”) and reinserts them in later chats. You can view, edit and delete them in settings.
• Projects: a folder with instructions and files placed in the context of every chat in the project. The best way to work on one topic over time (a client, a book, a course).

RAG: search before answering

When there’s too much information for the context (all of a company’s documents, all of a site’s pages) you use RAG, Retrieval-Augmented Generation:

1. documents are split into chunks and indexed;
2. when a question arrives, a search engine finds the most relevant chunks;
3. only those chunks go into the context, with the question;
4. the model answers from them and can cite them.

That’s exactly how Lumen’s chatbot works: when you ask about the site, it searches the relevant passages across all guides and pages, reads them and answers with links. For fresh news it does the same on four external sites.

Tools like Gemini Notebook (formerly NotebookLM) and Projects in ChatGPT and Claude apply the same idea to your files: upload PDFs, notes, videos, and the assistant answers only from those sources, showing where each claim comes from.

Caching: why it sometimes costs less

If via API you often send the same opening text (same instructions, same document), providers can cache the part already read and charge much less for it on later requests. Quality is unchanged, but for chatbots and automations it can cut costs significantly. Anthropic, for example, further reduced cache-read costs for Sonnet 5.5 in October 2026.

In practice

1. One chat per topic. Don’t use the same conversation for different things.
2. Restart with a summary when the chat gets long or the model starts slipping.
3. Instructions at the start, question also at the end when pasting long texts.
4. Use projects for recurring work, with instructions and reference documents.
5. Check the app’s memory now and then and delete what you don’t need.
6. For many documents, use RAG: Gemini Notebook, Projects or a dedicated system.

Sources

• ChatGPT — Plans and context limits
• Claude — Plans and context window
• Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (2023)
• IBM Technology — What is a Context Window?
• IBM Technology — What is Retrieval-Augmented Generation (RAG)?

Informational content, current as of the date shown. Prices, models and rules change often: always check official sources. Written with the support of AI tools and reviewed by the author.