Why ChatGPT Forgets What You Told It: Longer Chats Are Worse, Not Better
You explain the situation carefully at the top of a chat. Forty messages later, the AI is confidently doing the thing you told it not to do, and you find yourself typing “as I said earlier” to a computer.
It isn’t being lazy and it hasn’t stopped paying attention. It’s doing the only thing it can do, and once you know what that is, a handful of habits stop the problem happening.
ChatGPT is where most people meet this, but it isn’t a ChatGPT fault. Every assistant does it, Claude and Copilot and Gemini included, and the research below covers eighteen of them. It’s also not the “memory” feature some of them now advertise, which is a separate thing bolted on top. This is the mechanism underneath all of them.
AI has no memory: it has a context window
Here’s the part that surprises people: the model has no memory of your conversation at all. Between one message and the next it retains nothing.
What actually happens is that every time you hit send, the entire conversation gets handed to the model again from the beginning (your messages, its replies, any documents you pasted), and it reads the whole thing and writes the next response. Then it forgets again.
That bundle is its Everything the model can see at once: your messages, its own replies, and anything you’ve pasted or attached. It’s measured in tokens, roughly three-quarters of a word each. Nothing outside the window exists as far as the model is concerned: not last week’s chat, not the document you shared in a different conversation, not the correction you made before the window filled up and pushed it out. , and it’s the whole world. So “it forgot” almost always means one of two things: what you said fell out of the window, or it’s still in there and the model didn’t use it. The second one is more common, and much more interesting.
Lost in the middle: the part of your prompt AI skips
In a study called Lost in the Middle, Stanford researchers gave models a question and a stack of documents, exactly one of which contained the answer. Then they moved that document around.
Performance traced a U-shape. Answers near the beginning of the stack were found reliably. Answers near the end were found reliably. Answers in the middle were missed, and the effect held even on models specifically built for long inputs.
The sharpest result is worth stating plainly. GPT-3.5-Turbo answering from memory alone, with no documents at all, scored 56.1%. The same model given twenty or thirty documents, with the correct answer sitting in the middle of them, scored worse than that.
Read that twice. Handing it the material made it perform worse than handing it nothing. Not because the information wasn’t there. It was right there in the window. The problem was where in the pile it sat.
Context rot: why a 200,000-token window isn’t 200,000 usable
The obvious hope is that this got fixed as windows grew. Models now advertise 200,000 tokens and more, which is a small book.
In 2025 the team at Chroma ran a study on exactly that question across 18 models (Claude 4, GPT-4.1, Gemini 2.5, Qwen3 and others) and found every one of them degraded as inputs got longer. The important word in their finding is non-uniform: performance doesn’t decline gently and predictably towards the limit. It falls off unevenly, and often well before the number on the box.
Their explanation for why this went unnoticed is the most useful part. The standard industry test is a A benchmark where one odd sentence, the needle, is buried in a huge block of unrelated text, and the model is asked to find it. Models pass easily, which is where “we solved long context” came from. The catch is that it’s a word-matching exercise: the needle is conspicuous and there is nothing else in the haystack competing to be the answer. Real work is the opposite, with many passages that all look relevant and no exact phrase to match on. , and models ace it. When Chroma made the task more realistic, the weaknesses reappeared:
- Plausible alternatives hurt. Adding a single passage that looks like it answers the question dropped accuracy. Adding four dropped it further.
- Vaguer questions failed faster. When the wording of the question didn’t closely match the wording of the answer, long inputs degraded quicker.
- Coherent text did worse than shuffled text. This one is genuinely strange, and it’s the finding I’d least have predicted: a haystack of properly ordered, sensible prose scored lower than the same sentences in random order.
One caveat worth stating: Chroma sells a product for retrieving small relevant chunks of text, so “long context alone isn’t enough” is a conclusion that suits them. The result matches the older academic work, which is why it’s here, but it isn’t disinterested.
How to stop AI forgetting: four habits
None of this requires understanding a model. It’s four habits.
- Start fresh more often than feels natural. When a chat has wandered, don’t fight it. Open a new one and paste a short brief: the goal, the constraints, the current state. You’ll get a better answer in one message than in ten more of correcting.
- Put the important instruction last. The end of the window is prime real estate and it’s free. If something must not be ignored, it goes in your most recent message, not in message three.
- Shrink the haystack. Five relevant pages beat fifty. The instinct to paste everything “so it has full context” is the exact move the research says backfires. You’re adding plausible alternatives.
- Re-state, don’t refer back. “As I mentioned above” asks it to find something mid-window, which is the weakest thing it does. Just say the thing again. It costs you a sentence.
For anything you do repeatedly, the real fix isn’t a better chat at all. It’s a setup that starts with the right context every time, which is what Build a Reusable AI Assistant walks through. And if you want the companion skill, Giving AI Context: The Skill That Actually Makes It Useful is about what to put in the window; this post is about why it stops being used.
The takeaway
The model isn’t remembering your conversation, it’s re-reading it, and it reads the middle badly. That single fact explains the “as I said earlier” problem, why pasting more can make answers worse, and why a 200,000-token window doesn’t mean 200,000 usable tokens.
When a chat starts drifting, start a new one and re-state the goal. It feels like giving up and it’s the highest-return habit in this whole article.
Sources: the U-shaped position effect and the 56.1% closed-book comparison are from Lost in the Middle: How Language Models Use Long Contexts (opens in a new tab) (Liu et al., TACL 2024). Note that the headline figure is GPT-3.5-Turbo, a model now several generations old, which is part of why the follow-up work matters. The 18-model results are from Context Rot: How Increasing Input Tokens Impacts LLM Performance (opens in a new tab) (Hong, Troynikov and Huber, Chroma, July 2025), which is vendor research from a company selling retrieval infrastructure, as the post says. The four habits are mine. They follow from the findings, but no study tested “start a new chat,” and I’m offering them as sensible practice rather than a measured result. A widely repeated “30% drop” attached to the Liu paper does not appear in it, so it isn’t quoted here.
Related: Why AI Gives Different Answers to the Same Question is the other half of why the same prompt doesn’t behave the same way twice. How AI Agents Work: A Loop, Not a Brain applies all of this at speed: an agent runs the same re-reading loop twenty times without you there to notice it drifting.