When a Million Tokens Is Still Not Enough
Introduction
I still remember the day I went to the Gemma 2 launch. The product manager was very excited about the new model and kept telling us how “open” it was. I remember thinking that the definition of open felt a little slippery. But something else caught my attention: the context window was only 8k tokens.
At the time, models had no real long-term memory. There were no long-running conversations or agents that remembered what you had done yesterday. Anything outside the context window was simply gone. The leading models already offered 128k tokens, which was enough to fit real documents alongside the prompt. So an 8k context window was not very impressive. It made the model feel like a clever but very forgetful brain.
Today, 128k is the number people laugh at, and somehow we still run out of room.
The problem
How much do we expect an agent to remember?
Everything, of course.
We want it to remember what we did yesterday and what we did a month ago. If a human can remember something, we expect the agent to remember it too. At the same time, we want it to “see” the whole codebase, understand every little detail, and read every document connected to the code. Then it can make decisions with a complete understanding, or at least with everything related to the problem we are trying to solve.
This demand has pushed context windows towards 1 million tokens on frontier models. Even that does not seem to be enough for daily use. We still run into context overflow.
And overflow is only the visible part of the problem. Even when everything technically fits, agents gradually lose track of earlier instructions (paper). They overlook parts of the code, forget decisions already made, and repeat work they have already done. The final code might still work, but some of the effort was unnecessary. And filling a 1-million-token window is not cheap.
So everyone has started thinking about how to reduce context size.
It began with compaction. When the token count gets close to the limit, the agent summarises the conversation and starts again from that summary. Then compaction grew into memory systems. During a conversation, selected information is kept as short-term memory for the next few turns. After the conversation ends, it is compacted again, and the main points are stored for the next session.
That makes sense for human-like conversation. But what do we do with a codebase?
Code is not a memory. If we summarise it too aggressively, it loses the details that make it correct. A summary can tell us what a function is supposed to do, but not necessarily every edge case hidden inside it. It seems that the only practical option is to restrict how much code enters the context window.
But then think about the amount of code agents now generate, the size of modern diffs, and the giant monorepos many companies work with. How are we supposed to choose what the agent gets to see?
How companies are doing it
Companies running agents at scale have already hit the same wall, and each of them has developed its own way of managing it.
- Shopify sizes each hunting agent’s code partition to 20–30% of the context window. It learned that without a clear scope, models “fill up their context windows chasing an unordered list of potential findings” (Shopify).
- Grab ran into context overflow in production and addressed it with automatic summarisation (Grab).
- Rippling injects skills dynamically instead of loading everything at the beginning, reducing context size by 100–500x (Rippling, via LangChain).
- Meta uses progressive disclosure in its recipe system. The agent pulls in only the manual pages needed for the current step (Meta — DEmate).
- Cloudflare works on the input side, removing lockfiles and generated code from diffs before asking an agent to review them (Cloudflare).
For the broader picture, Anthropic discusses compaction, sub-agent handoffs, and the idea of treating context as a scarce budget in Effective context engineering for AI agents. Sourcegraph explains the retrieval side of the problem for very large codebases in Context engineering.
Everyone seems to have reached roughly the same conclusion: do not wait for a bigger context window. Give the agent less, and be much more careful about what it sees.
But the problem is not solved. Cloudflare still reports that costs scale badly on large diffs. Partitioning helps us manage the problem; it does not make the problem disappear.
Conclusion
Today, we use agents to write code, and those agents rely on LLMs. We compare models using coding benchmarks and talk about which one is best at Python, TypeScript, or software engineering in general. Because these models are trained or fine-tuned on existing code, they also develop their own preferences. When asked to build something, they often reach for familiar languages such as Python or TypeScript.
But I think the more interesting question is what happens when writing code is no longer mainly a human job. Will agents still need something code-like to read? What if they can understand a codebase directly through its underlying structure (its syntax tree, dependencies, data flow, and behaviour) instead of reading source files line by line? And if the code is written and consumed mostly by machines, do we still need so many programming languages? Do we even need source code in the form we know today?
Maybe one day, models will build software without producing a traditional codebase at all. Perhaps the software will be represented in a form designed for machines rather than humans. If that happens, “context window” may stop being a phrase we worry about.
Or perhaps we will simply find a new way to run out of room.