The problem it solves
A model cannot hold unlimited text in mind. There has to be a boundary on how much it processes in a single request, for reasons of both computing cost and how the underlying technology works. The context window is that boundary, made explicit.
Knowing it exists, and how big it is, is what lets you reason about real behavior: why a very long conversation starts to lose track of how it began, why an enormous document might not fit in one go, and why you sometimes have to be selective about what you give a model. Without this concept, those behaviors look like mysterious glitches. With it, they are predictable.
How it works
The context window is a fixed-size workspace, and everything the model needs for a response has to fit inside it.
Everything shares one budget. The window holds all of it together: the system instructions, the entire back-and-forth of the conversation, any retrieved documents or tool results, and the space reserved for the model’s own reply. They do not get separate allowances; they compete for the same total, counted in tokens.
When it fills up, the earliest material falls out. If a conversation grows longer than the window, the oldest parts drop out of view. This is why a long chat session can make a model seem to forget how it started: those early messages are simply no longer in the window. The model is not being forgetful; that content is outside what it can see.
Bigger is not automatically better. Context windows have grown enormously, and it is tempting to think a larger one is always an upgrade. But filling a huge window has costs: it is slower and more expensive, and models can actually lose accuracy when the input gets very large or cluttered. A smaller amount of well-chosen content often beats a giant pile of loosely relevant text. Fitting your task sensibly within the window usually matters more than having the biggest window available.
A concrete example
Imagine pasting a 300-page manual into a chat and asking a question.
If the manual fits within the context window, the model can answer from it directly. If it does not fit, something has to give: the model only ever sees the portion that fits, and the rest is invisible to it, no matter how relevant. This is exactly the situation where you would not dump the whole manual in, but instead retrieve only the relevant passages and place those in the window. The context window is the constraint that makes that decision necessary.
How it connects
The context window is the size limit on the context a model can use, and it is measured in tokens, which is why token counts and window sizes always appear together. It is also the reason RAG exists: when knowledge is too large to fit, you retrieve only the relevant pieces instead of trying to load everything.
For the building roles, respecting this limit is fundamental. An AI/LLM Developer manages what goes into the window; an AI Solutions Architect designs systems around its boundaries.