LLM Context Window Explained: Tokens, Limits, and How to Manage Them

What is a token, and how do you count one?

A token is a chunk of text the model reads, usually a short word, part of a longer word, or a piece of punctuation. Models do not see characters or whole words the way we do. They break text into tokens first, then process those. A rough rule of thumb for English is that one token is about four characters, or roughly three quarters of a word. So 1,000 tokens lands somewhere around 750 words.

That ratio shifts with the content. Common words often map to a single token, while rare words, names, code, and non-English text get split into several. The word 'cat' might be one token; a long technical term or a JSON blob can be many. Whitespace and formatting count too. This is why a tidy paragraph and a deeply nested JSON object of the same character length can cost very different amounts.

Tokens are the unit you are billed in and the unit your window is measured in, so it pays to have a feel for the conversion. A typical page of prose is around 500 to 700 tokens. A long support transcript can be several thousand. A full PDF report dumped in raw can be tens of thousands. When someone says a model has a 128K window, they mean roughly 96,000 words of combined input and output, which sounds enormous until you start adding tool definitions and retrieved documents.

What actually counts against your context window?

Plenty of things consume the window before the user's actual question ever shows up. People tend to picture the window as 'my prompt,' but in a real production setup the user message is often a small slice of the total. Here is what is competing for the same budget on every single request:

• The system prompt: the instructions that define the model's role, tone, rules, and guardrails. This is fixed overhead on every turn, and detailed prompts can run into the thousands of tokens.
• Tool and function definitions: when a model can call tools, the schema for each one (name, description, parameters) is injected into the context so the model knows what is available. Ten well-documented tools can quietly eat a few thousand tokens before any work happens.
• Conversation history: every prior user message and model reply in the thread is usually re-sent so the model has continuity. This grows turn over turn, and a long chat can dominate the window all by itself.
• Retrieved documents: chunks pulled from a knowledge base, CRM records, support tickets, or files. This is often the single largest and most variable consumer, and the easiest place to waste tokens by pasting more than the model needs.
• The current user message: the actual question or instruction, which is frequently the smallest piece of the whole payload.
• The model's output: the response is generated into the same window, so reserving room for a thorough answer means less room for input.

The practical takeaway is that your usable budget for retrieved knowledge is the full window minus all the fixed and growing overhead above. If you have a 128K window, a 4K system prompt, 3K of tool schemas, and a chat history that has grown to 40K, you are not working with 128K for your documents. You are working with roughly 80K, and shrinking with every turn.

The lost-in-the-middle effect: why placement matters

Models do not pay equal attention across the whole window. The well-documented lost-in-the-middle effect, first laid out in detail by Liu and colleagues in their 2023 paper of that name, shows that models recall information placed at the beginning or the end of the context far better than information stuck in the middle. In their multi-document question-answering tests, accuracy dropped sharply when the relevant passage sat in the middle of a long context, and the dip held even for models built specifically for long contexts.

The size of the drop is not trivial. Across various 2024 to 2026 evaluations, models commonly lose somewhere between 10 and 30 percent accuracy on facts buried mid-context compared to the same facts placed at the edges. The longer the context, the worse it gets, because a fixed attention budget gets spread thinner as the token count climbs. Information that the model attended to clearly at 2,000 tokens can be functionally ignored at 100,000.

There is a related effect often called context rot: as you pile in more tokens, including content that is similar to but not actually relevant, the distractors pull attention away from what matters. More context can make answers worse, not better. This is the core reason why a giant window is not a free win. A 1M-token window that you fill to the brim will reason less reliably over the middle 600K than a tight 30K prompt would over its whole contents.

The practical fix is placement and pruning. Put the instructions and the most important facts near the top. Put the actual question near the bottom. And cut anything the model does not need so the signal-to-noise ratio stays high.

Why a bigger context window is not always better

The marketing for million-token windows makes it sound like you can stop thinking about context entirely. In practice, the advertised window and the effective window are two different numbers. Microsoft Research and others have found that effective utilization tends to fall off well before the ceiling, with meaningful degradation past roughly 100K tokens on demanding retrieval tasks. The model technically accepts the tokens; it just stops using them well.

A concrete way to picture it: on hard multi-needle retrieval tests, even a top model can read a fact reliably at 256K tokens and then miss the same kind of fact at 1M. The window did not lie about its capacity, but capacity is not the same as comprehension. Stuffing a contract package, a full codebase, or a quarter of support tickets into one prompt 'because it fits' often produces vaguer, slower, and pricier answers than a focused prompt would.

Big windows are genuinely useful for bounded, coherent inputs where you truly need everything at once, like reasoning over a single large document or a tightly scoped repository. They are a poor substitute for selection when your real corpus is large and messy. The instinct to dump everything in and let the model sort it out is the most common and most expensive mistake teams make.

What does context actually cost?

Tokens are the billing unit, so every token in the window has a price, and you pay on both sides. Input tokens (your prompt, history, tools, retrieved docs) are billed at one rate, and output tokens (the generated answer) usually cost more per token. Because history and retrieved content keep growing, the cost of a conversation tends to climb turn over turn even if each new question is short.

The compounding is the sneaky part. In a long agent run that re-sends the full history every step, you are repeatedly paying for the same early messages over and over. A ten-step task that carries a 20K-token history is not a 20K bill; it can be many times that as the same context rides along on each call. Latency follows the same curve, since more input tokens take longer to process before the first word of the answer appears.

This is why context management is a cost lever, not just a quality lever. Trimming a bloated system prompt, retrieving three relevant chunks instead of twenty, and summarizing old turns can cut spend and speed up responses at the same time. The cheapest, fastest token is the one you never put in the window.

How context management works, step by step

Good context management is the discipline of deciding what enters the window on each turn. Rather than sending everything, you assemble a focused payload, run the model, and clean up afterward. Here is the typical flow for a production agent answering a question against a large knowledge base.

The loop matters more than any single step. An agent that runs for many turns will blow past its window unless it actively compresses history and re-selects context each time. Teams that get this right treat the window as a budget to be spent deliberately, not a bucket to be filled.

Three strategies to keep context tight

Most production context management comes down to three techniques, often used together. Each one trades a little upfront work for a window that stays focused and affordable.

• Retrieval over stuffing: instead of pasting an entire document set, index it and pull only the passages relevant to the current question. This keeps the window small and the signal high, which is exactly what fights the lost-in-the-middle effect. Retrieval is the right default whenever your corpus is larger than the window or larger than you want to pay for.
• Summarization of history: as a conversation grows, replace older turns with a compact running summary that preserves decisions and facts while dropping verbatim back-and-forth. This caps history growth so a long thread does not slowly consume the whole window, and it keeps per-turn cost roughly flat instead of compounding.
• Pruning and selection: actively drop content that is not pulling its weight: redundant tool definitions, stale retrieved chunks, boilerplate, and duplicated context. Fewer distractors means cleaner attention and better answers. Every token you remove is one less thing competing for the model's attention and one less thing on the bill.
• Strategic placement: order what remains so the highest-value content sits at the edges of the window. Instructions and constraints go first, the question goes last, and supporting material fills the middle, where a mistake costs the least if it gets under-attended.

A worked example: a support agent answering a billing question

Picture a support rep at a logistics company. A customer named Dana writes in: 'Why was I charged twice on my May invoice, and can you refund the duplicate?' The rep hands this to an AI agent that has access to the billing system and the customer's history.

A naive setup would dump everything into the window: the full system prompt, schemas for a dozen tools, the entire two-year chat history with Dana, and every invoice on the account pasted as raw JSON. That might total 90K tokens, most of it noise. The duplicate charge, the one fact that matters, is now sitting somewhere in the middle of a huge payload, exactly where the model is most likely to gloss over it. The answer comes back slow, expensive, and hand-wavy.

A well-managed setup does the opposite. It retrieves only the May invoice and the two charges in question, summarizes Dana's prior history into a two-line note ('long-time customer, no past refunds, autopay enabled'), and trims the tool list to just the billing lookup and refund tools. The assembled window is maybe 8K tokens. The instructions sit at the top, the two charges sit clearly in context, and Dana's question sits at the bottom. The agent spots the duplicate, confirms the amount, and proposes the refund.

In a governed platform like Onpilot, that refund does not just fire blindly. The agent drafts the action, a human approves it because refunds are gated, the agent executes against the billing system under least-privilege access, and the whole thing lands in an audit log. Tight context made the answer good; the approval and audit made it safe to act on. The result, a finished resolution, gets delivered back to the rep in Slack.

Common mistakes that waste your window

Teams tend to repeat the same handful of context errors. Watching for these will save you both accuracy and money.

• Treating the window as free storage: dumping whole documents 'just in case' buries the relevant fact in the middle and dilutes attention. More tokens is not more intelligence.
• Never summarizing history: letting a conversation accumulate every turn verbatim until it crowds out retrieved context and slows every response. Cap it with a running summary early.
• Over-broad retrieval: pulling twenty chunks when three would do, on the theory that recall beats precision. Extra chunks are distractors that the lost-in-the-middle effect punishes.
• Bloated system prompts and tool lists: carrying instructions and tool schemas you no longer use. This is fixed overhead on every single turn, so the waste compounds across a whole conversation.
• Ignoring the output reservation: requesting long answers without leaving room, so the model truncates or the request fails. Reserve output budget up front.
• Assuming a bigger window removes the need to manage context: it raises the ceiling, but the lost-in-the-middle effect and the cost curve do not go away. Discipline still wins.

A decision framework: stuff, retrieve, or summarize?

Choosing an approach is mostly about the size and shape of your input relative to the window. A simple rule of thumb covers most cases.

Stuff the full context when your input is bounded, coherent, and comfortably fits with room to spare: a single contract, one report, a small repo you need reasoned over as a whole. There is no point chunking a 12K-token document into a 128K window. Just send it, and put the question at the end.

Retrieve when your corpus is larger than the window, larger than you want to pay for, or full of irrelevant material. A knowledge base of thousands of articles, a CRM with millions of records, or years of tickets all demand selection before the model ever sees anything. Retrieval is the default for any real production system at scale.

Summarize when the growth is over time rather than over documents: long-running chats and multi-step agent tasks where history is the thing ballooning. Compress old turns so each new request stays lean. In practice you often combine all three: retrieve the documents, summarize the history, and let the model reason over a tight, well-ordered window. The goal is never to maximize tokens. It is to give the model the cleanest possible context to do the job.