Context Windows and Token Budgeting

The context window is a shared, finite budget for input and output tokens, not just the visible chat text.

Context & Reliability·Lesson 1 of 6·7 min

Every request to the Claude API operates inside a single finite budget: the context window. That budget is not "how much chat history fits on screen." It is the total token count across the system prompt, every tool definition you register, any retrieved documents or file contents you inject, the full conversation history, and the model's own output. All of it draws from the same pool. If you treat the context window as if it only holds user-visible messages, you will blow the budget in production long before you'd expect to, because tool schemas and retrieved context are frequently the largest consumers, not the dialogue itself.

What actually eats the budget

In a typical agentic system, the ranking of token consumers is often counterintuitive. A verbose system prompt with embedded few-shot examples can run several thousand tokens before a single user message arrives. Tool definitions compound this: each tool's name, description, and JSON schema for parameters costs tokens, and that cost is paid on every single request in the conversation, not once. An agent wired up with fifteen tools, each with a moderately detailed schema, can spend a meaningful fraction of a small context window before the model has read a single word from the user.

Retrieved data is the other major line item. A naive RAG pipeline that stuffs five full documents into context because a search returned five hits will exhaust budget fast, and it does so silently until the request fails or the model starts truncating its own reasoning to compensate. Budgeting is a design activity, not an afterthought: you have to account for system prompt tokens, tool schema tokens, expected retrieval tokens, expected history tokens, and reserved output tokens as separate line items that sum to less than the model's context window, with margin left over.

Output tokens count against the same budget

Because the context window is shared between input and output, a request that fills 95% of the window with input leaves almost no room for the model to respond, especially for tasks that require long-form output like generating a large code diff or a detailed report. Reliable systems reserve output budget explicitly rather than discovering the shortfall when a response gets cut off mid-sentence.

Targeted retrieval over reflexive whole-file reads

This budgeting discipline extends directly into codebase exploration. An agent given access to a large repository should not reflexively read entire files, let alone entire directories, when it needs one function signature or one config value. A targeted grep or search that narrows to the relevant lines costs a fraction of the tokens that a full-file read costs, and it leaves budget for everything else the agent needs to reason about. This is one of the most commonly tested judgment calls on the exam: given a scenario where an agent needs to find how a specific function is used across a codebase, the correct answer is almost always "search first, read narrowly," not "read the whole repository into context and then reason over it."

Exam trap A question describes an agent that "reads the entire file to be safe" before making a one-line edit, and asks whether this is good practice. The trap answer treats this as reasonable caution. The correct answer is that this wastes context budget for no proportional gain in correctness — targeted reads (grep for the function, read the surrounding 20 lines) achieve the same safety with a fraction of the token cost. Don't be swayed by framing that equates "reading more" with "being careful."

Scenario: An agent has a 200K-token context window, a system prompt of 3K tokens, twelve tool definitions totaling 4K tokens, and needs to reserve 8K tokens for output. A user request triggers a search that could return either three short snippets (600 tokens total) or the two full source files those snippets came from (22K tokens total). Which should the agent use, and why? The correct call is the snippets. The remaining budget after fixed costs is roughly 185K tokens, which sounds ample, but conversation history accumulates across turns, and burning 22K tokens on a single retrieval when 600 tokens would answer the question compounds badly over a multi-turn session. Budgeting is about the trajectory of usage across the whole conversation, not just whether a single request technically fits.

Try it

Take a real system prompt and tool set you use (or a sample one), and actually count the tokens in the system prompt plus all tool definitions using the Anthropic token counting endpoint or a local tokenizer. Compare that fixed cost against the model's total context window as a percentage. If it's over 10-15% before any conversation happens, identify which tool descriptions or system prompt sections could be trimmed or made conditional.