Context engineering: deciding what the model sees
The model sees only what is in its context window, and that window is limited, costly and easy to clutter. This guide covers what fills it, why more is not better, how a context goes wrong, and what caching, trimming, retrieval, compaction, memory files and sub-agents do about it.
For anyone who builds or configures an agent, or wants to understand why a long session gets worse. Reading the guide on how an agent works first helps; the terms are explained here as well.
- What fills the window of an agent request, which parts grow, and how to measure them
- Why long inputs degrade quality, why position matters, and the four typical ways a context fails
- How prompt caching works, its rules and prices, and which habits break it
- How trimming, retrieval, compaction, tool-result clearing, memory files and sub-agents each trade detail for space
- A short set of practices for keeping context small and useful
What fills the window
Everything the model can see in one step must fit into a fixed space called the context window. Instructions, tool descriptions, the conversation and tool results all compete for that space.
Context engineering is the work of deciding what goes into the context at each step of a run. The context is everything sent in one request: the system prompt, the tool definitions, instruction files, the message history and the tool results. The context window is all the text a model can take into account when it writes a reply, the reply included, measured in tokens, the small pieces of text that models read. Every current Claude model has a window of one million tokens, with up to 128,000 of them for the reply (October 2026); many smaller or older models offer 128,000 to 200,000.
Treat the window as a budget with hard rules. Everything counts: the system prompt, the tool definitions, every message including tool results, images and documents, and the reply with any thinking. Tokens served from a cache count as well; caching changes their price, not the space they take. If the input alone is larger than the window, the API refuses the request; if the reply runs into the limit, it stops with a stop reason that says so.
The parts behave differently, which decides how to manage them. Instructions and tool definitions are written once and sent again on every turn: in one Anthropic example, five MCP servers with 58 tools used about 55,000 tokens of definitions before any work started. History and tool results only grow, and tool results are often the largest part, because one file read or search can return thousands of tokens. Claude Code cuts a single tool response at 25,000 tokens by default.
You cannot manage what you do not measure. API replies report the input and output tokens of each request, and how many input tokens were read from or written to the cache (cache_read_input_tokens and cache_creation_input_tokens at Anthropic, cached_tokens at OpenAI). Logging these per turn, together with the size of each part of the request, shows quickly which part is crowding the window.
Why more context is not better
A model does not use a long input evenly. Facts in the middle are easily overlooked, and quality drops as the input grows, even when the window is far from full.
The model's attention mechanism relates every token to every other token, which Anthropic's engineers describe as n² pairwise relationships for n tokens. The work grows with the square of the length: double the input and the attention work roughly quadruples. More important than the cost is the dilution. Attention is a budget, every added token takes a share of it, and a relevant line competes with everything around it.
Lost in the middle is the best-known position effect. Liu and colleagues (2023) gave models 20 documents, one of which held the answer, and moved that document through the positions. Accuracy was highest at the start or the end and dropped in the middle; in one setting GPT-3.5-Turbo did worse with the answer in the middle (53.8 per cent) than with no documents at all (56.1 per cent). The study tested models available in 2023, so read the figures as a demonstration of the effect, not as a measure of current models. Anthropic's prompting guide advises putting long documents at the top and the question at the end, which it says can improve response quality by up to 30 per cent in tests.
Context rot is the broader decline. In Chroma's study (July 2025), 18 models, among them GPT-4.1, Claude 4, Gemini 2.5 and Qwen3, grew increasingly unreliable as the input grew, even on deliberately simple tasks. A single distractor, a passage that looks like the answer but is not, already lowered accuracy, and the loss grew faster when the question and the answer shared few words. Oddly, models did better on haystacks of shuffled sentences than on coherent text.
The working rule that follows is to aim for the smallest set of high-signal tokens that lets the model take the next step well. This is a rule of thumb, not a proven threshold, and the right size depends on the model and the task. Length also costs money and time: the whole input is processed on every turn, so a bloated context makes each turn slower and dearer.
Four ways a context goes wrong
Besides being too long, a context can be wrong in four typical ways: it contains an error, it drowns the task, it carries irrelevant material, or it contradicts itself. Each has its own fix.
Drew Breunig's essay on how long contexts fail (June 2025) names four patterns that recur in agent runs, and LangChain's guide to context engineering adopted them. They are useful because each points to a different fix, listed in the table.
The essay collects some numbers. In one study a small open model (Llama 3.1 8B) failed a task when it was offered 46 tools and succeeded with 19, although both sets fitted easily in its 16,000-token window. A Databricks study saw correctness begin to fall around 32,000 tokens for Llama 3.1 405B, and earlier for smaller models. When Microsoft and Salesforce researchers spread a task's information over several turns instead of giving it at once, results dropped by 39 per cent on average.
Two of these failures are made by the loop itself. Clash grows when a run keeps every failed attempt and every superseded plan in view. Removing every error is not the answer either: Manus reports that leaving failed attempts in the context helps the model avoid repeating them. The skill is to keep the lesson and drop the noise, for example by replacing a long error log with one line that names the error and the fix.
Order matters: prompt caching
Providers can reuse the work of reading the same beginning of a request. That makes repeated requests much cheaper and faster, but only if the beginning stays exactly the same.
When a model reads a request, it computes internal state for every token, the KV cache. Prompt caching keeps that state for the beginning of a request and reuses it when the next request starts with exactly the same tokens. A cache hit is cheaper and faster than a fresh read. Manus calls the cache hit rate the single most important metric for a production agent, because agents read far more than they write: about 100 input tokens for every output token in its case.
At Anthropic the cached beginning is built in a fixed order, tools, then system, then messages, and you mark up to four breakpoints with cache_control, or let automatic caching move one breakpoint forward as the conversation grows. An entry lives five minutes, and the timer restarts each time it is read; a one-hour entry is also available. Writing costs 1.25 times the input price (twice for an hour), reading a tenth of it by default and half of that on Claude Opus 5.5 and Sonnet 5.5 (October 2026). Below a minimum length, 512 tokens on the current models and more on older ones, nothing is cached and no error is returned.
OpenAI caches automatically from 1,024 tokens on its newest models, keeps a prefix for at least 30 minutes after its last use there, and charges a tenth of the input price or less for a read. The details differ between providers; the rule does not: only an identical beginning is reused. So put what never changes first and what changes last. In an agent loop this works well, since each turn repeats the previous request and adds a little at the end.
Anything that changes an earlier part invalidates everything after it. Common causes are a timestamp or request id near the top, tools added, removed or reordered mid-run, settings that alter the system prompt or the messages (see the table), JSON written with keys in a different order each time, and edits to earlier messages, including clearing old tool results. Manus therefore keeps its context append-only and hides unavailable tools by masking them during generation instead of removing their definitions.
Compaction rewrites the history, so the cache is rebuilt once after it; a breakpoint at the end of the system prompt keeps that part cached. A pause longer than the entry's lifetime means a full re-read on the next turn.
Keeping the window lean
Most of what crowds the window arrives through tools: their descriptions and what they return. Smaller tool sets, shorter results and instructions that load only when needed keep it clear.
Keep the tool set small and give each tool a precise description, because every definition is paid for on every turn. Claude's documentation notes that picking the right tool degrades beyond 30 to 50 tools. Tool search keeps rarely used definitions out of the context until the model looks for them, and loads only the three to five it needs. Anthropic reported that its tool search (November 2025) cut tokens spent on tool definitions by about 85 per cent and raised accuracy for Opus 4 on an MCP evaluation from 49 to 74 per cent. Because a found definition is added at the end of the conversation, the cached beginning stays intact.
Make tool results small by design. Anthropic's advice to tool authors is to combine pagination, ranges, filters and truncation with sensible defaults, to let the agent choose a concise or a detailed response (72 against 206 tokens in its example), and to return names rather than opaque ids. Strip noise such as navigation text from fetched pages before it enters the history.
A further step is to let the agent write code that calls tools and filters their output before anything enters the context. Anthropic reported that presenting MCP servers as code cut one workflow from about 150,000 to 2,000 tokens (November 2025).
For know-how, use progressive disclosure: skills keep only a name and a description in the context, about 100 tokens per skill, and load the full instructions, usually under 5,000 tokens, when a task needs them. Instruction files such as AGENTS.md are the opposite: loaded in full on every request. Keep them short and specific. A February 2026 study found that such files did not generally improve task success but raised inference cost by over 20 per cent on average; specific instructions were followed, while general overviews of the repository did not help.
Fetching only what is needed
When the model needs facts it was not trained on, the program fetches only the relevant pieces, either in advance from a prepared index or step by step with search tools.
A model cannot know facts that are outside its training or newer than its cut-off, so the harness must supply them, and it should supply only the relevant part. Retrieval-augmented generation (rag) does this ahead of the answer: documents are cut into chunks (chunking), each chunk is turned into an embedding (a vector of numbers that stands for its meaning), and the chunks closest in meaning to the question are placed in the context. Embeddings can miss exact names and codes, so good systems combine them with keyword search (hybrid-search) and sort the results again with a second model (reranking). Anthropic reported that adding a short context to each chunk plus keyword search cut failed retrievals by 49 per cent, and by 67 per cent with reranking (2024).
The alternative is agentic-search: the agent uses ordinary tools such as listing files, searching text and reading a range of lines, and decides step by step what to look at. It loads information just in time instead of in advance, and it can refine a query that failed. It costs more turns, but it needs no index and does not go stale. The team behind Claude Code tried retrieval over a code index and settled on agentic search, which in its creator's informal assessment outperformed everything, by a lot.
For a small collection, skip retrieval: Anthropic suggests that a knowledge base under about 200,000 tokens can go into the prompt whole, with caching to keep the repeated cost down. Whatever is retrieved, put it where position helps (long material first, the question last), mark each passage with its source, and drop near-duplicates, which act as distractors.
When the history gets long
When the conversation grows too long, older parts are either summarised or removed. Both free space, and both can lose a detail that a later step needs.
At some point the history itself is the problem. Compaction asks a model to summarise older turns (decisions made, open problems, files touched) and continues with the summary plus the latest turns word for word. The Claude API offers it on the server: when the input passes a threshold (150,000 tokens by default, at least 50,000), the reply carries a compaction block with the summary, and everything before that block is dropped from then on. Custom instructions can say what to keep, and they replace the default summary prompt entirely. The Claude Agent SDK compacts automatically near the limit, can run a hook first, for example to archive the full transcript, and re-sends project files such as CLAUDE.md with every request, so rules that live there survive.
Tool-result clearing is the lightest form, and Anthropic's engineers call it one of the safest. Once the prompt passes a trigger (100,000 input tokens by default), the API replaces the oldest tool results with a placeholder and keeps the three most recent. Results of named tools can be excluded, and a minimum amount to clear makes sure that breaking the cache is worth it. In a 100-turn web-search evaluation, Anthropic reported that context editing, which includes this clearing, cut token consumption by 84 per cent (September 2025).
Compaction is lossy, and its typical failures are forgetting a decision, repeating finished work and dropping an exact error message. Reduce the risk by telling the summariser what to preserve, keeping exact errors and identifiers, and writing key state to a file before the window fills, as the next part describes.
Memory outside the window
Anything written to a file survives when the conversation is summarised or a new session starts. Agents use such notes to keep the plan, the progress and lasting facts.
The model starts every session empty, and compaction forgets. Memory in an agent is therefore storage outside the context that the agent writes to and reads from through tools. Anthropic's memory tool shows the mechanism: the model asks for file operations in a /memories directory (view, create, replace text, insert, delete, rename), the application carries them out on its own storage, and the model checks that directory before it starts a task.
For one long job, the most useful memory is a progress file. Anthropic's harness for long coding runs has a first session write a list of features, all marked as failing, and a progress log; every later session reads the log and the version history, picks the next unfinished feature and commits when it is done. The list is kept in JSON because models were less likely to rewrite it carelessly than Markdown. A NOTES.md that the agent keeps up to date serves the same purpose in simpler setups.
Notes also steer attention. Manus has its agent rewrite a to-do list as it works, which puts the current goals at the end of the context, in the model's recent attention span, instead of in a message dozens of tool calls back; a typical Manus task takes about 50 tool calls.
Memory has the same failure modes as context, but they last longer. A wrong note poisons every later session, and a note planted by an attacker is a lasting prompt-injection. Check facts before writing them, keep notes short and dated, scope them to one user or project, and restrict paths: the memory tool's documentation warns that a path such as /memories/../../secrets.env must be rejected. Long-term memory across sessions is loaded selectively, like retrieved knowledge, not all at once.
Isolating work in sub-agents
A separate helper agent can do a messy sub-task in its own fresh context and hand back only a short result, so the main conversation stays small.
A sub-agent is a separate run with its own fresh context. The parent writes a brief, the sub-agent works through as many searches and reads as it needs, and only a short result returns. In Claude Code a sub-agent starts with its own system prompt and none of the parent's turns, and most also load the project's instruction files; Claude Code ships built-in ones, such as a read-only Explore agent and a Plan agent, which skip them. Anthropic's engineers report that a sub-agent may spend tens of thousands of tokens exploring and still return a summary of often 1,000 to 2,000 tokens. Sub-tasks can also run in parallel.
The costs are real. The sub-agent knows only what the brief says, so a vague brief produces work in the wrong direction. Parallel agents can make conflicting decisions because none sees the others' reasoning, a point Cognition made against parallel multi-agent designs in 2025. Total token use rises: Anthropic's multi-agent research system beat a single agent by 90.2 per cent on its internal evaluation but used about 15 times the tokens of a chat (2025). Anthropic also notes that such systems fit poorly where all agents need the same context or have many dependencies on each other, and that most coding tasks have fewer parallelizable parts than research. Use sub-agents for read-heavy exploration that can be summarised, and keep tightly coupled steps in one context.
Practices to carry away
Good context work comes down to four moves: store information outside the window, bring in only what the next step needs, shrink what stays, and split work into separate contexts.
Context engineering is often summed up as four moves: write context out to storage, select what to bring in, compress what stays, and isolate work in separate contexts. When a run goes wrong, read the exact request that the model received at the failing step before changing the prompt, because the cause is usually something that was missing or something that crowded the right line out.