How an agent works, from request to answer
One request followed from the first message to the final answer: how text becomes tokens, what the model is sent and what it sends back, how a tool call is shaped, how the loop runs, why it has to be stopped and what a long run costs.
For anyone who wants to know what really happens inside an AI agent. No programming is needed: the short JSON samples are explained in the text, and it helps to have used a chat assistant once.
- What a token is, and why every limit and price is counted in tokens
- What one request to a model contains, and why the whole thing is sent again on every turn
- How a tool call is shaped, what the stop reasons mean and why the model never runs anything itself
- How the loop is built, why it needs a step limit and which checks make it reliable
- Why a long job costs far more than one question, and how caching changes the bill
Tokens: the unit of everything
A model does not read letters or words. It reads tokens, small pieces of text, and every limit, price and speed in this guide is counted in them.
A language model works on numbers, not on letters. Before any text reaches it, a tokenizer, a small program that belongs to the model, cuts the text into tokens: pieces that are often a short word, part of a longer word, a word with its leading space, or a punctuation mark. Each piece is replaced by its number in a fixed vocabulary. Many tokenizers use byte-pair encoding (BPE): start from a base vocabulary of single characters, or of the 256 byte values in the byte-level variant, and keep merging the most frequent adjacent pair until the vocabulary reaches a chosen size, tens of thousands of pieces or more (Hugging Face documents 50,257 for GPT-2). Other methods such as Unigram and WordPiece exist. Common words end up as one token, rare words, names and long numbers as several.
There is no fixed exchange rate between words and tokens. IBM gives roughly 1.5 tokens per word as a general estimate, while OpenAI's rule of thumb for English text is about three quarters of a word per token, which is roughly 1.3 tokens per word. Both are rough. Other languages and rare words often need more tokens for the same content, and tokenizers differ: Anthropic reports that Claude models from version 4.7 onward produce about 30 per cent more tokens for the same text than earlier ones (October 2026). A budget measured on one model therefore does not carry over to another.
Tokens matter because everything is counted in them. The context window, the most that one request and its reply can hold, is a number of tokens: one million on every current Claude model, with up to 128,000 of them for the reply (October 2026). Prices are quoted per million input tokens and per million output tokens, and output costs more. Anthropic lists Claude Opus 5.5 at $4 per million input tokens and $20 per million output tokens (October 2026).
Tokenization also explains some odd behaviour: the model sees a word as one or a few numbered pieces, not as letters, so counting the letters in a word can trip it up.
What one request contains
Each time the program asks the model something, it sends one package: standing instructions, the list of tools and the whole conversation so far. The model remembers nothing between two packages.
An agent run starts with a request to a model through an api, a programming interface reached over the internet. The request is more than the person's message. It is the whole context: the system prompt (the developer's standing instructions), the definitions of the available tools, any instruction files that were loaded, and the message history, which at the start holds only the task. The model takes every one of these tokens into account each time it produces a new token.
The history is a list of messages, and each message has a role that says who wrote it. In Anthropic's Messages API the system prompt is a separate top-level field, the list holds user and assistant messages (current models also accept a system message after a user turn, for instructions that only become relevant later), and the results of tools travel back inside a user message. OpenAI's Responses API keeps tool calls and their outputs as separate items in the same list, and OpenAI's Model Spec ranks instructions in a chain of command, platform before developer before user, with instructions found inside tool results treated as information rather than commands by default. The names differ; the structure is the same everywhere: who said what, in what order.
The API is stateless. Anthropic's documentation puts it plainly: you always send the full conversational history. The model keeps nothing between calls, so the harness, the program that calls the model and runs everything around it, rebuilds the complete input every time. What feels like a continuous conversation is a list of messages sent again with one more entry at the end. Every cost and every limit later in this guide follows from that fact.
What the model sends back
The model writes its reply one token at a time and then states why it stopped: it finished, it ran out of room, it was refused, or it wants a tool to be run.
The model turns the whole input into a probability for every token in its vocabulary. A sampling step picks one, the token is appended to the input, and the process repeats until the model ends its turn or reaches a limit. Settings such as temperature make the pick more or less adventurous, but sampling is random by design, so the same request can produce different answers. That matters later, when an agent has to be tested. The input is processed in one pass, while the reply is produced token by token, so long replies take noticeably longer than long inputs.
The reply ends with a stop reason, a field that tells the harness what to do next. Anthropic documents seven values, listed in the table; other providers use other names for the same idea. The harness must handle all of them, not only the final answer and the tool request. A reply cut off at the output limit looks like a normal answer that simply ends early, so code that ignores the stop reason will save half a sentence or half a JSON object without noticing.
A reasoning model first writes intermediate thinking tokens, a form of chain-of-thought, and then the answer. These tokens are extra computation at answer time. OpenAI's reasoning tokens stay hidden but occupy the context and are billed as output. Current Claude models decide adaptively how much to think, and an effort setting from low to max is the main control (October 2026).
The model looks nothing up. With no tool in the request, it continues the text with the most plausible words, so a missing fact is replaced by a believable one, which is a hallucination. Nothing in the output marks the difference.
How a tool call is shaped
A model cannot do anything by itself. It can only ask, in a fixed format, for a named tool to be run with certain inputs. The program around it decides whether to run it.
A tool is described to the model by a name, a description and a JSON Schema for its inputs: a machine-readable list of the fields, their types and which of them are required. These descriptions are part of the context. When the model wants to use a tool, it does not run anything. It emits a structured tool call that names the tool and fills in the inputs, and the reply stops with the stop reason tool_use. This mechanism is called function-calling.
The harness parses the call, validates the inputs against the schema, runs the real function and appends the output to the history as a tool result that carries the id of the call. If the tool fails, the result says so (is_error: true in Anthropic's format) with a short explanation, so that the model can try another way instead of guessing. The next request contains both the call and its result, and the model reads the result as ordinary input. Every effect on the outside world passes through code that a developer wrote and can restrict.
One reply can contain several tool calls. Claude 4 and later models make such parallel calls by default when the calls are independent, for example three fare searches for three dates. The harness may run them at the same time: the Claude Agent SDK runs read-only tools concurrently and tools that change something one after the other. All results go back together in the next user message, each matched to its call by id. Sending each result as a separate message is a documented mistake, because it teaches the model to stop making parallel calls.
Two settings shape the call. tool_choice decides whether the model may answer without a tool (auto, the default) or must not use one (none), and on models that support it, whether it must call some tool or one named tool. Strict mode makes the provider guarantee that every call matches the schema exactly, which removes a whole class of parsing errors. OpenAI suggests keeping fewer than 20 functions available at the start of a turn.
The Model Context Protocol standardises how a harness reaches tools it did not write. An MCP client in the harness talks to an mcp-server, which lists its tools and runs them on request. On the wire to the model the shapes above stay the same.
The loop that ties it together
The agent is a simple cycle run by ordinary code: ask the model, run the tools it asks for, add the results, ask again, until the model answers without asking for a tool.
The agent loop is a cycle that the harness runs. It assembles the context, calls the model and reads the stop reason. If the model asked for tools, the harness checks the permission policy, runs them, appends the results and calls the model again. If the model returned plain text, the loop ends. One pass is a turn. The model chooses each next step; the harness does everything else.
The pattern was described in 2022 as ReAct, reasoning interleaved with actions. With only one or two examples in the prompt, it beat imitation and reinforcement-learning baselines by 34 and 10 percentage points on two interactive benchmarks. Almost every agent today runs a version of it.
Agent frameworks package the loop. In the Claude Agent SDK each full cycle of tool calls and results is one turn, and the loop continues until the model produces a response with no tool calls. The OpenAI Agents SDK runs the same cycle and adds handoffs, in which one agent passes the conversation to another. A system made of a model, its context, its tools and this loop is what the field calls an agent.
Why the loop must be stopped, and how it stays right
Nothing guarantees that a model will ever say it is done. The program around it must enforce limits on steps, money and time, and must check the work with something other than the model's own opinion.
Left alone, the loop ends only when the model decides to answer without a tool, and nothing guarantees that it will. A model can retry a failing tool with the same arguments, alternate between two states, or keep researching an open-ended goal such as “improve this”. Each of those turns is billed, and each costs more than the last, because the history keeps growing. A stop-condition enforced in code ends it: a turn cap, a token or money budget, a time limit, or an interruption by the person. The model's own sense of being done is useful, but it is not a limit.
Frameworks provide these limits but do not always switch them on. In the Claude Agent SDK, max_turns counts tool-use turns and max_budget_usd stops at an estimated spend; both default to no limit, and hitting one ends the run with a result such as error_max_turns. Spend is checked after each response, so a run can overshoot the cap by the cost of one reply. The OpenAI Agents SDK raises a MaxTurnsExceeded error when its cap is passed. Treat a hit limit as a result to report, not as a crash, and save the state so that the run can resume.
Errors also compound. If every step is right 95 per cent of the time, a ten-step job is right only about 60 per cent of the time (0.95 to the tenth power). A long job therefore needs checks along the way, and the strongest checks come from outside the model. A model that reviews its own work in the same context has no new information; research on self-correction found that models often fail to fix their own reasoning without external feedback. Claude Code's best practices put it simply: give the agent a check it can run, such as tests, a build or a screenshot to compare.
Where the guardrails sit
Rules that must always hold are enforced by the program around the model, not by asking the model nicely: permission rules, an isolated sandbox, and scripts that run at fixed moments.
Permissions sit in the Act step. The harness matches each tool call against a policy with three outcomes: allow, ask the person, or deny, and deny wins when rules conflict. Because this is code, the model cannot talk its way past it. A sandbox adds isolation underneath: the tool runs in a container or virtual machine with limited files and a restricted network.
Hooks are scripts that the harness runs at fixed events, such as before a tool call, after an edit, before compaction or when the agent tries to finish; Claude Code documents 33 such events (October 2026). A hook receives the event as JSON and can block the action, commonly by exiting with code 2, rewrite its input or demand that tests pass before the run may end. Hooks run every time, whatever the model decided. A rule that must always hold belongs in a hook or a permission. A sentence in the system prompt is a request that the model usually follows.
A person in the loop is a guardrail only if they read what they approve. Anthropic reports that Claude Code users approve 93 per cent of permission prompts (March 2026), which it calls approval fatigue. Ask only for actions that are risky or cannot be undone, and let rules handle the routine.
Prompt injection exists because the model reads instructions and data as one stream of tokens, so text in a fetched page can steer it. The lethal-trifecta names the dangerous combination: access to private data, exposure to untrusted content and a way to send data out. Remove one of the three, narrow the privileges, and put a person in front of sensitive actions.
When the job outgrows one context
A long job produces more text than one context can hold. The program then summarises old turns, keeps notes in files, or hands parts of the job to helpers that each start with a fresh context.
Every turn adds the model's reply and the tool results to the history, so the context only grows. When it nears the window, the harness has three options. Compaction replaces older turns with a summary and keeps recent turns word for word; the Claude Agent SDK does this automatically near the limit and marks the point in its message stream. Memory writes facts and progress to files outside the context, to be read back later. A sub-agent takes one sub-task in a fresh context and returns a short summary, so the main context never receives its intermediate results.
Each option loses something. A summary can drop a detail that a later step needs, such as an exact error message, so rules that must survive belong in a project file such as CLAUDE.md, which the SDK sends again with every request. A sub-agent knows only what its brief says. Multi-agent runs also use many more tokens: Anthropic's research system, a lead agent with parallel sub-agents, beat a single agent by 90.2 per cent on an internal research evaluation but used about 15 times the tokens of a chat (2025). They pay off for work that splits into independent parts, not for strictly sequential work.
For jobs that span many sessions, Anthropic's harness for long coding runs keeps a feature list and a progress log that every new session reads first. It guards against two documented failures: trying to build everything at once and declaring victory too early.
What a run costs, and how to know it works
Because the whole conversation is sent again on every step, the amount of text processed grows much faster than the conversation itself. Caching cuts the bill, and repeated tests on recorded tasks show whether the agent works.
Because the whole history is re-sent on every turn, the input processed grows much faster than the final context. Suppose the first request is 5,000 tokens and each turn adds 2,000. Ten turns process 140,000 input tokens in total, although the last request is only 23,000 tokens long. The total grows with the square of the number of turns: a hundred turns process 10.4 million tokens. The table puts a price on it at $4 per million input tokens.
Prompt caching is the main remedy. The provider keeps the processed form of a request's beginning, the KV cache, for a few minutes, and a new request that starts with exactly the same tokens reads it back at a fraction of the price. At Anthropic a cache read costs a tenth of the normal input price or less, and writing an entry costs 1.25 times for the five-minute lifetime (October 2026). In the ten-turn example, 117,000 of the 140,000 tokens are reads and 23,000 are writes, which, at a tenth of the input price for reads, bills like about 40,000 tokens if every turn hits the cache. Agents read far more than they write: Manus reports about 100 input tokens for every output token, so the cache hit rate decides most of the bill.
Evals measure a run against tasks with a defined pass or fail rule, scored by code, by a model following a rubric, or by a person. Anthropic's guide (January 2026) suggests starting with 20 to 50 tasks taken from real failures and grading the outcome rather than the exact path. Each task runs several times, because results vary: pass@k counts a task as solved if any of k tries works, pass^k only if all k do.
A trace, the timeline of every model call and tool call with arguments, results, token counts and timing, shows which step failed and what each step cost. See cost and governance for the money and the rules around a live agent.
A working mental model
An agent is a model that remembers nothing, inside a program that rebuilds its input, runs its requests and decides when to stop. Most design questions are about that program.
An agent is a stateless model inside a program that rebuilds its input, executes its requests and decides when to stop. Most design questions reduce to what that program puts in the input and what it refuses to execute.