Chapter 2: How LLM Inference Works

Chapter 1 established what an agent is: a system that runs a plan-act-observe loop to produce a trajectory. But every step of that loop bottoms out in the same primitive operation — handing some text to a language model and getting text back. That operation is inference, and it is the substrate every agent is built on.

You will almost never train a foundation model. Frontier labs spend tens of millions of dollars and months of cluster time doing that, and the result is a frozen set of weights you reach through an API. What you will do, constantly, is run inference against those weights — and nearly every design decision you make as an agent builder is shaped by how inference behaves: how much you can fit in the context window, how the model decides which token comes next, how it signals that it wants to call a tool, what a call costs, how long it takes, and how it fails.

This chapter is about that substrate. We start with how a model turns a prompt into tokens, then work through the levers you actually control at inference time — the context window, sampling, structured output and function calling, and the economics of latency and cost. We cover reasoning models and the test-time-compute trade they make, the failure modes every harness has to defend against, and where inference is heading. None of this requires understanding how the model was trained. It requires understanding how the model runs — because that is what you are building on top of.

1. From Prompt to Token

A language model does exactly one thing: given a sequence of tokens, it predicts the next one. Everything an agent does is built out of that single operation, repeated.

A token is not a word. It’s a chunk of text the model’s tokenizer has learned to treat as a unit — sometimes a whole word (apple), often a word fragment (token, ization), sometimes punctuation or whitespace. The rule of thumb worth memorizing: for English text, one token is roughly four characters, or about ¾ of a word, so 1,000 tokens is roughly 750 words. Code, JSON, and non-English languages tokenize less efficiently — a curly brace, an indent, and a variable name can each cost their own token, which is why a screenful of code burns through your budget faster than a screenful of prose.

When you send a prompt, the first step is tokenization: your text is chopped into tokens, and each token is swapped for the integer ID that stands for it in the model’s vocabulary. So before the model does any “thinking,” your sentence has become a list of numbers.

The model reads that list and produces its output for just the next position — but not as a single word. What it actually emits is a logit for every token in its vocabulary: one raw score per candidate token, saying how strongly the model favors that token coming next. Vocabularies are large — tens of thousands of tokens for smaller models, 200,000+ for the biggest — so this is a very long list of scores, one entry for every token the model could possibly say next.

Those raw scores aren’t probabilities yet; they’re just unbounded numbers. A softmax fixes that, squashing the whole list into a proper probability distribution that sums to 100%: maybe token X at 31%, token Y at 18%, and a long tail of increasingly unlikely options. Now the model has a ranked set of candidates with real odds attached.

From that distribution it picks a single token (how it picks — greedily, or by sampling — is §3). It appends that token to the sequence, and then does the entire thing over again: the now-slightly-longer sequence goes back in, out come fresh logits for the next position, softmax, pick. And again, and again, one token per pass.

This loop is what’s meant by autoregressive generation: each token is produced in its own forward pass, conditioned on everything before it — including the tokens the model just generated itself. The model is always predicting the next token given every token so far, whether those tokens came from you or from its own earlier output.

Two consequences fall out of this that matter for everything downstream.

First, the model has no plan for its output. It is not composing a paragraph and then writing it down; it is choosing one token at a time, and a single early token can steer the rest of the generation. This is why prompting an agent to “think step by step” works at all (Chapter 1) — getting the right early tokens onto the page changes the distribution over every token that follows.

Second, generation is inherently sequential and one token at a time. Reading your prompt can be done in parallel — the model sees all of it at once — but producing the answer cannot be parallelized, because token N+1 depends on token N. This asymmetry between reading and writing is the root of the cost and latency story in §5, and it’s the single most important performance fact about inference. Keep it in mind; we’ll return to it.

2. The Context Window

The context window is the maximum number of tokens the model can attend to in a single forward pass — prompt plus generated output, counted together. It is the model’s entire working memory for a request. Anything outside it does not exist as far as that inference call is concerned.

Modern models advertise large windows — 200K tokens is common at the time of writing, with some models reaching 1M.¹ That sounds enormous, but for an agent it fills up faster than you’d expect, because the window has to hold everything the model needs to do its job on this turn:

• The system prompt — instructions, persona, the rules the agent operates under.
• The tool definitions — the name, description, and argument schema of every tool the agent can call. Ten tools with rich descriptions can be several thousand tokens before the conversation even starts.
• The conversation history — every prior user message, model response, and (critically) every tool result the agent has seen.
• The current turn — the latest input plus room to generate a response.

In a long-running agent, the third item dominates. Tool results are the silent budget-killer: a single file read, API response, or search result can be thousands of tokens, and an agent on a 50-step task accumulates dozens of them. This is exactly the problem Chapter 8 (the execution harness) spends its energy on — compaction, context editing, sub-agent isolation — and it’s why Chapter 3 (memory) exists at all. The context window is not free storage; it is a tight, expensive budget you actively manage.

It also helps to know that the window is processed in two distinct phases, because they have very different performance characteristics:

• Prefill — the model reads your entire prompt and builds up its internal state. This happens in parallel across all input tokens, so it’s fast per token, but it’s where most of your input cost lands.
• Decode — the model generates the output, one token at a time, each step reusing the state from prefill. This is sequential and is where output latency lives.

Two more things every builder should internalize. First, bigger is not automatically better: models exhibit a well-documented “lost in the middle” effect, where information buried in the center of a long context is recalled less reliably than the same information at the start or end.² A 200K window stuffed with marginally-relevant material can perform worse than a 20K window of tightly relevant context. Chapter 3 treats this in depth; for now, treat context as signal-to-noise, not raw capacity. Second, overflow is a hard wall, not a soft one — when prompt plus requested output exceeds the window, the request errors or silently truncates. Designing for that wall is the harness’s job (§7).

3. Sampling and Determinism

Once the model has produced its probability distribution over the next token, something has to choose. That choice is sampling, and it’s one of the few inference knobs you control directly on every request. Understanding it is the difference between an agent that behaves predictably and one that surprises you in production.

The simplest strategy is greedy decoding: always take the single highest-probability token. Deterministic, but it tends to produce flat, repetitive text and can get stuck in loops. Real systems sample from the distribution instead, shaped by a few parameters:

• Temperature rescales the distribution before sampling. At temperature 0 you get (near-)greedy behavior — the model almost always takes its top choice. As temperature rises toward 1.0 and beyond, the distribution flattens: lower-probability tokens get a real chance, output gets more varied and “creative,” and at high enough values it degrades into incoherence.
• Top-p (nucleus sampling) keeps only the smallest set of tokens whose probabilities sum to p (say 0.9), then samples from that set. It adapts to the model’s confidence: when the model is sure, the nucleus is tiny; when it’s uncertain, the nucleus widens.
• Top-k is the blunter cousin: keep the k most likely tokens, discard the rest, sample from what remains.

The practitioner takeaway is sharper than the theory. For most agentic work, you want low temperature — often 0, or close to it. An agent calling tools, following a format, or executing a multi-step plan is doing work where you want the most probable, most reliable continuation, not surprise. Creative variety is a liability when the next token is a tool name or a JSON key. Save higher temperatures for genuinely open-ended generation (brainstorming, drafting prose), and even then know that you’re trading reliability for diversity.

One caveat that trips up newcomers: temperature 0 is not a determinism guarantee. You might expect that “always take the top token” gives you identical output for identical input every time. In practice it usually doesn’t, for reasons that have nothing to do with sampling — floating-point non-associativity, GPU kernel scheduling, batching effects on the inference server, and silent model updates behind an API endpoint all introduce variation.³ This matters for agents in two concrete ways: you cannot rely on caching responses by exact input match across model versions, and you cannot write tests that assert on exact model output. Test on behavior and structure (did it call the right tool with valid arguments?), not on verbatim strings.

4. Structured Output and Function Calling

Here is where inference stops being “text in, text out” and starts being the mechanical foundation of agency. An agent is only an agent because it can act — call a tool, query an API, run code (Chapter 1). The bridge from a text-predicting model to a tool-calling agent is function calling, and it’s worth understanding what’s actually happening under the hood, because it’s less magical than it looks.

When you give a model a set of tools, you’re really doing two things. You’re putting the tool definitions — names, descriptions, argument schemas — into the context (§2), and you’re relying on the model, trained for this, to emit output in a structured form the runtime can recognize as “I want to call tool X with these arguments” rather than as a message for the user. The model doesn’t execute anything. It produces text that represents a call; your harness parses that, runs the actual function, and feeds the result back in as the next observation. The loop from Chapter 1 runs on exactly this mechanism.

The mechanism that makes this reliable is constrained decoding. Recall from §1 that at each step the model has a probability distribution over the whole vocabulary. Constrained decoding masks out every token that would violate the required structure — if the grammar says the next token must be a { or a valid JSON key, every other token’s probability is forced to zero before sampling.⁴ The model literally cannot emit malformed JSON when this is enforced, because the illegal tokens are never on the table. This is how providers offer “guaranteed valid JSON” or schema-constrained output: not by asking nicely, but by restricting what the sampler is allowed to choose at each step.

This has direct consequences for how you build:

• Prefer real function-calling APIs over parsing free text. Asking a model to “respond in JSON” in the prompt and then regex-ing the result is the old, fragile way. The model can wrap it in prose, add a markdown fence, or hallucinate a field. Schema-constrained tool calling moves that guarantee from your error-handling code into the decoder.
• Your schemas are part of your prompt. Tool names and descriptions are the only thing the model sees when deciding what to call. A vaguely-described tool gets misused; the fix for “the agent keeps calling the wrong tool” is usually a better description, not a better model. This is the inference-level reason the Model Context Protocol (Chapter 5) standardizes how tools are described.
• Constrained ≠ correct. Constrained decoding guarantees the call is well-formed, not that it’s sensible. The model can still emit valid JSON with a hallucinated file path or a nonsensical argument value (§7). Structure is enforced; meaning is not.

This is the single most important inference concept for an agent builder, because it’s the seam where “a model that talks” becomes “a system that acts.” Chapters 5 and 8 build directly on it.

5. The Economics of Inference: Latency and Cost

Every agent design decision eventually collides with two numbers: how long inference takes and what it costs. Both come straight out of the prefill/decode asymmetry from §2, so once you understand that, the economics follow.

The KV cache is the central object. During prefill, the model computes intermediate state (the “keys” and “values” of the attention mechanism) for every input token. Rather than recompute that on every subsequent decode step, it caches it. This is what makes generation tractable — without it, generating token 1,000 would mean reprocessing the entire prompt 1,000 times. But the cache lives in GPU memory and grows with context length, which is one of the practical reasons very long contexts are expensive and rate-limited: you’re not just paying for compute, you’re occupying scarce memory for the duration of the request.

Latency splits into two numbers you should track separately:

• Time to first token (TTFT) — how long until the model starts responding. This is dominated by prefill, so it scales with how long your prompt is. A 100K-token context means a noticeable wait before the first token appears, no matter how short the answer.
• Inter-token latency / tokens per second — how fast output streams once it starts. This is the decode phase, and because decode is sequential (§1), a long answer is slow to produce regardless of prompt size.

For an agent, this reframes the whole performance picture. A multi-step agent makes many inference calls, and the user waits through the TTFT of every one. A big system prompt and a long tool-result history don’t just cost tokens — they inflate TTFT on every single turn. Trimming context (Chapter 8) is a latency optimization as much as a cost one.

On cost, the rule that matters most: input and output tokens are priced separately, and output is several times more expensive than input — commonly 3–5×, and higher for some models (Gemini 2.5 runs ~8×). This isn’t arbitrary; it falls out of the asymmetry. Input is processed in parallel during prefill (cheap per token); output is generated sequentially during decode, holding GPU resources the whole time (expensive per token). The implications for agent design are concrete:

• A reasoning model that “thinks” for 5,000 tokens before answering (§6) is spending output-priced tokens to do it. Verbosity has a real bill.
• A verbose tool result you stuff back into context is input-priced — cheaper, but it’s now in the window for every future turn of the conversation, so you pay to re-read it on each call.

Prompt caching is the lever that changes this math. Because the expensive part of a long prompt is the prefill, providers let you cache the KV state of a stable prefix — your system prompt, tool definitions, few-shot examples — so subsequent requests reuse it at a steep discount (the cache-read discount is provider-specific — roughly 50% off with OpenAI, up to ~90% off with Anthropic and Google) and with much lower TTFT.⁵ For agents this is enormous: the system prompt and tool schemas are identical on every turn of a session, so structuring your context as [stable, cacheable prefix] + [changing suffix] can cut both cost and latency dramatically. Put the parts that never change first. This single layout decision is one of the highest-leverage things you can do for a production agent’s economics. (Chapter 15 returns to cost and latency as production concerns.)

6. Reasoning Models and Test-Time Compute

For most of the LLM era, the way to get a more capable model was to make it bigger and train it longer — pay at training time. Reasoning models opened a second axis: spend more compute at inference time, on the specific problem in front of you, and get better answers. This is test-time compute, and it’s reshaping how the planning loop from Chapter 1 actually runs.

The mechanism is simpler than the marketing suggests. A reasoning model is trained to produce an extended internal chain of work — call them reasoning tokens or “thinking” — before it commits to a final answer.⁶ Where a standard model might jump straight to a response, a reasoning model first generates a long stretch of intermediate steps: exploring approaches, checking its own work, backtracking. You often don’t see these tokens (some providers hide them), but you pay for them, and they’re what drives the quality gain on hard, multi-step problems — math, complex coding, intricate planning.

This connects directly to Chapter 1’s chain-of-thought. CoT showed that prompting a model to reason step by step improves multi-step performance. Reasoning models bake that behavior into the weights through training and let the model decide how much to reason — sometimes exposed to you as an adjustable “thinking budget.”

The trade is explicit, and you have to make it deliberately:

• More thinking costs more and takes longer. Those reasoning tokens are output-priced (§5) and generated sequentially, so a reasoning model can be markedly slower and pricier per call. On a hard problem that’s worth it; on a simple one it’s pure waste.
• It’s not always a win. For straightforward tasks — a routine tool call, a simple extraction, a format conversion — a reasoning model can overthink, burning latency and tokens to arrive where a fast standard model would have landed instantly.

The practitioner move is task-appropriate model selection, often within the same agent. Route the hard planning step to a reasoning model and the dozens of routine tool-calling steps to a fast, cheap one. Treating “which model, with how much thinking” as a per-step decision rather than a global setting is one of the clearest signs of a mature agent design. Chapter 10 develops this through the lens of model tiers (Opus/Sonnet/Haiku), and Chapter 16 returns to what cheaper reasoning means for the field.

7. Inference Failure Modes

An agent runs hundreds of inference calls, often unattended. At that volume, failure is not an edge case — it’s a steady drizzle you design for. The harness (Chapter 8) exists largely to absorb these, but you can’t defend against what you can’t name. Here are the inference-level failures every builder runs into.

Truncation at the token limit. Every request has a maximum output length. If the model hits it mid-generation, you get a response cut off in the middle — a half-written sentence, or worse, half-written JSON that fails to parse. The fix is to check the stop reason the API returns (did it finish naturally, or hit the length cap?) and handle the capped case explicitly rather than feeding a truncated string downstream as if it were complete.

Malformed tool calls. Even with constrained decoding (§4) reducing this, you’ll see calls that don’t match the schema — especially when a tool result truncates mid-call, or when you’re parsing free-text JSON instead of using a real function-calling API. Your harness needs a parse-failure path: reject the call, return a clear error to the model, and let it retry. Crashing the whole run because one tool call didn’t parse is the most common rookie mistake.

Hallucinated arguments. This is the failure constrained decoding cannot catch, and the one to worry about most. The model emits a perfectly well-formed call — valid JSON, correct schema — with an argument it invented: a file path that doesn’t exist, an ID it never saw, a config value it imagined. The structure is flawless; the content is fiction. The defense isn’t at the model layer at all — it’s making your tools validate their inputs and return informative errors, so the model gets told “no file at that path” and can correct, rather than the system proceeding on a fabricated value. (This is the concrete version of Chapter 1’s point: an agent that reads the file beats one that confidently reports a value it imagined.)

Nondeterminism. As covered in §3, identical inputs don’t guarantee identical outputs, even at temperature 0. Don’t build logic that assumes the model will make the same choice twice, and don’t write tests that assert on exact output strings.

Context overflow. The hard wall from §2. When accumulated history plus the new turn exceeds the window, the request fails. This isn’t a maybe — it’s a certainty on any long-running agent, and “manage the context before it overflows” is a core harness responsibility (Chapter 8), not an optional optimization.

The throughline: inference fails in structured, predictable ways, and a production agent treats every model call as something that can fail. The model is one component in a larger system, and the system’s reliability comes from the code around the model — validating, retrying, bounding, and recovering — not from hoping the model gets it right every time.

8. Where Inference Is Heading

The inference substrate is moving fast, and a few directions are worth watching because they’ll change how you build.

Longer, cheaper context. Windows keep growing, and per-token prices keep falling. But the “lost in the middle” problem (§2) hasn’t vanished — a bigger window doesn’t automatically mean better recall across it. Expect the capacity to keep outrunning the effective use of that capacity, which keeps memory and retrieval (Chapters 3 and 4) relevant even as raw window sizes climb. Bet on better context management, not on the window getting big enough to stop caring.

Faster decode. Because decode is the sequential bottleneck (§1), most inference-speed research targets it. Speculative decoding is the most prominent technique: a small, fast “draft” model proposes several tokens ahead, and the large model verifies them in a single parallel pass, accepting the run when the draft was right.⁷ The user-visible effect is the same text, produced faster — which for a multi-call agent compounds across every step.

Cheaper reasoning. Test-time compute (§6) is expensive today, which is why model selection matters so much. As reasoning gets cheaper and more controllable — finer-grained thinking budgets, better routing — the calculus shifts toward using it more liberally. Chapter 16 takes up what that means for the planning loop and for agent design broadly.

The constant across all of it is the shape from §1: a model predicts one token at a time, reads in parallel and writes sequentially, and exposes a handful of levers — context, sampling, structure, and compute budget — that you control on every call. The hardware and the numbers will keep changing. That shape, and the engineering discipline of treating the model as a fallible component in a larger system, is what the rest of this book builds on.

Summary

Agents consume models through inference, not training — and nearly every agent design decision is shaped by how inference behaves. A model does one thing: predict the next token, autoregressively, reading its input in parallel but generating output one token at a time. That single asymmetry drives the cost, latency, and capability story.

The levers you control on every call are few but consequential. The context window is a tight, actively-managed budget, not free storage — and bigger isn’t automatically better. Sampling (temperature, top-p, top-k) shapes the output; for agentic work you usually want low temperature, and you should never assume determinism, even at zero. Function calling, made reliable by constrained decoding, is the mechanical bridge from a model that talks to a system that acts — it guarantees well-formed calls but not sensible ones. The economics follow from prefill versus decode: input and output are priced separately, output costs more, and prompt caching a stable prefix is one of the highest-leverage optimizations available. Reasoning models add a test-time-compute axis — pay more per call for better answers on hard problems — making per-step model selection a real design decision.

Inference fails in predictable ways — truncation, malformed and hallucinated tool calls, nondeterminism, context overflow — and a production agent is built to absorb all of them. The model is a fallible component; reliability lives in the code around it. That principle, and the prompt-to-token shape underneath it, is the foundation the memory, retrieval, protocol, and harness chapters all build on.

Notes