Chapter 1: What Is an Agentic AI System?

Ask a language model a question and it answers. Ask it again and it answers again, independently, with no memory of the first exchange, no awareness that you asked something related five minutes ago, and no ability to go check whether its answer was actually correct. This is the defining shape of a chatbot: a stateless input-output function, sophisticated in its language understanding, but fundamentally passive. It receives a prompt. It produces a response. It stops.

An agentic AI system does something different. It receives a goal. It breaks that goal into steps. It takes actions — querying a database, running code, calling an API, reading a file, sending a message — and observes what those actions return. It uses what it observes to decide what to do next. It persists across multiple steps, potentially across multiple sessions, adjusting its plan as new information arrives. And it does all of this with meaningful autonomy: the human does not need to specify each action in advance.

The difference sounds simple stated this way. It’s less simple to build, and getting the concepts precise — not just intuitive — is what the rest of this book depends on. This chapter lays that groundwork: what actually separates a chatbot from an agent, the vocabulary you’ll meet in every framework and toolkit, the handful of architectural patterns that make agents work, where these ideas came from, and where the technology honestly stands today — capabilities and limits both.

1. The Conceptual Divide

1.1 Responses vs. Trajectories

The most useful way to understand the chatbot-to-agent transition is not as a capability upgrade — “agents are better chatbots” — but as a structural one. Chatbots produce responses. Agents produce trajectories.

A response is a single artifact: text generated in one forward pass, ending when the output token stream reaches its stopping condition. A trajectory is a sequence of decisions, actions, and observations that unfolds across time. Each step in the trajectory changes the state of the world in some way — a file is written, a query returns results, a tool call succeeds or fails — and those changes inform the next step.

This structural difference has a cascade of consequences. A response can be evaluated at a single point in time: did the model produce the right text? A trajectory must be evaluated across its entire unfolding: did the agent’s sequence of decisions lead to the right outcome? A response cannot recover from a wrong first move, because there is no second move. A trajectory can — an agent that sees that its last action returned an error can try a different approach.

The field has converged on two related vocabularies for talking about trajectories. One comes from reinforcement learning, and one comes from cognitive science. Both are worth knowing, because the literature uses both.

1.2 The Reinforcement Learning Vocabulary

Reinforcement learning treats any decision-making system as an agent interacting with an environment. The agent observes the current state of the environment, selects an action from its action space, executes that action, and receives two things in return: a new state (the environment’s response to the action) and a reward (a scalar signal encoding how good or bad the action was). The agent’s goal is to learn a policy — a mapping from states to actions — that maximizes cumulative reward over time. The sequence of states, actions, and observations that the agent traces through during a task is its trajectory.

This language predates LLMs by decades — it was built for game-playing systems, robot controllers, and the like. But it fits LLM agents almost perfectly, even though they’re built completely differently, because it describes the same underlying activity: observe a situation, pick an action, see what happens, repeat. That’s why agent frameworks and docs lean on these terms, and why they’re worth knowing.

One distinction is especially useful in practice: an agent’s actions come in two flavors. Internal actions operate on the agent’s own context and memory — retrieving a fact, updating a plan, jotting something in a scratchpad. External actions reach outside the model — running a search, executing code, calling an API, messaging a user. The split matters because the two behave differently: external actions are slower, can fail for reasons outside the agent’s control, and are where most of your error handling and guardrails will live.

1.3 The Cognitive Architecture Vocabulary

The second vocabulary comes from cognitive science, by way of a paper called CoALA (“Cognitive Architectures for Language Agents,” Sumers et al., 2023). It describes each turn of the loop as four steps: propose candidate actions, evaluate them, select one, and execute it. The loop runs until the task is done or the agent gives up. If you’ve seen the plan-act-observe loop described elsewhere, this is the same idea spelled out in finer grain.

CoALA also names four kinds of memory an agent draws on: working memory (what’s in the active context right now), episodic memory (records of past experiences), semantic memory (general knowledge), and procedural memory (how to do things). Chapter 3 covers all of this in depth — for now, just note that the decision loop runs over all four at once, and getting their interaction right is much of what makes an agent good or bad.

1.4 What the Loop Actually Looks Like

In practice, the plan-act-observe loop of a language agent looks something like this:

1. The agent receives a goal: “Find the bug causing the production issue in the payment processing service and fix it.”
2. It generates a plan: check the logs, identify the error, locate the relevant code, understand the bug, write a fix, test it.
3. It takes its first action: search the logs. It calls a tool that returns the last 500 lines of production logs.
4. It observes the result: there is a null pointer exception at line 847 of PaymentProcessor.java.
5. It updates its model of the situation and takes the next action: read the relevant code.
6. It observes the code, identifies the bug (a missing null check), writes a fix, runs the tests, observes the test results, and either commits the fix or iterates if tests fail.

Each step is a cycle of the loop. The agent is not executing a pre-specified script — it is making decisions at each step based on what it has observed so far. This is what makes it an agent rather than a chatbot: the actions it takes are conditioned on a state that evolves as the task unfolds.

2. The Core Architectures

2.1 Chain-of-Thought: The Prerequisite

Before agents could reason about multi-step tasks, language models needed to be able to reason at all. The technique that unlocked this — chain-of-thought prompting — was introduced by Wei et al. in 2022.

The idea is simple: instead of asking a model for an answer directly, you prompt it to write out its reasoning step by step before committing to a conclusion. “What’s 17 × 24?” gets a wrong guess; “What’s 17 × 24? Think step by step” gets the model to break it down — and it lands on the right answer far more often. The same trick works for any multi-step problem.

Why does this help? A multi-step problem is hard to compress into a single token prediction. When the model writes out intermediate steps, each step becomes part of the input for the next one — effectively giving the model more room to compute. The reasoning is visible in the output rather than crammed into one leap.

Chain-of-thought matters for agents because planning is multi-step reasoning. An agent that can’t reason through a problem can’t plan a solution to it. CoT is the floor that everything else in this chapter builds on.

2.2 ReAct: Reasoning and Acting Interleaved

Chain-of-thought has a critical weakness: the model reasons entirely from what it already knows. If it doesn’t know something — or thinks it does but is wrong — there’s no way to catch the error. The reasoning looks coherent but can be completely detached from reality. This is the hallucination problem, and for anything that touches real systems or current data, it’s disqualifying.

ReAct, introduced by Yao et al. in 2022, fixes this by letting the model act between reasoning steps. It structures the agent’s work as a repeating cycle of three moves:

• Thought: a reasoning step — the agent plans what to do next.
• Act: a concrete action, usually a tool call.
• Observation: what the action returned.

The cycle repeats: the observation feeds the next thought, which drives the next action. The crucial shift is that the reasoning is no longer happening in a vacuum — every step is anchored to real information the agent just fetched.

In the original paper, this consistently beat both plain chain-of-thought and “just act, don’t reason” approaches across question-answering and simulated task environments. But the headline benchmark numbers matter less than why it won: grounding the reasoning in retrieved facts sharply reduced hallucination. On HotpotQA, hallucination was chain-of-thought’s single largest failure mode — 56% of its failures — and even when both approaches reached an answer, pure CoT returned false ones more than twice as often as ReAct (14% vs. 6%). When the agent had to look something up before reasoning about it, it largely stopped making things up.

That is the practical payoff of tool use, and it’s worth being precise about it. The value isn’t just that the agent can reach information it didn’t have. It’s that the act of reaching for that information forces the reasoning to rest on facts you can verify, instead of plausible-sounding claims you can’t. For an engineer, this is the difference between an agent that confidently reports a config value it imagined and one that reads the file first.

The ReAct Prompt

To make this concrete, consider a simplified example of what ReAct traces look like on a factual question:

Question: Were Pavel Durov and Sam Altman born in the same country?

Thought: I need to find the birthplaces of both Pavel Durov and Sam Altman.
Act: Search[Pavel Durov birthplace]
Observation: Pavel Durov was born in Leningrad, Soviet Union (now Saint Petersburg, Russia).

Thought: Durov was born in Russia. Now I need to find where Sam Altman was born.
Act: Search[Sam Altman birthplace]
Observation: Sam Altman was born in Chicago, Illinois, United States.

Thought: Durov was born in Russia and Altman was born in the United States. They were not born in the same country.
Act: Finish[No]

Now compare that to a pure chain-of-thought response on the same question — one that never leaves the model’s parametric memory. It might recall both birthplaces correctly, or it might confabulate one and answer confidently anyway, with no mechanism to catch the error. Step through both side by side below: same question, one path reasoning from memory, the other checking the world first.

2.3 Teaching Models to Use Tools: Toolformer

ReAct shows that reasoning plus tool use makes better agents — but in ReAct, you have to tell the model which tools exist and coach it to use them through examples in the prompt. That raises a natural question: can a model learn to use tools on its own, so reaching for the right one becomes second nature rather than something you prompt for every time?

Toolformer, from Schick et al. in 2023, showed it can. The team had the model teach itself: it generated its own examples of where an API call would help, kept only the ones that actually improved its predictions, and fine-tuned on those. No hand-labeled dataset required — just a few demonstrations per tool to get started. The resulting model could reach for a calculator, search engine, translator, or calendar at the right moment, without being told to.

The standout result: a relatively small model that had learned tool use beat GPT-3 — a model more than 25 times larger — on math problems. The reason is intuitive. Arithmetic is exactly the kind of thing language models are bad at and calculators are perfect at. The model learned when to stop trusting itself and offload to a tool that wouldn’t get it wrong.

The lasting lesson for builders is the principle, not the benchmark: tool use is a capability a model can acquire, not just a behavior you prompt for. Where the line falls between “the model answers from memory” and “the model calls out to the world” isn’t fixed — it can be deliberately shaped. That’s a design lever you’ll use constantly when deciding which capabilities to bake in versus prompt for.

2.4 Reflection Loops: Learning Without Weight Updates

ReAct and CoT with tools give an agent the ability to ground its reasoning in external facts during a task. But what about across tasks? If an agent makes a mistake — misunderstands a requirement, takes a wrong action, gets stuck in a loop — how does it improve? Traditional machine learning answers this with gradient descent: the model’s weights are updated to make the error less likely in the future. But weight updates require training infrastructure, labeled data, and time. They cannot happen in the middle of an agent deployment.

Reflexion, from Shinn et al. in 2023, offers a cheaper path: let the agent improve through words instead of weights. The loop is straightforward. The agent attempts a task and gets feedback — even just pass/fail. It then writes a short note to itself about what went wrong and what to try differently. That note gets saved, and the agent reads it before its next attempt.

Nothing in the model changes. What changes is the context the agent carries into the next try. The interactive widget below walks through an illustrative example: an agent that fails a task, writes itself a one-line lesson about what went wrong, and completes the same task in far fewer steps on the next attempt. In the original paper, this let a Reflexion-equipped agent solve 130 of 134 AlfWorld tasks, improving across repeated trials.

In the original work, this produced large gains on tasks where the agent got multiple tries — but not everywhere. The honest takeaway for practitioners: reflection helps most when failures are diagnosable in words. If the agent can articulate what went wrong (“I searched the wrong directory”), it can fix it next time. If the failure is something it can’t see or name, writing a reflection about it does little.

A close cousin, Self-Refine (Madaan et al., 2023) applies the same idea within a single task instead of across attempts. One model plays three roles in a loop: it drafts an output, critiques its own draft, and revises based on that critique — repeating until good enough. No training, no extra infrastructure, just prompting.

It’s an elegant pattern, but it comes with a serious caveat you should internalize before relying on it. Self-critique only works if the model can actually tell good output from bad. A model that’s confidently wrong will often be just as confident that its wrong answer is correct — so it “refines” nothing. What makes these loops reliable is an external signal the model can’t talk its way around: a failing test, a fact that doesn’t match a retrieved source, a human saying no. Without one, a reflection loop can just as easily reinforce a mistake as fix it. When you build these loops, anchor them to something real — tests, validators, ground-truth checks — not the model’s opinion of its own work.

3. The Vocabulary Map

The patterns above share a common vocabulary — the same handful of terms shows up in nearly every agent framework, paper, and docs page you’ll encounter. Knowing them precisely saves you from re-deriving the concepts every time you read a new tool’s documentation. Here are the ones that matter.

Agent: a system that perceives its environment, selects actions, and executes them in pursuit of a goal. In the LLM context, the agent’s core reasoning component is a language model, but the agent as a whole includes tools, memory systems, and execution infrastructure that extend far beyond the model itself.

Environment: everything the agent interacts with that is not the agent itself. For a software engineering agent, the environment includes the codebase, the terminal, the test runner, and any APIs or databases the agent calls. For a research agent, it includes the web, document stores, and any other information sources the agent retrieves from.

State: the current configuration of the environment, as observable by the agent. In many agentic tasks, the full state of the environment is not directly observable — the agent sees only a partial view, typically through the observations returned by its tool calls.

Action space: the set of possible actions available to the agent. This includes both internal actions (reasoning, memory operations) and external actions (tool calls, messages). The design of the action space is one of the most consequential architectural decisions in building an agent — too small and the agent cannot accomplish its goals; too large and it becomes difficult to search effectively.

Observation: the information the agent receives from the environment in response to an action. In a text-based agent, observations are typically text — tool call return values, API responses, error messages.

Policy: the mapping from state to action that the agent uses to decide what to do. For LLM-based agents, the policy is not a learned function in the RL sense; it is implicit in the model’s parameters and the prompting strategy. In practice, the policy is shaped by the system prompt, the few-shot examples, the tools available, and the agent’s current context.

Trajectory: the sequence of states, actions, and observations that the agent traces from task start to task completion (or failure). Trajectories are the natural unit of analysis for agentic systems — understanding why an agent succeeded or failed requires reading the trajectory, not just the final output.

Reward: in classical RL, a scalar feedback signal. In LLM-based agents, reward is often implicit or sparse: the task either succeeds or it does not. Some systems substitute verifiers (test suites, checkers, human evaluators) for a traditional reward signal.

3.1 The Brain-Perception-Action Framework

Beyond this RL-derived vocabulary, a widely cited 2023 survey by Xi et al. offers a simpler three-box model for the architecture of an agent:

• Brain: the central LLM, responsible for reasoning, planning, and decision-making. The brain is where chain-of-thought, tool selection, and reflection happen.
• Perception: the systems through which the agent receives information from its environment — text parsers, image encoders, structured data processors. Perception determines what the agent can observe.
• Action: the systems through which the agent acts on its environment — tool execution, code runners, web browsers, message sending. Action determines what the agent can do.

graph LR
    ENV["🌐 Environment<br/>(tools, APIs, files, web)"]

    subgraph AGENT["LLM-Based Agent"]
        direction TB
        P["👁 Perception<br/><i>text, images, structured data</i>"]
        B["🧠 Brain (LLM)<br/><i>CoT · ReAct · Reflexion<br/>Working memory · Planning</i>"]
        A["⚡ Action<br/><i>tool calls · code · messages</i>"]
        P --> B --> A
    end

    ENV -->|observations| P
    A -->|actions| ENV

This framework is less a theoretical claim than a useful way of thinking about the architecture of an agent system: if the agent is failing, the failure is usually in one of these three components. The brain reasons incorrectly; the perception system presents information in a form the brain cannot use effectively; the action system has a capability gap that prevents the brain’s plan from executing.

4. Historical Context

4.1 Classical AI Planning: STRIPS and PDDL

The core idea behind agentic AI — a system that pursues goals by planning and executing actions — isn’t new. It goes back to 1971 and STRIPS, the first influential automated planner. Knowing this history is useful because the limitations that sank earlier approaches explain why the LLM-based version works where they didn’t.

STRIPS worked on a symbolic model of the world: facts were logical statements, and each action declared its preconditions (what must be true to do it) and effects (what it changes). Planning meant searching for a sequence of actions that turned the starting state into the goal state. It was elegant, and by the early 2000s — with a standard problem-description language, PDDL, in wide use — planners could handle problems with hundreds of actions.

The catch was the model itself. Classical planning assumes the world is fully known, actions always do exactly what they say, and goals can be written in formal logic. Real tasks break all three assumptions constantly: you rarely have complete information, actions fail or surprise you, and “fix the production bug” doesn’t translate into clean predicate logic. The moment reality stopped matching the symbolic model, these systems broke — which is why they never escaped narrow, well-defined domains.

4.2 Reinforcement Learning: Acting Without a Perfect Model

Reinforcement learning took a different tack: instead of handing the agent a model of the world, let it learn what works by trying things and seeing what gets rewarded. This freed agents from needing a perfect world model, and it powered the famous results of the 2010s — systems that learned to play Atari from raw pixels, AlphaGo beating world champions, superhuman game-play from self-play.

RL handled messy, unpredictable environments that would have broken a classical planner. But it had its own dealbreakers, and they’re worth remembering because LLM agents sidestep them. RL needed staggering amounts of trial-and-error — often billions of attempts — and a clear reward signal on every step, which most real tasks don’t provide. Worse, what it learned didn’t transfer: an agent trained on one game knew nothing about the next. You couldn’t just describe a new task to it; you had to retrain from scratch.

4.3 The LLM Turn: From Chatbots to Agents

Large language models did not enter the picture as agents. They arrived as language models — systems trained to predict the next token in a sequence, on internet-scale text corpora, using the transformer architecture (Vaswani et al., 2017). GPT-2 (2019) demonstrated surprising capability at text generation and few-shot tasks. GPT-3 (2020) showed that capability scaled dramatically with model size, and that sufficiently large models could perform many tasks from natural language instructions without task-specific fine-tuning. ChatGPT (2022) brought this capability to the public as a conversational chatbot.

The shift from language model to agent happened in stages, and you’ve now seen each one. First, models learned to reason in steps (chain-of-thought). Then that reasoning was wired up to tools, so it could be grounded in real information instead of guesswork (ReAct, Toolformer). Then agents gained a way to learn from their own mistakes without retraining (Reflexion, Self-Refine). Each step added a capability the one before it lacked.

Put together, they add up to a genuine change in kind. A language model produces text; an agent pursues goals. The engine underneath — a transformer trained on text — is the same, but the system you build around it, how you evaluate it, and how it fails are all different. That difference is the whole subject of this book.

5. The Spectrum and the State of the Art

5.1 Where the Lines Are Not Clean

It’s tempting to draw a bright line between chatbots and agents. Don’t — there isn’t one. “Agent” is a position on a spectrum, not a category you either qualify for or you don’t, and arguing about whether a given system “is really an agent” is mostly a waste of time. What’s useful is knowing where a system sits and why.

The spectrum runs like this. At one end, a bare language model — no tools, no memory, no actions beyond producing text — is purely reactive. It answers; it doesn’t do. At the other end, a fully autonomous system has a rich tool set, memory that persists across sessions, the ability to spawn sub-agents, and the capacity to learn from feedback over long-running tasks. Everything real lives in between, and where a system lands is determined by three things: what it can see, what it can do, and how much it decides on its own versus checking with a human. The slider below lets you walk that spectrum tier by tier.

Most production systems today sit somewhere in the middle of this spectrum. They have access to a specific set of tools (search, code execution, file access, API calls). They maintain some form of state across a task session, though often not across sessions. They make autonomous decisions about how to sequence actions within a task, but they ask for human input at defined checkpoints. They are agents in the sense that matters — they pursue goals through multi-step action sequences — but they are not autonomous in the science-fiction sense.

5.2 Agent-Computer Interfaces: A New Design Space

Here’s a practical question that’s easy to overlook: when you give an agent tools, should those tools be built the same way they’d be built for a human? The answer turns out to be no, and it has real consequences for how you design agent systems.

The SWE-agent project (Yang et al., 2024) made the case directly. An agent is a different kind of user than a human, with different constraints. It can’t skim past irrelevant output the way you do — every line of text it sees costs tokens and competes for its attention. It can’t easily back out of a bad state. It needs guardrails so one wrong move doesn’t cascade. So the team built tools for the agent — a file editor, search, and navigation commands shaped around how a model actually works — rather than just handing it a raw shell.

The result was dramatic. On SWE-bench Lite, a benchmark of real GitHub issues that an agent has to fix with working code, the same base model (GPT-4 Turbo) resolved 11.0% of tasks through a plain Linux shell but 18.0% through the purpose-built interface — a 64% relative jump from the interface alone. And the pieces mattered individually: stripping out just the custom file-editor command dropped the score to 10.3%, below even the shell. The interface was doing as much work as the model.

The specific numbers have since been blown past — by mid-2025, frontier coding agents resolve a large fraction of SWE-bench, and the benchmark is now a standard yardstick for agentic coding. But the lesson is durable and worth internalizing: agent performance is a product of the model and the interface you give it. A weaker model with well-designed tools routinely beats a stronger model fumbling with bad ones. When your agent underperforms, the tools are often the thing to fix first — not the model.

5.3 Open Challenges

The architecture described in this chapter — plan-act-observe loops, tool use grounded in external facts, reflection and self-critique, CoALA-style memory management — represents the current consensus on how to build capable agentic systems. It is an architecture that works, at meaningful scale, on real tasks. But it is not a solved architecture. Several challenges remain open.

Long-horizon reliability. Current agents perform well on tasks that can be completed in tens of steps. Tasks requiring hundreds or thousands of steps — long-horizon software development projects, extended research tasks, persistent autonomous operation over days or weeks — remain substantially harder. The compounding of errors across many steps, and the difficulty of maintaining coherent plans over long trajectories, are not yet solved by any of the architectural patterns described above.

Self-critique accuracy. Reflection loops work when the agent can reliably evaluate its own outputs. They fail when the model is confidently wrong — when it generates an incorrect output and just as confidently evaluates that output as correct. The conditions under which LLM self-critique reliably improves outputs versus confidently reinforcing errors are not yet well understood.

Generalizing interface design. SWE-agent showed that ACI design matters for software engineering. It is not yet clear which ACI design principles generalize across domains and which are task-specific. An agent built for web research requires a different interface than one built for data analysis, which requires a different interface than one built for system administration. The field does not yet have a science of agent interface design.

The autonomy-oversight tradeoff. The more autonomous an agent is, the more useful it is — and the more consequential its mistakes. Chapter 9 will address this tradeoff in depth. For now, the point is that increasing autonomy is not unambiguously good. Every step toward more autonomous operation is also a step toward larger blast radius when the agent goes wrong.

None of these are solved. If you build agents today, you will run into all four — and the rest of this book is largely about how to work within these limits rather than pretend they don’t exist. Each gets its own treatment in a later chapter.

Summary

An agentic AI system is distinguished from a chatbot by the plan-act-observe loop: the agent takes actions in an environment, observes the results, and uses those observations to decide what to do next. This structure produces trajectories rather than responses — sequences of decisions that unfold across time — and enables multi-step autonomy that a stateless input-output function cannot support.

The core architectural patterns for implementing this loop are: chain-of-thought prompting (Wei et al., 2022), which enables multi-step reasoning; ReAct (Yao et al., 2022), which interleaves that reasoning with external tool calls to ground it in retrieved facts; Toolformer (Schick et al., 2023), which established that tool use is a learnable capability; and reflection loops including Reflexion (Shinn et al., 2023) and Self-Refine (Madaan et al., 2023), which enable agents to improve through self-critique without weight updates.

The vocabulary for describing these systems comes from two places: reinforcement learning (state, action, observation, policy, trajectory) and cognitive science (working memory, episodic memory, the proposal-evaluation-selection-execution cycle). You’ll see both in tool documentation and framework APIs, so it’s worth being comfortable with each.

The ideas aren’t new — agentic AI descends from decades of work in classical planning and reinforcement learning — but the LLM-based version came together fast, with the key building blocks landing between 2022 and 2024. What’s left unsolved — reliability over long tasks, trustworthy self-correction, interface design, and oversight — is exactly what the rest of this book is about.