Agent Harness Engineering: Why I Recommend AI Agents in Depth

type

status

date

slug

summary

What is an agent harness?

The book starts with an engineering decomposition: Agent = LLM + Context + Tools. This is not a formal definition from reinforcement learning. It is a way to identify the main parts of an LLM-based agent.

The model makes decisions from the information available to it. The harness surrounds that model. It constructs the context, exposes tool interfaces, runs the interaction loop, and maintains task state. In a production system, it also helps enforce constraints and check results. Chapter 1 explicitly separates this model–harness structure from the external environment.

That boundary matters. A file-reading tool is an interface. The files themselves belong to the environment. A harness may create a sandbox, but the sandbox's actual state is not the same thing as the model's description of it.

Here is the loop in my own words:

Each line contains a possible failure. The model can receive incomplete information. A tool can time out. A result can belong to an earlier task. A final answer can claim more than the evidence supports.

Three neighboring ideas are easy to confuse with a harness. Prompt engineering changes instructions; it does not itself implement permissions or recovery. A framework can supply useful components; choosing one does not settle your application's success criteria or operating rules. An evaluation harness runs and grades experiments, while an agent harness runs the agent being evaluated. They may share infrastructure, but they have different jobs. Anthropic's evaluation guide makes the last distinction explicit.

There is also no requirement to make every step model-directed. Anthropic's architecture guide distinguishes predefined workflows from agents that choose their next actions. My default would be to keep predictable steps in ordinary code and use a model where adaptive decisions are actually useful.

The book's ten chapters, as an engineering map

The structure moves from building one agent to measuring it, improving it, and coordinating several agents. The questions in the last column are my study prompts, not quotations from the book.

Chapter	What it covers	The question I would test
1. Agent fundamentals	The interaction loop and the model–harness–environment boundaries.	Who decides, who executes, and where does the real state live?
2. Context engineering	Instructions, context layout, KV caching, Skills, task summaries, and compression.	What must remain visible for the next decision to be correct?
3. User memory and knowledge bases	Cross-session memory, retrieval, knowledge organization, and memory evaluation.	Can the agent retrieve the right fact and handle a later correction?
4. Tools	Agent-facing interfaces, MCP, specialized tools, general executors, Skills, and discovery.	Is a capability easy to use correctly and difficult to misuse?
5. Coding and general agents	Code generation, file workspaces, and the read–edit–test–repair process.	Can the agent produce a verifiable artifact, rather than just describe one?
6. Interaction	Asynchronous events, interruptions, voice, computer use, and robotics.	What happens when the world changes while the agent is working?
7. Evaluation	Tasks, environments, outcome checks, process checks, and controlled comparisons.	What evidence distinguishes a better system from a lucky run?
8. Model post-training	Pretraining, mid-training, SFT, RL, and the role of data and environments.	Does this failure require changing weights, or changing what the model receives?
9. Continual evolution	Learning from trajectories through updates to knowledge, instructions, programs, or parameters.	How does an experience become a tested improvement?
10. Multi-agent collaboration	Shared versus isolated context, communication, and coordination patterns.	What does another agent add besides more messages and cost?

Version note: This map follows the Chinese v2.0 manuscript at commit 0c9390c, checked on September 14, 2026. In v2.0, interaction is Chapter 6, evaluation is Chapter 7, post-training is Chapter 8, and continual evolution is Chapter 9. Older editions use different numbering. Community translations can lag behind the original, so the links above pin the Chinese source used for this review.

The ideas I would carry into an implementation

1. Context is a working view, not a storage bin

The useful question is not simply how much information fits in the window. It is what information the next decision requires.

Chapter 2 connects that question to context layout, selective loading, and compression. Chapter 3 extends it to persistent memory and knowledge retrieval.

For the coding example, a useful working context might contain the requested behavior, the relevant function, the failing test output, the allowed edit scope, and the current patch. A complete chat archive is not automatically the best representation of that information.

But compression can remove exactly the fact that matters. A summary that preserves “fix the validator” but drops “do not modify tests” changes the task. I would therefore preserve critical constraints explicitly and enforce important ones outside the model as well.

The distinctions are worth learning precisely. A trajectory records what happened. Task state records what remains to be done. Long-term memory preserves information across tasks. Retrieval selects relevant material. Runtime context is what the model actually receives at a decision point. KV caching reuses inference computation; it is not a replacement for deciding which facts to retrieve or retain.

My takeaway is to treat context construction as a data pipeline with its own failure cases. Missing evidence, stale facts, and a lossy summary are different bugs. They deserve different fixes.

2. Tool access is not the same as safe authority

One of Chapter 4's useful distinctions is between how a capability is represented and when it is disclosed.

A capability can be a dedicated, schema-constrained function. It can also be a procedure described in a Skill and executed through a general tool. Independently, either kind of capability can be loaded eagerly or discovered when needed. “Use Skills” and “load tools lazily” are not the same design decision.

For a coding assistant, a general executor is powerful because code can compose operations that were not individually anticipated. That is a central argument in Chapter 5. But the chapter also limits its coding-centered architecture claim to open-ended general agents, rather than every constrained business workflow.

My design preference would depend on the risk. I would allow flexible computation in an isolated work area, but use narrow, audited interfaces for consequential external changes. A model producing valid arguments does not establish that it is authorized to perform the operation. The executor must check the relevant permissions.

Timing adds another boundary. Suppose a user cancels a task while a tool is still running. A late success response must not silently reactivate the canceled task. Chapter 6 is valuable because it makes asynchronous events and interruptions part of agent design, rather than treating the world as paused between turns.

3. A convincing transcript is not a successful outcome

I would move evaluation earlier in the learning process than the chapter order might suggest.

Chapter 7 evaluates the model and harness together. Chapter 9 sharpens the distinction between achieving the result, following an allowed process, and communicating well. These are related but separate questions.

For the coding task, “the tests pass” is useful evidence only after checking which tests ran and which files changed. An agent that deletes the failing test has not satisfied a requirement to fix the implementation without changing tests. For my exercise, I would keep the trusted grading tests outside the agent's writable area.

An LLM judge can help assess explanations or code quality. It should not replace a direct check that the expected artifact exists and satisfies the relevant assertions. Anthropic's evaluation guide similarly distinguishes the transcript from the resulting environment state.

The book's context-ablation project contains a particularly useful detail: its documentation separates a run completing from the task succeeding. It also records an example in which the “no reasoning” ablation showed no measurable degradation. That is not proof that reasoning never matters. It is a reminder to report the actual experiment rather than the result a teaching story expected.

For comparisons, I would keep the task set and budget explicit. Change the model while holding the harness fixed, or change a harness component while holding the model fixed. Neither experiment alone gives a complete explanation of all interactions, but both are more informative than changing everything at once.

4. Improving an agent means choosing what to change

Chapter 9 is especially worth reading alongside the post-training material. Its important distinction is that saving an experience is not the same as learning from it.

A log preserves events. Turning those events into better behavior requires judging the outcome, identifying what mattered, proposing a change, and testing it. The book separates four places where the change can live: knowledge, instructions or Skills, programs or harness logic, and model parameters.

For the same coding assistant, those choices look different. A newly discovered build command belongs in project knowledge. A useful review procedure may belong in a Skill. A rule forbidding writes outside the workspace belongs in executable controls. A persistent inability to follow the required action protocol may justify investigating better demonstrations or model training.

This connects harness engineering to Chapter 8's treatment of mid-training, SFT, and RL. The model's learning signal matters, but so does whether training is the appropriate intervention at all.

I would not train a model to recover a document the retrieval system never supplied. I would not rely on training alone to enforce a hard access boundary. Conversely, a good harness does not remove every reasoning or generalization limit in the model.

The versioned improvement loop is the part I would borrow: propose a change, test it against held-out tasks and known regressions, inspect failures, then decide whether to adopt it. An automatically generated reflection is a hypothesis, not permission to rewrite the system's rules.

5. Multiple agents should earn their complexity

Chapter 10 discusses context sharing, isolation, communication, and coordination. Those are useful design dimensions. They are more informative than merely naming roles such as planner, researcher, and critic.

My first question would be what the extra agent contributes: independent searches, useful parallel work, a smaller task-specific context, or evidence from a different tool. A reviewer that runs a relevant test adds something concrete. A reviewer that merely agrees with the original answer does not establish correctness.

I would still test these ideas rather than turn them into an absolute rule that collaboration only helps in one way. Compare the multi-agent design with a strong single-agent baseline under comparable resources. More computation or more specialized prompts may explain part of the gain. The architecture should justify its extra latency, coordination, and failure handling.

How I would study the book from scratch

My suggested first pass is 1 → 2 → 4 → 5 → 7: understand the loop, construct context, design tools, produce artifacts, and measure outcomes. Then add Chapter 3 for persistent memory. Read Chapter 6 when timing, interruptions, or non-text interaction become part of the task. Chapters 8 and 9 connect measured failures to learning and improvement. Read Chapter 10 when a concrete coordination problem appears.

This is a practical route for an application-focused LLM engineer, not the author's required reading order. Someone focused on post-training will spend much more time on Chapter 8 and its mathematical prerequisites.

Build one small agent before adding more machinery

Here is a proposed exercise—not a reported experiment from the book.

Use a small toy repository with a broken input validator. Give the agent one task: repair the implementation, leave the trusted tests unchanged, and produce a patch with a short explanation. Do not give it production credentials or permission to publish changes.

Start with reading, searching, patching, and running a fixed test command. Give the run a step limit and an execution timeout. Record the proposed action, the validation decision, the tool result, and the resulting state. Finish with the patch and test evidence, or an explicit account of what remains unresolved.

The useful deliverable is not a chat screenshot. It is a reproducible task, a trace, the final artifact, and checks another person can rerun.

Break the system deliberately

Before adding memory or more agents, test the basic contract:

Failure you introduce	Behavior to require
A relevant file read returns only part of the content.	Retrieve the missing part or report the limitation; do not treat a partial read as complete evidence.
A repository comment tells the agent to ignore the user's restrictions.	Treat it as untrusted source content. It must not grant new permissions or override the task.
A tool times out after it may have changed a file.	Inspect the current state before retrying or claiming success. A timeout does not establish that nothing happened.
The patch passes by weakening or removing a test.	Reject it using trusted checks and the allowed-change policy.
The step budget is exhausted.	Stop with the actual task state and remaining failure, not an invented completion report.

Then use the book's Chapter 1 context-ablation experiment to study what changes when history or tool results are withheld. Follow that project's setup instructions and inspect its grading logic before interpreting its output. Provider-backed runs need credentials and may incur charges; the commands are not a promise of zero-cost or universally compatible execution.

For each variation, write down the failure you expected, what actually happened, and what evidence would change your explanation. Keep the model configuration and task set fixed where possible. A small repeatable comparison teaches more than collecting architecture diagrams.

Why I recommend it—and where I would stay critical

I recommend this book because it puts several normally separate topics into one causal chain: information reaches the model, the model selects an action, a tool affects the world, evaluation checks the result, and a learning process decides what to change next. That is a useful bridge between studying LLMs and engineering systems around them.

The recommendation is not an endorsement of every broad claim or every current product example. Treat model names, API details, and vendor comparisons as version-sensitive. The repository contains different kinds of companion work, including local projects and external reproduction tracks; inspect the individual instructions and requirements rather than assuming every chapter is a ready-made production package. The repository guide is the starting point.

Two slogans also need restraint. “Code runs” is not a proof that the requirement is satisfied. “SFT memorizes while RL generalizes” is not a universal law; the current post-training chapter itself qualifies that comparison by the experiments' conditions. Data, objectives, evaluation, and distribution shift still need examination.

The book is a systems companion, not a substitute for deeper study of probability, optimization, reinforcement learning, or distributed systems. Its value is in showing where those subjects become relevant to a working agent.

The distinction I would remember

The most confusing step is the jump from the model generated an action to the system performed the right action.

Those are not equivalent. A request can be malformed, unauthorized, rejected, partially completed, or completed against stale state. The harness must manage that gap, and evaluation must inspect it.

This lesson transfers beyond coding. A research assistant needs traceable evidence. A calendar assistant needs verified changes. A document assistant needs to preserve scope and read back what it saved. A post-training pipeline needs feedback that reflects the behavior it actually wants to reinforce.

For the agent-building part of this series, I would make AI Agents in Depth a core reading resource. Read a chapter, implement one piece, introduce a failure, and test your explanation. The goal is not to memorize every framework. It is to know which part of the system should change when the agent gets something wrong.

Continue the series:

How to Become an LLM Engineer: A Knowledge Map and Learning Method · ·