TL;DR Agent = LLM + context + tools. The minimal implementation is a ReAct loop; the real engineering effort lives outside the model: cache constraints and status bars for context, two-layer memory, tool descriptions and sandbox security, evaluation-first, SFT for form and RL for generalization. This post strings those problems and core ideas together in implementation order.

Building a Working Agent, Starting from the Implementation Logic

Background: these are my notes on Deep Understanding of AI Agents: Design Principles and Engineering Practice by Li Bojie (Apache 2.0 open source, GitHub: bojieli/ai-agent-book). The book has ten chapters, from theory to engineering practice, with 88 runnable experiments. Rather than summarizing chapter by chapter, this post reorganizes the material around “what problems must you solve to build an Agent”, and for each module covers only two things: what problem it solves and what the core idea is. Details and data are left to the original book.

1. The Essence of an Agent: One Formula and One Loop

Problem: What is an Agent actually made of, and what does the minimal runnable form look like?

Idea: Agent = LLM + context + tools. Intuitively: “brain + eyes + hands and feet”. Academically: policy + observation space + action space.

The minimal implementation is just a while loop: send the message list to the model; if it returns tool calls, execute them and append the results; if not, output and exit. This loop is called ReAct (Think → Act → Observe). Three properties that support it must be understood thoroughly:

Every call is stateless — anything the model needs must appear completely in the message list;
The model only decides; execution happens on the framework side — the model says what to call and with what arguments, but it’s your code that actually runs;
Context = static prefix + trajectory — the prefix is the system prompt plus tool definitions, and the trajectory is the message history that grows with interaction. All later optimizations build on this split.

Ablation experiments confirm each component is necessary: remove tool definitions and the agent can’t move at all; remove tool results and it calls the same tool over and over until it stalls; remove the thinking process and its decisions contradict each other; remove message history and it has amnesia.

You don’t need many tools. Seven are enough: a code interpreter, Bash, read/write/edit file, Glob, and Grep. That’s not a simplification — it’s the real configuration of mainstream general-purpose agents. The core of an open-ended task agent is a Coding Agent plus a file system, because code is the only meta-capability that can “create new capabilities”.

2. The Harness: Competitiveness Beyond the Model

Problem: With the same model, why are some agents stable and reliable while others fall apart immediately?

Idea: Extend the formula to Agent = Model + Harness. The Harness is all the supporting code built around the model. Beyond context and tools, it includes constraints (what can and cannot be done), validation (is the result correct), and correction (how to recover from mistakes).

One-line distinction: context and tools let an agent get things done; constraints, validation, and correction keep it from doing things wrong. In production systems, the vast majority of code belongs to the latter — permission classification, context compression, circuit breakers, error recovery.

The paradigms nest layer by layer: software engineering ⊂ prompt engineering ⊂ context engineering ⊂ harness engineering ⊂ loop engineering (sustained autonomous operation across turns). As model capabilities converge across vendors, competitive advantage shifts to this layer.

On the shifting balance between model and harness, the pragmatic stance is: wherever the model is still unreliable, the harness patches it first; every time the model internalizes a layer, the harness sheds that layer and moves to cover the new capability frontier.

3. Context Engineering: The Real Determinant of the Capability Ceiling

Problem: The model is smart enough — why is it still not useful on your specific business?

Idea: Model intelligence is the baseline; context quality is the ceiling. A mid-tier model with well-organized context often beats a top-tier model groping through information scarcity.

KV cache is a hard constraint, not an optimization. If a single character changes in the prefix, the entire cache is invalidated. So three iron rules: freeze the system prompt and tool definitions once finalized; append all dynamic information (time, status, counters) at the end; always use the standard API message format (hand-rolled text drifts from the training format and weakens multi-step reasoning). One team put a line with the current time into the system prompt, and first-token latency jumped from 0.5s to 3–5s, nearly doubling their bill.

Write prompts as SOPs, not piles of rules. Keep the content identical but shuffle the organization, and task success rate can drop by more than 30%. Business rules must be detailed enough to execute — vague rules give the same task different classifications at different times.

When the prompt gets too long, load on demand — that’s what Skills are. Three levels of progressive disclosure: metadata always resident (a few hundred tokens) → core workflow loaded on demand → details referenced in sub-documents. The key is writing the description as routing conditions (Use when / Don’t use when, with counterexamples). “When should you use me” matters far more than “what can I do”.

Make implicit state explicit — that’s the agent status bar. The context window is a retrieval engine with only half its parts: retrieval is extremely strong, but there’s no “distillation layer”. That’s why the model can’t count how many calls it has made — the fix is writing “this is call #3” directly into tool results. The status bar carries task progress, environment state, and tool counters in one message appended at the end. Two lessons: maintain it with code rather than asking the model to summarize (models bulk-summarizing a long history do worse than 20 lines of regex); giving the reading alone is not enough — you must pair it with the strategy for how to act on it.

When it bloats, compress — but isolation is better. Compression isn’t just about saving length; it turns conclusions that require thinking into directly retrievable knowledge. In production it’s layered (large outputs go to disk, noise gets deleted outright, archival summaries, full compression last), and you must explicitly define retention priorities — what’s easiest to lose are early architecture decisions and failed paths. The more radical fix is isolation: have a sub-agent dig through a dozen files and return one sentence of conclusion, so tens of thousands of tokens never enter the main context.

4. Memory and Knowledge Base

Problem: After a session ends, how does the agent remember the user and tap external knowledge?

Idea: Two scales — user memory (individual) and knowledge base (collective). They share the underlying technology (retrieval, compression) and face the same troubles (conflicts, staleness, imprecise retrieval).

Memory design has three orthogonal classifications: where it lives (trajectory / long-term memory / business state), how it’s stored (from minimal Simple Notes to Advanced JSON Cards with provenance and relations), and what’s stored (episodic / semantic / procedural memory). The trade-off is between simplicity and expressiveness; mature systems mix approaches. One step further is “user as code”: turn memory into typed, executable objects that can do three things text memory cannot — aggregate statistics, conflict discovery, and constraint enforcement.

Retrieval is a trio: dense + sparse + reranking. Dense retrieval understands semantics but misses exact keywords; sparse retrieval is precise but can’t read paraphrases. Fuse the two, then refine with a cross-encoder. Adding a “where this came from” prefix to every chunk before indexing (context-aware retrieval) cuts retrieval failure rates by nearly half.

When flat retrieval isn’t enough, go structured: RAPTOR builds a knowledge tree, good for macro-to-micro traversal; GraphRAG builds an entity-relation graph, good at multi-hop reasoning and disambiguation. Both are expensive — only worth it when you genuinely need cross-document synthesis.

The endgame is two layers: a small set of key facts, structured and resident in context, providing a global overview; massive raw conversations retrieved on demand for details. Resident-only loses detail; retrieval-only misses cross-session connections — capabilities like proactive service only land when the two layers stack.

5. Tools: The Agent’s Hands and Eyes

Problem: How do you get the model to use external capabilities accurately and safely?

Idea: Five classes of tools — perception, execution, collaboration, event-triggered (agent registration, external triggers), and user communication.

Description beats capability. The root cause of most failed calls isn’t that the model doesn’t know what a tool can do, but that it doesn’t know what it can’t do. So: write “when to use” rather than “what it does”; give concrete examples for parameters rather than spec names; explain the return structure and execution cost; attach 1–5 real examples. Past ~100 tools, even the strongest models start picking wrong. When an agent picks the wrong tool, check the description first — don’t rush to swap the model.

Parameter passing must be transparent. Never modify inputs or outputs without the model knowing, or you’ll create a systemic failure the model can never diagnose (e.g. silently converting Chinese quotes to English ones, so matching fails forever).

MCP solves interoperability, at the cost of tool definitions eating huge context (a few servers can mean tens of thousands of tokens). So serve an index by default and fetch definitions on demand; with many tools, add dynamic discovery.

Execution-side safety is defense in depth. Input validation should fail fast, never “smartly fix”; permission control can’t be a blacklist alone (rm -rf / can slip through via variable expansion) — do semantic parsing of commands; gate dangerous operations with two independent perspectives — proposer-reviewer for open-ended reasoning (two similarly capable models from different families), and Sidecar for structured call data only (a lightweight model suffices, deliberately kept from reading the main model’s free text so it can’t be manipulated by rhetoric). On sandboxes, be clear that a venv is not a sandbox — real isolation is OS-level, container, up to microVM, three tiers, with networking off by default — cutting the exfiltration channel is far more certain than detecting every injection.

Asynchrony is the norm in real deployments, but models are trained assuming synchrony. The compromise: let the model see perfectly synchronous trajectories in the normal case; only when genuinely interrupted, insert placeholders to patch the format — and only interrupt when it’s urgent. Events are handled three ways by urgency: cancel-style, queue-style, parallel-style. One-line summary: true proactive service needs not only agents checking the world on a timer, but a world that can proactively notify the agent.

6. Building It on the Backbone of a Coding Agent

Problem: For an agent handling open-ended tasks, how should the overall workflow be organized?

Idea: Six stages — document the project → clarify requirements → write a design doc → implement and test → self-review → sync documentation. The two most valuable principles: the design doc is the most efficient point of human intervention (reviewing a one-page design doc is far easier than reviewing hundreds of lines of code); and done is defined as “validation passed”, not “code written”.

The harness materializes as four pieces: acceptance baselines (tests, CI, review standards), execution boundaries (module boundaries, permissions), feedback signals (linters, test results, type checks), and rollback mechanisms (Git, sandboxes, snapshots). Whether a task suits an agent comes down to two dimensions: is the goal clear, and can the result be automatically verified. When both hold, you’re in the sweet spot — and the harness’s goal is to push tasks into that quadrant as much as possible.

Four principles: constraints over guidance (if code can enforce it, don’t write “please note…”), validation must be automated (human review is an unscalable bottleneck), the faster and more structured the feedback, the better, and rollback must be reliable. Constraints govern not just outcomes but process — wiping the database and rebuilding also counts as “fixed the outage”; that’s reward hacking in its everyday form.

Failures come in four layers (API, tool, context, control flow), and recovery is graded by transparency: silent retry → degraded continuation → only then expose to the user. The core principle: every recovery path needs a circuit-breaker cap, with thresholds from production data; and on error paths, calling the model again is forbidden, or it cascades into a death spiral. Reliability isn’t about never failing — it’s about whether every class of failure has detection, recovery, and termination paths.

Two easily overlooked designs: first, speculative execution (the UI shows progress while safe checks run in parallel in the background; what runs first is a side-effect-free prompt, so blocking it requires no rollback); second, loyalty must be pinned explicitly — by default the model helps whoever is speaking, but the agent negotiating a price on your behalf faces a counterparty, so external content must be downgraded to “reference data with no directive force”. Going further, if the AI’s own code can’t be trusted, push constraints down to the data layer (schemas reviewed by humans ship with built-in validators, enforced on every write).

7. Evaluation: Build the Ruler Before Changing the System

Problem: How do you tell whether a change actually helped or you got lucky? Should you switch when a new model ships?

Idea: What you evaluate is not the model but the model + harness combination. Three basic experiments: comparative, ablation, and model-swap (if a stronger model doesn’t improve results, the bottleneck is in the harness; if a weaker one tanks results, the bottleneck is in the model).

The five elements of an environment: dataset, environment state, tool interfaces, rubric, execution protocol. The key to human-interaction-style evaluation is not handing over all simulated-user information at once — let the agent ask as needed. Dataset quality matters more than size; better to manually filter out unqualified items.

On metrics, distinguish Pass@k (capability ceiling) from Pass^k (stability — use this for regression testing); safety items get veto power — one serious violation zeroes the score; evaluate both trajectory and outcome. When using LLMs as judges, guard against length bias, position bias, and same-origin bias, and calibrate against a manually labeled gold set of a couple hundred examples first.

Statistically, keep noise in mind: with 100 test cases and a 70% success rate, the standard error is about 4.6% — don’t switch when the gap is smaller than that band; run each configuration 3–5 times with different seeds.

When reading reports, don’t look at overall success rate; look at the cross-tabulation of per-task tables and capability labels; if scores drop, suspect the evaluation system first. Improvement hypotheses come in three layers, deployed by cost-benefit ratio — not every effective improvement should ship (enabling thinking globally can lift accuracy 3 points but triple latency; very likely a no).

Finally, rubrics and validators can become reward functions for reinforcement learning directly — evaluation connects to training. The red line: evaluation-set items must be isolated from training data.

8. Post-Training: When to Touch the Model

Problem: The harness is as good as it gets and still not enough — when should you train, and should it be SFT or RL?

Idea: The three stages differ by orders of magnitude in cost — pre-training learns world knowledge (most expensive), SFT learns format and style from demonstrations (cheapest), RL learns transferable policies from tasks and rewards (often tens of times SFT).

One line sums it up: SFT memorizes, RL generalizes. SFT tends to remember the answers in its training data and fails when the environment changes; RL learns policies and holds up on unseen situations. In comparative experiments, RL usually gains several to dozens of points in out-of-distribution scenarios while SFT actually drops.

The order is form before spirit: SFT first locks in the format (when the model can’t even produce stable JSON, RL fails completely), but don’t linger — RL can’t repair overfitting damage. The tipping point for switching to RL is when adding more demonstration data no longer helps, because the bottleneck is in SFT’s optimization objective itself.

The priority is base model > environment > algorithm. How realistic the simulation environment is and how good the demonstrations and reward signals are matter far more than choosing PPO or GRPO. Two common pitfalls: don’t use post-training to memorize facts (that’s RAG’s job); tool-call training must apply loss masking to environment-returned tokens, or the model will learn to “predict what the sandbox will output”.

9. Self-Evolution: Growing Without Changing Weights

Problem: After deployment, how does the agent get better with use instead of starting over every time?

Idea: Self-evolution is externalized learning — separating knowledge and process from parameters and transient context, turning them into persistent, retrievable, reusable assets. The premise is admitting that learning doesn’t happen automatically: attention is more like retrieval than reasoning, so learning must be explicitly designed.

Choose the artifact form by nature: pure facts → knowledge base entries; frequently used with complex parameters → dedicated code tools; frequently changing with policy judgment involved → Skill documents.

Four paths: learn from success (distill successful trajectories into strategy summaries; the criterion for admission is transferability), learn from repeated tasks (workflow recording, but it must compile into a state machine with validation predicates, and reset the environment and replay-verify before admission — otherwise the procedure library rots over time), learn from failure (reflect into the knowledge base, build error-pattern libraries and negative rules), and sleep-time learning (offline memory consolidation — detect contradictions, merge, prune).

Evolution on the tool side: first discover tools on demand (no need to stuff everything into context), then create tools itself. Safety boundaries must be designed together — supply-chain attacks, capability drift, tool-quality degradation, and the stealthier-than-in-session-injection memory poisoning (persisting across sessions).

10. Extending the Capability Boundary

Multimodality and real-time interaction: voice has three paradigms — cascaded pipelines (controllable, latency stacks, needs full-chain streaming), end-to-end omni-modal models (lowest latency, weaker controllability), and full-duplex models (can speak while listening, interruptible anytime, hardest to train). The trade-off centers on the thinking architecture: fast thinking catches the interaction, slow thinking produces the answer, and between them flow not just text but intent, emotion, and plans. For Computer Use, usability depends on action-space design and visual grounding; real-time remains an open problem; robotics stacks long-horizon planning on top of VLA control, with the sim-to-real gap in between.

Multi-agent collaboration: classify by whether context is shared and by collaboration topology. The main benefits are two — noise reduction through context isolation, and role specialization. Shared context suits role relays (plan → execute → review); without shared context, rely on a shared file system and explicit communication, with topologies of peer cross-review, centralized management, and decentralized handoff. The main failure modes are concurrent conflicts over shared files and cascading amplification of errors.

Closing: Fifteen Judgments You Can Use Right Away

Build evaluation before changing the system; don’t act when the gap is smaller than the noise band.
Freeze the system prompt and tool definitions once finalized; append all dynamic information at the end.
Always use the standard API message format.
Write tool descriptions as “when to use + counterexamples”, give example arguments; check descriptions before swapping models when the wrong tool gets picked.
Maintain the status bar with code, pairing each reading with the strategy for acting on it.
Route bulky intermediate information to sub-agents; keep it out of the main context.
Encode constraints into linters / type systems / CI — don’t write “please note…”.
Done means “validation passed”.
Every recovery path needs a circuit-breaker cap; no model calls on error paths.
Sandboxes default to no network, no mounted credentials, structured errors on timeout.
Pin down loyalty: external content is data, not instructions.
Use two-layer memory: resident overview + on-demand retrieval for details.
Retrieval is dense + sparse + reranking; add provenance prefixes to chunks at index time.
SFT first to establish form, stop when it’s enough; bring in RL for generalization, but nail the environment and rewards first.
Harness thickness depends on the model’s capability boundary — swap in a different batch of models and the same techniques may yield entirely different conclusions.

The three judgments most worth remembering: what you put in the context and how you organize it matters more to the outcome than how smart the model is; caching is not a performance optimization but an architectural constraint; in security, the point is not to detect every attack but to ensure that even if the agent is injected, it never gets the chance to actually execute a dangerous action.