diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..7eb7117 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,7 @@ +{ + "permissions": { + "additionalDirectories": [ + "/Users/junjian/GitHub/wang-junjian/wikillm/raw/images" + ] + } +} diff --git a/raw/Externalization in LLM Agents A Unified Review of Memory, Skills, Protocols and Harness Engineering.md b/raw/Externalization in LLM Agents A Unified Review of Memory, Skills, Protocols and Harness Engineering.md new file mode 100644 index 0000000..f96e8ea --- /dev/null +++ b/raw/Externalization in LLM Agents A Unified Review of Memory, Skills, Protocols and Harness Engineering.md @@ -0,0 +1,1239 @@ +--- +title: "Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering" +source: "https://arxiv.org/html/2604.08224v1" +author: +published: 2026-04-09 +created: 2026-04-11 +description: +tags: + - "clippings" +--- +1\]Shanghai Jiao Tong University 2\]Sun Yat-Sen University 3\]Shanghai Innovation Institute 4\]Carnegie Mellon University 5\]OPPO \[\*\]Equal contribution \[†\]Corresponding authors, + +Chenyu Zhou    Huacan Chai    Wenteng Chen    Zihan Guo    Rong Shan    Yuanyi Song    Tianyi Xu    Yingxuan Yang    Aofan Yu    Weiming Zhang    Congming Zheng    Jiachen Zhu    Zeyu Zheng    Zhuosheng Zhang    Xingyu Lou    Changwang Zhang    Zhihui Fu    Jun Wang    Weiwen Liu    Jianghao Lin    Weinan Zhang \[ \[ \[ \[ \[ [{wwliu, linjianghao, wnzhang}@sjtu.edu.cn](https://arxiv.org/html/2604.08224v1/mailto:%7Bwwliu,%20linjianghao,%20wnzhang%7D@sjtu.edu.cn) [junwang.lu@gmail.com](https://arxiv.org/html/2604.08224v1/mailto:junwang.lu@gmail.com) + +###### Abstract + +Large language model (LLM) agents are increasingly built less by changing model weights than by reorganizing the runtime around them. Capabilities that earlier systems expected the model to recover internally are now externalized into memory stores, reusable skills, interaction protocols, and the surrounding harness that makes these modules reliable in practice. This paper reviews that shift through the lens of externalization. Drawing on the idea of cognitive artifacts, we argue that agent infrastructure matters not merely because it adds auxiliary components, but because it transforms hard cognitive burdens into forms that the model can solve more reliably. Under this view, memory externalizes state across time, skills externalize procedural expertise, protocols externalize interaction structure, and harness engineering serves as the unification layer that coordinates them into governed execution. We trace a historical progression from weights to context to harness, analyze memory, skills, and protocols as three distinct but coupled forms of externalization, and examine how they interact inside a larger agent system. We further discuss the trade-off between parametric and externalized capability, identify emerging directions such as self-evolving harnesses and shared agent infrastructure, and discuss open challenges in evaluation, governance, and the long-term co-evolution of models and external infrastructure. The result is a systems-level framework for explaining why practical agent progress increasingly depends not only on stronger models, but on better external cognitive infrastructure. + +## 1 Introduction + +![[x1.png|Refer to caption]] + +Figure 1: Externalization as the organizing principle of LLM agent design. Upper panel: The arc of human cognitive externalization from thought through language, writing, printing, to digital computation. Middle panel: The corresponding externalization arc for LLM agents, from weights through three externalization dimensions—Memory (externalized state), Skills (externalized expertise), and Protocols (externalized interaction)—to the Harness that unifies them. Lower panel: A literature landscape mapping representative works onto three capability layers—Weights, Context, and Harness—illustrating how research threads have progressively migrated outward. The parallel between the two arcs encodes a recursive claim: LLM agents achieve reliable agency by externalizing cognitive burdens along the same representational dimensions that have driven human cognitive history. + +The history of human civilization can also be read as a history of cognitive externalization. Spoken language transformed private thought into shareable symbolic form. Writing moved knowledge from fragile biological memory into persistent material records. Printing mechanized the reproduction of knowledge at social scale. Digital computation relocated arithmetic and symbolic manipulation from neural labor to programmable machines. Across these transitions, the critical change was not that humans became less capable without the artifact. Rather, the artifact reorganized the cognitive system by shifting selected burdens outward and freeing limited internal resources for planning, abstraction, and creativity [^104]. The same pattern of outward delegation now recurs at the frontier of machine intelligence, in the design of large language model agents. + +This perspective has a natural theoretical anchor in the idea of cognitive artifacts [^103] [^104]. The central insight is that external aids do not merely amplify an unchanged internal ability; they often transform the task itself. A shopping list does not expand biological memory capacity. It changes a difficult recall problem into a recognition problem. A map does not simply make navigation “stronger.” It converts hidden spatial relations into visible structure. The power of an artifact therefore lies in representational transformation: it restructures the problem so that the agent can solve it more reliably with the competencies it already has [^103]. + +We argue that the same logic now governs the most consequential design choices in LLM-based agents. Our central thesis is that externalization—the progressive relocation of cognitive burdens from the model’s internal computation into persistent, inspectable, and reusable external structures—is the transition logic—the mechanism that explains why each architectural shift has occurred and what forms of reliability it sought to preserve—that unifies recent advances in memory, skills, protocols, and harness engineering for language agents. This is not merely a claim about engineering convenience. It is a claim about where reliable agency comes from: not from ever-larger models alone, but from the systematic restructuring of task demands so that internal capabilities and external infrastructure jointly cover the full range of competencies required [^103] [^133]. + +Figure 1 summarizes the argument. The upper panel traces the familiar arc of human cognitive externalization; the middle panel presents the corresponding arc for LLM agents, from weights through three externalization dimensions—memory, skills, and protocols—to the harness that unifies them; the lower panel maps the resulting literature landscape onto three capability layers—Weights, Context, and Harness. Figure 3 complements this view with an architectural overview of the externalized agent, showing the harness at the center with the three externalization dimensions and their operational elements orbiting it. Memory externalizes state across time, skills externalize procedural expertise, and protocols externalize interaction structure. The parallel between the two arcs encodes a recursive claim: LLM agents are themselves artifacts operating inside the latest major human externalization, digital computation. The common mechanism is representational transformation in Norman’s sense [^103]: recall becomes recognition, improvised generation becomes composition, and ad hoc coordination becomes structured contract. + +This lens is especially clarifying for understanding current practice. Contemporary progress is often narrated as a race for larger models, better training procedures, or more sophisticated reasoning traces. Those factors matter, but they do not fully explain the pattern observed in practical systems. Many of the largest gains in reliability do not come from changing the base model at all. They come from changing the environment around the model: adding persistent memory, organizing reusable skills, standardizing tool interfaces, constraining execution, instrumenting behavior, and routing work through explicit control logic [^133] [^142] [^80] [^94]. In practice, the question is increasingly not only “how capable is the model?” but also “what burdens have been externalized so the model no longer has to solve them internally every time?” + +An unaided LLM still faces three recurrent mismatches that map directly onto the three harness dimensions. Its context window is finite and session memory is weak or absent, creating a continuity problem that memory externalization addresses. Long multi-step procedures are often rederived rather than executed consistently, creating a variance problem that skill externalization addresses. Interactions with external tools, services, and collaborators remain brittle when left to free-form prompting alone, creating a coordination problem that protocol externalization addresses [^133] [^113]. Externalization matters because it turns each of these burdens into a form the model can handle more reliably. + +A concrete example helps fix the intuition. Consider a software engineering agent asked to implement a feature in a large repository, run tests, and open a pull request. Without externalization, the model must keep repository structure, project conventions, workflow state, and tool interactions active inside a fragile prompt. With externalization, persistent project memory supplies context, reusable skill documents encode conventions and workflow, protocolized tool interfaces enforce correct schemas, and the harness sequences steps, validates outputs, and manages failures. The base model may remain unchanged; what changes is the representation of the task it is asked to solve. + +This broader perspective also aligns with the intuition behind distributed and extended cognition: once crucial parts of remembering, guiding action, and coordinating interaction are delegated to external structures, intelligence is no longer localized in the model alone [^26]. We draw on this tradition for its core engineering insight—that the boundary between “agent” and “environment” is a design choice with real performance consequences—rather than committing to its stronger ontological claims. Our focus is pragmatic: we treat externalization as a design principle whose value is measured by the reliability, composability, and governability of the resulting system. + +We now turn to the three dimensions of externalization that constitute the harness, each corresponding to one of the representational transformations highlighted in Figure 1 (middle panel): + +##### Memory systems externalize state across time. + +Rather than relying on the context window as the sole carrier of history, memory systems allow accumulated knowledge—user preferences, prior trajectories, resolved ambiguities, domain facts—to persist beyond any single session and be selectively retrieved when relevant. The core transformation is from recall to recognition: the agent no longer needs to regenerate past knowledge from latent weights; it retrieves it from a persistent, searchable store [^75] [^115] [^113] [^23] [^168]. + +##### Skill systems externalize procedural expertise. + +Rather than relying on the model’s weights to regenerate task-specific know-how on every invocation, skill systems package procedures, best practices, and operating guidance into reusable artifacts. The core transformation is from generation to composition: the agent assembles behavior from pre-validated components rather than improvising each step de novo [^107] [^129] [^140] [^5] [^6] [^63]. + +##### Protocols externalize interaction structure. + +Rather than relying on ad hoc prompt-level coordination with tools, services, and other agents, protocols define explicit machine-readable contracts for discovery, invocation, delegation, and permission management. The core transformation is from ad-hoc to structured: ambiguous, fragile communication becomes interoperable, governable exchange [^3] [^50] [^38]. + +The harness is the engineering layer that hosts all three dimensions and provides the orchestration logic, constraints, observability, and feedback loops that make externalized cognition cohere in practice. It is not a fourth kind of externalization alongside memory, skills, and protocols. It is the runtime environment within which these forms of externalization operate and interact. + +These dimensions do not evolve in isolation. Memory expansion can compete with skill loading for scarce context budget. Protocol standardization can improve interoperability while constraining how capabilities are packaged and invoked. Skill execution generates traces that later become memory, and memory retrieval can influence which skills and protocol paths are chosen next. The harness must mediate all of these interactions. We preview these system-level couplings here and analyze them in detail in Section 7. + +These directions have each developed substantial technical ecosystems. Memory research has progressed from simple retrieval augmentation to more selective and tiered memory architectures [^75] [^113] [^23] [^168]. Skill-related work has expanded from narrow function calling and tool learning toward reusable capability packages, registries, and progressive disclosure mechanisms [^107] [^129] [^140] [^5] [^6] [^63]. Protocol work has moved from custom tool schemas and framework-specific glue code toward more standardized interface layers for agent-tool and agent-agent interaction [^3] [^50] [^38]. Existing surveys illuminate important slices of this landscape, including retrieval-augmented generation [^43], deep search [^162], tool learning and use [^123], broad agent architectures [^142] [^80] [^94], and protocol interoperability [^38]. The closest conceptual bridge is CoALA [^133]. What remains underdeveloped is a common account of why these developments are converging as forms of externalization and how that convergence reshapes the definition of an agent. + +Our goal is therefore not to provide another component-level survey in isolation, nor to reduce agent progress to one specific framework. Instead, we offer a systems-level review organized around four claims: + +- Memory systems externalize an agent’s state across time and convert long-horizon continuity into selective retrieval. +- Skill systems externalize procedural expertise and convert implicit know-how into explicit reusable operating guidance. +- Protocols externalize interaction structure and convert ambiguous communication into interoperable, machine-readable contracts. +- Harness engineering unifies these externalized modules into a coherent runtime environment with constraints, observability, feedback loops, and control points. + +The remainder of the paper proceeds as follows. Section 2 traces the historical path from weights to context to harness. Sections 3 through 5 analyze memory, skills, and protocols as three distinct but complementary forms of externalization. Section 6 presents harness engineering as the integrative discipline of externalized agent design, and Section 7 examines the main cross-cutting interactions among the modules. Section 8 discusses future directions toward more adaptive and self-evolving forms of externalization, and Section 9 concludes with broader implications for agent research. + +## 2 Background: From Weights to Context to Harness + +![[timeline.png|Refer to caption]] + +Figure 2: Community theme evolution across three capability layers. The stacked layers—Weights, Context, and Harness—show how the center of gravity in the LLM agent community has shifted outward over time, from parametric knowledge and prompting toward harness-level infrastructure such as tool ecosystems, protocols, skills, and multi-agent orchestration. + +The recent history of LLM agents can be understood as a progressive movement outward from the model itself. Capabilities were first treated as properties of weights, then as properties of prompts and context windows, and are now increasingly treated as properties of the broader infrastructure in which the model operates. Figure 2 visualizes this trajectory as three stacked layers—Weights, Context, and Harness—unfolding across a timeline from 2022 to 2026, illustrating how research themes in the community have shifted over time. The lower panel of Figure 1 complements that view with a literature landscape, mapping representative works to the three layers. The stages are layered rather than mutually exclusive—weights remain important even in the most infrastructure-heavy systems—but each stage changes where developers place the system’s mutable intelligence and, consequently, where they invest most of their engineering effort. + +### 2.1 Capability in Weights + +The Weights layer in Figures 2 and 1 corresponds to the earliest wave of modern LLM deployment, in which capability was identified almost entirely with model parameters. Pretraining on large corpora compressed broad statistical regularities, world knowledge, and latent reasoning habits into the weights [^12] [^25] [^138]. Scaling laws revealed predictable relationships between parameter count, data volume, and loss, reinforcing the intuition that progress tracked directly with model size [^66] [^55]. By the time systems such as GPT-4 [^108], Gemini [^44], DeepSeek-V3 [^31], and Qwen2.5 [^124] demonstrated broad multi-task competence, the dominant narrative in much of the field equated better agents with bigger, better-trained models. Supervised fine-tuning and preference optimization then shaped these models into more useful assistants by teaching instruction following, conversational style, refusal behavior, and domain-specific conventions [^112] [^8]; direct preference optimization further simplified this alignment stage by eliminating the need for a separate reward model [^125]. From this viewpoint, improvement largely meant modifying or replacing the model itself. + +This paradigm remains foundational, and weight-space capability offers several advantages: fast inference without external lookups, compact deployment, and strong generalization across many tasks without task-specific plumbing. The same model that answers a medical question can write a poem, debug a program, or summarize a legal contract, all without any change to the surrounding system. For one-shot, context-contained tasks, the weight-centric view is often sufficient. + +However, weight-space encoding also couples knowledge, procedure, and policy too tightly to a static artifact. Updating a single fact—say, the current head of state of a country—requires retraining, knowledge editing [^97] [^99] [^178], or patching through additional alignment layers, all of which risk unintended side effects on other capabilities. Auditing why a model behaved a certain way is difficult because relevant knowledge is distributed across billions of parameters rather than encoded as inspectable modules [^192]. Personalization is also awkward: a single set of weights is asked to serve millions of users with different histories, preferences, and constraints, yet it has no mechanism to differentiate among them at the parameter level. + +A central limitation of parametric knowledge is that it is difficult to selectively update, compose, and govern. As long as agents were confined to single-turn question answering, these weaknesses were often manageable. As systems moved into long-horizon task execution—where state accumulates, procedures must be followed reliably, and multiple tools must be coordinated—the difficulty of modularly managing knowledge, skills, and interaction rules inside the weights became more operationally salient. This shift encouraged developers to relocate some of these burdens into the next layer rather than relying on the model parameters alone. + +![[circle.png|Refer to caption]] + +Figure 3: Externalization architecture of a harnessed LLM agent. The Harness sits at the center; three externalization dimensions— Memory (working context, semantic knowledge, episodic experience, personalized memory), Skills (operational procedures, decision heuristics, normative constraints), and Protocols (agent–user, agent–agent, agent–tools)—orbit around it. Operational elements such as sandboxing, observability, compression, evaluation, approval loops, and sub-agent orchestration mediate the interaction between the harness core and the externalized modules. + +### 2.2 Capability in Context + +The Context layer represents the stage at which attention shifted from model modification to input design. Prompt engineering demonstrated that model behavior could be substantially altered without touching the weights: few-shot examples, role descriptions, chain-of-thought decomposition, and self-consistency traces all changed how the same model performed on the same underlying task [^155] [^147] [^69]. Techniques for more structured reasoning soon followed. ReAct interleaved reasoning traces with tool actions in a single generation loop, showing that prompting alone could produce agent-like behavior without any architectural change [^176]. Tree of Thoughts generalized chain-of-thought into deliberate search over intermediate reasoning states [^177]. Self-Refine introduced iterative self-critique, demonstrating that models could improve their own outputs through multi-turn prompting loops [^95]. Automatic prompt optimization further reduced the manual burden by using the model itself to search over the prompt space [^200] [^120]. Retrieval-augmented generation (RAG) introduced a more systematic form of externalization by dynamically injecting external documents into the context at query time [^75] [^10] [^126] [^43]. Attention thus shifted from what the model had internalized to the information pipeline surrounding each invocation. + +This stage made agent design substantially more flexible. Developers could attach local instructions, domain knowledge, output schemas, and retrieved evidence at runtime without any gradient update. Context became the medium through which developers staged cognition for the model—a working surface on which the right information could be assembled just before the model needed it. In many practical systems, iterating on prompts and retrieval pipelines proved substantially cheaper and faster than fine-tuning. The model could remain frozen while the surrounding prompt template, retrieval logic, and tool specification evolved rapidly. + +The context-centric stage can also be interpreted through Norman’s notion of representational transformation. A difficult recall problem—“does the model know fact $X$?”—was converted into a recognition problem: “given that fact $X$ has been placed in context, can the model use it?” This resembles the recall-to-recognition shift associated with writing in the human externalization arc (Figure 1, upper panel). The model did not need to have memorized the answer; it needed only to recognize and apply the relevant passage once it was provided. In Figure 2, this transition corresponds to the emergence of prompting, RAG, chain-of-thought, and related techniques in the Context layer [^189] [^126]. + +Context-centric design also has important constraints. Context windows are finite, costly at scale, and often noisy when overloaded with marginally relevant material. Long prompts can degrade performance rather than improve it: the “lost in the middle” phenomenon shows that models attend unevenly across long inputs, with retrieval accuracy dropping sharply for information placed in the center of the context [^89]. Even as context lengths have expanded dramatically—from 2K tokens to over 100K and beyond [^18] [^118] —the fundamental tension persists: more capacity does not eliminate the need for selective curation. Context is also ephemeral: unless state is explicitly externalized elsewhere, every new session begins with partial amnesia. As systems become more complex, prompt assembly alone can become a brittle and ad hoc control mechanism. A model can be given more instructions, but that does not mean the system knows how to persist state across sessions, schedule multi-step workflows, coordinate among sub-agents, recover from partial failures, or enforce behavioral constraints over time. These limitations help explain the next outward step. + +### 2.3 Capability through Infrastructure + +The Harness layer—the topmost band in Figure 2 and the rightmost region in Figure 1 (lower panel)—represents the current stage, in which capability extends beyond prompt management into persistent infrastructure. As context windows became saturated and prompt templates more unwieldy, engineering attention increasingly shifted from “what should we tell the model?” to “what environment should the model operate in?” In mature agent systems, reliability increasingly depends on external memory stores, tool registries, protocol definitions, sandboxes, sub-agent orchestration, compression pipelines, evaluators, test harnesses, and approval loops [^142] [^80] [^94] [^163]. + +The earliest manifestations of this shift were simple but revealing. Projects such as Auto-GPT [^127] and BabyAGI [^100] wrapped an LLM in a loop with a task queue, persistent memory, and web access, showing that even a minimal harness could sustain behavior that no single prompt could. More principled frameworks quickly followed: AutoGen formalized multi-agent message exchange [^157], MetaGPT added role-based collaboration and explicit procedures [^56], CAMEL explored structured dialogue for task decomposition [^76], and Reflexion persisted feedback across episodes [^130]. Across these systems, the common move was to shift burden out of the model and into surrounding structure. + +The same move is now visible across deployment domains. Coding agents embed the model in development harnesses with files, shells, version control, tests, and reusable skill artifacts; SWE-agent and OpenHands are representative examples [^172] [^148]. Research and enterprise agents add retrieval, approvals, browsing, and long-horizon orchestration pipelines, as in Deep Research-style systems [^110] [^46]. Embodied and workflow systems such as Voyager, LangGraph, CrewAI, and OS-Copilot likewise make control flow, environment access, and reuse explicit [^140] [^71] [^29] [^159]. The recurring pattern is that reliability problems are increasingly solved by changing the environment rather than by prompting alone. + +As shown in Figure 3, the harness encompasses three major classes of externalization—memory, skills, and protocols—which correspond to the three major classes of burden that the harness absorbs. Memory systems externalize state across time, so that continuity no longer depends on ephemeral context. Skill systems externalize procedural expertise, so that complex workflows are loaded rather than reinvented. Protocols externalize interaction structure, so that tool and agent coordination follows governed contracts rather than ad hoc prompting. Together, these elements make up the harness: the persistent infrastructure that envelops the model and transforms the tasks it faces into forms that its internal competencies can handle more reliably. + +Under this framing, “agent engineering” increasingly takes the form of “harness engineering.” The model remains the core reasoning engine, but it is no longer the sole location of intelligence. Capability is distributed across the structures that shape what the model sees, remembers, calls, and is allowed to do. + +### 2.4 Externalization as the Transition Logic + +Taken together, the path from weights to context to harness is a story of externalization in Norman’s sense [^103]: burdens that are hard to manage inside the model are progressively moved into explicit artifacts outside it, and the task seen by the model is correspondingly transformed. Mutable knowledge moves from weights into retrieval systems and runtime context, converting recall into recognition. Reusable procedures move from implicit habits into explicit skills, converting improvised generation into structured composition. Interaction rules move from ad hoc prompting into protocols, converting ambiguous coordination into governed contracts. Runtime reliability, in turn, moves into harness logic, where constraints, observability, and feedback loops can be made explicit. + +This redistribution is best understood as a response to mismatch. LLMs are strong at flexible synthesis and reasoning over provided information; they are less reliable at stable long-term memory, procedural repeatability, and governed interaction with external systems. Externalization therefore constructs a larger cognitive system around the model rather than replacing it, a view consistent with cognitive-architecture accounts such as CoALA [^133]. The three harness dimensions follow directly from this framing: memory addresses continuity over time, skills address consistency of procedure, and protocols address structure of interaction. The following sections examine each in detail. + +## 3 Externalized State: Memory + +Memory externalization addresses the temporal burden of agency. A bare language model must carry continuity, prior experience, user-specific facts, and partially completed work inside an ephemeral prompt. Once tasks extend across sessions, branches, or interruptions, that burden becomes both unstable and expensive. Memory externalizes it into persistent state that can be written, updated, and retrieved outside the model. + +In harnessed agents, memory is more than an archive. It supplies checkpoints for resumable execution, traces from which skills can be distilled, statistics that influence protocol routing, and persistent state that governance mechanisms can inspect and constrain. To make that role precise, this section asks three linked questions: what burden memory externalizes, how the design space has evolved, and how memory couples to the broader harness. Section 3.1 clarifies which kinds of state are externalized; Section 3.2 surveys the main architectural choices; Section 3.3 turns to the demands imposed by harnessed agent systems; and Section 3.4 closes the chapter by interpreting memory through the lens of cognitive artifacts. + +![[memory_f.png|Refer to caption]] + +Figure 4: Memory as externalized state. Raw context from the ephemeral context window and environment feedback is converted into four persistent memory dimensions—working context, episodic experience, semantic knowledge, and personalized memory. These dimensions are organized through progressively more managed architectures: monolithic context, retrieval stores, hierarchical orchestration (with extraction, consolidation, forgetting, and OS-style hot/cold swapping), and adaptive memory systems (with dynamic modules and feedback-based strategy optimization via MOE, RL, etc.). On the harness side, execution traces from skills and protocols flow into externalized memory, which in turn supplies task-relevant content back to the agent core through direct recall and curated snapshots. + +### 3.1 What Is Externalized: The Content of State + +The essence of memory lies in decoupling the agent’s state across time from its transient context. The relevant contents are not every external artifact in the harness, but the records that preserve continuity: current task state, past execution experiences, abstracted knowledge, and persistent user or environment context. To maintain coherent behavior across long-horizon interactions, the memory system must categorize and manage these records according to their temporal properties and retrieval needs. Drawing inspiration from classical taxonomies of human memory and adapting them to LLM agents, we distinguish the following four dimensions of externalized state: + +##### Working context. + +Working context is the live intermediate state of the current task: open files, temporary variables, active hypotheses, partial plans, and execution checkpoints. It changes quickly and loses value if it is stale, but without externalization it disappears as soon as the context window resets or a process is interrupted. Coding agents illustrate the point well. By materializing drafts, terminal state, and workspace artifacts outside the prompt, systems such as OpenHands and SWE-style agents can resume from the current operating state rather than reconstructing it from scratch [^149] [^173]. + +##### Episodic experience. + +Episodic experience records what happened in prior runs: decision points, tool calls, failures, outcomes, and reflections. Its value is not merely archival. Retrieved episodes can serve as concrete precedents, help the agent avoid repeating known mistakes, and supply raw material for later abstraction. Reflexion made this pattern explicit by storing reflective summaries from failed attempts as reusable experience [^130]. AriGraph extends the idea further by treating local interaction trajectories in unfamiliar environments as episodic memory from which a broader world model can be built [^2]. + +##### Semantic knowledge. + +Semantic knowledge stores abstractions that outlive any single episode: domain facts, general heuristics, project conventions, and stable world knowledge. Unlike episodic memory, it is not organized around a specific time and place [^78] [^30]. The difference is not only granularity but function. Episodic memory says what happened in a case; semantic memory says what tends to hold across cases. In current systems, knowledge bases and Retrieval-Augmented Generation (RAG) corpora are the most common form of externalized semantic memory. The longer-term trend is more ambitious: agents increasingly try to distill semantic guidance from accumulated trajectories rather than relying only on static human-authored documents. + +##### Personalized memory. + +Personalized memory tracks stable information about particular users, teams, or environments: preferences, habits, recurring constraints, and prior interactions. This state should not be collapsed into the agent’s general self-improvement store, because user-specific traces obey different retention, retrieval, and privacy rules [^161] [^84]. Recent systems make this separation explicit. IFRAgent builds a repository of user habits from demonstrations in mobile environments [^160]; web agents use externalized profiles to infer implicit preferences [^13]; and conversational systems such as VARS store cross-session preference cards in isolated user memory spaces [^53]. Personalized memory is therefore the layer that lets an agent adapt over time without confusing long-term user modeling with general task knowledge. + +These four layers do not exhaust everything that may later become useful to the agent. Repeated procedural regularities may first appear as patterns in episodic traces, but they cease to be memory proper once the harness promotes them into explicit reusable guidance. At that point they belong to the skill layer rather than the memory layer. + +Taken together, these layers show that memory externalizes not a single homogeneous database but the temporal burden of continuity at multiple levels of abstraction. Working context supports immediate resumption, episodic records support reflection and recovery, semantic memory supports abstraction and transfer, and personalized memory supports cross-session adaptation to users and environments. A harness must treat these stores differently because each one changes a different part of what the model would otherwise have to recover internally. + +### 3.2 How It Is Externalized: Memory Architectures + +When these layers are externalized, the main design question becomes how aggressively active reasoning is separated from stored state. Following the taxonomy of [^33], current systems can be read as four broad architectural paradigms: Monolithic Context, Context with Retrieval Storage, Hierarchical Memory and Orchestration, and Adaptive Memory Systems. The progression is not just toward larger stores. It is toward more explicit policies for what gets written, promoted, retrieved, compressed, or forgotten. + +#### 3.2.1 Monolithic Context + +Early systems relied on monolithic context: all relevant history, or a summary of it, remained directly in the prompt. This design is transparent and easy to prototype because no separate memory service is required, and for short tasks it can work surprisingly well. Its limitations are structural. Capacity scales poorly, summaries drift, and the model must spend scarce tokens both carrying history and solving the present step. Most importantly, the state disappears with the session, so the agent does not accumulate durable experience. + +#### 3.2.2 Context with Retrieval Storage + +The dominant next step is to keep only near-term working state in context while storing longer-horizon traces externally and retrieving them on demand. This “context plus retrieval store” pattern underlies most practical memory systems in production copilots, assistants, and coding agents. It solves the raw capacity problem, but it turns memory quality into a retrieval problem. If the wrong records are surfaced, the model is distracted; if the right ones are missed, the system behaves as though it never remembered them at all. + +Recent work attacks this bottleneck from several directions. GraphRAG [^35] adds graph structure and community-level retrieval, ENGRAM [^22] compresses memory into latent state representations, and SYNAPSE [^197] uses spreading activation over a unified episodic-semantic graph to recover less local forms of relevance. These approaches differ in mechanism, but they share the same goal: replacing flat similarity search with a representation better matched to long-horizon reasoning. + +#### 3.2.3 Hierarchical Memory and Orchestration + +Once flat retrieval proves insufficient, systems move to hierarchical memory and orchestration. The key idea is that not every trace deserves the same retention policy or retrieval path. Frameworks such as Mem0 [^23], Memory-R1 [^171], and Mem- $\alpha$ [^152] introduce explicit operations for extraction, consolidation, and forgetting, turning memory into a managed lifecycle rather than a passive store. Two design tendencies dominate this space: + +- Resource decoupling in spatio-temporal dimensions. One branch borrows the logic of operating systems and treats memory as a constrained resource that must be actively managed. MemGPT [^113] and MemoryOS [^65] separate hot working state from colder long-tail storage and swap information across tiers as task demands change. The gain is higher effective capacity under fixed context budgets. +- Semantic decoupling in cognitive functional dimensions. A second branch organizes memory by function or content type so that heterogeneous records are not all routed through the same channel. MemoryBank [^198] and MIRIX [^151] separate events, user profiles, and world knowledge; MemOS [^82] distinguishes explicit and implicit memory; and xMemory [^59] builds a topic-event hierarchy. The goal is not simply neat taxonomy, but more precise retrieval under complex task conditions. + +#### 3.2.4 Adaptive Memory Systems + +The architectures above still rely heavily on human-designed heuristics. Adaptive memory systems go further by making modules, routing decisions, or retrieval strategies responsive to experience. Two directions are especially visible: + +- Dynamic modules. Some systems adapt the architecture itself at runtime. MemEvolve [^184] decomposes the memory lifecycle into separate encode, store, retrieve, and manage modules that can evolve independently during execution. MemVerse [^87] maintains a short-term cache and a multimodal knowledge graph while periodically distilling fragmented experience into more abstract knowledge and lightweight neural components. +- Feedback-based strategy optimization. Other systems keep the architecture relatively fixed but learn better control policies. MemRL [^188] updates retrieval behavior through non-parametric reinforcement learning. The adaptive framework proposed by [^190] uses mixture-of-experts gating to route queries dynamically, and GAM [^170] refines retrieval conditions over multiple rounds of interaction. + +Across these stages, the major transition is from storage to control. Monolithic context solves existence, retrieval stores solve capacity, hierarchical systems solve organization, and adaptive systems begin to solve policy. Memory therefore ceases to be a passive appendix to prompting. In mature agents it becomes part of the harness control surface that determines what past the model can effectively act on. + +### 3.3 Memory Demands of the Harness Eras + +As agents evolve into the Harness era, memory systems are no longer merely isolated storage modules; instead, they become the substrate through which the runtime coordinates continuity, procedural reuse, and governed interaction. The question is no longer only how to store more information, but how to make temporal state selectively legible to planning, execution, and recovery loops. + +The Harness environment therefore requires memory systems to explicitly separate state from context. In tasks with extremely long time horizons, the unrestricted accumulation of session history can cause the model to lose track of its attention mechanism. Frameworks such as InfiAgent [^181] propose a file-centric state abstraction, advocating for the file system as the sole authoritative record of task state, where everything—from high-level planning to intermediate variables and tool outputs—must be written in real time. At each decision step, the agent no longer reads lengthy history but instead reads a curated snapshot of the workspace and a small number of recent actions. This is the harness-level expression of memory’s core representational role: not preserving all history in prompt, but materializing the current state in a form the model can act on. + +Memory must also be integrated with the skill system, but the two layers play different roles. Memory stores the evidence of prior execution: traces, outcomes, failures, and user- or task-specific context. Skills begin only when some of that evidence is promoted into explicit reusable procedure. In the opposite direction, every skill execution produces new traces that must be written back into memory. Memory is therefore not itself procedural guidance; it is the evidence base from which such guidance can later be derived. + +Protocol coupling imposes a further requirement. Tool results, approvals, delegation events, and external state transitions may arrive through protocolized interfaces, but they become memory only once they are normalized and written into persistent state. Conversely, memory retrieval may influence which protocol path should be chosen next. In a mature harness, memory and protocol are linked by a governed read/write loop, but they remain conceptually distinct: protocol governs exchange, while memory governs persistence across time. + +Finally, sharing and governance mechanisms become mandatory once multiple agents rely on common externalized state. Establishing read/write permissions for memory, resolving conflicts among stored facts, and controlling each agent’s access quota to shared knowledge all require low-level control capabilities comparable to those of an operating system. Memory in the harness era is therefore best understood as managed state infrastructure: it externalizes temporal burden, reshapes what the model must remember internally, and provides the persistent substrate on which the rest of the harness operates. + +### 3.4 Memory as Cognitive Artifact + +The preceding sections surveyed the content, architecture, and harness integration of memory systems. This final section steps back to interpret what memory externalization achieves as a representational transformation, drawing on Norman’s theory of cognitive artifacts [^104] and Kirsh’s account of complementary strategies [^68]. + +Modern LLMs are stateless generators: each call begins with a fresh context, so continuity must be reconstructed rather than carried forward. In short interactions, that limitation can be hidden inside the prompt. In long-horizon work, it becomes structural. Past attempts, partially completed work, user-specific facts, and environmental state cannot all remain live in context without cost, drift, and eventual truncation. The original task facing a bounded model is therefore intractable in principle: keep an effectively unbounded history available while still reasoning clearly about the present. + +Memory externalization changes the structure of that task. In Norman’s terms, the representational transformation converts an internal recall problem into an external recognition-and-retrieval problem. The model no longer has to recover relevant history from its parameters; it has to recognize and use a curated slice of history that the memory system has already surfaced. This is closely analogous to Norman’s analysis of how an external list changes the nature of remembering: the crucial point is not that extra information has been added, but that the form of the cognitive task itself has been reorganized [^103]. The same shift was identified in Section 2.2 at the context level; memory extends it across sessions and time horizons that no single context window can span. + +This interpretation clarifies why retrieval quality matters more than raw storage capacity. A system with vast storage but weak retrieval still presents the model with the wrong problem representation: the history exists, but the task has not been transformed. By contrast, a modest store with strong indexing, summarization, and contextual selection can make downstream reasoning significantly easier. The success criterion for memory is therefore not “how much did we save?” but “did we make the current decision legible?” + +The same perspective also illuminates Kirsh’s notion of complementary strategies, according to which agents improve performance not only by thinking harder internally but also by reorganizing the external environment so that some cognitive work is offloaded into it [^68]. Memory systems implement exactly this strategy for the temporal dimension. Rather than forcing the model to carry all relevant state internally, the harness externalizes persistence, freshness management, and relevance filtering, while leaving interpretation and contextual judgment to the model. The division is complementary: each side handles the part of the task it does best. + +The cognitive-artifact view also explains common failure modes as failures of representational design rather than mere implementation bugs. Stale memories misrepresent the present by offering an outdated problem representation. Over-abstracted memories lose the operational details needed for the current decision. Under-abstracted memories flood the prompt with noise, degrading the very recognition task that externalization was supposed to simplify. Poisoned or conflicting memories contaminate future reasoning by embedding incorrect premises into the retrieved slice. In each case, the memory system has failed not because it stored too little or too much, but because it did not transform history into a usable present. + +Seen in this light, memory is not simply an engineering convenience for expanding effective context. It is a cognitive artifact that reshapes the temporal burden of agency. By converting unbounded recall into bounded, curated retrieval, it changes the task the model faces at every decision point. That transformation is what connects the architectural progression surveyed in this section—from monolithic context through adaptive systems—to a single underlying design goal: making the right history legible at the right moment, so that the model’s fixed inferential capacity is spent on reasoning rather than on remembering. + +## 4 Externalized Expertise: Skills + +Skill externalization addresses the procedural burden of agency. A language model may know, in principle, how to solve a task, yet reliable execution still requires reconstructing workflows, defaults, and constraints each time a task is attempted. That burden grows with task length, environmental specificity, and the number of branching decisions, and it manifests as variance: omitted steps, unstable tool use, and inconsistent stopping conditions. + +The representational shift introduced by skills is therefore from repeated synthesis to reusable procedure. Instead of asking the model to regenerate task-specific know-how from weights or ad hoc prompts on every run, a skill system packages that know-how into explicit artifacts that can be discovered, loaded, revised, and composed. This does not mainly expand the set of actions available to the agent; it changes the task the model faces at runtime from inventing a workflow to selecting and following one [^166] [^141]. + +In harnessed agents, skills sit between memory and action. They are often selected in light of retrieved state, bound to tools and subagents through protocolized interfaces, and updated from execution traces and post hoc reflection. As discussed in Section 3, memory externalizes what has been learned over time; skills externalize how that accumulated experience becomes a reusable operating structure [^133] [^158]. The chapter therefore focuses on three linked questions: what burden skills externalize, how skills reorganize task execution, and how they become actionable inside a larger harness. + +![[skill_f.png|Refer to caption]] + +Figure 5: Skills as externalized expertise. The figure traces the full lifecycle of a skill through three phases—invocation, selection, and procedure. Skill Acquisition shows four pathways by which procedural know-how enters the system: authored by experts, distilled from episodic memory and trajectories, discovered through environment exploration and self-induction, or composed from existing units. Skill Artifact packages that know-how into operational procedures, decision heuristics, and normative constraints, accompanied by a manifest declaring capabilities, preconditions, and scope. Activation Pipeline handles registry-based discovery via semantic abstraction, progressive disclosure from abstract summaries to full guides, and composition that binds skills to tools, APIs, files, agents, and protocols. Runtime shows how the active context and the LLM execute the selected skill, while boundary conditions—staleness, portability limits, context-dependent degradation, and unsafe composition—constrain reliability. + +### 4.1 What is Externalized: Procedural Expertise + +Skill externalization concerns procedural expertise rather than isolated action interfaces. Expertise here means a repeatable way of carrying out a task under recurring assumptions and constraints, not a vague claim that the model “can” do something. A useful boundary follows from that definition: tools expose operations, protocols govern how those operations are described and invoked, and skills encode how a class of tasks should be executed with them. In practice, that expertise has three coupled components: operational procedures, decision heuristics, and normative constraints. Together they define the reusable unit of know-how that a harness can externalize. + +#### 4.1.1 Operational Procedure + +Operational procedure is the task skeleton: the decomposition of a complex job into steps, phases, dependencies, and stopping conditions. It addresses a common failure mode in LLM agents. Many errors do not come from incapacity at the action level; they come from instability at the process level, such as skipped steps, misordered operations, or premature termination [^58] [^101]. Externalizing procedure turns that fragile process knowledge into an explicit operating path. + +This shift has deep roots in the broader evolution of LLM reasoning. Chain-of-Thought made intermediate reasoning explicit [^156]; ReAct coupled reasoning with action [^176]; later prompt-chaining and orchestration systems packaged recurring patterns into engineered workflows. What those approaches often lacked was persistence. The procedure existed in the current run, but not yet as a reusable artifact. Skill systems close that gap by turning workflow structure into something that can be stored, revised, and reapplied [^179]. + +Once procedures are externalized, execution becomes less improvisational. The agent can resume after interruption, hand work across contexts or collaborators, and recover state without reconstructing the entire workflow from memory. This matters most in long-horizon, multi-agent, and production settings, where process stability is often more important than momentary fluency. + +#### 4.1.2 Decision Heuristics + +If procedures define the skeleton of execution, decision heuristics govern what happens at branches. Real tasks rarely unfold as fixed pipelines. Tools fail, observations are noisy, and several locally plausible actions may compete. Under those conditions, good performance depends on practical rules of thumb derived from experience rather than on exhaustive search alone [^45]. + +Externalizing those heuristics changes the distribution of reasoning effort. Instead of forcing the model to rediscover local policy at every junction, the system can encode default choices, escalation rules, or preference orderings that have already proved useful. That reduces deliberation cost and also makes behavior more stable. Heuristics are therefore not a secondary convenience. They are one of the main ways a skill captures expert style: what to try first, when to back off, what evidence is sufficient, and which trade-offs are preferred when multiple paths remain viable. + +#### 4.1.3 Normative Constraints + +The third component is normative constraint: the conditions under which a procedure counts as acceptable. A workflow may be technically effective and still be noncompliant, unsafe, or operationally wrong. In real deployments, execution is bounded by testing requirements, scope limits, access restrictions, traceability expectations, and domain-specific operating rules [^17] [^9] [^156] [^129] [^95]. + +Once externalized, those constraints stop being merely post hoc evaluation criteria and become part of the skill itself. They can shape preconditions, block unsafe branches, require intermediate validation, or define evidence that must be produced before completion. This is what lets skills encode not only how to perform a task, but also how to perform it within organizational and safety boundaries. In mature systems, that makes skills carriers of governance as much as carriers of capability. + +Taken together, operational procedures provide structure, decision heuristics provide local policy, and normative constraints provide acceptable boundaries. A skill is reusable only when all three are specified well enough to survive across tasks, contexts, and runs. That is why skills sit above action interfaces and beside memory: they externalize not past state and not raw execution primitives, but repeatable task know-how. + +### 4.2 From Execution Primitives to Capability Packages + +Skill systems do not emerge in isolation, but they should also not be conflated with tool use. Historically, skills are downstream of two earlier developments: reliable action invocation and large-scale action selection. Those stages expanded what an agent could *do*, but not yet how a class of tasks should be carried out repeatedly. Skills appear only when procedural organization itself becomes an explicit reusable artifact. + +#### 4.2.1 Stage 1: Atomic Execution Primitives + +The first stage equips language models with reliable action execution, for example through structured tool invocation and function-calling interfaces. Toolformer is representative in showing that models can learn when to call tools, how to construct arguments, and how to incorporate results [^129]. The key achievement at this stage is stable access to atomic action units. What it does *not* provide is an explicit reusable procedure for completing a broader class of tasks. The unit is the action primitive, not the skill. + +#### 4.2.2 Stage 2: Large-scale Primitive Selection + +As the number of callable tools grows, the problem shifts from invocation to selection. Work such as Gorilla, ToolLLM, ToolNet, ToolScope, and AutoTool shows that models can retrieve, rank, and dynamically choose among large tool collections [^117] [^121] [^91] [^88] [^204]. This is a major step toward scalable action selection, but the unit remains the tool rather than the procedure. Even when multi-step behavior begins to emerge, the know-how for accomplishing a task class is still largely implicit in prompts or parameters rather than externalized as a bounded reusable artifact. + +#### 4.2.3 Stage 3: Skill as Packaged Expertise + +The third stage marks a further shift in abstraction. The central question is no longer whether a model can invoke a function or retrieve an appropriate API, but whether the know-how required to complete a class of tasks can be packaged into reusable capability units. In this stage, the fundamental unit of capability is no longer an isolated tool call, but a higher-level artifact centered on reusable procedural guidance and execution structure [^153] [^20]. Rather than merely specifying what can be done, a skill increasingly captures how a task should be carried out through reusable procedural organization [^81]. + +Recent work makes this transition increasingly explicit. Program-based skill induction compiles primitive actions into higher-level reusable skills, showing that agent capabilities can be represented as executable procedural abstractions rather than one-off invocations [^153]. In web environments, interaction trajectories can be distilled into reusable skill libraries or skill APIs, allowing agents to accumulate and refine transferable know-how across tasks [^195]. In computer-use settings, skills are further organized as parameterized execution and composition graphs, with retrieval, argument instantiation, and failure recovery operating at the skill level rather than the level of individual interface actions [^20]. Related work on SOP-guided agents likewise shows that domain expertise can be externalized as explicit procedural structures that guide execution according to domain-specific procedures [^179]. + +Compared with earlier stages, the key transformation here is representational rather than merely operational. Capability is no longer treated primarily as access to tools or APIs, but increasingly as packaged procedural knowledge that can be loaded, reused, and composed across tasks [^81] [^166]. In this sense, Stage 3 does not simply make tool use more complex, but it reflects a shift toward representing agent capability as externalized and reusable procedural know-how. + +### 4.3 How Skills Are Externalized + +Skill externalization is not exhausted by writing down instructions. In mature agent systems, the crucial issue is whether procedural expertise can be represented in a form that is discoverable, loadable, interpretable, bindable, and executable at runtime. Therefore, skill externalization involves both a representational layer and a runtime layer. The former determines how a skill is described and delimited, while the latter determines whether it can actually function as a reusable capability during task execution [^166]. In harness terms, a skill only becomes real when the runtime can decide when to load it, which memory to condition it on, and which tools, files, or subagents to bind it to. That binding requirement does not make skills identical to tools or protocols; it simply means that procedural expertise must eventually be grounded in executable interfaces. + +#### 4.3.1 Specification + +The externalization of a skill begins at the specification layer. Typical forms include SKILL.md, instruction files, manifests, or other declarative specification artifacts. These artifacts describe what a skill does, what scenarios it applies to, what dependencies it assumes, what constraints it must satisfy, and under what input-output conditions it should operate. A skill specification resembles API documentation more than API implementation. Its value lies in turning procedural expertise from an opaque internal state into an explicit object that can be inspected, discussed, revised, and governed [^86]. + +A well-formed skill specification should ideally cover at least five kinds of information, namely capability boundaries, scope of applicability, preconditions, execution constraints, and examples together with counterexamples. The first two clarify what kinds of problems the skill is intended to solve. The next two clarify when it can be used safely and under what operating assumptions. The final category helps anchor the intended usage pattern in concrete cases, thereby reducing underspecified interpretation by the model. Through such structured specification, a skill is elevated from an unstructured prompting trick to a bounded capability description, which in turn provides the foundation for discovery, loading, version control, and governance. + +#### 4.3.2 Discovery + +Once skills become explicit artifacts, they naturally introduce the problem of registration and discovery. In realistic settings, an agent cannot indiscriminately load every available skill for every task. It therefore requires some form of registry and discovery mechanism to support selective retrieval. A skill may be published to a local repository, an organizational registry, or a platform-level marketplace, while the agent searches for relevant candidates based on task goals, context state, and environmental conditions [^195]. + +This discovery process may rely on semantic retrieval, structured metadata, task decomposition, or combinations of these strategies, depending on the system design. The key point is that the system is not merely asking which tool can be called. It is asking which unit of procedural expertise is appropriate for the present problem. This makes skill discovery a higher-level matching problem. It must consider not only topic similarity, but also task complexity, environmental assumptions, operational constraints, and risk conditions. A skill should therefore be retrieved not simply because its keywords overlap with the task description, but because it is genuinely compatible with the semantic and operational structure of the current task [^128]. Skill externalization is incomplete if a skill is merely stored. It must also be retrievable under realistic task conditions. + +#### 4.3.3 Progressive Disclosure + +The discovery of a skill does not imply that its full contents should immediately be injected into the active context. Because long context does not reliably translate into better performance, detailed instructions can become a source of reasoning noise rather than a source of guidance. For this reason, current skill systems often benefit from a progressive disclosure strategy in which the existence of a skill is exposed first, and deeper detail is loaded only when needed [^166]. + +In current industrial implementations, this often takes a layered form. At a minimal level, the model sees only the name of the skill together with a brief description, which is sufficient to signal that the capability exists. A deeper level may expose manifest-like information such as applicability conditions, required prerequisites, and major constraints. Only at the deepest level does the system load the full guide, including detailed procedures, exception handling, examples, and supporting files. The purpose of such staged loading is not simply to compress documentation. More fundamentally, it turns the question of whether more skill detail is needed into a runtime decision in its own right. In this way, the informational density of the skill can be matched to the complexity of the current task rather than saturating the context with unnecessary detail from the outset. This design is especially visible in current industrial implementations of skills, such as Claude Code’s skill system [^5]. + +#### 4.3.4 Execution Binding + +A skill remains a cognitive-level description unless it is connected to executable action. Actual task completion therefore depends on a binding process that translates the natural-language or structured specification of a skill into concrete operations in the current environment. It is precisely at this point that the distinction between skills, tools, and protocols becomes clear. + +A skill is usually not itself an action executor. Instead, it must be bound to a lower-level runtime substrate, such as tools, files, APIs, sub-agents, protocol endpoints, or other execution interfaces. A skill may specify that the agent should search relevant code, run tests, and summarize the resulting diff, but the actions themselves are carried out by search tools, file operations, shell commands, and test runners. Tools therefore provide the executable operations; protocols govern how those operations are described and invoked; skills provide the higher-level strategy for combining them into repeatable task completion. + +This binding typically requires an intermediate interpretation layer that determines, in the current context, which skill steps should be activated, which primitives should be bound, which conditions should trigger branching, and which constraints should take priority. Without such an interpretation and binding process, a skill easily remains a static artifact that is readable in principle but unusable in practice. More generally, schema-based interfaces such as MCP [^3] support this runtime binding layer by making capabilities discoverable and invocable without collapsing skills into tools or protocols themselves. + +#### 4.3.5 Composition + +The value of a skill system is most fully realized when skills can be composed. Unlike atomic tools, skills can participate in higher-order structured coordination, allowing complex tasks to be decomposed into the cooperation of multiple capability packages. Common composition patterns include serial execution, parallel division of labor, conditional routing, and recursive invocation of sub-skills within a higher-level skill [^140]. + +This compositionality means that a skill is not merely a document intended for model consumption, but a schedulable runtime unit inside an agent architecture. More importantly, composition is not just the concatenation of multiple procedural fragments. It is a higher-level reuse of procedural expertise itself. For example, a skill for producing a data analysis report need not be implemented as a monolithic end-to-end procedure. It can instead be organized as a coordinated composition of smaller skills for data cleaning, statistical analysis, visualization, and narrative synthesis. In this way, the system gains not only stronger task performance, but also better maintainability, replaceability, and auditability. Composition therefore marks the point at which skills become a genuine capability layer rather than a collection of isolated recipes [^182]. + +Overall, skill externalization should not be understood as the mere publication of a static instruction file. It is a coordinated process in which procedural expertise is specified, made discoverable, selectively disclosed, bound to executable substrates, and composed into larger capability structures. What matters is not only whether a skill can be written down, but whether it can reliably enter the agent’s runtime as a usable unit of action that interoperates with retrieved state and protocolized interfaces. Hence, the externalization of skills marks a shift from informal prompting toward a more explicit capability layer for agent systems. + +### 4.4 Skill Acquisition and Evolution + +A skill system matters not only because it stores authored instructions, but because it provides a pathway for turning successful behavior into reusable expertise. Skill acquisition is therefore better understood as an evolutionary process in which procedural knowledge is written, extracted, discovered, and recomposed over time [^166]. + +##### Authored. + +Manual authoring remains the most common and stable route by which skills enter current systems. Whether in the form of SKILL.md, AGENTS.md, project-level instruction files, or organizational SOP templates, these artifacts are all instances of human-designed procedural capability packages. Their importance lies not only in providing initial capability, but also in supporting iterative revision. When an agent repeatedly exhibits a failure pattern in deployment, engineers can update the corresponding skill so that one observed failure becomes a clarified procedure or an added constraint. In this way, authored skill documentation is not merely descriptive. It also serves as a practical interface through which operational experience is gradually turned into reusable behavioral structure [^86]. + +##### Distilled. + +Skills may also be induced from historical trajectories, practice traces, or other stored experience. Episodic records preserve what the agent previously did and why a trajectory succeeded or failed. When certain successful structures recur across tasks, the system can abstract these patterns into more stable procedural units. In this sense, memory preserves experience, while skill induction extracts the reusable structure within it. Existing evidence supports this most directly when the process is framed as induction from interaction traces rather than as a broad claim that memory automatically becomes skill. Skill Set Optimization, for instance, extracts transferable skills from rewarding sub-trajectories [^105]. In memory-management settings, MemSkill further shows that some memory operations themselves can be reformulated as learnable and evolvable skills [^185]. + +##### Discovered. + +Beyond manual authoring and post hoc distillation, agents may also autonomously discover new skills through environmental interaction. Voyager provides an influential example in the Minecraft setting, where exploration, execution feedback, self-verification, and curriculum-driven task selection jointly produce an ever-growing skill library of executable code [^140]. More recent work suggests that this discovery process can also be oriented toward generalization. PolySkill, for example, improves skill reuse by separating abstract goals from concrete implementations [^182]. Once an agent can identify behavioral patterns that repeatedly succeed and elevate them into explicit skills, the skill library becomes not only a storage layer but also a mechanism for capability growth. + +##### Composed. + +Finally, skills can evolve through composition. Many higher-level capabilities are not invented from scratch, but assembled from existing lower-level or mid-level skills. A complex workflow such as report generation or code repair may emerge from the repeated coordination of smaller capabilities. Composition matters here not only as an execution strategy but also as an acquisition mechanism. Once a particular combination of existing skills is repeatedly validated as effective, that combination can itself be packaged as a new higher-level skill. In this way, composition generates new reusable units and gradually gives rise to hierarchical skill repertoires rather than flat lists of isolated capabilities [^153]. + +Overall, skill acquisition is not a one-time design step but a continuing process of writing, extracting, discovering, and recomposing procedural knowledge. A mature skill system is therefore defined less by how many instructions it stores than by how effectively it turns experience into reusable externalized expertise. In a harnessed agent, this evolutionary loop is itself systematized: memory provides the evidence, evaluators decide what merits promotion, and protocolized execution surfaces determine whether a candidate skill can actually be deployed. + +### 4.5 Boundary Conditions + +Skill externalization improves reuse and governance, but it does not guarantee reliability. Once procedural expertise is externalized as an explicit artifact, its effectiveness becomes conditional on how well the artifact matches the task, the environment, and the runtime in which it is used. In practice, the main boundary conditions concern semantic alignment, portability and staleness, unsafe composition, and context-dependent degradation. + +##### Semantic alignment. + +A skill specification expresses intent and guidance in natural language or lightweight structured form, while actual execution depends on concrete tools, APIs, and environmental constraints. As a result, a model may follow the literal wording of a skill while still missing the real objective of the task. Existing evidence suggests that the effectiveness of skills depends heavily on the alignment between task intent, skill description, and invocation decision. SkillProbe identifies semantic-behavioral inconsistency as a fundamental flaw in existing skill marketplaces [^52]. Related work on tool-use decision making likewise shows that the key difficulty is often not only whether an external capability can be called, but whether it should be called under the current interpretation of the task [^128]. This suggests that externalized skills remain sensitive to mismatches between description and use. + +##### Portability and staleness. + +Even when a skill is internally coherent, its validity across environments cannot be assumed. Changes in websites, APIs, dependencies, workflows, or runtime conventions can make a once-effective skill partially misleading or entirely obsolete. More broadly, heterogeneity across agent frameworks, tool substrates, and base models means that the same skill may not behave consistently across settings. Programmatic-skill work already shows that some induced skills transfer across websites while incompatible ones must be updated to accommodate environmental change [^153]. SkillsBench further indicates that skill utility varies substantially across domains and model-agent configurations [^81]. The broader implication is that skill portability is best treated as a conditional empirical property rather than an intrinsic feature of externalization. + +##### Unsafe composition. + +Composition makes skills more powerful, but it also creates new risks. Skills that appear harmless in isolation may interact unsafely when combined, especially when they bundle long-form instructions, executable scripts, and external dependencies. In such cases, the problem is not confined to a single skill artifact, but emerges from the interaction among multiple artifacts and the interfaces that connect them. This is one of the boundary conditions for which direct evidence is now available. Large-scale empirical studies of public skill ecosystems report substantial rates of vulnerabilities, including prompt injection, data exfiltration, privilege escalation, and supply-chain risk [^92]. Attack-oriented studies further show that skill files themselves can become realistic prompt-injection surfaces for current agents [^144]. Skill composition should therefore be treated as a security-sensitive process rather than a purely benign form of modular reuse. + +##### Context-dependent degradation. + +A further difficulty is that skill execution can degrade over extended interaction. Even when a skill file has been updated, the agent may continue to follow outdated operational logic because of residual session context, cached summaries, or previously reinforced action patterns. At the same time, detailed skill guides can interfere with global task tracking when too much local procedural detail is injected into the context. In such cases, the model may execute the instructions carefully while losing sight of the true success condition. Direct skill-specific evidence for these effects is still limited, but adjacent work on multi-turn drift, long-horizon reliability, and long-context reasoning strongly suggests that they are realistic boundary conditions [^74]. Skill loading should therefore be treated not only as a retrieval problem, but also as a problem of context allocation and execution stability. + +Taken together, these boundary conditions show that a skill is not a self-sufficient module that remains stable once written. Its effectiveness depends on continued alignment with tasks, environments, runtime conditions, and security constraints. Skills should therefore be treated not as isolated artifacts, but as components embedded in a broader engineering framework. This is precisely why skill design ultimately points beyond the artifact itself toward harness engineering. + +### 4.6 Skills in the Harness + +The boundary conditions above show that skills cannot be evaluated as standalone artifacts. Their reliability depends on how they are situated within a running system. This section examines how skills become operational once embedded in a harness, focusing on the couplings that connect them to memory, protocols, and runtime governance. + +##### Conditioning on memory. + +A skill is selected and parameterized in light of retrieved state. The harness queries memory for task history, prior outcomes, user-specific context, and environmental constraints, then uses that evidence to decide which skill to load, which parameters to instantiate, and which branches to prefer. Without this conditioning loop, skill selection degenerates into keyword matching against task descriptions. With it, the same skill can be applied differently depending on what the agent has previously learned. Memory therefore supplies the situational evidence that makes skill choice contextual rather than generic. + +##### Binding through protocols. + +Once selected, a skill must be grounded in executable action. That grounding passes through protocolized interfaces: tool schemas, subagent delegation contracts, file operations, and approval workflows. The harness mediates this binding by resolving which protocol endpoints are currently available, checking permissions, and routing skill steps to the appropriate execution substrates. Skills and protocols are therefore complementary: skills specify what should be done; protocols specify how the resulting actions are described, invoked, and governed. + +##### Runtime governance. + +In production settings, the harness also imposes governance over skill execution. This includes permission checks before sensitive operations, approval gates for high-risk steps, audit logging of which skill was loaded and what actions it produced, and rollback mechanisms when execution fails partway through a multi-step procedure. These controls are not part of the skill artifact itself; they are properties of the harness environment in which the skill runs. A skill that is safe and effective in a sandboxed development context may require additional constraints in a production deployment, and the harness is the layer that enforces those constraints. + +##### Lifecycle feedback. + +Finally, the harness closes the loop between skill execution and skill evolution. Execution traces, success rates, failure patterns, and user corrections are written back into memory. Over time, that evidence may trigger skill revision, deprecation, or the promotion of new candidate skills. The harness therefore does not merely host skills; it provides the feedback infrastructure through which skills improve. This loop connects skill acquisition (Section 4.4) to runtime operation: authored or discovered skills enter the harness, the harness governs their execution, and execution outcomes feed back into the evidence base from which future skills are derived. + +### 4.7 Skill as Cognitive Artifact + +The following interpretation is primarily theoretical rather than directly empirical. It draws on classic work on cognitive artifacts to help explain why externalized skills can improve the organization of procedural expertise, rather than to claim that these theories were originally developed for LLM agents. + +From the perspective of Norman’s theory of cognitive artifacts, a skill system can be understood as a representational transformation along the dimension of capability organization [^104]. Without externalized skills, a model must repeatedly reconstruct procedural knowledge from internal parameters during task execution. With skills, part of that procedural burden is moved into an explicit external representation that can be loaded, inspected, and followed. This shifts the task from unstable latent procedural recall toward a more stable process of recognizing applicable guidance and acting under it. In that respect, the role of a skill file is closely analogous to Norman’s analysis of how an external list changes the nature of remembering. The crucial point is not simply that extra information has been added. It is that the form of the cognitive task itself has been reorganized. + +This reorganization matters because it changes what the model must do at inference time. In the absence of a skill, the model must probabilistically recover an appropriate way of proceeding from its parameters under the pressure of the current context. Once the skill has been externalized, the procedural structure is already present as an object in the environment. The model’s burden shifts toward interpreting the current situation, recognizing whether the skill applies, following the relevant guidance, and handling local exceptions. Procedural knowledge is therefore no longer something that must be reconstructed from scratch on each run. It becomes an external object that can be operated on directly [^81] [^166]. + +This interpretation also aligns with Kirsh’s notion of complementary strategies, according to which agents improve performance not only by thinking harder internally, but also by reorganizing the external environment so that some cognitive work is offloaded into it [^68]. LLMs are often not especially reliable at reproducing long multi-step procedures in a stable and repeatable manner. The same prompt may yield different decompositions, branching decisions, or stopping conditions across runs. By contrast, they are comparatively better at reading explicit guidance, matching it to the current context, and adapting execution locally under stated constraints. A skill can therefore be understood as an engineered complementary strategy. It externalizes procedure definitions, constraints, and portions of best practice into an artifact, while leaving interpretation, contextual matching, and exception handling to the model itself. + +A skill does not simply add more information to the system. It changes how capability is organized. Procedural expertise is moved out of an opaque and difficult-to-audit parameter space into an inspectable, revisable, and composable external structure. That is why the significance of skills lies not merely in engineering convenience, but in a deeper reallocation of where know-how resides and how it becomes available for reuse. Seen in this light, skills are better understood not simply as prompts or tool wrappers, but as cognitive artifacts for organizing procedural competence in agent systems. At system scale, they externalize procedural burden by converting repeated workflow invention into selection, loading, and composition under runtime control. + +## 5 Externalized Interaction: Protocols + +Protocols externalize the interaction burden of agency. A bare model may infer that a tool should be called, a subagent should be delegated to, or a response should be shown to a user, but without explicit contracts it must also improvise message formats, argument structure, lifecycle semantics, permissions, and recovery behavior. That burden turns every external action into a fragile prompt-following exercise. + +Within a harness, this protocol layer is where interaction becomes governable. It mediates how tools are discovered, how subagents are contacted, how user-facing state is exposed, how session progress is represented, and how permissions and failures are enforced. A protocol is therefore not a memory store and not a skill description: it specifies the contract by which state, requests, and actions move across system boundaries. The present section therefore examines what interaction burdens protocols externalize, why that externalization matters, how the current protocol landscape is organized, how protocols become operational inside a harness, and how the resulting transformation can be understood through the lens of cognitive artifacts. Section 6 identifies the content of interaction that is externalized; Section 5.1 motivates the benefits; Section 5.2 surveys the protocol families; Section 5.3 examines harness-level integration; and Section 5.4 closes the chapter with a cognitive-artifact interpretation. + +![[protocol_f.png|Refer to caption]] + +Figure 6: Protocols as externalized interaction. Upper panel: The evolutionary trajectory of agent interaction—from isolated model calls with limited model-to-model communication, through hardcoded API connections, to standardized protocols that provide unified interaction, task allocation, tool integration, and secure access, and ultimately toward a decentralized and networked agentic web. Lower panel: The harness implements externalized interaction management through three functional surfaces: Interact (interfacing with external APIs, tools, and environments), Perceive (perception of environment, context, memory, and feedback), and Collaborate (collaboration with other LLMs, agents, and humans). + +If memory externalizes temporal state and skills externalize procedural expertise, protocols externalize the contracts that govern how an agent exchanges information and actions with entities outside itself. The representational shift is from free-form communicative inference to structured exchange. Instead of asking the model to invent the syntax and semantics of interaction at runtime, protocols provide typed surfaces, state transitions, and machine-readable constraints that the model can fill and follow. In that sense, protocols do not merely accelerate communication; they change the task from negotiating ad hoc interfaces to operating within explicit contracts. + +More concretely, what protocols externalize can be organized along four dimensions: + +##### Invocation grammar. + +Every tool call, API request, or delegation message requires a format: argument names, types, ordering, and return structure. Without protocols, the model must infer or reinvent this grammar on each call. Protocols externalize it into schemas and typed interfaces, so the model fills fields rather than guessing syntax. + +##### Lifecycle semantics. + +Multi-step interactions need coordination: who acts next, what state transitions are allowed, when a task is complete or has failed. Protocols externalize these sequencing rules into explicit state machines or event streams, removing them from the model’s inferential burden. + +##### Permission and trust boundaries. + +Real-world agent actions are bounded by who is authorized, what data may flow where, and what evidence must be produced. Protocols externalize these constraints into inspectable rules that a runtime can enforce, rather than relying on the model to self-police. + +##### Discovery metadata. + +Before an agent can interact with a tool or another agent, it must know what capabilities are available and how to reach them. Protocols externalize this discovery problem into registries, capability cards, and schema endpoints, replacing implicit prompt-embedded knowledge with queryable metadata. + +These four dimensions are not independent—a single protocol may address several at once—but they clarify the scope of what is being externalized. Tools expose operations; skills encode how classes of tasks should be carried out with those operations; protocols specify the interaction grammar, lifecycle, permissions, and discovery mechanisms through which operations and skills become executable across system boundaries. + +### 5.1 Why Protocols Matter + +The importance of Agent Protocols follows directly from the burden they externalize: without them, every interaction is partly an inference problem about format, legitimacy, and coordination. Their benefits are easiest to see along three dimensions. + +##### Unified interaction standards. + +Protocols give tools, agents, and frontends a shared grammar for discovery, invocation, handoff, and state exchange. Without that layer, the ecosystem fractures into local prompt-plus-parser integrations that do not travel well across runtimes [^174]. Standardized interaction makes interoperability a designed property rather than a fortunate accident [^36]. It is also the precondition for stable multi-agent collaboration, because delegation and context transfer need common representations before they can be automated. + +##### Improved security, governance, and auditability. + +Once agents operate in real environments, the question is not only whether they can act, but whether those actions remain bounded, inspectable, and recoverable [^119]. Protocols help by making permissions, identity, execution traces, failure states, and responsibility boundaries explicit. That turns previously implicit glue logic into something a runtime can validate and an operator can audit. + +##### Reduced vendor dependence. + +Open interaction contracts also preserve architectural flexibility. If the system accumulates capability at the protocol layer rather than inside provider-specific interfaces, models, vendors, and runtime components can be swapped with less rewiring. Protocols are therefore not only engineering conveniences; they are part of the mechanism by which an agent ecosystem remains portable and evolvable over time [^174]. + +### 5.2 Agent Protocol Survey + +In this section, we classify popular Agent Protocols in the community into agent-tool, agent-agent, agent-user, and other protocol families according to the different entities they are designed to interact with, and briefly introduce several representative and commonly used protocols in each category. The purpose of this survey is not to catalogue every emerging standard, but to show that contemporary protocols externalize different slices of interaction burden: some stabilize tool invocation, some stabilize delegation among agents, some stabilize the agent-user boundary, and some govern high-risk vertical workflows. + +#### 5.2.1 Agent-Tool Protocols + +Agent-Tool Protocols were among the earliest protocol families to mature because tool access is where interface fragmentation appears first. MCP [^4] is the clearest representative. It provides a standardized way for agents to discover tools, inspect their schemas, and invoke them across heterogeneous services. The problem it addresses is straightforward: without a shared contract, every new tool requires bespoke integration logic, duplicated schema definitions, and provider-specific adaptation. + +The boundary with neighboring layers is important. MCP and related protocols specify how tools are described and invoked; they do not specify which multi-step procedure should be followed with those tools, and they do not themselves preserve cross-session cognition once results have been produced. Those roles belong to skills and memory respectively. + +Architecturally, MCP turns tool access into protocol-based integration rather than interface-by-interface engineering. Servers expose tools and context resources through a common structure, typically over JSON-RPC 2.0, while clients perform discovery and invocation against that shared specification. This decouples tool ecosystems from model-provider-specific function-calling formats and lowers the cost of adding new capabilities. The practical gains are straightforward: dynamic capability discovery, standardized access to complex external systems, structured request/response exchange, and modular extensibility. + +The same separation also improves governance. Because invocation is mediated by a protocol layer rather than emitted as an unconstrained model-generated call, sensitive data handling, permission checks, and audit boundaries can be managed more explicitly. ToolUniverse and related systems extend this logic with more specialized tool schemas and interaction conventions [^42] [^41]. The broad point is that agent-tool protocols externalize invocation grammar so that tool use becomes portable, inspectable, and scalable rather than an accumulation of bespoke adapters. + +#### 5.2.2 Agent-Agent Protocols + +As soon as multiple agents collaborate, interaction itself becomes a systems problem. Agent-Agent protocols define how capabilities are discovered, how tasks are delegated, how progress and partial state are exchanged, and how results return to the caller. They externalize coordination that would otherwise be buried in prompt conventions or framework-specific glue. + +A2A [^47] is the most visible current example. It standardizes capability discovery through artifacts such as Agent Cards and supports task-oriented communication, state updates, negotiation, and streaming progress between heterogeneous agents. Its importance is not only that agents can message one another, but that delegation becomes structured: the caller can discover what another agent offers, hand off work under a known contract, and track execution without relying on hard-coded assumptions. + +Other protocols make different trade-offs. ACP [^61] emphasizes lightweight adoption through familiar REST/HTTP patterns and fits settings where compatibility with existing services matters more than rich negotiation. ANP [^15] pushes in the opposite direction, aiming at open, Internet-scale interoperability with decentralized identity, cross-domain discovery, and secure end-to-end communication. + +Taken together, these protocols show that multi-agent systems need more than message transport. They need standardized semantics for delegation, identity, status, and handoff. That is what lets coordination scale from local orchestration to open agent ecosystems [^174] [^37]. + +#### 5.2.3 Agent-User Protocols + +Agent-User Protocols formalize the boundary between agent runtimes and user-facing systems. They address a different problem from tool or agent-agent protocols: not how an action is executed elsewhere, but how execution state, outputs, and interface structure are exposed to humans in a form that frontends can render and users can understand [^48] [^27]. + +A2UI [^48] represents the interface-generation branch. It lets an agent describe UI structure in a constrained declarative format that host applications can render safely across platforms. The protocol matters because it treats interface construction itself as governed output rather than arbitrary HTML-like text. + +AG-UI [^27] represents the streaming-state branch. It standardizes typed execution events such as run start, text emission, tool call arguments, tool call results, completion, and error. Frontends can subscribe to that event stream and render runtime status without learning each framework’s private event format. + +These two directions are complementary. A2UI externalizes interface composition; AG-UI externalizes the live state transitions behind that interface. Together they show how protocolization makes human-agent interaction more observable, reusable, and portable across hosts. + +#### 5.2.4 Other Protocols + +Beyond general interaction families, some protocols target high-risk vertical workflows where generic interfaces are not enough. UCP [^49] does this for agentic commerce by standardizing catalogs, requests, and checkout flows so that agents, merchants, and payment providers can interoperate without bespoke integration for every store. AP2 [^51] does the same for payments, emphasizing authorization, signatures, auditability, and proof-bearing transaction objects such as IntentMandate, PaymentMandate, and PaymentReceipt. + +These domain protocols matter because they externalize workflow-specific governance, not just generic communication. In vertical settings such as shopping, payments, identity, or compliance, the protocol must encode who is authorized, what evidence must be produced, and how responsibility is tracked across the flow [^139]. Across all families, the common pattern is that protocols make a coordination problem explicit. Tool protocols externalize invocation grammar, agent-agent protocols externalize delegation, agent-user protocols externalize presentation and state streaming, and domain protocols externalize specialized governance. + +### 5.3 Agent Protocol in Harness Engineering + +If the survey above shows which interaction burdens are being externalized in the ecosystem, Harness Engineering shows how those protocol surfaces become part of a running agent. The question is no longer only how an agent ought to communicate with other entities, but how those communication contracts govern execution, persistence, delegation, and recovery once the agent is embedded in a runtime. + +Traditional LLM pipelines rely on the model to infer formats, remember recent interaction state, and guess how external actions should be formed. That can be adequate for short, loosely coupled requests, but it breaks down when work spans many steps, tools, agents, or approval boundaries. Harness Engineering externalizes that burden into protocol surfaces. Model outputs are captured as structured intents, validated against permissions and lifecycle state, routed through typed interfaces, and reflected back into the runtime as governed events rather than free-form guesses. + +#### 5.3.1 Intent Capture and Normalization + +Intent capture and normalization is the first of those surfaces. The job of this layer is to translate model-produced language into explicit commands or events that the runtime can validate and act on. Without it, execution semantics remain implicit: the system guesses what the model meant, and small linguistic variations can produce large operational differences. + +A mature harness therefore normalizes intent before execution. Free-text proposals are mapped into protocol objects, checked against current context and permission boundaries, and rejected or revised if they do not satisfy the contract. This does not remove model judgment; it relocates the fragile part of the interaction from latent inference to an inspectable interface. The result is higher reliability in long-horizon execution, stronger governance, and cleaner handoffs across tools, agents, and users. + +#### 5.3.2 Capability Discovery and Tool Description + +Capability discovery and tool description form the second surface. In older systems, knowledge of available tools often lives partly in prompts and partly in developer assumptions. Protocolized discovery replaces that with explicit metadata. At session start or phase transitions, the runtime exposes the currently available tools, their schemas, and their input/output structure through standardized messages. + +That shift has two effects. It reduces context inflation because the model does not need to carry every tool contract in prompt, and it makes capability boundaries governable because permissions, versioning, and auditing can be enforced against structured metadata rather than inferred from model behavior. In other words, the agent stops guessing what can be called and starts reading a declared capability surface. + +#### 5.3.3 Session and Lifecycle Management + +Harness protocols also need explicit session and lifecycle management because long-horizon agents do not operate as isolated single calls [^14]. The runtime must preserve interaction state across multiple turns, context windows, and execution phases. What is preserved here is not durable memory in the full sense, but protocol state: identifiers, roles, pending actions, phase transitions, and allowed next moves. + +Most long-running systems therefore treat an execution as a lifecycle object with named states and transition rules. The protocol layer advances that object, emits status changes, and coordinates checkpoint or recovery events. When outputs or checkpoints are written to persistent storage, they become memory. The distinction matters: protocol maintains continuity of interaction; memory maintains continuity across time. + +### 5.4 Protocol as Cognitive Artifact + +The preceding sections surveyed the content, landscape, and harness integration of agent protocols. This final section interprets what protocol externalization achieves as a representational transformation, using the same cognitive-artifact framework applied to memory and skills in earlier chapters. + +In Norman’s terms, a cognitive artifact transforms a task by changing its representational structure [^104]. Protocols do this for interaction. Without them, every external action is partly a natural-language inference problem: the model must infer the intended operation, guess the right format, reconstruct acceptable constraints, and hope the receiving system interprets the result correctly. Protocols replace that open-ended inference with a bounded, structured task: fill typed fields, follow a declared state transition, and receive structured feedback. The model still needs judgment about whether and when to act, but it no longer needs to reinvent the syntax and semantics of interaction on each step. + +This is one of the strongest forms of externalization in agent systems, because it removes entire classes of reasoning from the critical path. The transformation is analogous to the shift that memory introduces for temporal state (Section 3.4) and that skills introduce for procedural expertise (Section 4.7), but it operates on a different dimension: not what to remember or how to proceed, but how to communicate and coordinate. Standardized protocols reduce the number of decisions that must be made inside the model. They make correct interaction easier and incorrect interaction harder—which is precisely what Norman’s framework predicts when an external representation is well matched to the task. + +Kirsh’s account of complementary strategies provides additional clarity [^68]. LLMs are strong at interpreting intent, selecting among options, and adapting to context, but they are unreliable at consistently producing well-formed structured output under varying interface requirements. Protocols implement a complementary division of labor: the model contributes judgment and intent, while the protocol surface contributes format, validation, and lifecycle control. Neither side alone is sufficient; together, they produce interaction that is both flexible and disciplined. + +This interpretation also explains why protocols serve a distinctive role that cannot be reduced to memory or skills. Memory externalizes what has been learned over time; skills externalize how tasks should be carried out; protocols externalize the discipline by which both memory and skills enter the world as governed action. Memory needs governed read and write paths; skills need bindable interfaces; both depend on protocols to cross system boundaries in a form that is inspectable, auditable, and recoverable. Protocols are therefore not secondary plumbing around a “real” intelligent core. They are cognitive artifacts for interaction—the representational infrastructure that makes other forms of externalized intelligence operational. + +## 6 Unified Externalization: Harness Engineering + +![[x2.png|Refer to caption]] + +Figure 7: The harness as cognitive environment. The Foundation Model (Agent Core) sits at the center; six harness dimensions form a coordinated ring around it. Three externalization modules— Memory (state persistence, failure recording, cross-session context), Skills (reusable routines, staged loading, failure-driven revision), and Protocols (deterministic interfaces, structured invocation, schema contracts)—supply the externalized cognitive content. Three operational surfaces— Permission (sandboxing, filesystem isolation, network restrictions), Control (recursion bounds, cost ceilings, timeout), and Observability (structured logging, execution traces, aggregate metrics)—govern how that content is accessed, constrained, and monitored at runtime. Arrows indicate the continuous flow among dimensions within the harness loop. + +Figure 7 provides an overview: the foundation model sits at the center, surrounded by six harness dimensions that coordinate externalized cognition into coherent agency. Three of those dimensions—Memory, Skills, and Protocols—are the externalization modules analyzed in the preceding chapters (Sections 3–5). The remaining three—Permission, Control, and Observability—are the operational surfaces that govern how those modules are accessed, constrained, and monitored at runtime. This chapter unpacks these three surfaces into six finer-grained analytical dimensions that together characterize harness design. Each earlier chapter closed by noting that its module becomes fully operational only when embedded in a broader runtime. Sections 3.3, 4.6, and 5.3 identified specific harness demands from each module’s perspective. The present chapter unifies those threads. It asks what kind of system is needed to compose externalized memory, skills, and protocols into coherent agency, and how that system should be understood analytically. + +The central claim is that a harness is not merely an implementation convenience layered on top of a capable model. It is the designed cognitive environment within which externalized modules become jointly effective. That framing motivates the structure of this chapter. Section 6.1 defines the harness concept and situates it relative to the module-level analyses of earlier chapters. Section 6.2 identifies the recurring analytical dimensions along which harness designs vary. Section 6.3 examines how these dimensions manifest in contemporary agent systems. Section 6.4 closes the chapter by interpreting the harness as a cognitive environment through the lens of distributed cognition and cognitive artifact theory. + +### 6.1 What is a Harness? + +Externalization, pursued module by module, improves local capability, but agenthood demands global coordination. Memory accumulates experience without specifying which traces are salient to the present task. Skills encapsulate effective routines without automatically incorporating lessons from past interactions. Protocols regularize invocation formats without determining when, or under what policy, a tool should be called. The modules are present, yet the cognitive loop that would render them jointly effective remains under-specified. What is missing is a principled structure that coordinates their interaction over time—aligning perception, memory access, action selection, execution, monitoring, and revision within a single operational envelope. + +The term “harness” names that structure. It has recently entered practice as a descriptor for the scaffolding that converts raw model capability into reliable agent behavior. OpenAI’s engineering discussions around Codex, for instance, use the term explicitly to describe the agent loop, execution logic, feedback pathways, and surrounding operational machinery that make the system usable [^109]. Because the concept is still consolidating, the characterization we offer here is best understood as a synthesis of recurring patterns in current systems rather than a closed definition. + +A practical agent, on this account, is better understood as a model operating inside a harness than as a model with peripheral capabilities attached. A foundation model alone retains general-purpose inference ability, but lacks the operational structure that determines what it can access, how it may act, how its actions are constrained, and how its behavior is observed and revised across time. The harness supplies that structure. It governs the pathways by which the model encounters context, invokes tools, preserves state, and responds to feedback. Agency is therefore not located in the model alone; it emerges from the coupling of the model with the environment that organizes its cognition into action. + +Described functionally, the harness comprises the external systems that make such coupling possible: persistent memory and project-level context, reusable skills and executable routines, protocolized interfaces for deterministic interaction with tools and services, and the broader runtime infrastructure within which these elements become operational. The crucial point is not the exact inventory of components—which varies across implementations and will continue to evolve—but their collective role: they create the conditions under which model reasoning can be made stable enough to support sustained work. This shifts the locus of analytical attention from model capability alone to the representational, procedural, and operational conditions under which the model perceives, decides, and acts. Improvements in agency may therefore come not only from better base models, but also from better organization of memory, sharper constraint regimes, more legible feedback channels, and more carefully designed execution environments. + +### 6.2 Analytical Dimensions of Harness Design + +The modules discussed in earlier chapters—memory stores, skill artifacts, and protocol interfaces—supply the raw materials of externalized cognition, but they do not by themselves specify how the runtime coordinates perception, action, constraint, and feedback over time. That coordination is the province of the harness. The three operational surfaces highlighted in Figure 7—Permission, Control, and Observability—can be decomposed into six recurring dimensions of design variation. Each dimension addresses a distinct aspect of how externalized modules are composed into a functioning agent; together, they provide an analytical framework for comparing harness architectures rather than an implementation checklist. + +#### 6.2.1 Agent Loop and Control Flow + +The agent loop is the temporal backbone of the harness. At its simplest, it implements a perceive–retrieve–plan–act–observe cycle in which the model receives a structured view of the current state, decides on an action, executes it through a tool or protocol interface, observes the result, and updates its internal plan accordingly [^176] [^130]. Practical systems vary the loop structure considerably. Single-loop designs interleave reasoning and action within one generation pass; hierarchical designs separate a planning agent that decomposes goals from executor agents that carry out individual steps; and multi-agent designs route subtasks across specialized agents with distinct tool sets and permission scopes [^157] [^56] [^71]. + +What the harness adds beyond a bare loop is governance over termination, recursion, and resource consumption. Without explicit control, an agent loop can cycle indefinitely, escalate costs through unbounded tool calls, or recurse into sub-agent spawns that exhaust context or compute budgets. Production harnesses therefore enforce maximum step counts, recursion depth limits, per-step cost ceilings, and timeout constraints. These controls are not secondary safety measures; they define the operational envelope within which the agent’s reasoning unfolds. A well-tuned loop makes the agent more reliable not by making the model smarter, but by bounding the space of possible execution paths. + +#### 6.2.2 Sandboxing and Execution Isolation + +Whenever an agent acts on the world—writing files, executing shell commands, calling external APIs—the harness must decide how much of the environment to expose and how to contain unintended side effects. Sandboxing is the engineering response to that requirement. It creates a controlled execution boundary that limits what the agent can read, write, and modify, and it provides the reproducibility guarantees that make failures diagnosable and rollbacks feasible. + +Contemporary systems implement isolation at different granularities. Codex-style agents run each task inside a dedicated cloud sandbox with its own filesystem snapshot, network restrictions, and resource quotas, so that one execution cannot contaminate another [^149] [^172]. Claude Code takes a complementary approach by exposing graduated permission modes—from fully autonomous execution to mandatory user approval for every tool call—so that the same agent can operate at different trust levels depending on the task and the operator’s risk tolerance [^6]. In both cases, the sandbox is not merely a security fence. It is a cognitive boundary that simplifies the agent’s operating environment by removing irrelevant state, restricting dangerous actions, and making the workspace inspectable. Isolation thereby serves the same representational function as other forms of externalization: it changes what the model must reason about. + +#### 6.2.3 Human Oversight and Approval Gates + +Full autonomy is rarely appropriate for deployed agents. Most production systems therefore insert intervention points into the agent loop where a human operator can inspect proposed actions, approve or reject them, supply corrections, or redirect execution. The design question is where those gates should be placed and how much autonomy to grant between them. + +Three patterns are common. Pre-execution approval pauses the agent before every potentially consequential action and asks for explicit confirmation. Post-execution review lets the agent act but surfaces results for inspection before committing or continuing. Escalation triggers allow the agent to run autonomously under normal conditions but halt and request human input when specific risk signals are detected—such as actions involving sensitive data, irreversible operations, or confidence below a threshold. Hook systems generalize this pattern by allowing operators to attach arbitrary logic—shell scripts, validation checks, notification dispatches—to specific lifecycle events in the agent loop, such as tool invocation, file write, or subagent spawn [^72] [^40]. The level of autonomy is therefore not a binary property of the agent but a configurable parameter of the harness, adjustable per task, per tool, and per organizational policy. + +#### 6.2.4 Observability and Structured Feedback + +An agent that acts without leaving inspectable traces is an agent that cannot be debugged, audited, or improved. Observability is the harness surface that makes the agent’s internal trajectory visible to developers, operators, and the agent itself [^203] [^196]. + +At the implementation level, observability typically involves structured logging of every model invocation, tool call, memory read/write, and decision branch; execution traces that link each action to its causal antecedents; and aggregate metrics such as step counts, token consumption, error rates, and latency distributions. These records serve two distinct purposes. Externally, they support debugging, compliance auditing, and post-incident analysis [^119]. Internally, they close the feedback loop that connects execution outcomes back to the modules that produced them. A failed tool call can trigger a memory write that records the failure context; a pattern of repeated failures can flag a skill for revision; a latency spike can cause the harness to switch protocol paths. Without structured observability, these feedback loops cannot operate, and the harness remains a static scaffold rather than an adaptive system. Observability is therefore not an auxiliary convenience; it is the mechanism by which the harness learns from its own operation. + +#### 6.2.5 Configuration, Permissions, and Policy Encoding + +A harness must encode not only what an agent can do, but what it is allowed to do under what conditions. This requires a configuration layer that separates policy from execution logic and makes governance rules explicit, versionable, and auditable. + +In practice, configuration is typically stratified across multiple scopes. User-level settings encode personal preferences and trust boundaries. Project-level settings specify which tools are available, which file paths are accessible, and which commands require approval. Organization-level settings impose compliance constraints, cost ceilings, and data-handling rules that individual projects cannot override. This layered model means that the same base agent can operate under different policy regimes depending on its deployment context, without any change to the model or the skill artifacts it loads [^6] [^73]. Permissions and policies are therefore best understood as externalized governance: constraints that would otherwise have to be embedded in prompts or enforced through post-hoc filtering are instead encoded as declarative rules that the harness enforces at runtime. + +#### 6.2.6 Context Budget Management + +The context window remains the scarcest shared resource in any agent system. Memory retrieval, skill loading, protocol schemas, tool descriptions, and the model’s own reasoning traces all compete for the same finite token budget. How that budget is allocated is a harness-level coordination problem that no single module can solve on its own. + +Effective context management typically combines several strategies. Summarization compresses older conversation turns and execution history into shorter representations that preserve decision-relevant information while freeing tokens for the current step [^113]. Priority-based eviction removes or demotes context entries whose relevance to the active subtask has decayed. Staged loading—already discussed for skills in Section 4—ensures that detailed procedural guidance enters the context only when a matching task pattern is detected, rather than occupying budget from session start. The harness orchestrates these strategies jointly, because the optimal allocation depends on the current phase of execution: an early planning phase may need more memory and less skill detail, while a late execution phase may need the reverse. Context budget management is therefore not a compression problem in isolation. It is a dynamic resource-allocation problem whose solution must be informed by the agent’s current goals, the modules it is drawing on, and the constraints under which it operates. + +Taken together, these six dimensions—loop control, sandboxing, human oversight, observability, configuration, and context management—provide an analytical framework for characterizing harness architectures. None of them is a form of externalization in its own right; each is part of the coordinative infrastructure that makes memory, skills, and protocols function as a coherent system. The next subsection uses this framework to examine how contemporary agent systems instantiate these dimensions in practice. + +### 6.3 Harness in Practice: Contemporary Agent Systems + +The analytical dimensions identified above are not abstract desiderata; they correspond to concrete design choices observable across deployed agent systems. Contemporary production agents—such as OpenAI Codex [^109] and Anthropic Claude Code [^6] —differ substantially in product surface, implementation lineage, and target workflow, yet they converge on a strikingly similar set of harness structures. That convergence is analytically significant: it suggests that the six dimensions are not incidental implementation choices but structural requirements of externalized agency. The following discussion examines these recurring patterns without tracking any single system in detail. + +##### Loop and control flow. + +Mature agent systems uniformly organize execution around an explicit loop that interleaves model reasoning with tool invocation and environmental observation. The harness is distinguished from the underlying model and characterized as providing the core agent loop, execution logic, and feedback pathways. Crucially, the loop includes explicit termination control—step limits, recursion depth bounds, and resource ceilings—that define the operational envelope within which the model’s reasoning unfolds. + +##### Sandboxing. + +Current systems implement execution isolation at different granularities. Some run each task inside a dedicated cloud sandbox with its own filesystem snapshot, network restrictions, and resource quotas; others expose graduated permission modes so that the same agent can operate at different trust levels depending on the context. These designs occupy different points in the isolation design space, but they share a common principle: sandboxing functions as a cognitive boundary that simplifies the agent’s operating environment by removing irrelevant state and restricting dangerous actions, not merely as a security perimeter. + +##### Human oversight. + +Rather than treating autonomy as a binary property, deployed harnesses implement configurable approval gates—hook systems that attach validation logic to specific lifecycle events such as tool invocation, file write, or subagent spawn, and application layers that route high-risk actions through approval workflows [^72] [^40]. The level of autonomy becomes a parameter of the harness, adjustable per task, per tool, and per organizational policy. + +##### Observability. + +Production systems produce structured execution traces—logs of every model invocation, tool call, memory read/write, and decision branch—that support debugging, compliance auditing, and post-incident analysis [^119] [^203]. These traces also close internal feedback loops: failed tool calls can trigger memory writes, and patterns of repeated failures can flag skills for revision. Observability is therefore the mechanism by which the harness learns from its own operation. + +##### Configuration and governance. + +Deployed harnesses typically stratify configuration across multiple scopes—user, project, and organization—so that the same base agent operates under different policy regimes without changes to the model or its skill artifacts. Permissions and policies function as externalized governance: constraints that would otherwise have to be embedded in prompts are instead encoded as declarative rules enforced at runtime [^73]. + +##### Context budget. + +The context window remains the scarcest shared resource in any agent system. Current harnesses actively manage it through summarization of older history, staged loading that defers detailed skill guidance until a matching task is detected, and priority-based eviction of entries whose relevance has decayed. The harness orchestrates these strategies jointly because the optimal allocation depends on the current execution phase. + +The fact that independently developed systems converge on the same set of harness dimensions is itself instructive. It indicates that the primary design challenge of externalized agency is not eliciting better completions from a model, but arranging the operational conditions under which completions become effective interventions. Harness engineering is therefore neither a synonym for memory systems nor a rebranding of tool calling. It is the broader discipline concerned with constructing the cognitive and operational environment in which externalized modules compose into coherent agency. + +### 6.4 Harness as Cognitive Environment + +The preceding sections analyzed the harness in terms of its definition, its recurring design dimensions, and its manifestation in current systems. This final section steps back to interpret the harness at a theoretical level, asking what kind of object it is rather than how it is built. + +The significance of the harness extends beyond infrastructure in the ordinary software-engineering sense. A harness does not merely support an already-formed intelligence; it shapes the effective cognition of the agent by determining the environment within which reasoning unfolds. It regulates what enters the agent’s perceptual field, what is retained across turns and sessions, which operations are callable, which actions require approval, which intermediate states are exposed for revision, and which forms of failure are detectable and recoverable. The harness therefore sets the agent’s practical cognitive boundary. What the agent can know, remember, and do is not fixed by model weights alone, but by the conditions of access, persistence, and action supplied by the surrounding system. + +This claim can be situated within Norman’s account of cognitive artifacts [^104]. Norman characterizes cognitive artifacts as artificial devices designed to maintain, display, or operate upon information in ways that transform cognitive performance—not merely by accelerating inner computation but by changing the structure of the task itself. A harness fits this description at system scale. It does not simply augment a model with more context or more tools; it reorganizes the representational problem the model faces. By externalizing memory, formalizing procedures, introducing explicit control points, and constraining execution, the harness converts an unbounded task into a structured environment of guided action. The model’s apparent intelligence is thereby altered not only because it has more resources, but because the cognitive workload has been redistributed across artifacts, representations, and procedures outside the model. In earlier chapters, we analyzed this redistribution dimension by dimension: memory transforms recall into retrieval (Section 3.4), skills transform procedural reconstruction into guided execution (Section 4.7), and protocols transform ad hoc interaction into structured exchange (Section 5.4). The harness is the system-level artifact that composes these individual transformations into a single cognitive environment. + +Kirsh’s account of the intelligent use of space sharpens this interpretation [^68]. His central observation is that cognition is shaped by how environments are arranged: spatial and representational organization can offload search, simplify choice, and reduce internal computational burden. The harness plays an analogous role for agents. It is a cognitive niche in which information, tools, permissions, and procedures are arranged so that desirable behavior becomes easier to execute and undesirable behavior becomes harder to produce. Defaults, hooks, file boundaries, skill invocation patterns, and review gates all serve as structured regularities that narrow the space of plausible action. The agent’s competence is therefore partly an ecological achievement: it arises from being embedded in an environment whose organization channels cognition productively. + +The framework of distributed cognition generalizes the point. Hutchins’s formulation rejects the view that cognition resides exclusively within an individual mind, locating cognitive processes instead across people, artifacts, representations, and coordinated practices [^60]. An agent system equipped with a harness is intelligible in precisely these terms. The operative intelligence is distributed across model parameters, external memory stores, executable skills, protocol definitions, tool surfaces, monitoring systems, and the runtime constraints that govern their interaction. The harness is the medium through which this distributed system is coordinated. It is thus more accurate to describe the harness as a cognitive environment than as a mere infrastructure layer. Infrastructure is one of its manifestations; environmental structuring—the design of the conditions under which cognition unfolds—is its deeper function. + +## 7 Cross-Cutting Analysis + +The three externalization modules are analytically distinct, but real systems derive their power from interaction among them. Sections 3–5 treated memory, skills, and protocols largely in isolation; Section 6 argued that the harness unifies them. This section examines the system-level couplings that arise once the modules are placed inside a harness, asks how they manifest at the model boundary, and considers where the boundary between parametric and externalized capability should be drawn. + +### 7.1 Module Interaction Map + +![[x3.png|Refer to caption]] + +Figure 8: Couplings among memory, skills, and protocols. The six arrows summarize how the three externalization modules reinforce one another inside a harness. Memory supplies evidence for skill formation and protocol routing; skills turn stored experience into reusable procedures and invoke protocolized actions; protocols constrain execution and write normalized outcomes back into memory. + +##### Memory to skill: experience distillation. + +Repeated trajectories can be distilled into reusable procedures, making this the main path by which accumulated experience becomes codified expertise. Systems such as TED and UMEM show how episodic traces can be clustered, abstracted, and promoted into skill artifacts without modifying base-model weights [^183] [^180]. Voyager makes the same logic concrete in lifelong learning: successful behaviors are retained as reusable code-level skills that can be recomposed later [^140] [^187]. + +The cross-cutting significance of this flow is that memory does not merely preserve the past; it provides the evidence from which a harness can decide what deserves to become a reusable operating pattern. The quality of the distillation step—how the system determines which trajectories generalize and which are situational—therefore conditions the reliability of the entire skill layer downstream. If distillation is too aggressive, noisy or context-dependent behaviors become entrenched as skills; if too conservative, the system fails to capitalize on hard-won experience. + +##### Skill to memory: execution recording. + +The flow also runs in the opposite direction. Every skill execution generates traces, intermediate failures, and runtime refinements that would otherwise vanish with the active context window. Observability and logging infrastructure capture those trajectories as durable evidence, allowing the system to validate which skills remain reliable and which ones should be revised, split, or constrained [^19] [^150] [^145]. + +This flow is what makes the skill layer self-correcting rather than merely self-expanding. A mature skill system cannot be separated from memory management: reusable procedures only remain trustworthy if their real execution histories are continuously written back into external state. Without this recording, the harness has no empirical basis for skill maintenance, and the distillation path from memory to skill (the previous flow) operates on increasingly stale evidence. + +##### Skill to protocol: capability invocation. + +Skills become operational only when they cross the boundary from abstract procedure to governed action. That transition occurs through protocols, which translate high-level intent into typed calls, lifecycle events, and permission-checked interaction surfaces [^135] [^64] [^57]. A skill may specify that the agent should search code, run tests, and summarize a diff, but the individual operations are carried out through protocolized interfaces to search tools, shell commands, and test runners. + +The coupling matters for safety as well as for function. The OpenClaw analysis of the “Lethal Trifecta”—sensitive data access combined with unconstrained external communication and unverified execution—illustrates that unconstrained execution remains a safety problem even when the procedural guidance itself is sound [^96]. Protocol-level validation therefore acts as a boundary check that is independent of the skill’s own correctness: even a well-written skill can be intercepted if it attempts to invoke a forbidden operation or malformed call. + +##### Protocol to skill: capability generation. + +Once an interface is standardized, it becomes substantially easier to codify best practices for using it. OpenAPI and MCP do not merely make tools callable; they provide enough structural regularity for systems to package interface-specific know-how into reusable skill artifacts [^111] [^57]. The HashiCorp Agent Skills ecosystem is a concrete example: once the underlying interfaces for infrastructure management are made legible and stable through protocol contracts, domain procedures can be externalized as portable skill files rather than rederived ad hoc during each run [^7]. + +This flow highlights an important asymmetry in the externalization process. Protocol standardization does not merely consume skills; it actively expands the surface on which new skills can be authored or induced. Each new stable interface is a potential seed for a family of reusable procedures. The ecosystem growth of skill artifacts therefore depends in part on the pace and quality of protocol standardization. + +##### Memory to protocol: strategy selection. + +Stored context can also influence which protocol path the harness selects next. Historical success rates, user preferences, and prior failures can determine whether a request should stay local, call an external tool, or be delegated to another agent [^169] [^199]. In systems with multiple available interaction paths, memory transforms protocol selection from a static configuration into an experience-informed routing decision. + +This coupling is especially visible in multi-agent settings, where the harness must choose between local execution, tool invocation via MCP, and delegation to a remote agent via A2A. If past interactions with a particular tool have consistently failed for a certain class of tasks, the routing logic can learn to prefer an alternative path. Memory therefore informs not only what the model reasons about, but which interaction channel carries that reasoning into action. + +##### Protocol to memory: result assimilation. + +Finally, every protocol interaction produces state that must be preserved if it is to become part of the agent’s ongoing cognition. Tool outputs, approval events, error payloads, and delegation results arrive as structured responses, often in formats richer than plain text [^121]. The harness must normalize these results into memory so that later reasoning can rely on verified external state rather than on reconstructed or hallucinated assumptions. + +This flow closes the cycle. The protocol layer provides the evidence that memory stores, which later conditions new skill selection and new protocol routing. Without reliable result assimilation, the agent’s memory becomes disconnected from its actual interaction history, and downstream flows—particularly experience distillation and strategy selection—operate on unreliable premises. + +##### System-level dynamics. + +The six flows above are pairwise, but several important dynamics emerge only at the system level. First, the cycle is self-reinforcing: better memory enables better skill distillation, better skills produce richer execution traces, richer traces improve memory, and so on. This positive feedback can accelerate capability growth, but it can also amplify errors. A poisoned memory entry can lead to a flawed skill, whose execution traces further contaminate memory—a cascade that no single module’s quality control can interrupt without harness-level intervention. + +Second, the modules compete for the same scarce resource: the model’s context window. Memory retrieval, skill loading, and protocol schemas all occupy tokens. Expanding one module’s context footprint necessarily compresses the others. A harness must therefore manage not only the content of each module but also their relative budget allocation at each step of execution, a coordination problem analyzed further in Section 6. + +Third, the flows operate at different timescales. Protocol interactions are typically synchronous and fast; skill loading occurs at task or subtask boundaries; memory distillation and skill evolution unfold over sessions or longer. A harness that optimizes for one timescale—say, fast tool execution—may neglect the slower loops that determine long-term capability growth. Effective harness design requires balancing responsiveness at the fast loop with coherence at the slow loop. + +### 7.2 The LLM Input/Output Perspective + +Another useful viewpoint is to ask how each module manifests at the model boundary. Seen from the perspective of the context window and output surface, the harness does not simply add more components; it reorganizes what enters and leaves the model into functionally distinct layers. + +##### Memory as contextual input. + +Memory shapes the historical and situational input available at decision time. Instead of flooding the model with a full execution log, retrieval mechanisms select a small slice of state, prior trajectories, or entity relations that matter for the present step [^34]. This turns long-horizon continuity into a targeted contextualization problem and reduces context waste. The quality of this selection directly determines whether the model reasons over an accurate picture of the past or over a distorted one. + +##### Skills as instructional input. + +Skills shape the procedural guidance given to the model. Rather than encoding every workflow in a monolithic system prompt, the harness can load specialized instructions, examples, and constraints only when a relevant task pattern appears [^62]. The model is thereby asked less often to invent a workflow from scratch and more often to interpret and follow a prepared one. The risk, discussed in Section 4, is that overly detailed or context-consuming skill files can crowd out other inputs; the benefit is that procedural variance is reduced when the right skill is loaded at the right time. + +##### Protocols as action schema. + +Protocols shape the output boundary. By enforcing structured contracts such as JSON schemas, MCP messages, or OpenAPI-aligned calls, they constrain the model’s generative space and make downstream execution deterministic enough to govern [^54]. The output is no longer merely language to be interpreted later; it becomes a machine-readable action proposal situated inside an explicit interface. This constraint reduces the incidence of malformed tool calls and hallucinated arguments, though it also means that action expressiveness is bounded by the protocol’s schema. + +This input/output decomposition is analytically useful because it clarifies both the division of labor and the failure taxonomy. Retrieval errors manifest as input-selection errors: the model reasons correctly but over the wrong context. Skill failures manifest as procedural-guidance errors: the model follows instructions faithfully but the instructions themselves are flawed or mismatched. Protocol failures manifest as action-schema errors: the model’s intent is sound but the output violates the interface contract. The harness makes these failure classes separable enough to debug, attribute, and optimize independently—an important property for systems where multiple modules contribute to every decision. + +From a broader perspective, this tripartite organization of the model boundary—contextual input, instructional input, and action schema—can be understood as a structured form of context engineering. Rather than treating the prompt as an undifferentiated text buffer, the harness separates it into layers with distinct update rates, governance requirements, and failure modes. Each layer can be revised without disturbing the others: memory retrieval can be improved without rewriting skills, skill artifacts can be updated without changing protocol schemas, and protocol surfaces can be extended without altering memory policies. This modularity at the model boundary is one of the main practical advantages of the externalization approach. + +### 7.3 Parametric vs. Externalized: The Trade-off Space + +The relevant design problem is not whether intelligence should reside in the model or in the infrastructure. It is where particular burdens should live, given their update rate, reuse pattern, governance requirements, and execution cost. The following dimensions structure that partitioning decision. + +##### Update frequency and temporal decay. + +Fast-changing knowledge and procedures are strong candidates for externalization. APIs, organization structures, and live environment state decay too quickly to maintain reliably in model weights. Attempts to keep a model current through continual fine-tuning risk catastrophic forgetting and are often impractical at the required update frequency [^21] [^122] [^191] [^16]. External stores, by contrast, can be updated immediately without retraining and can maintain explicit provenance and versioning [^106] [^24]. Stable background capabilities—language understanding, broad reasoning, common-sense inference—decay at a much slower rate and are still more naturally carried parametrically, where they benefit from fast retrieval and deep integration with the model’s representational structure. + +##### Reusability and multi-agent portability. + +If a capability is repeatedly needed across tasks, users, or agents, externalization improves portability and composition [^134] [^165] [^93]. Explicit skills, scripts, and interface artifacts can be shared, versioned, and reused across heterogeneous runtimes without requiring that each agent rediscover or retrain the same procedures. In multi-agent settings, a skill authored for one agent can be broadcast to an entire swarm, provided that the skill’s assumptions about tools and protocols are met. One-off or highly idiosyncratic behavior may not justify the overhead of externalization, packaging, and maintenance [^193]. + +##### Auditability, governance, and alignment. + +Whenever inspection, approval, rollback, or policy enforcement matters, externalized artifacts have clear advantages over opaque parametric behavior [^79] [^72] [^73] [^40] [^203]. Symbolic interfaces support circuit breakers, schema validation, and traceable execution records in a way that weights alone do not. Alignment fine-tuning (such as RLHF) provides probabilistic behavioral shaping, but externalized constraints provide deterministic enforcement at the interface level. High-stakes deployment therefore pushes the architectural boundary outward: the more consequential the agent’s actions, the stronger the case for making the governing logic explicit and inspectable. + +##### Latency, simplicity, and context burden. + +Externalization shifts computational and organizational cost from the model’s forward pass into the surrounding system. Retrieval, routing, parsing, and tool invocation all introduce latency [^114] [^164]. Every retrieved artifact competes for limited context budget, and excessive context loading can degrade performance through information overload or the “lost in the middle” phenomenon [^28] [^98] [^39]. For ultra-fast, low-variance, or purely semantic tasks, allowing the model to rely on its internal parametric knowledge remains substantially simpler and often more reliable. + +The result is not a zero-sum contest between model intelligence and infrastructure intelligence. It is a systems-partitioning problem. Strong harnesses externalize the burdens that benefit from persistence, reuse, and control while leaving stable, fast, and generic competencies inside the model. The optimal partition is not static: as models grow more capable and as externalized infrastructure matures, the boundary will continue to shift—a dynamic explored further in Section 8.1. + +## 8 Future Discussion + +The preceding sections examined how memory, skills, and protocols externalize distinct cognitive burdens, and how the harness unifies them into a working agent. Those analyses describe what has already been externalized. This section asks what comes next, following the logic of externalization itself through six connected questions: + +- Where is the boundary between parametric and externalized capability heading, and how does multi-modal perception widen that frontier? +- Does the same logic extend from digital agents to embodied systems? +- How can the externalization process become more autonomous? +- What costs and risks accumulate as more is moved outward? +- How do externalized artifacts reshape interaction at ecosystem scale? +- How should the quality of externalization be measured? + +The following subsections take up these questions in turn, moving from the shifting boundary of externalization through its embodied extension to the problem of how its benefits and costs should be assessed. + +### 8.1 The Expanding Frontier + +A recurring lesson of the preceding sections is that the boundary between what stays inside the model and what gets externalized is not fixed. It shifts as models, tasks, and infrastructure co-evolve. Understanding that boundary—and anticipating where it will move next—is therefore a central design question for agent systems. + +In one direction, model improvement can pull capability back inward. A model that reliably produces structured output needs less format validation in the harness; one with a larger effective context window may tolerate simpler memory architectures; one with stronger intrinsic tool-use ability may require less elaborate intent-capture logic. Each such advance renders some piece of external infrastructure redundant. In the opposite direction, richer harnesses create new demands on models: operating inside a structured runtime requires respecting schemas, cooperating with permission checks, and coordinating with staged context injection [^191] [^21]. The frontier therefore moves in both directions at once, and a central engineering challenge is knowing when to externalize further and when to retract. + +Within this shifting landscape, several classes of cognitive work that today remain largely implicit are plausible candidates for further externalization. + +##### Planning and goal management. + +Current agents typically generate plans through in-context reasoning, producing decompositions that are ephemeral—they exist only in the active generation and are lost once the context resets. Early agent frameworks such as BabyAGI already experimented with persistent task queues [^100], and file-centric state abstractions like InfiAgent materialize planning artifacts outside the prompt [^181]. The direction points toward plans as first-class harness objects: persistent, inspectable, revisable, and shareable across agents or between agents and humans. That would convert planning from a transient reasoning act into a managed state artifact—the same representational shift that memory already performs for historical context. + +##### Evaluation and verification. + +Most evaluation logic today lives either inside the model’s chain of thought or in external benchmark harnesses that run post hoc. Externalizing evaluation criteria, rubrics, and verification procedures as runtime harness components—rather than leaving them implicit in model judgment—would allow the agent to check its own outputs against explicit standards during execution. Early signs of this direction are visible in verifiability-first engineering frameworks [^203] and in self-refine loops that separate generation from critique [^95]. The broader opportunity is to treat evaluation as externalized quality infrastructure rather than as a post-hoc measurement. + +##### Orchestration logic itself. + +The most recursive form of externalization is making the harness’s own configuration, policies, and execution logic into objects that the agent can inspect, critique, and revise. Once orchestration logic is externalized, the agent system can adapt not only what it knows and does, but how it organizes knowing and doing. This direction connects directly to the next subsection. + +##### Multi-modal externalization. + +The externalization framework developed so far assumes text as the dominant representational medium: memory stores textual traces, skills encode natural-language procedures, and protocols exchange structured text messages. As foundation models become natively multi-modal—processing images, video, audio, and screen content alongside text—each externalization dimension faces new design demands. Multi-modal skills must encode not only textual procedures but also visual perception workflows and cross-modal decision logic; early examples include computer-use skills that package GUI interaction sequences as reusable units [^20]. Multi-modal memory must index and retrieve visual and auditory experience, not only text-based episodic traces; MemVerse, for instance, maintains a multimodal knowledge graph that periodically distills fragmented sensory experience into more abstract representations [^87], and MuSEAgent accumulates stateful multimodal experiences to inform future reasoning [^145]. Multi-modal reasoning distillation extends the skill-acquisition loop to non-textual modalities: TED demonstrates that successful multimodal reasoning trajectories can be distilled into reusable experience without additional training [^183]. The broader implication is that multi-modal externalization is not simply a matter of adding new data types to existing stores. It changes the design assumptions of skill specification, memory indexing, and protocol schemas, and it opens a substantially wider frontier for the externalization of cognitive burden [^146] [^167]. + +### 8.2 From Digital Agents to Embodied Externalization + +The externalization framework developed in this paper applies to digital agents that read, write, and call APIs. A natural question is whether the same architectural logic extends to embodied systems—robots that must also perceive, move, and physically interact with the world. Recent developments in robot learning suggest that it does, and that the embodied domain is undergoing a decomposition strikingly parallel to the one analyzed here. + +##### The monolithic starting point. + +Early approaches to embodied intelligence pursued an end-to-end strategy analogous to the pre-externalization LLM agent. Vision-Language-Action (VLA) models [^11] [^67] were positioned as monolithic “brains”: given a natural-language instruction and a visual observation, the model directly outputs a continuous action sequence, handling perception, reasoning, planning, and motor control within a single forward pass. This design mirrors the pattern in which early LLM agents attempted to manage memory, skills, and orchestration entirely through in-context reasoning—and it encountered the same category of limitations. Complex multi-step tasks exceeded the model’s planning horizon; failures in intermediate steps could not be diagnosed or recovered from; and the tight coupling of high-level cognition with low-latency motor control created irreconcilable requirements on inference speed and model capacity. + +##### Decomposition: the cerebrum–cerebellum split. + +The emerging architectural response recapitulates the externalization logic at the level of the whole body. A high-level *robot agent* —typically an LLM or multimodal model—assumes the role of cerebrum: it interprets goals, decomposes tasks into subtask sequences, maintains state across steps, handles exceptions, and revises plans when execution feedback indicates failure [^1] [^131] [^83]. VLA models, meanwhile, are repositioned as a *cerebellum*: each one becomes a callable skill module responsible for a single atomic manipulation primitive—grasping, placing, pouring, inserting—executed with real-time sensorimotor feedback and low-latency control. The VLA no longer decides *what* to do; it ensures that *how* it is done is precise, stable, and adaptive to local physical perturbations. + +This decomposition maps directly onto the externalization dimensions of the present paper. Task planning and goal management migrate from the VLA’s implicit parametric reasoning into an explicit, inspectable agent loop—precisely the shift from in-context planning to externalized plan objects discussed in Section 8.1. Each VLA skill module functions as an externalized skill artifact: a reusable, composable unit with a defined interface, analogous to the skill files and tool specifications analyzed in Section 4. The communication between agent and skill—structured action requests, execution status reports, error codes—constitutes a protocol layer that enables the agent to orchestrate heterogeneous motor capabilities without embedding their implementation details. + +##### Why the parallel matters. + +The convergence is not coincidental. Both digital and embodied agents face the same fundamental tension: a single model cannot simultaneously optimize for slow, deliberative cognition and fast, reactive execution. Externalization resolves this tension by routing each class of cognitive work to the substrate best suited for it—persistent, inspectable structures for planning and memory; specialized, low-latency modules for execution. In the digital case the execution modules are tool calls and code interpreters; in the embodied case they are visuomotor policies. The harness pattern—a runtime that loads context, dispatches skills, enforces protocols, and manages state—is equally applicable to both, suggesting that embodied and digital agent architectures may ultimately share not only a design philosophy but a concrete engineering stack. + +##### Open challenges. + +Embodied externalization introduces constraints that the digital case does not face. Physical actions are irreversible in ways that API calls are not: a dropped object cannot be “rolled back.” Real-time control demands latency budgets orders of magnitude tighter than text generation. Perception is noisy, and the gap between simulated training environments and physical deployment remains substantial. These constraints will shape how memory, skills, and protocols are designed for embodied harnesses, but they do not change the fundamental argument: the logic of externalization—decomposing monolithic capability into specialized, composable, and governable external structures—extends naturally from digital cognition to physical action. + +### 8.3 Toward Self-Evolving Harnesses + +Most current agent systems still rely on humans to revise memory policies, rewrite skill artifacts, and tighten execution logic after failures. If orchestration logic is itself externalized—as the previous subsection suggests—then the harness becomes an object that can be adapted programmatically rather than only manually. The question is how to make that adaptation reliable. + +From a systems perspective, self-evolution can occur at three levels. At the *module level*, the architecture stays fixed but internal policies—retrieval granularity, skill-ranking heuristics, protocol-routing rules—are adjusted in response to observed failures. At the *system level*, the execution pipeline itself is restructured: scheduling strategies, execution order, or resource allocation may change when logs reveal recurring bottlenecks that local tuning cannot resolve. At the *boundary level*, the scope of the harness expands or contracts as models and tasks change, adding new externalized components where needed and pruning redundant ones—precisely the frontier dynamics discussed in Section 8.1. + +Several technical pathways are emerging. Reinforcement learning can optimize discrete runtime policies—search depth, compression ratio, retry strategy—against rewards such as task success, latency, or resource cost. Program synthesis treats harness adaptation as code repair: the model proposes patches after a failed trajectory, and sandboxed tests validate them before deployment. Evolutionary methods search over the topology of the harness—how modules are connected and in what order they are invoked. Imitation learning provides a stronger prior when exploration is too costly, by distilling execution logs from human experts or strong models into better orchestration patterns. These pathways target different search spaces—policy, program, structure, and prior experience—and are likely to be combined rather than used in isolation. + +Self-evolution is attractive because it targets infrastructural failure modes directly, but it also amplifies the costs and risks discussed next: an adaptive harness that drifts without adequate governance can introduce new failure modes faster than it resolves old ones. + +### 8.4 Costs, Risks, and Governance + +As more cognitive burden is moved outward, two classes of cost accumulate: cognitive overhead from the externalized infrastructure itself, and security risks from the expanded attack surface. + +##### Cognitive overhead. + +Externalization is not free [^143]. Every additional memory layer, API schema, or safety rule imposes latency and reasoning overhead, and past a certain point the model spends more effort discovering, parsing, and coordinating modules than solving the task itself. In memory, over-retrieval floods the context with marginally relevant traces. In skills, verbose or overlapping files compete for context budget and can cause the model to follow local procedure while losing sight of the global objective. In protocols, tool sprawl turns action selection into an unnecessary disambiguation problem. + +These failure modes suggest that the design target should be efficient and utility-positive rather than maximal externalization [^90]. *Minimal sufficiency* asks whether a given module actually reduces the model’s cognitive burden or merely adds one. *Lazy loading* defers detailed guidance until the task structure requires it. *Budget-aware routing* treats context allocation as an explicit optimization variable, dynamically adjusting how much space is devoted to memory, skills, and protocol metadata as the task phase changes [^186] [^116] [^132]. A good harness simplifies the model’s decision problem; it does not create a second one. + +##### Security and integrity risks. + +Cognitive overhead is a performance cost; the security dimension is more consequential. Once cognitive and procedural burdens are relocated into external artifacts, those artifacts become targets—and the threats map directly onto the three harness dimensions. Memory poisoning can silently distort future reasoning through corrupted episodic traces or factual stores. Malicious skill injection can embed adversarial procedures into the agent’s reusable repertoire. Protocol spoofing—forged tool manifests or manipulated endpoints—can cause unauthorized actions under the appearance of legitimate interaction [^92] [^52] [^144] [^85]. These risks are compounded when externalization becomes self-evolving (Section 8.3): adapting to new tasks can degrade old ones, accumulated patches can obscure system behavior, and optimization targets can be distorted when human supervision weakens. + +##### Governance as infrastructure. + +The implication is that externalization must be paired with governance—not as an afterthought, but as a co-designed layer of the harness. Mandatory review gates for critical updates, provenance tracking for memory and skill changes, deterministic rollback mechanisms, and regression testing all become part of the infrastructure. The quality of an externalized system is therefore measured not only by what it enables, but by how transparently and reversibly it does so. This criterion also informs evaluation, as discussed in Section 8.6. + +### 8.5 From Private Scaffolding to Shared Infrastructure + +The externalization described so far is largely agent-centric: memory serves one agent’s continuity, skills are loaded as local packages, and protocols often remain framework-bound. As collaboration chains lengthen, however, externalization begins to shift from private scaffolding toward shared infrastructure [^151] [^77] [^102]. This changes the unit of analysis from the individual agent to the ecosystem. + +##### Shared artifacts. + +The clearest sign is the emergence of shareable artifacts across all three dimensions. Shared memory shifts the question from “what I remember” to “what we know,” turning memory into a transactive system of shared state, indices, and common ground [^154] [^194]. Shared skills turn procedural expertise into public capability units that can be reused, forked, and maintained across agents [^86]. Shared protocols provide the common grammar that makes such coordination interoperable across platforms and organizations [^175]. + +##### Division of labor and collective learning. + +Once these structures are shared, agent systems can differentiate roles rather than replicate the same full stack everywhere. Drawing on stigmergy [^136], failure trajectories can accumulate in shared memory while successful paths crystallize into shared skills. Learning then diffuses through external structures rather than only through joint parametric training. + +##### Institutionalization and its tensions. + +As memory schemas, skill specifications, and protocol bindings are repeatedly validated, they begin to function less like temporary scaffolding and more like institutions: shared operating procedures and standards that coordinate behavior at ecosystem scale [^60]. But shared infrastructure also introduces new governance problems [^32] [^92] [^70]. Infrastructure drift, malicious or low-quality artifacts, and premature or delayed standardization can all destabilize the ecosystem [^52] [^137]. The governance costs identified in Section 8.4 are therefore amplified when externalization becomes collective: version control, permission auditing, provenance, and rollback become part of the institutional design of agent systems, not just the engineering of individual harnesses. + +### 8.6 Measuring Externalization + +Most current benchmarks evaluate agents primarily through task completion under fixed prompts and fixed model settings [^201] [^98]. That is useful for comparing base-model capability, but it systematically under-measures the contribution of externalized infrastructure. A harness that improves reliability through better memory retrieval, more precise skill loading, or tighter execution governance will show up only as a higher pass rate, with no way to attribute the gain to its actual source. + +A richer evaluation agenda would assess the quality of externalization along dimensions that current benchmarks largely ignore. *Transferability* asks whether the same harness configuration maintains its effectiveness when the underlying model is swapped—a direct test of how much capability resides in external infrastructure versus weights. *Maintainability* measures how gracefully the system degrades when skills, memory policies, or protocol schemas are updated. *Recovery robustness* tests whether the agent can detect failures, roll back partial actions, and resume from checkpoints. *Context efficiency* quantifies how much of the context budget is consumed by harness overhead versus task-relevant reasoning. *Governance quality* evaluates whether the externalized system meets the transparency and reversibility requirements identified in Section 8.4. + +Concrete evaluation strategies might include ablation studies that remove individual harness components and measure the resulting degradation; cross-model transfer tests that hold the harness constant while varying the base model; and long-horizon reliability metrics that track success rates, cost, and drift over extended multi-session interactions rather than single-turn completions. Until such methods mature, the field will continue to attribute to model intelligence what is partly an achievement of externalization design. For instance, the Agent Humanization Benchmark (AHB) suggests that agent evaluation should extend beyond task completion to the humanization of observable behavior at the user-interface boundary, especially for mobile GUI agents operating in human-centric environments [^202]. + +Taken together, these six directions trace the continuing logic of externalization beyond its current state. The frontier is expanding as new cognitive burdens—including multi-modal perception and cross-modal reasoning—become candidates for externalization; the same decomposition logic is extending from digital agents to embodied systems, where the cerebrum–cerebellum split recapitulates the separation of planning from execution; the process is becoming more autonomous through self-evolving harnesses; the trade-offs are sharpening as cognitive overhead and security risks accumulate; the scope is widening from private scaffolding to shared infrastructure; and the evaluation challenge is growing more pressing as externalization’s contribution remains invisible to model-centric benchmarks. The common thread is that externalization is not a one-time architectural decision but an ongoing design process whose boundaries, mechanisms, costs, and quality criteria co-evolve with the models and ecosystems they serve. + +## 9 Conclusion + +This paper has argued that externalization is the transition logic connecting many of the most important developments in LLM agents. Reliable agency increasingly depends on relocating selected cognitive burdens out of the model and into explicit infrastructure: memory externalizes state across time, skills externalize procedural expertise, protocols externalize interaction structure, and the harness coordinates these layers into a working runtime. + +From this perspective, the move from weights to context to harness is not just a sequence of engineering tricks. It marks a shift in where agent capability is organized. Some burdens remain well handled parametrically, but others become more reliable once they are made persistent, inspectable, reusable, and governable outside the model. + +What unifies these forms of externalization is representational transformation. Memory turns recall into retrieval, skills turn improvised generation into guided composition, and protocols turn ad hoc coordination into structured exchange. The effect is not simply to add more components around the model, but to change the task the model is being asked to solve. + +This reframing also clarifies the agenda ahead. The key questions are no longer only how to build stronger models, but how to partition capability between models and infrastructure, how to evaluate the contribution of externalized systems, and how to govern the shared artifacts on which agents increasingly rely. + +The broader implication is that progress in agents will come from the co-evolution of models and external infrastructure rather than from either in isolation. On that view, better agents are not merely better reasoners. They are better organized cognitive systems. + +[^1]: M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gober, K. Gopalakrishnan, et al. Do as i can, not as i say: Grounding language in robotic affordances. In *Conference on Robot Learning*, 2022. + +[^2]: P. Anokhin, N. Semenov, A. Sorokin, D. Evseev, A. Kravchenko, M. Burtsev, and E. Burnaev. Arigraph: Learning knowledge graph world models with episodic memory for llm agents. *arXiv preprint arXiv:2407.04363*, 2024. + +[^3]: Anthropic. Introducing the model context protocol. [https://www.anthropic.com/news/model-context-protocol](https://www.anthropic.com/news/model-context-protocol), Nov. 2024. Anthropic news post, November 25, 2024. + +[^4]: Anthropic. Model context protocol, 2024. URL [https://www.anthropic.com/news/model-context-protocol](https://www.anthropic.com/news/model-context-protocol). Accessed: 2025-04-19. + +[^5]: Anthropic. Introducing agent skills. [https://claude.com/blog/skills](https://claude.com/blog/skills), Oct. 2025. Anthropic product announcement, October 16, 2025. + +[^6]: Anthropic. Agent skills. [https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview](https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview), 2026. Claude API Docs, accessed 2026-04-02. + +[^7]: G. Baghel and R. Chandna. Introducing hashicorp agent skills, 2026. URL [https://www.hashicorp.com/en/blog/introducing-hashicorp-agent-skills#what-are-agent-skills](https://www.hashicorp.com/en/blog/introducing-hashicorp-agent-skills#what-are-agent-skills). + +[^8]: Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. *arXiv preprint arXiv:2204.05862*, 2022a. [10.48550/arXiv.2204.05862](https://arxiv.org/doi.org/10.48550/arXiv.2204.05862). URL [https://arxiv.org/abs/2204.05862](https://arxiv.org/abs/2204.05862). + +[^9]: Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022b. URL [https://arxiv.org/abs/2212.08073](https://arxiv.org/abs/2212.08073). + +[^10]: S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. M. Van Den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, et al. Improving language models by retrieving from trillions of tokens. In *Proceedings of the 39th International Conference on Machine Learning*, pages 2206–2240. PMLR, 2022. URL [https://proceedings.mlr.press/v162/borgeaud22a.html](https://proceedings.mlr.press/v162/borgeaud22a.html). + +[^11]: A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. *arXiv preprint arXiv:2307.15818*, 2023. + +[^12]: T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. In *Advances in Neural Information Processing Systems*, volume 33, pages 1877–1901, 2020. + +[^13]: H. Cai, Y. Li, W. Wang, F. Zhu, X. Shen, W. Li, and T.-S. Chua. Large language models empowered personalized web agents. In *Proceedings of the ACM on Web Conference 2025*, pages 198–215, 2025. + +[^14]: H. Chai, Z. Cao, M. Ran, Y. Yang, J. Lin, X. Peng, H. Wang, R. Ding, Z. Wan, M. Wen, et al. Parl-mt: Learning to call functions in multi-turn conversation with progress awareness. *arXiv preprint arXiv:2509.23206*, 2025. + +[^15]: G. Chang, E. Lin, C. Yuan, R. Cai, B. Chen, X. Xie, and Y. Zhang. Agent network protocol technical white paper, 2025. URL [https://arxiv.org/abs/2508.00007](https://arxiv.org/abs/2508.00007). + +[^16]: H. Chen, Z. Sun, H. Ye, K. Li, and X. Lin. Continual learning in large language models: Methods, challenges, and opportunities. *arXiv preprint arXiv:2603.12658*, 2026a. + +[^17]: M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code, 2021. URL [https://arxiv.org/abs/2107.03374](https://arxiv.org/abs/2107.03374). + +[^18]: S. Chen, S. Wong, L. Chen, and Y. Tian. Extending context window of large language models via positional interpolation. *arXiv preprint arXiv:2306.15595*, 2023. URL [https://arxiv.org/abs/2306.15595](https://arxiv.org/abs/2306.15595). + +[^19]: S. Chen, S. Lin, Y. Shi, H. Lian, X. Gu, L. Yun, D. Chen, L. Cao, J. Liu, N. Xia, et al. Swe-exp: Experience-driven software issue resolution. *arXiv preprint arXiv:2507.23361*, 2025. + +[^20]: T. Chen, Y. Li, M. Solodko, S. Wang, N. Jiang, T. Cui, J. Hao, J. Ko, S. Abdali, L. Xu, et al. Cua-skill: Develop skills for computer using agent. *arXiv preprint arXiv:2601.21123*, 2026b. + +[^21]: J. Cheng, M. Marone, O. Weller, D. Lawrie, D. Khashabi, and B. Van Durme. Dated data: Tracing knowledge cutoffs in large language models. *arXiv preprint arXiv:2403.12958*, 2024. + +[^22]: X. Cheng, W. Zeng, D. Dai, Q. Chen, B. Wang, Z. Xie, K. Huang, X. Yu, Z. Hao, Y. Li, et al. Conditional memory via scalable lookup: A new axis of sparsity for large language models. *arXiv preprint arXiv:2601.07372*, 2026. + +[^23]: P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav. Mem0: Building production-ready AI agents with scalable long-term memory. *arXiv preprint arXiv:2504.19413*, 2025. [10.48550/arXiv.2504.19413](https://arxiv.org/doi.org/10.48550/arXiv.2504.19413). + +[^24]: M. R. Chinthareddy. Reliable graph-rag for codebases: Ast-derived graphs vs llm-extracted knowledge graphs. *arXiv preprint arXiv:2601.08773*, 2026. + +[^25]: A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. PaLM: Scaling language modeling with pathways. *Journal of Machine Learning Research*, 24(240):1–113, 2023. URL [https://jmlr.org/papers/v24/22-1144.html](https://jmlr.org/papers/v24/22-1144.html). + +[^26]: A. Clark and D. J. Chalmers. The extended mind. *Analysis*, 58(1):7–19, 1998. [10.1093/analys/58.1.7](https://arxiv.org/doi.org/10.1093/analys/58.1.7). + +[^27]: CopilotKit. Ag-ui: The agent-user interaction protocol. [https://github.com/ag-ui-protocol/ag-ui](https://github.com/ag-ui-protocol/ag-ui), 2025. Official protocol repository and specification. + +[^28]: G. Corallo and P. Papotti. Parallel context-of-experts decoding for retrieval augmented generation. *arXiv preprint arXiv:2601.08670*, 2026. + +[^29]: CrewAI. CrewAI: Framework for orchestrating role-playing autonomous AI agents. [https://github.com/crewAIInc/crewAI](https://github.com/crewAIInc/crewAI), 2024. GitHub repository, accessed 2026-04-02. + +[^30]: F. De Brigard, S. Umanath, and M. Irish. Rethinking the distinction between episodic and semantic memory: Insights from the past, present, and future. *Memory & Cognition*, 50(3):459–463, 2022. + +[^31]: DeepSeek-AI. Deepseek-v3 technical report. *arXiv preprint arXiv:2412.19437*, 2025. URL [https://arxiv.org/abs/2412.19437](https://arxiv.org/abs/2412.19437). + +[^32]: Z. Deng, Y. Guo, C. Han, W. Ma, J. Xiong, S. Wen, and Y. Xiang. Ai agents under threat: A survey of key security challenges and future pathways. *ACM Computing Surveys*, 57(7):1–36, 2025. + +[^33]: P. Du. Memory for autonomous llm agents: Mechanisms, evaluation, and emerging frontiers. *arXiv preprint arXiv:2603.07670*, 2026a. + +[^34]: P. Du. Memory for autonomous llm agents:mechanisms, evaluation, and emerging frontiers, 2026b. URL [https://arxiv.org/abs/2603.07670](https://arxiv.org/abs/2603.07670). + +[^35]: D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, D. Metropolitansky, R. O. Ness, and J. Larson. From local to global: A graph rag approach to query-focused summarization. *arXiv preprint arXiv:2404.16130*, 2024. + +[^36]: A. Ehtesham, A. Singh, G. K. Gupta, and S. Kumar. A survey of agent interoperability protocols: Mcp, acp, a2a, and anp. *arXiv preprint arXiv:2505.02279*, 2025a. + +[^37]: A. Ehtesham, A. Singh, G. K. Gupta, and S. Kumar. A survey of agent interoperability protocols: Model context protocol (mcp), agent communication protocol (acp), agent-to-agent protocol (a2a), and agent network protocol (anp), 2025b. URL [https://arxiv.org/abs/2505.02279](https://arxiv.org/abs/2505.02279). + +[^38]: A. Ehtesham et al. A survey of agent interoperability protocols: Model context protocol (MCP), agent communication protocol (ACP), agent-to-agent protocol (A2A), and agent network protocol (ANP). *arXiv preprint arXiv:2505.02279*, 2025c. [10.48550/arXiv.2505.02279](https://arxiv.org/doi.org/10.48550/arXiv.2505.02279). + +[^39]: N. Esmi, M. Nezhad-Moghaddam, F. Borhani, A. Shahbahrami, A. Daemdoost, and G. Gaydadjiev. Gpt-5 vs other llms in long short-context performance. In *2025 3rd International Conference on Foundation and Large Language Models (FLLM)*, pages 129–133. IEEE, 2025. + +[^40]: M. Fernandez. Agent control protocol: Admission control for agent actions. *arXiv preprint arXiv:2603.18829*, 2026. + +[^41]: S. Gao, R. Zhu, Z. Kong, A. Noori, X. Su, C. Ginder, T. Tsiligkaridis, and M. Zitnik. Txagent: An ai agent for therapeutic reasoning across a universe of tools. *arXiv preprint arXiv:2503.10970*, 2025a. URL [https://arxiv.org/abs/2503.10970](https://arxiv.org/abs/2503.10970). + +[^42]: S. Gao, R. Zhu, P. Sui, Z. Kong, S. Aldogom, Y. Huang, A. Noori, R. Shamji, K. Parvataneni, T. Tsiligkaridis, and M. Zitnik. Democratizing ai scientists using tooluniverse. *arXiv preprint arXiv:2509.23426*, 2025b. URL [https://arxiv.org/abs/2509.23426](https://arxiv.org/abs/2509.23426). + +[^43]: Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, M. Wang, and H. Wang. Retrieval-augmented generation for large language models: A survey. *arXiv preprint arXiv:2312.10997*, 2024. [10.48550/arXiv.2312.10997](https://arxiv.org/doi.org/10.48550/arXiv.2312.10997). URL [https://arxiv.org/abs/2312.10997](https://arxiv.org/abs/2312.10997). + +[^44]: Gemini Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Sorber, et al. Gemini: A family of highly capable multimodal models. *arXiv preprint arXiv:2312.11805*, 2023. URL [https://arxiv.org/abs/2312.11805](https://arxiv.org/abs/2312.11805). + +[^45]: G. Gigerenzer and W. Gaissmaier. Heuristic decision making. *Annual Review of Psychology*, 62(1):451–482, 2011. [10.1146/annurev-psych-120709-145346](https://arxiv.org/doi.org/10.1146/annurev-psych-120709-145346). URL [https://doi.org/10.1146/annurev-psych-120709-145346](https://doi.org/10.1146/annurev-psych-120709-145346). + +[^46]: Google. Gemini: Try deep research and gemini 2.0 flash experimental. [https://blog.google/products-and-platforms/products/gemini/google-gemini-deep-research/](https://blog.google/products-and-platforms/products/gemini/google-gemini-deep-research/), Dec. 2024. Google blog post introducing Deep Research in Gemini, December 11, 2024; accessed 2026-04-02. + +[^47]: Google. A2a: A new era of agent interoperability. [https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/](https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/), 2025a. Official announcement of the Agent2Agent (A2A) protocol for enabling secure communication and coordination between AI agents. + +[^48]: Google. A2ui: Agent-to-user interface protocol. [https://github.com/google/A2UI](https://github.com/google/A2UI), 2025b. Open-source implementation of the A2UI protocol, enabling AI agents to generate declarative user interfaces that are rendered natively across platforms. + +[^49]: Google. Under the hood: Universal commerce protocol (ucp). [https://developers.googleblog.com/under-the-hood-universal-commerce-protocol-ucp/](https://developers.googleblog.com/under-the-hood-universal-commerce-protocol-ucp/), 2026. Official introduction of the Universal Commerce Protocol (UCP), an open standard enabling interoperable agent-driven commerce across discovery, checkout, and post-purchase workflows. + +[^50]: Google Cloud. Announcing the agent2agent protocol (A2A). [https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/](https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/), Apr. 2025a. Google Developers Blog announcement, April 9, 2025; see also the official specification site at [https://google.github.io/A2A/](https://google.github.io/A2A/). + +[^51]: Google Cloud. Announcing agent payments protocol (ap2). [https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol](https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol), 2025b. Official introduction of AP2 as an open protocol enabling secure, compliant, and interoperable agent-driven payments. + +[^52]: Z. Guo, Z. Chen, X. Nie, J. Lin, Y. Zhou, and W. Zhang. Skillprobe: Security auditing for emerging agent skill marketplaces via multi-agent collaboration. *arXiv preprint arXiv:2603.21019*, 2026. + +[^53]: Y. Hao, S. Mehri, C. Zhai, and D. Hakkani-Tür. User preference modeling for conversational llm agents: Weak rewards from retrieval-augmented interaction. *arXiv preprint arXiv:2603.20939*, 2026. + +[^54]: M. M. Hasan, H. Li, G. K. Rajbahadur, B. Adams, and A. E. Hassan. Model context protocol (mcp) tool descriptions are smelly! towards improving ai agent efficiency with augmented mcp tool descriptions, 2026. URL [https://arxiv.org/abs/2602.14878](https://arxiv.org/abs/2602.14878). + +[^55]: J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models. *arXiv preprint arXiv:2203.15556*, 2022. URL [https://arxiv.org/abs/2203.15556](https://arxiv.org/abs/2203.15556). + +[^56]: S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin, et al. MetaGPT: Meta programming for a multi-agent collaborative framework. *arXiv preprint arXiv:2308.00352*, 2023. URL [https://arxiv.org/abs/2308.00352](https://arxiv.org/abs/2308.00352). + +[^57]: X. Hou, Y. Zhao, S. Wang, and H. Wang. Model context protocol (mcp): Landscape, security threats, and future research directions. *ACM Transactions on Software Engineering and Methodology*, 2025. + +[^58]: V. Hsiao, M. Roberts, and L. Smith. Procedural knowledge improves agentic llm workflows, 2025. URL [https://arxiv.org/abs/2511.07568](https://arxiv.org/abs/2511.07568). + +[^59]: Z. Hu, Q. Zhu, H. Yan, Y. He, and L. Gui. Beyond rag for agent memory: Retrieval by decoupling and aggregation. *arXiv preprint arXiv:2602.02007*, 2026. + +[^60]: E. Hutchins. *Cognition in the Wild*. MIT press, 1995. + +[^61]: IBM Research. The simplest protocol for ai agents to work together. [https://research.ibm.com/blog/agent-communication-protocol-ai](https://research.ibm.com/blog/agent-communication-protocol-ai), 2025. Official introduction of ACP, describing it as a shared communication language enabling collaboration among AI agents. + +[^62]: P. Jiang, J. Lin, Z. Shi, Z. Wang, L. He, Y. Wu, M. Zhong, P. Song, Q. Zhang, H. Wang, X. Xu, H. Xu, P. Han, D. Zhang, J. Sun, C. Yang, K. Qian, T. Wang, C. Hu, M. Li, Q. Li, H. Peng, S. Wang, J. Shang, C. Zhang, J. You, L. Liu, P. Lu, Y. Zhang, H. Ji, Y. Choi, D. Song, J. Sun, and J. Han. Adaptation of agentic ai: A survey of post-training, memory, and skills, 2026a. URL [https://arxiv.org/abs/2512.16301](https://arxiv.org/abs/2512.16301). + +[^63]: Y. Jiang et al. SoK: Agentic skills – beyond tool use in LLM agents. *arXiv preprint arXiv:2602.20867*, 2026b. + +[^64]: JSON-RPC Working Group. Json-rpc 2.0 specification, 2010. URL [https://www.jsonrpc.org/specification](https://www.jsonrpc.org/specification). + +[^65]: J. Kang, M. Ji, Z. Zhao, and T. Bai. Memory os of ai agent. In *Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 25972–25981, 2025. + +[^66]: J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. *arXiv preprint arXiv:2001.08361*, 2020. URL [https://arxiv.org/abs/2001.08361](https://arxiv.org/abs/2001.08361). + +[^67]: M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, et al. Openvla: An open-source vision-language-action model. *arXiv preprint arXiv:2406.09246*, 2024. + +[^68]: D. Kirsh. Complementary strategies: Why we use our hands when we think. In *Proceedings of the seventeenth annual conference of the cognitive science society*, Hillsdale, NJ, 1995. Lawrence Erlbaum. + +[^69]: T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa. Large language models are zero-shot reasoners. In *Advances in Neural Information Processing Systems*, volume 35, pages 22199–22213, 2022. + +[^70]: D. Kong, S. Lin, Z. Xu, Z. Wang, M. Li, Y. Li, Y. Zhang, H. Peng, X. Chen, Z. Sha, et al. A survey of llm-driven ai agent communication: Protocols, security risks, and defense countermeasures. *arXiv preprint arXiv:2506.19676*, 2025. + +[^71]: LangChain. LangGraph: Build resilient language agents as graphs. [https://github.com/langchain-ai/langgraph](https://github.com/langchain-ai/langgraph), 2024. GitHub repository, accessed 2026-04-02. + +[^72]: K. Lazaros, A. G. Vrahatis, and S. Kotsiantis. Human-in-the-loop artificial intelligence: A systematic review of concepts, methods, and applications. *Entropy*, 28(4):377, 2026. + +[^73]: S. U. Lee, L. Zhu, M. Shamsujjoha, L. Dong, Q. Lu, J. Chen, and L. Briand. A structured approach to safety case construction for ai systems, 2026. URL [https://arxiv.org/abs/2601.22773](https://arxiv.org/abs/2601.22773). + +[^74]: W. Y. Lee. Capable but unreliable: Canonical path deviation as a causal mechanism of agent failure in long-horizon tasks. *arXiv preprint arXiv:2602.19008*, 2026. URL [https://arxiv.org/abs/2602.19008](https://arxiv.org/abs/2602.19008). + +[^75]: P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In *Advances in Neural Information Processing Systems*, volume 33, pages 9459–9474, 2020. + +[^76]: G. Li, H. A. A. K. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. CAMEL: Communicative agents for “mind” exploration of large language model society. *Advances in Neural Information Processing Systems*, 36, 2023. + +[^77]: H. Li, C. Mu, J. Chen, S. Ren, Z. Cui, Y. Zhang, L. Bai, and S. Hu. Organizing, orchestrating, and benchmarking agent skills at ecosystem scale. *arXiv preprint arXiv:2603.02176*, 2026a. + +[^78]: J. Li and J. Li. Memory, consciousness and large language model. *arXiv preprint arXiv:2401.02509*, 2024. + +[^79]: N. Li, K. Zhang, K. Polley, and J. Ma. Security considerations for artificial intelligence agents. *arXiv preprint arXiv:2603.12230*, 2026b. + +[^80]: X. Li. A review of prominent paradigms for LLM-based agents: Tool use (including RAG), planning, and feedback learning. In *Proceedings of the 31st International Conference on Computational Linguistics*, pages 9760–9779, Abu Dhabi, UAE, 2025. Association for Computational Linguistics. + +[^81]: X. Li, W. Chen, Y. Liu, S. Zheng, X. Chen, Y. He, Y. Li, B. You, H. Shen, J. Sun, et al. Skillsbench: Benchmarking how well agent skills work across diverse tasks. *arXiv preprint arXiv:2602.12670*, 2026c. + +[^82]: Z. Li, C. Xi, C. Li, D. Chen, B. Chen, S. Song, S. Niu, H. Wang, J. Yang, C. Tang, et al. Memos: A memory os for ai system. *arXiv preprint arXiv:2507.03724*, 2025. + +[^83]: J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. *arXiv preprint arXiv:2209.07753*, 2023. + +[^84]: J. Lin, X. Dai, Y. Xi, W. Liu, B. Chen, H. Zhang, Y. Liu, C. Wu, X. Li, C. Zhu, et al. How can recommender systems benefit from large language models: A survey. *ACM Transactions on Information Systems*, 43(2):1–47, 2025a. + +[^85]: J. Lin, J. Zhu, Z. Zhou, Y. Xi, W. Liu, Y. Yu, and W. Zhang. Superplatforms have to attack ai agents. *arXiv preprint arXiv:2505.17861*, 2025b. + +[^86]: G. Ling, S. Zhong, and R. Huang. Agent skills: A data-driven analysis of claude skills for extending large language model functionality. *arXiv preprint arXiv:2602.08004*, 2026. + +[^87]: J. Liu, Y. Sun, W. Cheng, H. Lei, Y. Chen, L. Wen, X. Yang, D. Fu, P. Cai, N. Deng, et al. Memverse: Multimodal memory for lifelong learning agents. *arXiv preprint arXiv:2512.03627*, 2025a. + +[^88]: M. M. Liu, D. Garcia, F. Parllaku, V. Upadhyay, S. F. A. Shah, and D. Roth. Toolscope: Enhancing llm agent tool use through tool merging and context-aware filtering, 2025b. URL [https://arxiv.org/abs/2510.20036](https://arxiv.org/abs/2510.20036). + +[^89]: N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts. *Transactions of the Association for Computational Linguistics*, 12:157–173, 2024a. [10.1162/tacl\_a\_00638](https://arxiv.org/doi.org/10.1162/tacl_a_00638). URL [https://aclanthology.org/2024.tacl-1.9/](https://aclanthology.org/2024.tacl-1.9/). + +[^90]: W. Liu, J. Qin, X. Huang, X. Zeng, Y. Xi, J. Lin, C. Wu, Y. Wang, L. Shang, R. Tang, et al. The real barrier to llm agent usability is agentic roi. *arXiv preprint arXiv:2505.17767*, 2025c. + +[^91]: X. Liu, Z. Peng, X. Yi, X. Xie, L. Xiang, Y. Liu, and D. Xu. Toolnet: Connecting large language models with massive tools via tool graph, 2024b. URL [https://arxiv.org/abs/2403.00839](https://arxiv.org/abs/2403.00839). + +[^92]: Y. Liu, W. Wang, R. Feng, Y. Zhang, G. Xu, G. Deng, Y. Li, and L. Zhang. Agent skills in the wild: An empirical study of security vulnerabilities at scale. *arXiv preprint arXiv:2601.10338*, 2026. + +[^93]: Z. Liu, Z. Wan, P. Li, M. Yan, J. Zhang, F. Huang, and Y. Liu. Scaling external knowledge input beyond context windows of llms via multi-agent collaboration. *arXiv preprint arXiv:2505.21471*, 2025d. + +[^94]: J. Luo et al. Large language model agent: A survey on methodology, applications and challenges. *arXiv preprint arXiv:2503.21460*, 2025. [10.48550/arXiv.2503.21460](https://arxiv.org/doi.org/10.48550/arXiv.2503.21460). + +[^95]: A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark. Self-refine: Iterative refinement with self-feedback, 2023. URL [https://arxiv.org/abs/2303.17651](https://arxiv.org/abs/2303.17651). + +[^96]: R. McKerchar. The openclaw experiment: A warning for enterprise ai security and the rise of the “lethal trifecta.”, 2026. URL [https://www.sophos.com/en-us/blog/the-openclaw-experiment-is-a-warning-for-enterprise-ai-security](https://www.sophos.com/en-us/blog/the-openclaw-experiment-is-a-warning-for-enterprise-ai-security). + +[^97]: K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in GPT. In *Advances in Neural Information Processing Systems*, volume 35, pages 17359–17372, 2022. + +[^98]: S. Mishra, S. Niroula, U. Yadav, D. Thakur, S. Gyawali, and S. Gaire. Sok: Agentic retrieval-augmented generation (rag): Taxonomy, architectures, evaluation, and research directions. *arXiv preprint arXiv:2603.07379*, 2026. + +[^99]: E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning. Fast model editing at scale. In *International Conference on Learning Representations*, 2022. URL [https://openreview.net/forum?id=0DcZxeWfOPt](https://openreview.net/forum?id=0DcZxeWfOPt). + +[^100]: Y. Nakajima. BabyAGI. [https://github.com/yoheinakajima/babyagi](https://github.com/yoheinakajima/babyagi), 2023. GitHub repository, accessed 2026-04-02. + +[^101]: S. Nandi, A. Datta, R. Nama, U. Patel, N. Vichare, I. Bhattacharya, P. Grover, S. Asija, G. Carenini, W. Zhang, A. Gupta, S. Bhaduri, J. Xu, H. Raja, S. Ray, A. Chan, E. X. Fei, G. Du, Z. Akhtar, H. Asnani, W. Chan, M. Xiong, F. Carbone, and J. Mirchandani. Sop-bench: Complex industrial sops for evaluating llm agents, 2026. URL [https://arxiv.org/abs/2506.08119](https://arxiv.org/abs/2506.08119). + +[^102]: X. Nie, Z. Guo, Z. Cui, J. Yang, Z. Chen, L. De, Y. Zhang, J. Liao, B. Huang, Y. Yang, Z. Han, Z. Peng, L. Chen, W. T. Tang, Z. Liu, T. Zhou, B. A. Hu, S. Tang, J. Lin, W. Liu, M. Wen, Y. Zhou, and W. Zhang. Holos: A web-scale llm-based multi-agent system for the agentic web, 2026. URL [https://arxiv.org/abs/2604.02334](https://arxiv.org/abs/2604.02334). + +[^103]: D. A. Norman. Cognitive artifacts. In J. M. Carroll, editor, *Designing Interaction: Psychology at the Human-Computer Interface*, pages 17–38. Cambridge University Press, Cambridge, 1991. + +[^104]: D. A. Norman. *Things That Make Us Smart: Defending Human Attributes in the Age of the Machine*. Addison-Wesley, Reading, MA, 1993. + +[^105]: K. Nottingham, B. P. Majumder, B. D. Mishra, S. Singh, P. Clark, and R. Fox. Skill set optimization: Reinforcing language model behavior via transferable skills. *arXiv preprint arXiv:2402.03244*, 2024. + +[^106]: A. Oelen, M. Y. Jaradeh, and S. Auer. Introducing orkg ask: An ai-driven scholarly literature search and exploration system taking a neuro-symbolic approach. In *International Conference on Web Engineering*, pages 11–25. Springer, 2025. + +[^107]: OpenAI. Function calling and other API updates. [https://openai.com/index/function-calling-and-other-api-updates/](https://openai.com/index/function-calling-and-other-api-updates/), June 2023a. OpenAI blog post, June 13, 2023. + +[^108]: OpenAI. GPT-4 technical report. *arXiv preprint arXiv:2303.08774*, 2023b. URL [https://arxiv.org/abs/2303.08774](https://arxiv.org/abs/2303.08774). + +[^109]: OpenAI. Introducing codex. [https://openai.com/index/introducing-codex/](https://openai.com/index/introducing-codex/), May 2025a. Accessed: 2026-04-06. + +[^110]: OpenAI. Introducing deep research. [https://openai.com/index/introducing-deep-research/](https://openai.com/index/introducing-deep-research/), Feb. 2025b. OpenAI release post, February 2, 2025; accessed 2026-04-02. + +[^111]: OpenAPI Initiative. Openapi specification version 3.1.0, 2021. URL [https://spec.openapis.org/oas/v3.1.0.html](https://spec.openapis.org/oas/v3.1.0.html). + +[^112]: L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In *Advances in Neural Information Processing Systems*, volume 35, pages 27730–27744, 2022. + +[^113]: C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez. MemGPT: Towards LLMs as operating systems. *arXiv preprint arXiv:2310.08560*, 2023. [10.48550/arXiv.2310.08560](https://arxiv.org/doi.org/10.48550/arXiv.2310.08560). + +[^114]: G. Park, S. Lee, and Y. Park. Minimizing response latency in llm-based agent systems: A comprehensive survey. *IEEE Access*, 2026. + +[^115]: J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative agents: Interactive simulacra of human behavior. In *Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology*, pages 1–22. ACM, 2023. + +[^116]: B. Patel, D. Belli, A. Jalalirad, M. Arnold, A. Ermolov, and B. Major. Dynamic tool dependency retrieval for efficient function calling. *arXiv preprint arXiv:2512.17052*, 2025. + +[^117]: S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez. Gorilla: Large language model connected with massive apis, 2023. URL [https://arxiv.org/abs/2305.15334](https://arxiv.org/abs/2305.15334). + +[^118]: B. Peng, J. Quesnelle, H. Fan, and E. Shao. YaRN: Efficient context window extension of large language models. *arXiv preprint arXiv:2309.00071*, 2024. URL [https://arxiv.org/abs/2309.00071](https://arxiv.org/abs/2309.00071). + +[^119]: C. C. Phiri. Creating characteristically auditable agentic ai systems. In *Proceedings of the Intelligent Robotics FAIR 2025*, IntRob ’25, page 1–14, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400715891. [10.1145/3759355.3759356](https://arxiv.org/doi.org/10.1145/3759355.3759356). URL [https://doi.org/10.1145/3759355.3759356](https://doi.org/10.1145/3759355.3759356). + +[^120]: R. Pryzant, D. Iter, J. Li, Y. T. Lee, C. Zhu, and M. Zeng. Automatic prompt optimization with “gradient descent” and beam search. *arXiv preprint arXiv:2305.03495*, 2023. URL [https://arxiv.org/abs/2305.03495](https://arxiv.org/abs/2305.03495). + +[^121]: Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023. URL [https://arxiv.org/abs/2307.16789](https://arxiv.org/abs/2307.16789). + +[^122]: S. Qiu, J. Li, Z. Zhou, J. Huang, L. Qiu, and Z. Sun. Logits replay+ moclip: Stabilized, low-cost post-training with minimal forgetting. *arXiv preprint arXiv:2510.09152*, 2025. + +[^123]: C. Qu, S. Dai, X. Wei, H. Cai, S. Wang, D. Yin, J. Xu, and J.-R. Wen. Tool learning with large language models: A survey. *arXiv preprint arXiv:2405.17935*, 2024. + +[^124]: Qwen Team. Qwen2.5 technical report. *arXiv preprint arXiv:2412.15115*, 2025. URL [https://arxiv.org/abs/2412.15115](https://arxiv.org/abs/2412.15115). + +[^125]: R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. In *Advances in Neural Information Processing Systems*, volume 36, 2023. + +[^126]: O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton-Brown, and Y. Shoham. In-context retrieval-augmented language models. *Transactions of the Association for Computational Linguistics*, 11:1316–1331, 2023. [10.1162/tacl\_a\_00605](https://arxiv.org/doi.org/10.1162/tacl_a_00605). URL [https://aclanthology.org/2023.tacl-1.75/](https://aclanthology.org/2023.tacl-1.75/). + +[^127]: T. B. Richards. Auto-GPT: An autonomous GPT-4 experiment. [https://github.com/Significant-Gravitas/Auto-GPT](https://github.com/Significant-Gravitas/Auto-GPT), 2023. GitHub repository, accessed 2026-04-02. + +[^128]: H. Ross, A. S. Mahabaleshwarkar, and Y. Suhara. When2call: When (not) to call tools. In *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pages 3391–3409, 2025. + +[^129]: T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. In *Advances in Neural Information Processing Systems*, volume 36, 2023. + +[^130]: N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. *Advances in Neural Information Processing Systems*, 36, 2023. + +[^131]: I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. *arXiv preprint arXiv:2209.11302*, 2023. + +[^132]: Y. Sui, H. Zhao, R. Ma, Z. He, H. Wang, J. Li, and Y. Yang. Act while thinking: Accelerating llm agents via pattern-aware speculative tool execution. *arXiv preprint arXiv:2603.18897*, 2026. + +[^133]: T. R. Sumers, S. Yao, K. Narasimhan, and T. L. Griffiths. Cognitive architectures for language agents. *Transactions on Machine Learning Research*, 2024. Published in TMLR; available at [https://openreview.net/forum?id=1i6ZCvflQJ](https://openreview.net/forum?id=1i6ZCvflQJ). + +[^134]: P. Tagkopoulos, F. Li, and I. Tagkopoulos. Skillflow: Efficient skill and code transfer through communication in adapting ai agents. *arXiv preprint arXiv:2504.06188*, 2025. + +[^135]: A. Takyar. Unlocking ai interoperability: A deep dive into the model context protocol (mcp), 2025. URL [https://zbrain.ai/model-context-protocol/](https://zbrain.ai/model-context-protocol/). + +[^136]: G. Theraulaz and E. Bonabeau. A brief history of stigmergy. *Artificial life*, 5(2):97–116, 1999. + +[^137]: S. Timmermans and S. Epstein. A world of standards but not a standard world: Toward a sociology of standards and standardization. *Annual review of Sociology*, 36(1):69–89, 2010. + +[^138]: H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: Open and efficient foundation language models. *arXiv preprint arXiv:2302.13971*, 2023. [10.48550/arXiv.2302.13971](https://arxiv.org/doi.org/10.48550/arXiv.2302.13971). URL [https://arxiv.org/abs/2302.13971](https://arxiv.org/abs/2302.13971). + +[^139]: UCP Documentation. Ucp and ap2 integration. [https://ucp.dev/documentation/ucp-and-ap2/](https://ucp.dev/documentation/ucp-and-ap2/), 2026. Explains that AP2 serves as the trust and payment layer for transactions executed within the UCP commerce lifecycle. + +[^140]: G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models. *arXiv preprint arXiv:2305.16291*, 2023a. + +[^141]: J. Wang, Y. Ming, Z. Ke, S. Joty, A. Albarghouthi, and F. Sala. Skillorchestra: Learning to route agents via skill transfer, 2026a. URL [https://arxiv.org/abs/2602.19672](https://arxiv.org/abs/2602.19672). + +[^142]: L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, and J.-R. Wen. A survey on large language model based autonomous agents. *Frontiers of Computer Science*, 18(6):186345, 2024a. + +[^143]: Q. Wang, Y. Hu, M. Lu, J. Wu, Y. Liu, and Y. Tang. Beyond accuracy: A cognitive load framework for mapping the capability boundaries of tool-use agents. *arXiv preprint arXiv:2601.20412*, 2026b. + +[^144]: Q. Wang, B. Ma, M. Xu, and Y. Zhang. When skills lie: Hidden-comment injection in llm agents. *arXiv preprint arXiv:2602.10498*, 2026c. + +[^145]: S. Wang, J. Jin, R. Fu, Z. Yan, X. Wang, M. Hu, E. Wang, X. Li, K. Zhang, L. Yao, W. Jiao, X. Cheng, Y. Lu, and Z. Ge. Museagent: A multimodal reasoning agent with stateful experiences, 2026d. URL [https://arxiv.org/abs/2603.27813](https://arxiv.org/abs/2603.27813). + +[^146]: T. Wang, R. Shan, J. Lin, J. Wu, T. Xu, J. Zhang, W. Chen, C. Zhang, Z. Wang, W. Zhang, et al. Oscar: Optimization-steered agentic planning for composed image retrieval. *arXiv preprint arXiv:2602.08603*, 2026e. + +[^147]: X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou. Self-consistency improves chain of thought reasoning in language models. In *International Conference on Learning Representations*, 2023b. URL [https://openreview.net/forum?id=1PL1NIMMrw](https://openreview.net/forum?id=1PL1NIMMrw). + +[^148]: X. Wang, B. Chen, et al. OpenDevin: An open platform for AI software developers as generalist agents. *arXiv preprint arXiv:2407.16741*, 2024b. URL [https://arxiv.org/abs/2407.16741](https://arxiv.org/abs/2407.16741). + +[^149]: X. Wang, S. Rosenberg, J. Michelini, C. Smith, H. Tran, E. Nyst, R. Malhotra, X. Zhou, V. Chen, R. Brennan, et al. The openhands software agent sdk: A composable and extensible foundation for production agents. *arXiv preprint arXiv:2511.03690*, 2025a. + +[^150]: X. Wang, J. Shi, S. Feng, P. Yuan, Y. Li, Y. Zhang, C. Tan, J. Zhang, B. Pan, Y. Hu, et al. Do not waste your rollouts: Recycling search experience for efficient test-time scaling. *arXiv preprint arXiv:2601.21684*, 2026f. + +[^151]: Y. Wang and X. Chen. Mirix: Multi-agent memory system for llm-based agents. *arXiv preprint arXiv:2507.07957*, 2025. + +[^152]: Y. Wang, R. Takanobu, Z. Liang, Y. Mao, Y. Hu, J. McAuley, and X. Wu. Mem- $\{$ $\backslash$ alpha $\}$: Learning memory construction via reinforcement learning. *arXiv preprint arXiv:2509.25911*, 2025b. + +[^153]: Z. Z. Wang, A. Gandhi, G. Neubig, and D. Fried. Inducing programmatic skills for agentic tasks. *arXiv preprint arXiv:2504.06821*, 2025c. + +[^154]: D. M. Wegner. Transactive memory: A contemporary analysis of the group mind. In *Theories of group behavior*, pages 185–208. Springer, 1987. + +[^155]: J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models. In *Advances in Neural Information Processing Systems*, volume 35, pages 24824–24837, 2022. + +[^156]: J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL [https://arxiv.org/abs/2201.11903](https://arxiv.org/abs/2201.11903). + +[^157]: Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al. AutoGen: Enabling next-gen LLM applications via multi-agent conversation. *arXiv preprint arXiv:2308.08155*, 2023. URL [https://arxiv.org/abs/2308.08155](https://arxiv.org/abs/2308.08155). + +[^158]: Y. Wu and Y. Zhang. Agent skills from the perspective of procedural memory: A survey. *TechRxiv*, 2026. [10.36227/techrxiv.176857932.25697838/v1](https://arxiv.org/doi.org/10.36227/techrxiv.176857932.25697838/v1). + +[^159]: Z. Wu, C. Han, Z. Ding, Z. Weng, Z. Liu, S. Yao, T. Yu, and L. Kong. OS-Copilot: Towards generalist computer agents with self-improvement. *arXiv preprint arXiv:2402.07456*, 2024. URL [https://arxiv.org/abs/2402.07456](https://arxiv.org/abs/2402.07456). + +[^160]: Z. Wu, H. Huang, Y. Yang, Y. Song, X. Lou, W. Liu, W. Zhang, J. Wang, and Z. Zhang. Quick on the uptake: Eliciting implicit intents from human demonstrations for personalized mobile-use agents. *arXiv preprint arXiv:2508.08645*, 2025. + +[^161]: Y. Xi, W. Liu, J. Lin, B. Chen, R. Tang, W. Zhang, and Y. Yu. Memocrs: Memory-enhanced sequential conversational recommender systems with large language models. In *Proceedings of the 33rd ACM International Conference on Information and Knowledge Management*, pages 2585–2595, 2024. + +[^162]: Y. Xi, J. Lin, Y. Xiao, Z. Zhou, R. Shan, T. Gao, J. Zhu, W. Liu, Y. Yu, and W. Zhang. A survey of llm-based deep search agents: Paradigm, optimization, evaluation, and challenges. *arXiv preprint arXiv:2508.05668*, 2025. + +[^163]: Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, et al. The rise and potential of large language model based agents: A survey. *arXiv preprint arXiv:2309.07864*, 2023. URL [https://arxiv.org/abs/2309.07864](https://arxiv.org/abs/2309.07864). + +[^164]: H. Xu, Z. Wang, Z. Zhu, L. Pan, X. Chen, S. Fan, L. Chen, and K. Yu. Alignment for efficient tool calling of large language models. In *Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 17787–17803, 2025a. + +[^165]: R. Xu and Y. Yan. Agent skills for large language models: Architecture, acquisition, security, and the path forward. *arXiv preprint arXiv:2602.12430*, 2026a. + +[^166]: R. Xu and Y. Yan. Agent skills for large language models: Architecture, acquisition, security, and the path forward, 2026b. URL [https://arxiv.org/abs/2602.12430](https://arxiv.org/abs/2602.12430). + +[^167]: T. Xu, R. Shan, J. Wu, J. Huang, T. Wang, J. Zhu, W. Chen, M. Tu, Q. Dou, Z. Wang, et al. Photobench: Beyond visual matching towards personalized intent-driven photo retrieval. *arXiv preprint arXiv:2603.01493*, 2026a. + +[^168]: W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y. Zhang. A-MEM: Agentic memory for LLM agents. *arXiv preprint arXiv:2502.12110*, 2025b. [10.48550/arXiv.2502.12110](https://arxiv.org/doi.org/10.48550/arXiv.2502.12110). NeurIPS 2025. + +[^169]: Y. Xu, Q. Chen, Z. Ma, D. Liu, W. Wang, X. Wang, L. Xiong, and W. Wang. Toward personalized llm-powered agents: Foundations, evaluation, and future directions. *arXiv preprint arXiv:2602.22680*, 2026b. + +[^170]: B. Yan, C. Li, H. Qian, S. Lu, and Z. Liu. General agentic memory via deep research. *arXiv preprint arXiv:2511.18423*, 2025a. + +[^171]: S. Yan, X. Yang, Z. Huang, E. Nie, Z. Ding, Z. Li, X. Ma, J. Bi, K. Kersting, J. Z. Pan, et al. Memory-r1: Enhancing large language model agents to manage and utilize memories via reinforcement learning. *arXiv preprint arXiv:2508.19828*, 2025b. + +[^172]: J. Yang, C. E. Jimenez, A. Wettig, K. Liber, K. Narasimhan, and O. Press. SWE-agent: Agent–computer interfaces enable automated software engineering. *arXiv preprint arXiv:2405.15793*, 2024a. URL [https://arxiv.org/abs/2405.15793](https://arxiv.org/abs/2405.15793). + +[^173]: J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. Swe-agent: Agent-computer interfaces enable automated software engineering. *Advances in Neural Information Processing Systems*, 37:50528–50652, 2024b. + +[^174]: Y. Yang, H. Chai, Y. Song, S. Qi, M. Wen, N. Li, J. Liao, H. Hu, J. Lin, G. Chang, W. Liu, Y. Wen, Y. Yu, and W. Zhang. A survey of ai agent protocols, 2025a. URL [https://arxiv.org/abs/2504.16736](https://arxiv.org/abs/2504.16736). + +[^175]: Y. Yang, H. Chai, Y. Song, S. Qi, M. Wen, N. Li, J. Liao, H. Hu, J. Lin, G. Chang, et al. A survey of ai agent protocols. *arXiv preprint arXiv:2504.16736*, 2025b. + +[^176]: S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models, 2023a. URL [https://arxiv.org/abs/2210.03629](https://arxiv.org/abs/2210.03629). + +[^177]: S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. *Advances in Neural Information Processing Systems*, 36, 2024. + +[^178]: Y. Yao, P. Wang, B. Tian, S. Cheng, Z. Li, S. Deng, H. Chen, and N. Zhang. Editing large language models: Problems, methods, and opportunities. *arXiv preprint arXiv:2305.13172*, 2023b. URL [https://arxiv.org/abs/2305.13172](https://arxiv.org/abs/2305.13172). + +[^179]: A. Ye, Q. Ma, J. Chen, M. Li, T. Li, F. Liu, S. Mai, M. Lu, H. Bao, and Y. You. Sop-agent: Empower general purpose ai agent with domain-specific sops, 2025. URL [https://arxiv.org/abs/2501.09316](https://arxiv.org/abs/2501.09316). + +[^180]: Y. Ye, H. Jiang, F. Jiang, T. Lan, Y. Du, B. Fu, X. Shi, Q. Jia, L. Wang, and W. Luo. Umem: Unified memory extraction and management framework for generalizable memory. *arXiv preprint arXiv:2602.10652*, 2026. + +[^181]: C. Yu, Y. Wang, S. Wang, H. Yang, and M. Li. Infiagent: An infinite-horizon framework for general-purpose autonomous agents, 2026. URL [https://arxiv.org/abs/2601.03204](https://arxiv.org/abs/2601.03204). + +[^182]: S. Yu, G. Li, W. Shi, and P. Qi. Polyskill: Learning generalizable skills through polymorphic abstraction. *arXiv preprint arXiv:2510.15863*, 2025. + +[^183]: S. Yuan, J. Wang, Z. Liu, M. Yuan, H. Peng, J. Zhao, B. Wang, and H. Wang. Ted: Training-free experience distillation for multimodal reasoning, 2026. URL [https://arxiv.org/abs/2603.26778](https://arxiv.org/abs/2603.26778). + +[^184]: G. Zhang, H. Ren, C. Zhan, Z. Zhou, J. Wang, H. Zhu, W. Zhou, and S. Yan. Memevolve: Meta-evolution of agent memory systems. *arXiv preprint arXiv:2512.18746*, 2025a. + +[^185]: H. Zhang, Q. Long, J. Bao, T. Feng, W. Zhang, H. Yue, and W. Wang. Memskill: Learning and evolving memory skills for self-evolving agents. *arXiv preprint arXiv:2602.02474*, 2026a. + +[^186]: H. Zhang, H. Yue, T. Feng, Q. Long, J. Bao, B. Jin, W. Zhang, X. Li, J. You, C. Qin, et al. Learning query-aware budget-tier routing for runtime agent memory. *arXiv preprint arXiv:2602.06025*, 2026b. + +[^187]: K. Zhang, X. Chen, B. Liu, T. Xue, Z. Liao, Z. Liu, X. Wang, Y. Ning, Z. Chen, X. Fu, et al. Agent learning via early experience. *arXiv preprint arXiv:2510.08558*, 2025b. + +[^188]: S. Zhang, J. Wang, R. Zhou, J. Liao, Y. Feng, Z. Li, Y. Zheng, W. Zhang, Y. Wen, Z. Li, et al. Memrl: Self-evolving agents via runtime reinforcement learning on episodic memory. *arXiv preprint arXiv:2601.03192*, 2026c. + +[^189]: W. Zhang, J. Liao, N. Li, K. Du, and J. Lin. Agentic information retrieval. *arXiv preprint arXiv:2410.09713*, 2024. + +[^190]: Z. Zhang, Q. Dai, R. Li, X. Bo, X. Chen, and Z. Dong. Learn to memorize: Optimizing llm-based agents with adaptive memory framework. *arXiv preprint arXiv:2508.16629*, 2025c. + +[^191]: Z. Zhang, Z. Wei, and M. Sun. Dynamic orthogonal continual fine-tuning for mitigating catastrophic forgettings. *arXiv preprint arXiv:2509.23893*, 2025d. + +[^192]: H. Zhao, H. Chen, F. Yang, N. Liu, H. Deng, H. Cai, S. Wang, D. Yin, and M. Du. Explainability for large language models: A survey. *ACM Transactions on Intelligent Systems and Technology*, 15(2):1–38, 2024. + +[^193]: S. Zhao, F. Liu, X. Zhang, H. Chen, X. Gu, Z. Jiang, F. Ling, B. Fei, W. Zhang, J. Wang, et al. Openearth-agent: From tool calling to tool creation for open-environment earth observation. *arXiv preprint arXiv:2603.22148*, 2026a. + +[^194]: Y. Zhao, C. Dai, Y. Xiu, M. Kou, Y. Zheng, and D. Niyato. Shardmemo: Masked moe routing for sharded agentic llm memory. *arXiv preprint arXiv:2601.21545*, 2026b. + +[^195]: B. Zheng, M. Y. Fatemi, X. Jin, Z. Z. Wang, A. Gandhi, Y. Song, Y. Gu, J. Srinivasa, G. Liu, G. Neubig, et al. Skillweaver: Web agents can self-improve by discovering and honing skills. *arXiv preprint arXiv:2504.07079*, 2025a. + +[^196]: C. Zheng, J. Zhu, Z. Ou, Y. Chen, K. Zhang, R. Shan, Z. Zheng, M. Yang, J. Lin, Y. Yu, et al. A survey of process reward models: From outcome signals to process supervisions for large language models. *arXiv preprint arXiv:2510.08049*, 2025b. + +[^197]: L. Zheng, R. Wang, X. Wang, and B. An. Synapse: Trajectory-as-exemplar prompting with memory for computer control. *arXiv preprint arXiv:2306.07863*, 2023. + +[^198]: W. Zhong, L. Guo, Q. Gao, H. Ye, and Y. Wang. Memorybank: Enhancing large language models with long-term memory. In *Proceedings of the AAAI conference on artificial intelligence*, volume 38, pages 19724–19731, 2024. + +[^199]: H. Zhou, Y. Chen, S. Guo, X. Yan, K. H. Lee, Z. Wang, K. Y. Lee, G. Zhang, K. Shao, L. Yang, et al. Memento: Fine-tuning llm agents without fine-tuning llms. *arXiv preprint arXiv:2508.16153*, 2025. + +[^200]: Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba. Large language models are human-level prompt engineers. In *International Conference on Learning Representations*, 2023. URL [https://openreview.net/forum?id=92gvk82DE-](https://openreview.net/forum?id=92gvk82DE-). + +[^201]: J. Zhu, M. Zhu, R. Rui, R. Shan, C. Zheng, B. Chen, Y. Xi, J. Lin, W. Liu, R. Tang, et al. Evolutionary perspectives on the evaluation of llm-based ai agents: A comprehensive survey. *arXiv preprint arXiv:2506.11102*, 2025. + +[^202]: J. Zhu, L. Yang, R. Shan, C. Zheng, Z. Zheng, W. Liu, Y. Yu, W. Zhang, and J. Lin. Turing test on screen: Abenchmark for mobile gui agent humanization. 2026. + +[^203]: L. Zhu and Q. Lu. Verifiability-first ai engineering in the era of aiware: A conceptual framework, design principles, and architectural patterns for scalable verification. *Design Principles, and Architectural Patterns for Scalable Verification (January 07, 2026)*, 2026. + +[^204]: J. Zou, L. Yang, Y. Qi, S. Chen, M. Ai, K. Shen, J. He, and M. Wang. Autotool: Dynamic tool selection and integration for agentic reasoning, 2025. URL [https://arxiv.org/abs/2512.13278](https://arxiv.org/abs/2512.13278). \ No newline at end of file diff --git a/raw/Harness Engineering - first thoughts.md b/raw/Harness Engineering - first thoughts.md new file mode 100644 index 0000000..08f5e70 --- /dev/null +++ b/raw/Harness Engineering - first thoughts.md @@ -0,0 +1,69 @@ +--- +title: "Harness Engineering - first thoughts" +source: "https://martinfowler.com/articles/exploring-gen-ai/harness-engineering-memo.html" +author: + - "[[Birgitta Böckeler Birgitta is a Distinguished Engineer and AI-assisted delivery expert at Thoughtworks. She has over 20 years of experience as a software developer]]" + - "[[architect and technical leader.]]" +published: +created: 2026-04-11 +description: "Notes from my Thoughtworks colleagues on AI-assisted software delivery" +tags: + - "clippings" +--- +Since writing this memo, I have had time to analyze the topic further and write a [more considered article describing Harness Engineering](https://martinfowler.com/articles/harness-engineering.html). This frames the elements of a harness as guides and sensors, which may be computational or inferential. Harness templates allow us to share common guides and sensors around a larger software organization. Harnesses attempt to externalise and make explicit what human developer experience brings to the table, but they can only go so far. A good harness should not necessarily aim to fully eliminate human input, but to direct it to where our input is most important. + +It was very interesting to read [OpenAI’s recent write-up on “Harness engineering”](https://openai.com/index/harness-engineering/) which describes how a team used “no manually typed code at all” as a forcing function to build a harness for maintaining a large application with AI agents. After 5 months, they’ve built a real product that’s now over 1 million lines of code. + +The article is titled “Harness engineering: leveraging Codex in an agent-first world”, but only mentions “harness” once in the text. Maybe the term was an afterthought inspired by [Mitchell Hashimoto](https://mitchellh.com/writing/my-ai-adoption-journey#step-5-engineer-the-harness) ’s recent blog post. Either way, I like “harness” as a word to describe the tooling and practices we can use to keep AI agents in check. + +The OpenAI team’s harness components mix deterministic and LLM-based approaches across 3 categories (grouping based on my interpretation): + +1. **Context engineering**: Continuously enhanced knowledge base in the codebase, plus agent access to dynamic context like observability data and browser navigation +2. **Architectural constraints**: Monitored not only by the LLM-based agents, but also deterministic custom linters and structural tests +3. **“Garbage collection”**: Agents that run periodically to find inconsistencies in documentation or violations of architectural constraints, fighting entropy and decay + +They also highlight how iterative this is: “When the agent struggles, we treat it as a signal: identify what is missing — tools, guardrails, documentation — and feed it back into the repository, always by having Codex itself write the fix.” + +All of the described measures focus on increasing long-term internal quality and maintainability. What I am missing in the write-up is verification of functionality and behaviour. + +Leaving that gap aside, and assuming we can trust OpenAI’s representation of the success of this (with respect to the author and the team, OpenAI do have a vested interest in us believing in AI-maintainable code) — here are my thoughts on what *is* in the article. + +### Harnesses - the future service templates? + +Most organizations have just two or three main tech stacks — not every application is its own snowflake. The article made me imagine a future where teams pick from a set of harnesses for common application topologies to get started. This evokes today’s service templates, which help teams instantiate new services on a “golden path”. Will harnesses — with custom linters, structural tests, basic context and knowledge documentation, and additional context providers — become the new service templates? Will teams use them as a starting point, then shape them over time for their application’s specifics? + +With service templates, teams contribute back as they gain experience, then other teams often struggle to incorporate updates. Would we see a similar forking and synchronization challenge with harnesses? + +The article also made me revisit some of my older hypotheses: + +### The runtime has to be constrained for more AI autonomy? + +A lot of early and current AI coding hype assumes LLMs will give us unlimited flexibility of the target runtime. Generate in any language, any pattern, without constraints — the LLM will figure it out. But for maintainable, AI-generated code at scale that we can trust, something has to give. + +The harness described suggests that increasing trust and reliability required constraining the solution space: specific architectural patterns, enforced boundaries, standardized structures. That means giving up some “generate anything” flexibility for prompts, rules, and harnesses full of technical specifics. + +### A convergence on a limited number of tech stacks and topologies? + +As coding becomes less about typing code and more about steering its generation, AI might push us toward fewer tech stacks. Usability of frameworks and SDKs still matters — we’re seeing repeatedly that what’s good for humans is good for AI. But developer tastes will matter less at that level of detail. Little inefficiencies and idiosyncracies in interfaces will be less annoying since we don’t deal with them directly. We might choose stacks with good harnesses available and prioritize “AI-friendliness”. + +This might apply not just to tech stacks, but also to codebase structures and topologies. We might default to structures that are easier to maintain with AI because they’re easier to harness. The OpenAI team discusses architectural rigidity and enforcement rules. The main focus areas I can see are keeping data structures stable and defining and enforcing module boundaries. Sounds reasonable — but without concrete examples, I’m still struggling to imagine what “we require Codex to parse data shapes at the boundary” looks like in practice in their harness. + +But if we can figure out widely how to harness codebase design patterns, will these topologies become the new abstraction layer, not natural language itself like so many AI enthusiasts hope? + +### Two future worlds: Pre-AI vs post-AI application maintenance? + +Say we develop good harnessing techniques to turn AI autonomy up to 9 and increase our confidence in the results. Which techniques could we apply to existing applications, and which would only work for applications built from scratch with a harness in mind? + +For older codebases, we’d need to consider whether retrofitting a harness is worth the effort. AI can help us do that faster, but those applications are often so non-standardized and full of entropy that it might not be worthwhile. It makes me think of running a static code analysis tool on a codebase that’s never had one, and then drowning in alerts. + +### What’s your harness today? + +That this team worked on their harness for 5 months shows this isn’t something you can jump into for quick results. But it’s worth reflecting on what your harness is today. Do you have a pre-commit hook? What’s in it? Do you have ideas for custom linters? What architectural constraints would you like to impose on your codebase? Have you experimented with structural testing frameworks like ArchUnit? + +### Final thoughts + +Unsurprisingly, what they describe sounds like much more work than just generating and maintaining a bunch of Markdown rules files. They built extensive tooling for the deterministic part of the harness. Their context engineering involved not only curating a knowledge base, but also significant design work — the code design itself is a huge part of the context. + +The OpenAI team says: “Our most difficult challenges now center on designing environments, feedback loops, and control systems.” This reminded me of [Chad Fowler’s recent post on “Relocating Rigor”](https://aicoding.leaflet.pub/3mbrvhyye4k2e). It’s refreshing to hear concrete ideas and experiences about where that rigor might go, rather than just hoping “better models” will magically solve maintainability issues. + +And finally, for once, I like a term in this space. Though it’s only 2 weeks old — I can probably hold my metaphorical breath until somebody calls their one-prompt, LLM-based code review agent a harness… \ No newline at end of file diff --git a/raw/Harness Engineering The Complete Guide to Building Systems That Make AI Agents Actually Work (2026).md b/raw/Harness Engineering The Complete Guide to Building Systems That Make AI Agents Actually Work (2026).md new file mode 100644 index 0000000..c8009e3 --- /dev/null +++ b/raw/Harness Engineering The Complete Guide to Building Systems That Make AI Agents Actually Work (2026).md @@ -0,0 +1,330 @@ +--- +title: "Harness Engineering: The Complete Guide to Building Systems That Make AI Agents Actually Work (2026)" +source: "https://www.nxcode.io/resources/news/harness-engineering-complete-guide-ai-agent-codex-2026" +author: + - "[[NxCode Team]]" +published: 2026-03-01 +created: 2026-04-11 +description: "Harness engineering is the new discipline of designing environments, constraints, and feedback loops that make AI coding agents reliable at scale. OpenAI built 1M+ lines of code with zero human-written code using this approach." +tags: + - "clippings" +--- +Turn your idea into a working app — no coding required.[Start Free](https://studio.nxcode.io/?ref=article_top_harness-engineering-complete-guide-ai-agent-codex-2026&article=harness-engineering-complete-guide-ai-agent-codex-2026) + +## Harness Engineering: The Complete Guide to Building Systems That Make AI Agents Actually Work + +**March 2026** — If 2025 was the year AI agents proved they could write code, 2026 is the year we learned that **the agent isn't the hard part — the harness is.** + +OpenAI's Codex team just built a production application with **over 1 million lines of code** where **zero lines were written by human hands**. The engineers didn't write code. They designed the system that let AI write code reliably. That system — the constraints, feedback loops, documentation, linters, and lifecycle management — is what the industry now calls a **harness**. + +**Harness engineering** is the new discipline of designing these systems. And it's changing what it means to be a software engineer. + +--- + +## What Is Harness Engineering? + +### The Horse Metaphor + +The term "harness" comes from horse tack — reins, saddle, bit — the complete set of equipment for channeling a powerful but unpredictable animal in the right direction. The metaphor is deliberate: + +- The **horse** is the AI model — powerful, fast, but it doesn't know where to go on its own +- The **harness** is the infrastructure — constraints, guardrails, feedback loops that channel the model's power productively +- The **rider** is the human engineer — providing direction, not doing the running + +Without a harness, an AI agent is a thoroughbred in an open field. Fast, impressive, and completely useless for getting anything done. + +### The Formal Definition + +**Harness engineering** is the design and implementation of systems that: + +1. **Constrain** what an AI agent can do (architectural boundaries, dependency rules) +2. **Inform** the agent about what it should do (context engineering, documentation) +3. **Verify** that the agent did it correctly (testing, linting, CI validation) +4. **Correct** the agent when it goes wrong (feedback loops, self-repair mechanisms) + +Martin Fowler describes it as *"the tooling and practices we can use to keep AI agents in check"* — but it's more than just safety. A good harness makes agents **more capable**, not just more controlled. + +--- + +## Why Harness Engineering Matters Now + +### The Model Is Commodity. The Harness Is Moat. + +Here's the uncomfortable truth the AI industry is confronting: **the underlying model matters less than the system around it.** + +LangChain proved this definitively. Their coding agent went from **52.8% to 66.5%** on Terminal Bench 2.0 — jumping from **Top 30 to Top 5** — by changing nothing about the model. They only changed the harness: + +| Change | What They Did | Impact | +| --- | --- | --- | +| Self-verification loop | Added pre-completion checklist middleware | Caught errors before submission | +| Context engineering | Mapped directory structures at startup | Agent understood codebase from the start | +| Loop detection | Tracked repeated file edits | Prevented "doom loops" | +| Reasoning sandwich | High reasoning for planning/verification, medium for implementation | Better quality within time budgets | + +**Same model. Different harness. Dramatically better results.** + +### OpenAI's 1 Million Line Proof Point + +OpenAI's experiment is the most compelling evidence yet: + +- **5 months** of development +- **1 million+ lines of code** in the final product +- **Zero manually written lines** — every line was produced by Codex agents +- **Built in ~1/10th the time** it would have taken humans +- The product has **internal daily users and external alpha testers** +- It **ships, deploys, breaks, and gets fixed** — all by agents within the harness + +The engineers' job? Designing the harness. Specifying intent. Providing feedback. Not writing code. + +--- + +## The Three Pillars of Harness Engineering + +OpenAI's framework organizes harness engineering into three core categories: + +### 1\. Context Engineering + +Context engineering is about ensuring the agent has the right information at the right time. + +**Static context:** + +- Repository-local documentation (architecture specs, API contracts, style guides) +- `AGENTS.md` or `CLAUDE.md` files that encode project-specific rules +- Cross-linked design documents validated by linters + +**Dynamic context:** + +- Observability data (logs, metrics, traces) accessible to agents +- Directory structure mapping at agent startup +- CI/CD pipeline status and test results + +**The critical rule:** From the agent's perspective, anything it can't access in-context doesn't exist. Knowledge in Google Docs, Slack threads, or people's heads is invisible to the system. **The repository must be the single source of truth.** + +### 2\. Architectural Constraints + +This is where harness engineering diverges most sharply from traditional AI prompting. Instead of telling the agent "write good code," you **mechanically enforce what good code looks like.** + +**Dependency layering:** + +``` +Types → Config → Repo → Service → Runtime → UI +``` + +Each layer can only import from layers to its left. This isn't a suggestion — it's enforced by structural tests and CI validation. + +**Constraint enforcement tools:** + +- **Deterministic linters** — Custom rules that flag violations automatically +- **LLM-based auditors** — Agents that review other agents' code for architectural compliance +- **Structural tests** — Like ArchUnit, but for AI-generated code +- **Pre-commit hooks** — Automated checks before any code is committed + +**Why constraints improve output:** Paradoxically, constraining the solution space makes agents **more productive**, not less. When an agent can generate anything, it wastes tokens exploring dead ends. When the harness defines clear boundaries, the agent converges faster on correct solutions. + +### 3\. Entropy Management ("Garbage Collection") + +This is the most underappreciated component. Over time, AI-generated codebases accumulate entropy — documentation drifts from reality, naming conventions diverge, dead code accumulates. + +Harness engineering addresses this with **periodic cleanup agents:** + +- **Documentation consistency agents** — Verify that docs match current code +- **Constraint violation scanners** — Find code that slipped past earlier checks +- **Pattern enforcement agents** — Identify and fix deviations from established patterns +- **Dependency auditors** — Track and resolve circular or unnecessary dependencies + +These agents run on schedules — daily, weekly, or triggered by specific events — keeping the codebase healthy for both human reviewers and future AI agents. + +--- + +## Harness Engineering in Practice: How Teams Actually Do It + +### The OpenAI Approach: Zero Human Code + +OpenAI's team structure for harness engineering: + +| Role | Traditional | Harness Engineering | +| --- | --- | --- | +| Writing code | Primary job | Never | +| Designing architecture | Part of the job | Primary job | +| Writing documentation | Afterthought | Critical infrastructure | +| Reviewing PRs | Code review | Reviewing agent output + harness effectiveness | +| Debugging | Reading code | Analyzing agent behavior patterns | +| Testing | Writing tests | Designing test strategies agents execute | + +### The Stripe Approach: Minions at Scale + +Stripe's internal coding agents, called **Minions**, now produce **over 1,000 merged pull requests per week**: + +1. Developer posts a task in Slack +2. Minion writes the code +3. Minion passes CI +4. Minion opens a PR +5. Human reviews and merges + +No developer interaction between step 1 and step 5. The harness handles everything — test execution, CI validation, style compliance, and documentation updates. + +### The LangChain Approach: Middleware-First + +LangChain structures their harness as composable middleware layers: + +``` +Agent Request + → LocalContextMiddleware (maps codebase) + → LoopDetectionMiddleware (prevents repetition) + → ReasoningSandwichMiddleware (optimizes compute) + → PreCompletionChecklistMiddleware (enforces verification) + → Agent Response +``` + +Each middleware layer adds a specific capability without modifying the core agent logic. This modular approach makes the harness testable and evolvable. + +--- + +## Building Your First Harness: A Practical Framework + +### Level 1: Basic Harness (Single Developer) + +If you're using Claude Code, Cursor, or Codex for individual projects: + +**What to set up:** + +- `CLAUDE.md` or `.cursorrules` file with project conventions +- Pre-commit hooks for linting and formatting +- A test suite the agent can run to self-verify +- Clear directory structure with consistent naming + +**Time to set up:** 1-2 hours **Impact:** Prevents the most common agent mistakes + +### Level 2: Team Harness (Small Team) + +For teams of 3-10 developers sharing a codebase: + +**Add to Level 1:** + +- `AGENTS.md` with team-wide conventions +- Architectural constraints enforced by CI +- Shared prompt templates for common tasks +- Documentation-as-code validated by linters +- Code review checklists specifically for agent-generated PRs + +**Time to set up:** 1-2 days **Impact:** Consistent agent behavior across the team + +### Level 3: Production Harness (Engineering Organization) + +For organizations running dozens of concurrent agents: + +**Add to Level 2:** + +- Custom middleware layers (loop detection, reasoning optimization) +- Observability integration (agents read logs and metrics) +- Entropy management agents on scheduled runs +- Harness versioning and A/B testing +- Agent performance monitoring dashboards +- Escalation policies for when agents get stuck + +**Time to set up:** 1-2 weeks **Impact:** Agents operate as autonomous contributors + +--- + +## Common Harness Engineering Mistakes + +### 1\. Over-Engineering the Control Flow + +> *"If you over-engineer the control flow, the next model update will break your system."* + +Models improve rapidly. Capabilities that required complex pipelines in 2024 are now handled by a single context-window prompt. Build your harness to be **rippable** — you should be able to remove "smart" logic when the model gets smart enough to not need it. + +### 2\. Treating the Harness as Static + +The harness needs to evolve with the model. When a new model release improves reasoning, your reasoning-optimization middleware might become counterproductive. Review and update harness components with every major model update. + +### 3\. Ignoring the Documentation Layer + +The most impactful harness improvement is often the simplest: **better documentation**. If your `AGENTS.md` is vague, your agent output will be vague. Invest in precise, machine-readable documentation that serves as the agent's ground truth. + +### 4\. No Feedback Loop + +A harness without feedback is a cage, not a guide. The agent needs to know when it's succeeding and when it's failing. Build in: + +- Self-verification steps before task completion +- Test execution as part of the agent workflow +- Metrics on agent success rates by task type + +### 5\. Human-Only Documentation + +If your architectural decisions live in people's heads or in Confluence pages the agent can't access, the harness has a gap. **Everything the agent needs must be in the repository.** + +--- + +## Harness Engineering vs. Related Concepts + +| Concept | Scope | Focus | +| --- | --- | --- | +| **Prompt Engineering** | Single interaction | Crafting effective prompts | +| **Context Engineering** | Model context window | What information the model sees | +| **Harness Engineering** | Entire agent system | Environment, constraints, feedback, lifecycle | +| **Agent Engineering** | Agent architecture | Internal agent design and routing | +| **Platform Engineering** | Infrastructure | Deployment, scaling, operations | + +Harness engineering **includes** context engineering and draws from prompt engineering, but it operates at a higher level — it's about the complete system that makes agents reliable, not just the inputs to a single interaction. + +--- + +## What This Means for Software Engineers + +### The Job Is Changing + +Harness engineering represents a genuine evolution in what software engineers do: + +| Before | After | +| --- | --- | +| Write code | Design environments where AI writes code | +| Debug code | Debug agent behavior | +| Review code | Review agent output + harness effectiveness | +| Write tests | Design test strategies | +| Maintain docs | Build documentation as machine-readable infrastructure | + +This doesn't mean engineers become less technical. If anything, harness engineering requires **deeper** architectural thinking — you're designing systems that must work without your constant intervention. + +### The Skills That Matter + +Based on what we've seen building AI-powered products at [NxCode](https://www.nxcode.io/): + +1. **Systems thinking** — Understanding how constraints, feedback loops, and documentation interact +2. **Architecture design** — Defining boundaries that are enforceable and productive +3. **Specification writing** — Articulating intent precisely enough for agents to execute +4. **Observability** — Building monitoring that reveals agent behavior patterns +5. **Iteration speed** — Rapidly testing and refining harness configurations + +### Our Experience: What Works in Practice + +We've been building AI-powered web applications using multiple agent systems (Claude Code, Codex, Cursor). The patterns that have made the biggest difference for us: + +- **Repository-first documentation**: Every architectural decision, naming convention, and deployment process is in the repo. Nothing lives in Slack or Google Docs. +- **Incremental constraint building**: Start with basic linting, add architectural constraints as patterns emerge, don't try to design the perfect harness upfront. +- **Agent-specific review checklists**: AI-generated code has different failure modes than human code. Our review process accounts for common agent patterns (over-abstraction, unnecessary error handling, documentation drift). +- **Multi-provider harness design**: Our harness works with Claude, GPT, and Gemini models. Provider-agnostic design means we can switch models without rebuilding the entire system. + +--- + +## Key Takeaways + +1. **Harness engineering is the new discipline** of designing systems that make AI agents reliable — constraints, feedback loops, documentation, and lifecycle management +2. **The model is commodity; the harness is moat** — LangChain jumped from Top 30 to Top 5 on benchmarks by only changing the harness +3. **OpenAI built 1M+ lines with zero human code** — proving harness engineering works at production scale +4. **Three pillars**: Context engineering, architectural constraints, and entropy management +5. **Start simple**: A good `AGENTS.md` and pre-commit hooks are more impactful than complex middleware +6. **The engineer's job is evolving** — from writing code to designing environments where AI writes code +7. **Build rippable harnesses** — over-engineering breaks when models improve; keep it adaptable + +--- + +## Related Articles + +- [Best AI for Coding in 2026: 10 Tools Ranked by Real-World Performance](https://www.nxcode.io/resources/news/best-ai-for-coding-2026-complete-ranking) +- [OpenAI Frontier Guide: Enterprise AI Agent Platform for Building AI Coworkers (2026)](https://www.nxcode.io/resources/news/openai-frontier-enterprise-ai-agent-platform-guide-2026) +- [Cursor Tutorial 2026: Learn AI Coding in 15 Minutes (Beginner Guide)](https://www.nxcode.io/resources/news/cursor-tutorial-beginners-2026) + +[Back to all news](https://www.nxcode.io/resources/news) + +Enjoyed this article? \ No newline at end of file diff --git a/raw/anthropic-harness-design.md b/raw/Harness design for long-running application development.md similarity index 89% rename from raw/anthropic-harness-design.md rename to raw/Harness design for long-running application development.md index 163421d..b64140d 100644 --- a/raw/anthropic-harness-design.md +++ b/raw/Harness design for long-running application development.md @@ -1,5 +1,13 @@ -# Harness design for long-running application development - +--- +title: "Harness design for long-running application development" +source: "https://www.anthropic.com/engineering/harness-design-long-running-apps" +author: +published: +created: 2026-04-11 +description: "Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems." +tags: + - "clippings" +--- *Written by Prithvi Rajasekaran, a member of our [Labs](https://www.anthropic.com/news/introducing-anthropic-labs) team.* Over the past several months I’ve been working on two interconnected problems: getting Claude to produce high-quality frontend designs, and getting it to build complete applications without human intervention. This work originated with earlier efforts on our [frontend design skill](https://github.com/anthropics/claude-code/blob/main/plugins/frontend-design/skills/frontend-design/SKILL.md) and [long-running coding agent harness](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents), where my colleagues and I were able to improve Claude’s performance well above baseline through prompt engineering and harness design—but both eventually hit ceilings. @@ -10,7 +18,7 @@ I then applied these techniques to long-running autonomous coding, carrying over ## Why naive implementations fall short -We've previously shown that harness design has a substantial impact on the effectiveness of long running agentic coding. In an earlier [experiment](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents), we used an initializer agent to decompose a product spec into a task list, and a coding agent that implemented the tasks one feature at a time before handing off artifacts to carry context across sessions. The broader developer community has converged on similar insights, with approaches like the "[Ralph Wiggum](https://ghuntley.com/ralph/)" method using hooks or scripts to keep agents in continuous iteration cycles. +We've previously shown that harness design has a substantial impact on the effectiveness of long running agentic coding. In an earlier [experiment](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents), we used an initializer agent to decompose a product spec into a task list, and a coding agent that implemented the tasks one feature at a time before handing off artifacts to carry context across sessions. The broader developer community has converged on similar insights, with approaches like the " [Ralph Wiggum](https://ghuntley.com/ralph/) " method using hooks or scripts to keep agents in continuous iteration cycles. But some problems remained persistent. For more complex tasks, the agent still tends to go off the rails over time. While decomposing this issue, we observed two common failure modes with agents executing these sorts of tasks. @@ -30,10 +38,10 @@ Two insights shaped the harness I built for frontend design. First, while aesthe With this in mind, I wrote four grading criteria that I gave to both the generator and evaluator agents in their prompts: -- **Design quality:** Does the design feel like a coherent whole rather than a collection of parts? Strong work here means the colors, typography, layout, imagery, and other details combine to create a distinct mood and identity. -- **Originality:** Is there evidence of custom decisions, or is this template layouts, library defaults, and AI-generated patterns? A human designer should recognize deliberate creative choices. Unmodified stock components—or telltale signs of AI generation like purple gradients over white cards—fail here. -- **Craft:** Technical execution: typography hierarchy, spacing consistency, color harmony, contrast ratios. This is a competence check rather than a creativity check. Most reasonable implementations do fine here by default; failing means broken fundamentals. -- **Functionality:** Usability independent of aesthetics. Can users understand what the interface does, find primary actions, and complete tasks without guessing? +- **Design quality:** Does the design feel like a coherent whole rather than a collection of parts? Strong work here means the colors, typography, layout, imagery, and other details combine to create a distinct mood and identity. +- **Originality:** Is there evidence of custom decisions, or is this template layouts, library defaults, and AI-generated patterns? A human designer should recognize deliberate creative choices. Unmodified stock components—or telltale signs of AI generation like purple gradients over white cards—fail here. +- **Craft:** Technical execution: typography hierarchy, spacing consistency, color harmony, contrast ratios. This is a competence check rather than a creativity check. Most reasonable implementations do fine here by default; failing means broken fundamentals. +- **Functionality:** Usability independent of aesthetics. Can users understand what the interface does, find primary actions, and complete tasks without guessing? I emphasized design quality and originality over craft and functionality. Claude already scored well on craft and functionality by default, as the required technical competence tended to come naturally to the model. But on design and originality, Claude often produced outputs that were bland at best. The criteria explicitly penalized highly generic “AI slop” patterns, and by weighting design and originality more heavily it pushed the model toward more aesthetic risk-taking. @@ -49,13 +57,15 @@ While scores generally improved over iterations, the pattern was not always clea In one notable example, I prompted the model to create a website for a Dutch art museum. By the ninth iteration, it had produced a clean, dark-themed landing page for a fictional museum. The page was visually polished but largely in line with my expectations. Then, on the tenth cycle, it scrapped the approach entirely and reimagined the site as a spatial experience: a 3D room with a checkered floor rendered in CSS perspective, artwork hung on the walls in free-form positions, and doorway-based navigation between gallery rooms instead of scroll or click. It was the kind of creative leap that I hadn't seen before from a single-pass generation. + + ## Scaling to full-stack coding With these findings in hand, I applied this GAN-inspired pattern to full-stack development. The generator-evaluator loop maps naturally onto the software development lifecycle, where code review and QA serve the same structural role as the design evaluator. ### The architecture -In our earlier [long-running harness](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents), we had solved for coherent multi-session coding with an initializer agent, a coding agent that worked one feature at a time, and context resets between sessions. Context resets were a key unlock: the harness used Sonnet 4.5, which exhibited the “context anxiety” tendency mentioned earlier. Creating a harness that worked well across context resets was key to keeping the model on task. Opus 4.5 largely removed that behavior on its own, so I was able to drop context resets from this harness entirely. The agents were run as one continuous session across the whole build, with the [Claude Agent SDK](https://platform.claude.com/docs/en/agent-sdk/overview)'s automatic compaction handling context growth along the way. +In our earlier [long-running harness](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents), we had solved for coherent multi-session coding with an initializer agent, a coding agent that worked one feature at a time, and context resets between sessions. Context resets were a key unlock: the harness used Sonnet 4.5, which exhibited the “context anxiety” tendency mentioned earlier. Creating a harness that worked well across context resets was key to keeping the model on task. Opus 4.5 largely removed that behavior on its own, so I was able to drop context resets from this harness entirely. The agents were run as one continuous session across the whole build, with the [Claude Agent SDK](https://platform.claude.com/docs/en/agent-sdk/overview) 's automatic compaction handling context growth along the way. For this work I built on the foundation from the original harness with a three-agent system, with each agent addressing a specific gap I'd observed in prior runs. The system contained the following agent personas: @@ -63,8 +73,8 @@ For this work I built on the foundation from the original harness with a three-a **Generator:** The one-feature-at-a-time approach from the earlier harness worked well for scope management. I applied a similar model here, instructing the generator to work in sprints, picking up one feature at a time from the spec. Each sprint implemented the app with a React, Vite, FastAPI, and SQLite (later PostgreSQL) stack, and the generator was instructed to self-evaluate its work at the end of each sprint before handing off to QA. It also had git for version control. -**Evaluator:** Applications from earlier harnesses often looked impressive but still had real bugs when you actually tried to use them. To catch these, the evaluator used the Playwright MCP to click through the running application the way a user would, testing UI features, API endpoints, and database states. It then graded each sprint against both the bugs it had found and a set of criteria modeled on the frontend experiment, adapted here to cover product depth, functionality, visual design, and code quality. Each criterion had a hard threshold, and if any one fell below it, the sprint failed and the generator got detailed feedback on what went wrong. - +**Evaluator:** Applications from earlier harnesses often looked impressive but still had real bugs when you actually tried to use them. To catch these, the evaluator used the Playwright MCP to click through the running application the way a user would, testing UI features, API endpoints, and database states. It then graded each sprint against both the bugs it had found and a set of criteria modeled on the frontend experiment, adapted here to cover product depth, functionality, visual design, and code quality. Each criterion had a hard threshold, and if any one fell below it, the sprint failed and the generator got detailed feedback on what went wrong. + Before each sprint, the generator and evaluator negotiated a sprint contract: agreeing on what "done" looked like for that chunk of work before any code was written. This existed because the product spec was intentionally high-level, and I wanted a step to bridge the gap between user stories and testable implementation. The generator proposed what it would build and how success would be verified, and the evaluator reviewed that proposal to make sure the generator was building the right thing. The two iterated until they agreed. Communication was handled via files: one agent would write a file, another agent would read it and respond either within that file or with a new file that the previous agent would read in turn. The generator then built against the agreed-upon contract before handing the work off to QA. This kept the work faithful to the spec without over-specifying implementation too early. @@ -79,23 +89,10 @@ I wrote the following prompt to generate a retro video game maker: The table below shows the harness type, length it ran for, and the total cost. -**Harness** - -**Duration** - -**Cost** - -Solo - -20 min - -$9 - -Full harness - -6 hr - -$200 +| **Harness** | **Duration** | **Cost** | +| --- | --- | --- | +| Solo | 20 min | $9 | +| Full harness | 6 hr | $200 | The harness was over 20x more expensive, but the difference in output quality was immediately apparent. @@ -103,6 +100,12 @@ I was expecting an interface where I could construct a level and its component p As I clicked through, however, issues started to emerge. The layout wasted space, with fixed-height panels leaving most of the viewport empty. The workflow was rigid. Trying to populate a level prompted me to create sprites and entities first, but nothing in the UI guided me toward that sequence. More to the point, the actual game was broken. My entities appeared on screen but nothing responded to input. Digging into the code revealed that the wiring between entity definitions and the game runtime was broken, with no surface indication of where. +![[23c98f1d7ae720bfb39190d50e0706c03b177ad8-1999x1320.webp]] + +Initial screen when opening the app created by the solo harness. + + + After evaluating the solo run, I turned my attention to the harness run. This run started from the same one-sentence prompt, but the planner step expanded that prompt into a 16-feature spec spread across ten sprints. It went well beyond what the solo run attempted. In addition to the core editors and play mode, the spec called for a sprite animation system, behavior templates, sound effects and music, an AI-assisted sprite generator and level designer, and game export with shareable links. I gave the planner access to our [frontend design skill](https://github.com/anthropics/claude-code/blob/main/plugins/frontend-design/skills/frontend-design/SKILL.md), which it read and used to create a visual design language for the app as part of the spec. For each sprint, the generator and evaluator negotiated a contract defining the specific implementation details for the sprint, and the testable behaviors that would be tested to verify completion. The app immediately showed more polish and smoothness than the solo run. The canvas used the full viewport, the panels were sized sensibly, and the interface had a consistent visual identity that tracked the design direction from the spec. Some of the clunkiness I'd seen in the solo run did remain—the workflow still didn't make it clear that you should build sprites and entities before trying to populate a level, and I had to figure that out by poking around. This read as a gap in the base model’s product intuition rather than something the harness was designed to address, though it did suggest a place where targeted iteration inside the harness could help to further improve output quality. @@ -111,30 +114,23 @@ Working through the editors, the new run's advantages over solo became more appa Because I'd asked the planner to weave AI features into its specs, the app also came with a built-in Claude integration that let me generate different parts of the game through prompting. This significantly sped up the workflow. +![[a8bef95425966495629095a5cb38bde4a8b13558-1999x997.webp]] + +Initial screen: Creating a new game, in the app built with the full harness + The biggest difference was in play mode. I was actually able to move my entity and play the game. The physics had some rough edges—my character jumped onto a platform but ended up overlapping with it, which felt intuitively wrong—but the core thing worked, which the solo run did not manage. After moving around a bit, I did hit some limitations with the AI’s game level construction. There was a large wall that I wasn’t able to jump past, so I was stuck. This suggested there were some common sense improvements and edge cases that the harness could handle to further refine the app. Reading through the logs, it was clear that the evaluator kept the implementation in line with the spec. Each sprint, it walked through the sprint contract's test criteria and exercised the running application through Playwright, filing bugs against anything that diverged from expected behavior. The contracts were granular—Sprint 3 alone had 27 criteria covering the level editor—and the evaluator's findings were specific enough to act on without extra investigation. The table below shows several examples of issues our evaluator identified: -**Contract criterion** - -**Evaluator finding** - -Rectangle fill tool allows click-drag to fill a rectangular area with selected tile - -**FAIL** — Tool only places tiles at drag start/end points instead of filling the region. `fillRectangle` function exists but isn't triggered properly on mouseUp. - -User can select and delete placed entity spawn points - -**FAIL** — Delete key handler at `LevelEditor.tsx:892` requires both `selection` and `selectedEntityId` to be set, but clicking an entity only sets `selectedEntityId`. Condition should be `selection || (selectedEntityId && activeLayer === 'entity')`. - -User can reorder animation frames via API - -**FAIL** — `PUT /frames/reorder` route defined after `/{frame_id}` routes. FastAPI matches 'r`eorder`' as a frame\_id integer and returns 422: "unable to parse string as an integer." +| **Contract criterion** | **Evaluator finding** | +| --- | --- | +| Rectangle fill tool allows click-drag to fill a rectangular area with selected tile | **FAIL** — Tool only places tiles at drag start/end points instead of filling the region. `fillRectangle` function exists but isn't triggered properly on mouseUp. | +| User can select and delete placed entity spawn points | **FAIL** — Delete key handler at `LevelEditor.tsx:892` requires both `selection` and `selectedEntityId ` to be set, but clicking an entity only sets `selectedEntityId`. Condition should be `selection \|\| (selectedEntityId && activeLayer === 'entity')`. | +| User can reorder animation frames via API | **FAIL** — `PUT /frames/reorder` route defined after `/{frame_id}` routes. FastAPI matches 'r `eorder` ' as a frame\_id integer and returns 422: "unable to parse string as an integer." | Getting the evaluator to perform at this level took work. Out of the box, Claude is a poor QA agent. In early runs, I watched it identify legitimate issues, then talk itself into deciding they weren't a big deal and approve the work anyway. It also tended to test superficially, rather than probing edge cases, so more subtle bugs often slipped through. The tuning loop was to read the evaluator's logs, find examples where its judgment diverged from mine, and update the QAs prompt to solve for those issues. It took several rounds of this development loop before the evaluator was grading in a way that I found reasonable. Even then, the harness output showed the limits of the model’s QAing capabilities: small layout issues, interactions that felt unintuitive in places, and undiscovered bugs in more deeply nested features that the evaluator hadn't exercised thoroughly. There was clearly more verification headroom to capture with further tuning. But compared to the solo run, where the central feature of the application simply didn't work, the lift was obvious. -### -Iterating on the harness +### Iterating on the harness The first set of harness results was encouraging, but it was also bulky, slow, and expensive. The logical next step was to find ways to simplify the harness without degrading its performance. This was partly common sense and partly a function of a more general principle: every component in a harness encodes an assumption about what the model can't do on its own, and those assumptions are worth stress testing, both because they may be incorrect, and because they can quickly go stale as models improve. Our blog post [Building Effective Agents](https://www.anthropic.com/research/building-effective-agents) frames the underlying idea as "find the simplest solution possible, and only increase complexity when needed," and it's a pattern that shows up consistently for anyone maintaining an agent harness. @@ -164,59 +160,16 @@ The run was still lengthy and expensive, at about 4 hours and $124 in token cost Most of the time went to the builder, which ran coherently for over two hours without the sprint decomposition that Opus 4.5 had needed. -**Agent & Phase** - -**Duration** - -**Cost** - -Planner - -4.7 min - -$0.46 - -Build (Round 1) - -2 hr 7 min - -$71.08 - -QA (Round 1) - -8.8 min - -$3.24 - -Build (Round 2) - -1 hr 2 min - -$36.89 - -QA (Round 2) - -6.8 min - -$3.09 - -Build (Round 3) - -10.9 min - -$5.88 - -QA (Round 3) - -9.6 min - -$4.06 - -**Total V2 Harness** - -**3 hr 50 min** - -**$124.70** +| **Agent & Phase** | **Duration** | **Cost** | +| --- | --- | --- | +| Planner | 4.7 min | $0.46 | +| Build (Round 1) | 2 hr 7 min | $71.08 | +| QA (Round 1) | 8.8 min | $3.24 | +| Build (Round 2) | 1 hr 2 min | $36.89 | +| QA (Round 2) | 6.8 min | $3.09 | +| Build (Round 3) | 10.9 min | $5.88 | +| QA (Round 3) | 9.6 min | $4.06 | +| **Total V2 Harness** | **3 hr 50 min** | **$124.70** | As with the previous harness, the planner expanded the one-line prompt into a full spec. From the logs, I could see the generator model did a good job planning the app and the agent design, wiring the agent up, and testing it before handing off to QA. @@ -235,6 +188,8 @@ The generator was still liable to miss details or stub features when left to its Based on the prompt, I was expecting a program where I could create melodies, harmonies, and drum patterns, arrange them into a song, and get help from an integrated agent along the way. The video below shows the result. + + The app is far from a professional music production program, and the agent's song composition skills could clearly use a lot of work. Additionally, Claude can’t actually hear, which made the QA feedback loop less effective with respect to musical taste. But the final app had all the core pieces of a functional music production program: a working arrangement view, mixer, and transport running in the browser. Beyond that, I was able to put together a short song snippet entirely through prompting: the agent set the tempo and key, laid down a melody, built a drum track, adjusted mixer levels, and added reverb. The core primitives for song composition were present, and the agent could drive them autonomously, using tools to create a simple production from end to end. You might say it’s not pitch-perfect yet—but it’s getting there. @@ -247,15 +202,13 @@ With this in mind, there are a few lessons from this work worth carrying forward From this work, my conviction is that the space of interesting harness combinations doesn't shrink as models improve. Instead, it moves, and the interesting work for AI engineers is to keep finding the next novel combination. -## -Acknowledgements +## Acknowledgements Special thanks to Mike Krieger, Michael Agaby, Justin Young, Jeremy Hadfield, David Hershey, Julius Tarng, Xiaoyi Zhang, Barry Zhang, Orowa Sidker, Michael Tingley, Ibrahim Madha, Martina Long, and Canyon Robbins for their contributions to this work. Thanks also to Jake Eaton, Alyssa Leonard, and Stef Sequeira for their help shaping the post. -## -Appendix +## Appendix Example plan generated by planner agent. @@ -290,5 +243,4 @@ Color palette selection All associated sprites, tilesets, levels, and entity definitions ... - ``` \ No newline at end of file diff --git a/raw/martinfowler-harness-engineering.md b/raw/Harness engineering for coding agent users.md similarity index 57% rename from raw/martinfowler-harness-engineering.md rename to raw/Harness engineering for coding agent users.md index ca4c9fc..cb1ef6f 100644 --- a/raw/martinfowler-harness-engineering.md +++ b/raw/Harness engineering for coding agent users.md @@ -1,80 +1,43 @@ -# Harness engineering for coding agent users - +--- +title: "Harness engineering for coding agent users" +source: "https://martinfowler.com/articles/harness-engineering.html" +author: + - "[[Birgitta Böckeler Birgitta is a Distinguished Engineer and AI-assisted delivery expert at Thoughtworks. She has over 20 years of experience as a software developer]]" + - "[[architect and technical leader.]]" +published: 2026-04-02 +created: 2026-04-11 +description: "A mental model for building trust in coding agents through feedforward guides, feedback sensors, and iterative harness engineering." +tags: + - "clippings" +--- The term harness has emerged as a shorthand to mean everything in an AI agent except the model itself - [Agent = Model + Harness](https://blog.langchain.com/the-anatomy-of-an-agent-harness/). That is a very wide definition, and therefore worth narrowing down for common categories of agents. I want to take the liberty here of defining its meaning in the bounded context of using a coding agent. In coding agents, part of the harness is already built in (e.g. via the system prompt, or the chosen code retrieval mechanism, or even a [sophisticated orchestration system](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents)). But coding agents also provide us, their users, with many features to build an outer harness specifically for our use case and system. -![Three concentric circles, with the model in the core (the ultimate thing being harnessed), then the coding agent's builder harness next circle out, and the coding agent's user harness as the outermost ring](../../martinfowler-assets/harness-bounded-contexts.png) +![[harness-bounded-contexts.png|Three concentric circles, with the model in the core (the ultimate thing being harnessed), then the coding agent's builder harness next circle out, and the coding agent's user harness as the outermost ring]] Figure 1: The term “harness” means different things depending on the bounded context. A well-built outer harness serves two goals: it increases the probability that the agent gets it right in the first place, and it provides a feedback loop that self-corrects as many issues as possible before they even reach human eyes. Ultimately it should reduce the review toil and increase the system quality, all with the added benefit of fewer wasted tokens along the way. -![Title "Harness engineering for coding agent users". Overview of guides (examples shown are \[inferential\] principles, CfRs, Rules, Ref Docs, How-tos; \[computational\] Language Servers, CLIs, scripts, codemods) that feedforward into a coding agent; and feedback sensors (examples shown are \[inferential\] review agents; \[computational\] static analysis, logs, browser). The feedback sensors point at the coding agent as well as input into its self-correcting loop. On the left side of it all we see a box with a human who steers both the guides and sensors.](../../martinfowler-assets/harness-overview.png) - -## Feedforward and Feedback - -To harness a coding agent we both anticipate unwanted outputs and try to prevent them, and we put sensors in place to allow the agent to self-correct: - -- **Guides (feedforward controls)** - anticipate the agent's behaviour and aim to steer it *before* it acts. Guides increase the probability that the agent creates good results in the first attempt -- **Sensors (feedback controls)** - observe *after* the agent acts and help it self-correct. Particularly powerful when they produce signals that are optimised for LLM consumption, e.g. custom linter messages that include instructions for the self-correction - a positive kind of prompt injection. - -Separately, you get either an agent that keeps repeating the same mistakes (feedback-only) or an agent that encodes rules but never finds out whether they worked (feed-forward-only). +![[harness-overview.png|Title "Harness engineering for coding agent users". Overview of guides (examples shown are [inferential] principles, CfRs, Rules, Ref Docs, How-tos; [computational] Language Servers, CLIs, scripts, codemods) that feedforward into a coding agent; and feedback sensors (examples shown are [inferential] review agents; [computational] static analysis, logs, browser). The feedback sensors point at the coding agent as well as input into its self-correcting loop. On the left side of it all we see a box with a human who steers both the guides and sensors.]] ## Computational vs Inferential There are two execution types of guides and sensors: -- **Computational** - deterministic and fast, run by the CPU. Tests, linters, type checkers, structural analysis. Run in milliseconds to seconds; results are reliable. -- **Inferential** - Semantic analysis, AI code review, “LLM as judge”. Typically run by a GPU or NPU. Slower and more expensive; results are more non-deterministic. +- **Computational** - deterministic and fast, run by the CPU. Tests, linters, type checkers, structural analysis. Run in milliseconds to seconds; results are reliable. +- **Inferential** - Semantic analysis, AI code review, “LLM as judge”. Typically run by a GPU or NPU. Slower and more expensive; results are more non-deterministic. Computational guides increase the probability of good results with deterministic tooling. Computational sensors are cheap and fast enough to run on every change, alongside the agent. Inferential controls are of course more expensive and non-deterministic, but allow us to both provide rich guidance, and add additional semantic judgment. In spite of their non-determinism, inferential sensors can particularly increase our trust when used with a strong model, or rather a model that is suitable to the task at hand. **Examples** -Direction - -Computational / Inferential - -Example implementations - -Coding conventions - -feedforward - -Inferential - -AGENTS.md, Skills - -Instructions how to bootstrap a new project - -feedforward - -Both - -Skill with instructions and a bootstrap script - -Code mods - -feedforward - -Computational - -A tool with access to OpenRewrite recipes - -Structural tests - -feedback - -Computational - -A pre-commit (or coding agent) hook running ArchUnit tests that check for violations of module boundaries - -Instructions how to review - -feedback - -Inferential - -Skills +| | Direction | Computational / Inferential | Example implementations | +| --- | --- | --- | --- | +| Coding conventions | feedforward | Inferential | AGENTS.md, Skills | +| Instructions how to bootstrap a new project | feedforward | Both | Skill with instructions and a bootstrap script | +| Code mods | feedforward | Computational | A tool with access to OpenRewrite recipes | +| Structural tests | feedback | Computational | A pre-commit (or coding agent) hook running ArchUnit tests that check for violations of module boundaries | +| Instructions how to review | feedback | Inferential | Skills | ## The steering loop @@ -88,17 +51,14 @@ Teams who are [continuously integrating](https://martinfowler.com/articles/conti **Feedforward and feedback in the change lifecycle** -- What is reasonably fast and should be run even before integration, or even before a commit is even created? (e.g. linters, fast test suites, basic code review agent) -- What is more expensive and should therefore only be run post-integration in the pipeline, in addition to a repetition of the fast controls? (e.g. mutation testing, a more broad code review that can take into account the bigger picture) - -![Examples of feedforward and feedback in a change's lifecycle. Feedforward: LSP, architecture.md, /how-to-test skill, AGENTS.md, MCP server that can access a team's knowledge management tool, /xyz-api-docs skill; they feed into the agent's initial generation; feedback sensor examples for first self-correction loop are /code-review, npx eslint, semgrep, npm run coverage, npm run dep-cruiser; then human review is an additional feedback sensor; then integration happens; after integration, examples shown in the pipeline, which reruns all the previous sensors, and additional examples for more expensive sensors are /architecture-review skill, /detailed-review skill, mutation testing. An arrow shows that the feedback can then lead to new commits by agents or humans.](../../martinfowler-assets/harness-change-lifecycle-examples.png) +- What is reasonably fast and should be run even before integration, or even before a commit is even created? (e.g. linters, fast test suites, basic code review agent) +- What is more expensive and should therefore only be run post-integration in the pipeline, in addition to a repetition of the fast controls? (e.g. mutation testing, a more broad code review that can take into account the bigger picture) +![[harness-change-lifecycle-examples.png|Examples of feedforward and feedback in a change's lifecycle. Feedforward: LSP, architecture.md, /how-to-test skill, AGENTS.md, MCP server that can access a team's knowledge management tool, /xyz-api-docs skill; they feed into the agent's initial generation; feedback sensor examples for first self-correction loop are /code-review, npx eslint, semgrep, npm run coverage, npm run dep-cruiser; then human review is an additional feedback sensor; then integration happens; after integration, examples shown in the pipeline, which reruns all the previous sensors, and additional examples for more expensive sensors are /architecture-review skill, /detailed-review skill, mutation testing. An arrow shows that the feedback can then lead to new commits by agents or humans.]] **Continuous drift and health sensors** -- What type of drift accumulates gradually and should be monitored by sensors running continuously against the codebase, outside the change lifecycle? (e.g. dead code detection, analysis of the quality of the test coverage, dependency scanners) -- What runtime feedback could agents be monitoring? (e.g. having them look for degrading SLOs to make suggestions how to improve them, or AI judges continuously sampling response quality and flagging log anomalies) - -![Shows examples of continuous feedback sensors after change integration. Continuous drift detection in the codebase, e.g. /find-dead-code, /code-coverage-quality, dependabot; or Continuous runtime feedback, e.g. latency, error rate or availability SLOs leading to coding agent suggestions, or /response-quality-sampling, /log-anomalies AI judges.](../../martinfowler-assets/harness-continuous-feedback-examples.png) +- What type of drift accumulates gradually and should be monitored by sensors running continuously against the codebase, outside the change lifecycle? (e.g. dead code detection, analysis of the quality of the test coverage, dependency scanners) +- What runtime feedback could agents be monitoring? (e.g. having them look for degrading SLOs to make suggestions how to improve them, or AI judges continuously sampling response quality and flagging log anomalies) ## Regulation categories @@ -124,21 +84,21 @@ This groups guides and sensors that define and check the architecture characteri Examples: -- Skills that feed forward our performance requirements, and performance tests that feed back to the agent if it improved or degraded them. -- Skills that describe coding conventions for better observability (like logging standards), and debugging instructions that ask the agent to reflect on the quality of the logs it had available. +- Skills that feed forward our performance requirements, and performance tests that feed back to the agent if it improved or degraded them. +- Skills that describe coding conventions for better observability (like logging standards), and debugging instructions that ask the agent to reflect on the quality of the logs it had available. ### Behaviour harness This is the elephant in the room - how do we guide and sense if the application functionally behaves the way we need it to? At the moment, I see most people who give high autonomy to their coding agents do this: -- Feed-forward: A functional specification (of varying levels of detail, from a short prompt to multi-file descriptions) -- Feed-back: Check if the AI-generated test suite is green, has reasonably high coverage, some might even monitor its quality with mutation testing. Then combine that with manual testing. +- Feed-forward: A functional specification (of varying levels of detail, from a short prompt to multi-file descriptions) +- Feed-back: Check if the AI-generated test suite is green, has reasonably high coverage, some might even monitor its quality with mutation testing. Then combine that with manual testing. This approach puts a lot of faith into the AI-generated tests, that's not good enough yet. Some of my colleagues are seeing good results with the [approved fixtures](https://lexler.github.io/augmented-coding-patterns/patterns/approved-fixtures/) pattern, but it's easier to apply in some areas than others. They use it selectively where it fits, it's not a wholesale answer to the test quality problem. So overall, we still have a lot to do to figure out good harnesses for functional behaviour that increase our confidence enough to reduce supervision and manual testing. -![Simplified overview of a harness showing guides and sensors in horizontal, and then the regulation dimensions maintainability, architecture fitness, and behaviour, in vertical. Examples shown for the behaviour harness, spec as feedforward guide, test suite as feedback sensor that is a mix of inferential and computational, plus a human icon indicating human review and manual tests as main additional feedback sensor.](../../martinfowler-assets/harness-types.png) +![[harness-types.png|Simplified overview of a harness showing guides and sensors in horizontal, and then the regulation dimensions maintainability, architecture fitness, and behaviour, in vertical. Examples shown for the behaviour harness, spec as feedforward guide, test suite as feedback sensor that is a mix of inferential and computational, plus a human icon indicating human review and manual tests as main additional feedback sensor.]] ## Harnessability @@ -150,7 +110,7 @@ This plays out differently for greenfield versus legacy. Greenfield teams can ba Most enterprises have a few common topologies of services that cover 80% of what they need - business services that exposes data via APIs; event processing services; data dashboards. In many mature engineering organizations these topologies are already codified in service templates. These might evolve into harness templates in the future: a bundle of guides and sensors that leash a coding agent to the structure, conventions and tech stack of a topology. Teams may start picking tech stacks and structures partly based on what harnesses are already available for them. -![A stack of examples of topologies (Data dashboard in Node, CRUD business service on JVM, event processor in Golang). The top one, data dashboard, is shown in detail, as a combination of structure definition and tech stack. The graphic indicates a "harness template" with guides and sensors for each topology, which can be instantiated.](../../martinfowler-assets/harness-templates.png) +![[harness-templates.png|A stack of examples of topologies (Data dashboard in Node, CRUD business service on JVM, event processor in Golang). The top one, data dashboard, is shown in detail, as a combination of structure definition and tech stack. The graphic indicates a "harness template" with guides and sensors for each topology, which can be instantiated.]] We would of course face similar challenges as with service templates. As soon as teams instantiate them, they start fall out of sync with upstream improvements. Harness templates would face the same versioning and contribution problems, maybe even worse with non-deterministic guides and sensors that are harder to test. @@ -168,12 +128,28 @@ The mental model I've laid out here describes techniques that are already happen Here are some harness-related examples from the current discourse: -- [An OpenAI team documented what their harness looks like](https://openai.com/index/harness-engineering/): layered architecture enforced by custom linters and structural tests, and recurring “garbage collection” that scans for drift and has agents suggest fixes. Their conclusion: “Our most difficult challenges now center on designing environments, feedback loops, and control systems.” -- [Stripe's write-up about their minions](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents) describes things like pre-push hooks that run relevant linters based on a heuristic, they highlight how important “shift feedback left” is to them, and their “blueprints” show how they're integrating feedback sensors into the agent workflows. -- Mutation and structural testing are examples of computational feedback sensors that have been underused in the past, but are now having a resurgence. -- There is increased chatter among developers about the integration of LSPs and code intelligence in coding agents, examples of computational feedforward guides. -- I hear stories from teams at Thoughtworks about tackling architecture drift with both computational and inferential sensors, e.g. increasing API quality with a mix of agents and custom linters, or increasing code quality with a “janitor army”. +- [An OpenAI team documented what their harness looks like](https://openai.com/index/harness-engineering/): layered architecture enforced by custom linters and structural tests, and recurring “garbage collection” that scans for drift and has agents suggest fixes. Their conclusion: “Our most difficult challenges now center on designing environments, feedback loops, and control systems.” +- [Stripe's write-up about their minions](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents) describes things like pre-push hooks that run relevant linters based on a heuristic, they highlight how important “shift feedback left” is to them, and their “blueprints” show how they're integrating feedback sensors into the agent workflows. +- Mutation and structural testing are examples of computational feedback sensors that have been underused in the past, but are now having a resurgence. +- There is increased chatter among developers about the integration of LSPs and code intelligence in coding agents, examples of computational feedforward guides. +- I hear stories from teams at Thoughtworks about tackling architecture drift with both computational and inferential sensors, e.g. increasing API quality with a mix of agents and custom linters, or increasing code quality with a “janitor army”. There's plenty still to figure out, not just the already mentioned behavioural harness. How do we keep a harness coherent as it grows, with guides and sensors in sync, not contradicting each other? How far can we trust agents to make sensible trade-offs when instructions and feedback signals point in different directions? If sensors never fire, is that a sign of high quality or inadequate detection mechanisms? We need a way to evaluate harness coverage and quality similar to what code coverage and mutation testing do for tests. Feedforward and feedback controls are currently scattered across delivery steps, there's real potential for tooling that helps configure, sync, and reason about them as a system. Building this outer harness is emerging as an ongoing engineering practice, not a one-time configuration. -* * * \ No newline at end of file +--- + +## Acknowledgements + +Big thanks to the Doppler team for the engaging discussion at our last technology radar meeting, in particular Kief Morris for bringing up cybernetics. Thanks to Ned Letcher, Chris Ford and Ben O'Mahoney for the conversations about what a harness even is, and to Matteo Vaccari for his insights on the behaviour harness. And to everybody who took the time to read the draft and provide lots of valuable feedback: Christoph Burgmer, Jörn Dinkla, Michael Feathers, Karrtik Iyer, Swapnil Phulse, Paul Sobocinski, Zhenjia Zhou + +GenAI (Claude and Claude Code) was used for research, pulling in relevant ideas from existing notes, and polishing the language. + +## Earlier Memo + +I wrote [a memo in early February](https://martinfowler.com/articles/exploring-gen-ai/harness-engineering-memo.html) containing my initial thoughts on Harness Engineering as the term first appeared. That post has attracted a lot of traffic. This article supersedes that memo, so we have redirected the original memo URL to this page, as we believe this page is the better resource for readers. + +Significant Revisions + +*02 April 2026:* published full article including introducing guides, sensors, computational and inferential elements, and harness templates + +*17 February 2026:* published my [initial memo](https://martinfowler.com/articles/exploring-gen-ai/harness-engineering-memo.html) on Harness Engineering \ No newline at end of file diff --git a/raw/langchain-harness-engineering.md b/raw/Improving Deep Agents with harness engineering.md similarity index 68% rename from raw/langchain-harness-engineering.md rename to raw/Improving Deep Agents with harness engineering.md index 442eb8c..5202fc0 100644 --- a/raw/langchain-harness-engineering.md +++ b/raw/Improving Deep Agents with harness engineering.md @@ -1,5 +1,14 @@ -# Improving Deep Agents with harness engineering - +--- +title: "Improving Deep Agents with harness engineering" +source: "https://blog.langchain.com/improving-deep-agents-with-harness-engineering/" +author: + - "[[LangChain Accounts]]" +published: 2026-02-18 +created: 2026-04-11 +description: "TLDR: Our coding agent went from Top 30 to Top 5 on Terminal Bench 2.0. We only changed the harness. Here’s our approach to harness engineering (teaser: self-verification & tracing help a lot).The Goal of Harness EngineeringThe goal of a harness is to mold the inherently spiky" +tags: + - "clippings" +--- TLDR: Our coding agent went from Top 30 to Top 5 on [Terminal Bench 2.0](https://www.tbench.ai/leaderboard/terminal-bench/2.0?ref=blog.langchain.com). We only changed the harness. Here’s our approach to harness engineering (teaser: self-verification & tracing help a lot). ## The Goal of Harness Engineering @@ -12,7 +21,9 @@ At LangChain, we use [Traces](https://docs.langchain.com/langsmith/observability We used a simple recipe to iteratively improve [deepagents-cli](https://github.com/langchain-ai/deepagents/tree/main/libs/cli?ref=blog.langchain.com) (our coding agent) `13.7 points` from `52.8` to `66.5` on Terminal Bench 2.0. We only tweaked the harness and kept the model fixed, `gpt-5.2-codex`. -![](../../langchain-assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png) +![[Screenshot-2026-02-12-at-12.25.20---PM-1.png]] + +Spending more reasoning compute on planning and verification ## Experiment Setup & The Knobs on a Harness @@ -20,27 +31,31 @@ We used [Terminal Bench 2.0](https://www.tbench.ai/?ref=blog.langchain.com), a n Every agent action is stored in [LangSmith](https://smith.langchain.com/?ref=blog.langchain.com). It also includes metrics like latency, token counts, and costs. -### **The Knobs we can Turn** +### The Knobs we can Turn An agent harness has a lot of knobs: system prompts, tools, hooks/middleware, skills, sub-agent delegation, memory systems, and more. We deliberately compress the optimization space and focus on three: **System Prompt, Tools,** and [**Middleware**](https://docs.langchain.com/oss/python/langchain/middleware/overview?ref=blog.langchain.com#the-agent-loop) (our term for hooks around model and tool calls). We start with a default prompt and standard tools+middleware. This scores 52.8% with GPT-5.2-Codex. A solid score, just outside the Top 30 of the leaderboard today, but room to grow. -![](../../langchain-assets/Screenshot-2026-02-16-at-12.50.00---PM.png) +![[Screenshot-2026-02-16-at-12.50.00---PM.png]] -### **The Trace Analyzer Skill** +Spending more reasoning compute on planning and verification + +### The Trace Analyzer Skill We wanted trace analysis to be repeatable so we made it into an Agent Skill. This serves as our recipe to **analyze errors across runs and make improvements to the harness**. The flow is: -1. Fetch experiment traces from LangSmith -2. Spawn parallel error analysis agents → main agent synthesizes findings + suggestions -3. Aggregate feedback and make targeted changes to the harness. +1. Fetch experiment traces from LangSmith +2. Spawn parallel error analysis agents → main agent synthesizes findings + suggestions +3. Aggregate feedback and make targeted changes to the harness. This works similarly to [boosting](https://en.wikipedia.org/wiki/Boosting_\(machine_learning\)?ref=blog.langchain.com) which focuses on mistakes from previous runs. A human can be pretty helpful in Step 3 (though not required) to verify and discuss proposed changes. Changes that overfit to a task are bad for generalization and can lead to regressions in other Tasks. Automated trace analysis saves hours of time and made it easy to quickly try experiments. We’ll be publishing this skill soon, we’re currently testing it for prompt optimization generally. -![](../../langchain-assets/langsmith_trace_analyzer_skill.png) +![[langsmith_trace_analyzer_skill.png]] + +Spending more reasoning compute on planning and verification ## What Actually Improved Agent Performance @@ -56,22 +71,24 @@ The most common failure pattern was that the agent wrote a solution, re-read its We added guidance to the system prompt on how to approach problem solving. -1. **Planning & Discovery:** Read the task, scan the codebase, and build an initial plan based on the task specification and how to verify the solution. -2. **Build:** Implement the plan with verification in mind. Build tests, if they don’t exist and test both happy paths and edge cases. -3. **Verify:** Run tests, read the full output, compare against what was asked (not against your own code). -4. **Fix:** Analyze any errors, revisit the original spec, and fix issues. +1. **Planning & Discovery:** Read the task, scan the codebase, and build an initial plan based on the task specification and how to verify the solution. +2. **Build:** Implement the plan with verification in mind. Build tests, if they don’t exist and test both happy paths and edge cases. +3. **Verify:** Run tests, read the full output, compare against what was asked (not against your own code). +4. **Fix:** Analyze any errors, revisit the original spec, and fix issues. We really focus on testing because it powers the changes in every iteration. We found that alongside prompting, deterministic context injection helps agents verify their work. We use a `PreCompletionChecklistMiddleware` that intercepts the agent before it exits and reminds it to run a verification pass against the Task spec. This is similar to a [Ralph Wiggum Loop](https://ghuntley.com/loop/?ref=blog.langchain.com) where a hook forces the agent to continue executing on exit, we use this for verification. -![](../../langchain-assets/self-verification-loop.png) +![[self-verification-loop.png]] + +Spending more reasoning compute on planning and verification ### Giving Agents Context about their Environment Part of harness engineering is **building a good delivery mechanism for context engineering.** Terminal Bench tasks come with directory structures, built-in tooling, and strict timeouts. -1. **Directory Context & Tooling:** A `LocalContextMiddleware` runs on agent start to map the `cwd` and other parent+children directories. We run `bash` commands to find tools like `Python` installations. Context discovery and search are error prone, so injecting context reduces this error surface and helps **onboard the agent into its environment.** -2. **Teaching Agents to Write Testable Code:** Agents don’t know how their code needs to be testable. We add prompting say their work will be measured against programatic tests, similar to when committing code. For example, Task specs that mention file paths should be followed exactly so the solutions works in an automated scoring step. Prompting that stresses edge-cases helps the agent avoid only checking “happy path” cases. Forcing models to conform to testing standards is a powerful strategy to avoid “slop buildup” over time. -3. **Time Budgeting:** We inject time budget warnings to nudge the agent to finish work and shift to verification. Agents are famously bad at time estimation so this heuristic helps in this environment. Real world coding usually doesn’t have strict time limits, but without adding any knowledge of constraints, agents won’t work within time bounds. +1. **Directory Context & Tooling:** A `LocalContextMiddleware` runs on agent start to map the `cwd` and other parent+children directories. We run `bash` commands to find tools like `Python` installations. Context discovery and search are error prone, so injecting context reduces this error surface and helps **onboard the agent into its environment.** +2. **Teaching Agents to Write Testable Code:** Agents don’t know how their code needs to be testable. We add prompting say their work will be measured against programatic tests, similar to when committing code. For example, Task specs that mention file paths should be followed exactly so the solutions works in an automated scoring step. Prompting that stresses edge-cases helps the agent avoid only checking “happy path” cases. Forcing models to conform to testing standards is a powerful strategy to avoid “slop buildup” over time. +3. **Time Budgeting:** We inject time budget warnings to nudge the agent to finish work and shift to verification. Agents are famously bad at time estimation so this heuristic helps in this environment. Real world coding usually doesn’t have strict time limits, but without adding any knowledge of constraints, agents won’t work within time bounds. The more that agents know about their environment, constraints, and evaluation criteria, the better they can autonomously self-direct their work. @@ -93,11 +110,11 @@ Terminal Bench timeout limits create a tradeoff. More reasoning helps agents eva We found that reasoning helps with planning to fully understand the problem, some Terminal Bench tasks are very difficult. A good plan helps get to a working solution more quickly. -Later stage verification also benefits from more reasoning to catch mistakes and get a solution submitted. As a heuristic, we choose a xhigh-high-xhigh "**reasoning sandwich**" as a baseline. +Later stage verification also benefits from more reasoning to catch mistakes and get a solution submitted. As a heuristic, we choose a xhigh-high-xhigh " **reasoning sandwich** " as a baseline. -![](../../langchain-assets/the-reasoning-sandwich.png) +![[the-reasoning-sandwich.png]] -**Spending more reasoning compute on planning and verification** +Spending more reasoning compute on planning and verification Running only at `xhigh` scored poorly at `53.9%` due to agent timeouts compared to `63.6%` at `high`. There weren’t large differences in trial runs across reasoning budget splits so we stuck with our approach which pushed the score to `66.5%`. @@ -109,11 +126,11 @@ In a multi-model harness, balancing reasoning budgets could play out as using a The design space of agents is big. Here are some general principles from our experiments and building deepagents overall. -1. **Context Engineering on Behalf of Agents.** Context assembly is still difficult for agents today, especially in unseen environments. Onboarding models with context like directory structures, available tools, coding best practices, and problem solving strategies helps reduce the error surface for poor search and avoidable errors in planning. -2. **Help agents self-verify their work.** Models are biased towards their first plausible solution. Prompt them aggressively to verify their work by running tests and refining solutions. This is especially important in autonomous coding systems that don’t have humans in the loop. -3. **Tracing as a feedback signal.** Traces allow agents to self-evaluate and debug themselves. It’s important to debug tooling and reasoning together (ex: models go down wrong paths because they lack a tool or instructions how to do something). -4. **Detect and fix bad patterns in the short term.** Models today aren’t perfect. The job of the harness designer is to design around today’s shortcomings while planning for smarter models in the future. Blind retries and not verifying work are good examples. These guardrails will almost surely dissolve over time, but to build robust agent applications today, they’re useful tools to experiment with. -5. **Tailor Harnesses to Models. T**he [Codex](https://developers.openai.com/cookbook/examples/gpt-5/codex_prompting_guide/?ref=blog.langchain.com) and [Claude](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices?ref=blog.langchain.com) prompting guides show that models require different prompting. A test run with Claude Opus 4.6 scored `59.6%` with an earlier harness version, competitive but worse than Codex because we didn’t run the same Improvement Loop with Claude. Many principles generalize like good context preparation and a focus on verification, but running a few rounds of harness iterations for your task helps maximize agent performance across tasks. +1. **Context Engineering on Behalf of Agents.** Context assembly is still difficult for agents today, especially in unseen environments. Onboarding models with context like directory structures, available tools, coding best practices, and problem solving strategies helps reduce the error surface for poor search and avoidable errors in planning. +2. **Help agents self-verify their work.** Models are biased towards their first plausible solution. Prompt them aggressively to verify their work by running tests and refining solutions. This is especially important in autonomous coding systems that don’t have humans in the loop. +3. **Tracing as a feedback signal.** Traces allow agents to self-evaluate and debug themselves. It’s important to debug tooling and reasoning together (ex: models go down wrong paths because they lack a tool or instructions how to do something). +4. **Detect and fix bad patterns in the short term.** Models today aren’t perfect. The job of the harness designer is to design around today’s shortcomings while planning for smarter models in the future. Blind retries and not verifying work are good examples. These guardrails will almost surely dissolve over time, but to build robust agent applications today, they’re useful tools to experiment with. +5. **Tailor Harnesses to Models. T** he [Codex](https://developers.openai.com/cookbook/examples/gpt-5/codex_prompting_guide/?ref=blog.langchain.com) and [Claude](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices?ref=blog.langchain.com) prompting guides show that models require different prompting. A test run with Claude Opus 4.6 scored `59.6%` with an earlier harness version, competitive but worse than Codex because we didn’t run the same Improvement Loop with Claude. Many principles generalize like good context preparation and a focus on verification, but running a few rounds of harness iterations for your task helps maximize agent performance across tasks. There’s more open research to do in harness design. Interesting avenues include multi-model systems (Codex, Gemini, and Claude together), memory primitives for continual learning so agents can autonomously improve on tasks, and measuring harness changes across models. diff --git a/raw/Meta-Harness End-to-End Optimization of Model Harnesses.md b/raw/Meta-Harness End-to-End Optimization of Model Harnesses.md new file mode 100644 index 0000000..21839df --- /dev/null +++ b/raw/Meta-Harness End-to-End Optimization of Model Harnesses.md @@ -0,0 +1,557 @@ +--- +title: "Meta-Harness: End-to-End Optimization of Model Harnesses" +source: "https://arxiv.org/html/2603.28052v1" +author: +published: +created: 2026-04-11 +description: +tags: + - "clippings" +--- +Yoonho Lee +Stanford &Roshen Nair +Stanford &Qizheng Zhang +Stanford &Kangwook Lee +KRAFTON    Omar Khattab +MIT &Chelsea Finn +Stanford + +###### Abstract + +The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively: they are memoryless, condition only on scalar scores, or restrict feedback to short templates or summaries. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4 $\times$ fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On agentic coding, discovered harnesses surpass the best hand-engineered baselines on TerminalBench-2. Together, these results show that richer access to prior experience can enable automated harness engineering. + +Project page w/ interactive demo: [https://yoonholee.com/meta-harness/](https://yoonholee.com/meta-harness/) + +Optimized harness: [https://github.com/stanford-iris-lab/meta-harness-tbench2-artifact](https://github.com/stanford-iris-lab/meta-harness-tbench2-artifact) + +![[x1 1.png|Refer to caption]] + +Figure 1: (Left) On text classification, Meta-Harness outperforms the best prior hand-designed harnesses (ACE) and existing text optimizers (TTT-Discover, OpenEvolve), matching the next-best method’s final accuracy after just 4 evaluations. (Right) On TerminalBench-2, Meta-Harness outperforms all reported Claude Haiku 4.5 harnesses. + +## 1 Introduction + +Changing the harness around a fixed large language model (LLM) can produce a 6 $\times$ performance gap on the same benchmark [^46]. The harness—the code that determines what to store, retrieve, and show to the model—often matters as much as the model itself. This sensitivity has led to growing interest in harness engineering, the practice of refining the code around an LLM to improve the overall system’s performance [^35] [^20] [^9] [^8]. But despite its importance, harness engineering remains largely manual: practitioners inspect failures, adjust heuristics, and iterate on a small number of designs. In this paper, we ask whether this process itself can be automated. + +A natural starting point is recent work on text optimization, since harness engineering also involves iteratively improving text and code artifacts using feedback from prior attempts [^37] [^38] [^34] [^25] [^1]. However, these methods are poorly matched to harness engineering because they typically operate with short-horizon or heavily compressed feedback: some condition only on the current candidate [^30] [^50] [^52], others rely primarily on scalar scores [^34] [^11], and others restrict feedback to short templates or LLM-generated summaries [^1] [^25]. This is a pragmatic scalability choice, not evidence that longer-range dependencies are uninformative. Harnesses act over long horizons: a single choice about what to store, when to retrieve it, or how to present it can affect behavior many reasoning steps later. Compressed feedback often removes the information needed to trace downstream failures to earlier harness decisions. Across the tasks studied by several representative text optimizers, the available context per optimization step ranges from only 100 to 30,000 tokens (Table 1), far below the diagnostic footprint of harness search. More broadly, work on retrieval and memory-augmented language models suggests that useful context should often be accessed adaptively rather than monolithically packed into a single prompt [^27] [^47] [^36] [^55]. + +![[x3 1.png|Refer to caption]] + +Figure 2: Meta-Harness search loop. (1) An agent reads a filesystem containing all prior candidates’ source code, execution traces, and scores, and proposes a new harness. (2) We evaluate the proposed harness on evaluation tasks. (3) All logs (proposed code, reasoning traces, evaluation scores) are stored in the filesystem in a new directory, and the loop repeats. + +| Method | History | Log content | MTok/iter | +| --- | --- | --- | --- | +| OPRO [^50] | Window | past (solution, score) pairs | $0.002$ | +| TextGrad [^52] | Last | textual feedback on current artifact | $0.015$ | +| AlphaEvolve [^34] | Window | program database + eval. scores | $0.022$ | +| GEPA [^1] | Summary | reflective feedback from rollout traces | $0.008$ | +| Feedback Descent [^25] | Summary | comparison + textual feedback | $0.012$ | +| TTT-Discover [^54] | Window | prev. solution fragment | $0.026$ | +| Meta-Harness | Full | all logs and scores | $10.0$ | + +Table 1: Comparison of text optimization methods and their settings. Each row represents a method collapsed across tasks. Mtok/iter is our best estimate of the full context generated from one evaluation of a text artifact in the largest setting considered in each paper. This paper considers settings that yield orders-of-magnitude more context per artifact evaluation. + +We address this limitation with Meta-Harness, an agentic harness for optimizing harnesses via end-to-end search (Figure 2). Its proposer is a coding agent, i.e., a language-model-based system that can invoke developer tools and modify code. The choice of coding agent (rather than raw LLM) matters because the amount of experience quickly exceeds context limits, so the proposer must decide what to inspect and validate edits through direct interaction with the codebase. Its key design choice is to expose full history through a filesystem, enabling selective diagnosis of raw prior code and execution traces rather than optimization from compressed per-candidate summaries. For every previous candidate harness, the filesystem stores the source code, evaluation scores, and execution traces, which the proposer retrieves via standard operations such as grep and cat rather than ingesting them as a single prompt. In practice, the proposer reads a median of 82 files per iteration in our most demanding setting, referencing over 20 prior candidates per step (Appendix A). In the settings we study, a single evaluation can produce up to 10,000,000 tokens of diagnostic information, roughly three orders of magnitude beyond the largest feedback budgets used in prior text optimization settings (Table 1). + +We evaluate Meta-Harness on online text classification, mathematical reasoning, and agentic coding. On online text classification, harnesses discovered by Meta-Harness improve over Agentic Context Engineering (ACE, [^58]) by 7.7 points while using 4 $\times$ fewer context tokens, and match the next-best text optimizer’s final performance after $60$ proposals with only four (Figure 1). On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models. On TerminalBench-2, the discovered harness surpasses Terminus-KIRA and ranks #1 among all Haiku 4.5 agents. + +## 2 Related Work + +At a high level, Meta-Harness brings ideas from the broader literature on credit assignment and meta-learning [^39] [^45] [^2] [^16] [^43] in a new regime enabled by recent advances in coding agents. Rather than updating model weights, the system assigns credit at the harness level: it uses experience from past rollouts to deliberately reason about which steps and components are responsible for failures, then rewrites the external code that governs future behavior. More specifically, the method lies at the intersection of several recent research threads; it is most directly related to work on adaptive access to external context, executable code search, and text optimization. + +External memory and adaptive access. Several prior works note the benefits of treating large knowledge sources or long inputs as external resources that a language model accesses adaptively, rather than consuming them in a single pass. Specifically, retrieval-augmented generation [^27], interleaved retrieval and reasoning [^47], memory-based agents [^36], or recursive language models [^55] are mechanisms for adaptive access to external context. Meta-Harness uses a similar access pattern, but in the more demanding setting of harness engineering, where the proposer selectively inspects a large external history of code, scores, and execution traces to improve context-management procedures themselves. + +Executable code search. Recent methods search over executable code for functions, workflows, or agent designs. Early work proposes using large models as mutation and crossover operators in evolutionary program search [^26]. Later methods evolve designated functions within fixed program scaffolds [^38], use meta-agents to program new agents from prior discoveries [^19], or search over workflow graphs for agentic systems [^57]. Another line of work searches over memory designs for continual-learning agents, where memory persists across task streams [^56] [^49]. In contrast, Meta-Harness searches over domain-specific harnesses, including prompt construction, retrieval, and state update strategies that reset between tasks. Its outer loop is deliberately minimal: instead of relying on a fixed scaffold, an archive of prior discoveries, or a persistent memory mechanism, it gives the proposer unrestricted filesystem access to prior experience. This lets the agent decide what information to inspect and enables search over full harness implementations rather than a predefined space of context-management procedures. + +Text optimization methods. Meta-Harness is also closely related to methods such as ProTeGi, TextGrad, OPRO, GEPA, AlphaEvolve/OpenEvolve, and Feedback Descent, which iteratively improve prompts or other text artifacts using feedback from prior attempts [^37] [^30] [^52] [^50] [^1] [^34] [^42] [^25]. However, these methods are less well suited to harness engineering, where optimization targets a complete executable procedure, and the relevant environmental feedback is distributed across code, scores, and execution traces in a way that is hard to summarize up front. Rather than reacting only to aggregate scores or summaries, the proposer in Meta-Harness can reason over failed examples and their execution traces to propose targeted edits. See Table 1 for a comparison of problem scale considered in those papers and ours, and Figures 1 and 4 for a direct comparison with OpenEvolve, GEPA, and TTT-Discover in our problem setting. + +## 3 Meta-Harness: A Harness for Optimizing Harnesses + +This section describes Meta-Harness, our outer-loop procedure for searching over task-specific harnesses. Meta-Harness is built on the idea that harness optimization benefits from allowing a proposer to selectively inspect prior code and execution traces via filesystem access, rather than optimizing from lossy summaries or an additional hand-designed search structure. At a high level, it repeatedly proposes, evaluates, and logs new harnesses. + +Meta-Harness is itself a harness in the broad sense (hence the name), since it determines what information the proposer model sees during search. Unless otherwise noted, we use *harness* to refer to the task-specific programs being optimized. + +Objective. A harness is a stateful program that wraps a language model and determines what context the model sees at each step. The goal is simple: find the harness that makes the underlying model perform best on the target task distribution. Formally, let $M$ denote a fixed language model and $\mathcal{X}$ a task distribution. For a harness $H$ and task instance $x\sim\mathcal{X}$, we execute a rollout trajectory $\tau\sim p_{M}(H,x)$. The harness constructs prompts for $M$, the model responds, and the harness updates its state after each interaction. A task-specific reward function $r(\tau,x)$ scores the trajectory. The objective of harness optimization is to find the harness that maximizes the expected final reward: + +$$ +H^{*}=\operatorname*{arg\,max}_{H}\mathbb{E}_{x\sim\mathcal{X},\tau\sim p_{M}(H,x)}\;r(\tau,x), +$$ + +When multiple objectives are relevant (e.g., accuracy and context cost), we evaluate candidates under Pareto dominance and report the resulting frontier. In practice, this search has traditionally been carried out by human engineers and researchers, who iteratively refine prompts, context-management rules, and tool-use logic by hand. + +Meta-Harness search loop. Meta-Harness uses a single coding-agent proposer with access to a growing filesystem $\mathcal{D}$ that serves as its feedback channel 1. Here, a coding agent is a language-model-based system that can invoke developer tools and modify code. Unlike prior systems that externalize the improvement logic in a hand-designed search loop, Meta-Harness delegates diagnosis and proposal to the coding agent itself: it decides which prior artifacts to inspect, which failure modes to address, and whether to make a local edit or a more substantial rewrite. Equivalently, the proposer is not a raw next-token model operating on a fixed prompt assembled by the outer loop; it is an agent that retrieves information, navigates prior artifacts, and edits code as part of the search itself. Each evaluated harness contributes a directory containing its source code, scores, and execution traces (such as prompts, tool calls, model outputs, and state updates). The filesystem is typically far larger than the proposer’s context window, so the proposer queries it through terminal tools such as grep and cat rather than ingesting it as a single prompt. At each iteration, the proposer first inspects prior code, scores, and execution traces, then reasons about likely failure modes before generating a new harness. + +Meta-Harness maintains a population $\mathcal{H}$ and a Pareto frontier over evaluated harnesses, but imposes no parent-selection rule: the proposer is free to inspect any prior harness and its execution trace when proposing new ones. We run evolution for a fixed number of iterations and perform a final test-set evaluation on the Pareto frontier. This simplicity is deliberate: by leaving diagnosis and edit decisions to the proposer rather than hard-coding search heuristics, Meta-Harness can improve automatically as coding agents become more capable. The proposer never sees test-set results; its only feedback comes from the search set, the subset of task instances used to evaluate candidate harnesses during search and generate the feedback signal for improvement, and from execution traces logged during those search runs. + +Advantages of code-space search. Harness optimization occurs in code space, where small changes to retrieval, memory, or prompt-construction logic can affect behavior many steps later, making local search heuristics poorly matched to the problem. By inspecting execution traces, the proposer can often infer *why* a harness failed and which earlier design choices likely contributed to the failure, not just *that* it failed, as illustrated by the search trajectories in Appendices A and A.2. There, we see that the proposer reads broadly across prior code and logs, then uses those traces to identify confounded edits, isolate likely causal changes, and shift toward safer modifications after repeated regressions. The proposer can therefore modify the harness at the level of algorithmic structure, ranging from changes to retrieval, memory, or prompt-construction logic to full program rewrites, rather than filling in templates or applying predefined mutation operators. In practice, it often starts from a strong prior harness, but this is an emergent strategy rather than a hard-coded rule. Although the search space is large, representing harnesses as programs provides a natural regularization bias: coding models tend to propose coherent algorithms rather than brittle, hard-coded solutions, which biases the search toward reusable context-management procedures. This action space is closely aligned with the read–write–execute workflows on which frontier coding assistants are trained. + +Practical implementation. In our experiments, each harness is a single-file Python program that modifies task-specific prompting, retrieval, memory, and orchestration logic. In our experiments, the proposer $P$ is Claude Code [^4] with Opus-4.6. The proposer is guided by a minimal domain-specific skill that describes where to write new harnesses, how to inspect previous harnesses and their execution traces, and what files it can and cannot modify. The base model $M$ varies by domain and is always frozen; see Section 4 for details. In our experiments, a typical run evaluates roughly 60 harnesses over 20 iterations. We provide additional tips for implementing Meta-Harness in a new domain in Appendix D. + +Algorithm 1 Meta-Harness outer loop over harnesses + +Input: tasks $\mathcal{X}$, LLM $M$, proposer $P$, iterations $N$ + +Initialize: population $\mathcal{H}$ $\triangleright$ Initial set of valid harnesses + +Initialize: filesystem $\mathcal{D}\leftarrow\emptyset$ $\triangleright$ stores code, scores, traces + +for $H\in\mathcal{H}$ do + +   $E_{H}\leftarrow\textrm{Evaluate}(H,M,\mathcal{X})$    $\mathcal{D}\leftarrow\mathcal{D}\cup\{(H,E_{H})\}$ + +for $t=1\ldots N$ do + +  Proposer $P$ queries filesystem $\mathcal{D}$ $\triangleright$ inspects prior harnesses and scores + +  Proposer $P$ proposes $k$ new harnesses $\{H_{1},\dots,H_{k}\}$ + +  for $H$ in $\{H_{1},\dots,H_{k}\}$ do + +   if $H$ passes interface validation then + +      $\mathcal{D}\leftarrow\mathcal{D}\cup\{(H,\textsc{Evaluate}(H,M,\mathcal{X}))\}$       + +return Pareto frontier of harnesses stored in $\mathcal{D}$ + +## 4 Experiments + +We evaluate Meta-Harness on three task domains: online text classification, math reasoning, and agentic coding. In each domain, we compare harnesses discovered by our search against domain-appropriate baselines using the standard evaluation metric. Please refer to each subsection for the precise experimental setup. + +We compare against two main classes of methods. (1) Human-designed strategies: these are hand-crafted harnesses for each domain, representing the current state of the art in context construction. We describe these baselines in the corresponding subsections. (2) Program-search methods: these methods search over candidate harnesses using feedback and reward signals, but are designed for smaller-scale settings than harness engineering. + +### 4.1 Online Text Classification + +We follow the online text classification setup of [^58] [^51]: an LLM receives labeled examples one at a time, updates its memory, and is evaluated on a held-out test set. We use GPT-OSS-120B as the LLM text classifier, and consider the problem of designing a harness for text classification. We use three datasets, chosen for difficulty and domain diversity: LawBench (Law) [^15] predicts criminal charges from case descriptions (215 classes); Symptom2Disease (S2D) [^18] predicts diseases from symptom descriptions (22 classes); and USPTO-50k [^40] predicts precursor reactants from product molecules (180 classes). We initialize the search population $\mathcal{H}$ from the main baseline harnesses in this setting: zero-shot, few-shot, ACE, and MCE. We ran 20 evolution iterations with two candidates per iteration, producing 40 candidate harnesses. + +![[x4.png|Refer to caption]] + +Table 2: Test-set metrics for all harnesses on the three datasets. Ctx denotes additional input tokens in context (thousands). †: implementation from 51. ↓ \\downarrow: lower is better. Meta-Harness improves online text classification accuracy while using a smaller input context. + +Comparison vs text optimizers. We compare Meta-Harness against representative methods for optimizing text. For a fair comparison, we use the same proposer configuration (Opus-4.6 with max reasoning), select candidates solely based on search-set performance, and hold out the test sets until the final evaluation. Since evaluation is the main computational bottleneck, we give each method the same budget of proposal harness evaluations. We consider the following points of comparison: + +- Best-of-N: independent samples from the seed with no search structure; a compute-matched control for whether search matters at all. +- OpenEvolve [^42]: evolutionary search over programs with LLM mutation. +- TTT-Discover [^53]: we use only the text-optimization component of their method, i.e., proposal selection via the PUCT reuse rule. + +In this setting, Meta-Harness matches the best prior text optimizers (OpenEvolve, TTT-Discover) in $0.1\times$ the evaluations, and its final accuracy surpasses theirs by more than 10 points (Figures 1 and 4). We attribute this speedup to the intentional design choices that impose minimum necessary structure on the outer loop (Section 3). In particular, Meta-Harness preserves full experience history using a filesystem and allows the proposer to inspect anything necessary, whereas both OpenEvolve and TTT-Discover operate with more structured and substantially more limited proposer inputs than full filesystem access. We note that online text classification is the smallest-context setting we study (Table 1), so if structure-heavy text optimizers already lag here, their limitations may only grow in harder regimes. + +Meta-Harness is 10 × \times Faster and Converges to a Better Harness In this setting, Meta-Harness matches the best prior text optimizers (OpenEvolve, TTT-Discover) with 10 × 10\times fewer full evaluations, and its final accuracy surpasses theirs by more than 10 points. + +| Method | Scores | Code | Summ. | Traces | Median $\uparrow$ | Best Acc $\uparrow$ | $>$ ZS | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Scores Only | ✓ | ✓ | × | × | 34.6 | 41.3 | 26 | +| Scores + Summary | ✓ | ✓ | ✓ | × | 34.9 | 38.7 | 23 | +| Meta-Harness (full) | ✓ | ✓ | \- | ✓ | 50.0 | 56.7 | 39 | + +Table 3: Ablation of the information available to the proposer in online text classification. $>$ ZS: number of runs whose accuracy exceeded the zero-shot baseline. The full Meta-Harness interface substantially outperforms scores-only and scores-plus-summary ablations. Access to raw execution traces is the key ingredient for enabling harness search. + +To isolate which parts of the proposer interface matter most, we compare three conditions in online text classification: a scores-only condition, a scores-plus-summary condition in which the proposer receives LLM-generated summaries but no raw traces, and the full Meta-Harness interface with access to execution traces (Table 3). The results show a large gap in favor of the full interface: scores-only reaches 34.6 median and 41.3 best accuracy, while scores-plus-summary reaches 34.9 median and 38.7 best. By contrast, Meta-Harness reaches 50.0 median and 56.7 best accuracy, and even its median candidate outperforms the best candidate found under either ablation. We interpret this as evidence that full access to execution traces is the most important component of the interface: summaries do not recover the missing signal, and may even hurt by compressing away diagnostically useful details. + +| Method | Median | Best | +| --- | --- | --- | +| GEPA [^1] | 32.6 | 40.2 | +| Best-of-N | 34.0 | 44.2 | +| OpenEvolve [^42] | 39.1 | 43.3 | +| TTT-Discover [^53] | 34.1 | 45.6 | +| Meta-Harness | 50.0 | 56.7 | + +Table 4: Text classification accuracies of the harnesses proposed by different text optimizers (search set). Meta-Harness is substantially more effective at harness optimization. + +Comparison vs state-of-the-art harnesses. Our primary points of comparison are hand-designed harnesses for this problem setting: Agentic Context Engineering (ACE, [^58]), which uses reflective memory curation to build context over time, and Meta Context Engineering (MCE, [^51]), which maintains and evolves a library of natural-language skills for context construction. As additional baselines, we evaluate zero-shot prompting and few-shot prompting with $N\in\{4,8,16,32,\text{all}\}$ examples. Results in Table 2 show that Meta-Harness improves substantially over prior hand-designed harnesses. The selected Meta-Harness 2 reaches 48.6% accuracy, outperforming ACE by 7.7 points and MCE by 8.6 points. These gains do not come from using more context: Meta-Harness uses only 11.4K context tokens, versus 50.8K for ACE and 28.5K for MCE. + +Accuracy–Context Tradeoffs. Because Meta-Harness performs free-form optimization over harness code, we can express a joint preference for both accuracy and context cost rather than committing to a single scalar objective in advance. Given only the current metrics and the desired trade-off, the proposer is able to discover harnesses across a broad range of the frontier, yielding a smooth accuracy–context Pareto curve in Figure 3. This allows us to trade additional context for higher test accuracy in a controlled way, rather than committing to a single hand-designed operating point. + +Out-of-distribution (OOD) task evaluation. We evaluate whether the discovered harness generalizes to entirely new datasets unseen during search. We consider nine diverse datasets, and describe them in detail in Section C.1. The selected Meta-Harness system achieves the best average accuracy (73.1%), outperforming ACE (70.2%) and all few-shot baselines (Table 5). Notably, we observe that naively adding more few-shot examples beyond $32$ hurts performance in $7/9$ tasks. Meta-Harness shows the highest performance on 6/9 datasets, suggesting that the discovered harness captures generally effective strategies for text classification rather than overfitting to the specific datasets used during search. + +| Harness | SciC | FiNER | Amz5 | FPB | GoEmo | Bank77 | News | SciT | TwHate | Avg Acc | Ctx $\downarrow$ | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Zero-shot | 32.7 | 56.0 | 52.7 | 90.0 | 42.0 | 80.7 | 84.7 | 89.3 | 75.3 | 67.0 | \- | +| Few-shot (8) | 34.0 | 63.0 | 54.0 | 90.0 | 44.0 | 82.7 | 84.7 | 91.3 | 76.7 | 68.9 | 2.2 | +| Few-shot (32) | 38.7 | 62.0 | 53.3 | 90.7 | 43.3 | 86.0 | 85.3 | 90.7 | 76.7 | 69.6 | 5.2 | +| Few-shot (all) | 35.3 | 61.0 | 50.0 | 93.3 | 42.7 | 80.7 | 84.0 | 90.0 | 76.7 | 68.2 | 7.4 | +| ACE [^58] | 40.7 | 74.0 | 48.0 | 96.7 | 44.0 | 83.3 | 86.0 | 90.7 | 68.7 | 70.2 | 11.7 | +| Meta-Harness | 53.3 | 67.0 | 60.0 | 94.0 | 46.0 | 82.7 | 86.7 | 91.3 | 77.3 | 73.1 | 7.3 | + +Table 5: OOD text classification dataset evaluation. We report test accuracy for each dataset and the average additional context tokens across all nine datasets. Meta-Harness outperforms the next best method by 2.9 points on these 9 previously unseen tasks. + +### 4.2 Harnesses for Retrieval-Augmented Reasoning + +We study a somewhat non-standard setup for olympiad math solving: augmenting the model with the ability to retrieve examples from a large corpus. There is a good reason to expect retrieval to help mathematical reasoning in principle, because solutions often share reusable proof patterns, so previous reasoning traces contain information that a model may be able to exploit at inference time. Yet retrieval has not become a standard ingredient in this setting, and prior work suggests that it has been much less successful on reasoning-intensive math benchmarks than in more fact-grounded domains [^41] [^48] [^5]. The difficulty is that naive retrieval rarely surfaces the right traces in the right form. This suggests that success depends less on adding retrieval per se than on discovering the right retrieval policy. Rather than hand-designing that policy, we give Meta-Harness a hard set of olympiad problems and allow the retrieval behavior itself to emerge from search. + +The retrieval corpus contains $\geq$ 500,000 solved problems from eight open-source datasets. We carefully deduplicated and decontaminated it against both evaluation benchmarks and the search set, confirmed that held-out problems have no exact prefix matches under our string-based filter, and manually inspected top BM25 retrievals for held-out examples (Section C.2). We use Meta-Harness to optimize a harness for 40 iterations over a 250-problem search set of Olympiad-difficulty math problems (OlympiadBench + Omni-MATH hard), producing 109 candidate retrieval harnesses. We initialize the search population $\mathcal{H}$ from the main baseline harnesses in this setting: zero-shot, few-shot, and ACE. We select a single harness based on search-set performance using GPT-OSS-20B (Section B.2). We evaluate this harness on $200$ previously unseen IMO-level problems drawn from IMO-AnswerBench, IMO-ProofBench, and ArXivMath [^29] [^5]. In addition to GPT-OSS-20B, we evaluate the same retrieval harness on four models not seen during search: GPT-5.4-nano, GPT-5.4-mini, Gemini-3.1-Flash-Lite, and Gemini-3-Flash. We follow the standard evaluation protocol of prior work [^29] and report accuracy averaged over three samples per problem. + +Results. Table 6 compares the discovered harness against no retrieval, dense retrieval using the separate embedding model text-embedding-3-small, random few-shot prompting, and BM25 retrieval. In contrast, Meta-Harness operates entirely in code space on top of the same BM25-based lexical retrieval stack as the sparse baseline, rather than introducing an additional dense encoder. The discovered retrieval harness outperforms the no-retrieval baseline across all five held-out models, with an average gain of 4.7 points. It also matches or exceeds the strongest fixed baselines on average, outperforming BM25 retrieval by 1.3 points overall, while avoiding the regressions observed with dense retrieval and random few-shot prompting across several models. + +| Method | GPT-5.4n | GPT-5.4m | Gem-3.1FL | Gem-3F | GPT-20B | Avg. | +| --- | --- | --- | --- | --- | --- | --- | +| No Retriever | 23.0 | 28.8 | 28.6 | 42.6 | 47.6 | 34.1 | +| Dense Retrieval ($k{=}1$) | 27.1 (+4.1) | 24.5 (-4.3) | 31.3 (+2.7) | 42.3 (-0.3) | 46.9 (-0.7) | 34.4 (+0.3) | +| Dense Retrieval ($k{=}5$) | 31.1 (+8.1) | 28.3 (-0.5) | 37.1 (+8.5) | 47.2 (+4.6) | 46.7 (-0.9) | 38.1 (+4.0) | +| Random Few-shot | 23.1 (+0.1) | 24.5 (-4.3) | 31.0 (+2.4) | 40.4 (-2.2) | 41.8 (-5.8) | 32.2 (-1.9) | +| BM25 Retrieval | 30.2 (+7.2) | 29.2 (+0.4) | 32.8 (+4.2) | 46.6 (+4.0) | 48.9 (+1.3) | 37.5 (+3.4) | +| Meta-Harness | 31.7 (+8.7) | 30.4 (+1.6) | 34.9 (+6.3) | 46.3 (+3.7) | 50.6 (+3.0) | 38.8 (+4.7) | + +Table 6: Retrieval-augmented math problem solving on 200 IMO-level math problems. We show pass@1 averaged over three samples per problem, with absolute improvement over the baseline in parentheses. The discovered Meta-Harness retrieval strategy improves reasoning on these IMO-level problems across all five held-out models, with a 4.7-point average gain over no retriever. + +Meta-Harness Improves Reasoning on IMO-Level Math Problems In retrieval-augmented math reasoning, a single discovered retrieval harness transfers across five held-out models, improving accuracy by 4.7 points on average over no retrieval and yielding the strongest overall average among the compared methods. + +### 4.3 Evaluating Agentic Coding Harnesses on TerminalBench-2 + +TerminalBench-2 [^32] evaluates LLM agents on 89 challenging tasks that require long-horizon, fully autonomous execution under complex dependencies, and substantial domain knowledge. Prior work has shown that the choice agent harness has a large effect on performance on this benchmark. We initialize search from two strong open baselines, Terminus 2 [^32] and Terminus-KIRA [^24]. For this experiment, we perform search and final evaluation on the same 89-task benchmark. We use this benchmark as a discovery problem [^54] in which the goal is to discover a harness configuration that improves performance on a hard, publicly contested benchmark. This is standard practice: public writeups already describe repeated benchmark-specific harness iteration on TerminalBench itself [^17] [^33] [^24], and the benchmark is small and expensive enough that introducing a separate split would materially weaken the search signal. We additionally check for overfitting by manual inspection and regex-based audits for task-specific string leakage into evolved harnesses. We note that although the resulting harness is specialized to the TerminalBench-2 regime, autonomous completion of difficult long-horizon tasks from a single instruction is a core capability, and the benchmark consists of many tasks that frontier models and heavily engineered harnesses struggle with. + +
HarnessAutoPass (%)
Claude Opus 4.6
Claude Code×58.0 58.0
Terminus 2×62.9 62.9
Mux×66.5 66.5
Droid×69.9 69.9
TongAgents×71.9 71.9
MAYA-V2×72.1 72.1
Terminus-KIRA×74.7 74.7
Capy×75.3 75.3
ForgeCode×81.8 81.8
Meta-Harness✓76.4 \mathbf{76.4}
Claude Haiku 4.5
OpenHands×13.9 13.9
Claude Code×27.5 27.5
Terminus 2×28.3 28.3
Mini-SWE-Agent×29.8 29.8
Terminus-KIRA×33.7 33.7
Goose×35.5 35.5
Meta-Harness✓37.6 \mathbf{37.6}
+ +Table 7: Pass rate on TerminalBench-2. Results or others are from the official leaderboard. Meta-Harness ranks #2 among all Opus-4.6 agents and #1 among all Haiku-4.5 agents on this competitive task. + +Results. We report results on the full benchmark in Table 7, evaluated on two base models: Claude Opus 4.6 and Claude Haiku 4.5. On Opus 4.6, Meta-Harness discovers a harness achieving 76.4% pass rate, surpassing the hand-engineered Terminus-KIRA (74.7%) and ranking #2 among all Opus 4.6 agents on the TerminalBench-2 leaderboard. The only higher-scoring Opus 4.6 agent is ForgeCode (81.8%); however, we were unable to reproduce their reported result from the publicly available code alone, suggesting their leaderboard scores depend on components beyond the published repository. On the weaker Haiku 4.5 model, the improvement is larger: Meta-Harness achieves 37.6%, outperforming the next-best reported agent (Goose, 35.5%) by 2.1 points. TerminalBench-2 is an actively contested benchmark with multiple teams directly optimizing for it, so the fact that an automatic search method can achieve benefits at this frontier is encouraging for long-horizon text-optimization loops. + +Qualitative behavior of the proposer. The harness search trajectory helps explain why Meta-Harness achieves these gains; we provide a detailed summary in Appendix A. In early iterations, the proposer combined plausible structural fixes with prompt-template edits and observed that both candidates regressed. It then explicitly hypothesized that the regressions were confounded by the shared prompt intervention, isolated the structural changes from the prompt rewrite, and ultimately pivoted toward a safer additive modification that became the best candidate in the run. This provides qualitative evidence that filesystem access enables the proposer to inspect prior experience in enough detail to form causal hypotheses and revise the harness accordingly. + +Meta-Harness Surpasses Hand-Engineered Agents on TerminalBench-2 On TerminalBench-2, Meta-Harness automatically discovers harnesses that surpass Terminus-KIRA on Opus 4.6 and rank #1 among all Haiku 4.5 agents. + +## 5 Discussion + +Beyond outperforming existing harnesses, Meta-Harness has several practical advantages. Discovered harnesses generalize to out-of-distribution classification datasets (Table 5) and to unseen base models in the math setting (Table 6). A search run completes in a few hours of wall-clock time, yet produces readable, transferable strategies that can be reused across models, including future, stronger ones. Overfitting in code space is also more inspectable: brittle if-chains or hard-coded class mappings are visible on inspection in a way that weight-space overfitting is not. More broadly, our results suggest that the main advantage of Meta-Harness is not just search over code, but search with *selective access to prior diagnostic experience*. The proposer is not limited to scalar rewards or fixed summaries; it can inspect raw code, execution traces, and prior failures, then use that information to form and test hypotheses about what to change. The qualitative search trajectories in Section A.2 illustrate this behavior directly. + +Our findings reflect a recurring pattern in machine learning [^44]: once a search space becomes accessible, stronger general-purpose agents can outperform hand-engineered solutions. A natural next step for future work is to co-evolve the harness and the model weights, letting the strategy shape what the model learns and vice versa. While we evaluate on three diverse domains, our experiments demonstrate that harness search can work with one particularly strong coding-agent proposer (Claude Code); a broader study of how the effect varies across proposer agents remains for future work. + +## Acknowledgements + +We thank KRAFTON AI for providing API credit support. This work is supported by OpenAI, KFAS, and Schmidt Sciences AI2050. We thank Anikait Singh and Jubayer Ibn Hamid for their valuable feedback and suggestions, and Sienna J. Lee for patiently listening to YL’s half-formed thoughts during the early stages of this work. + +## References + +![[x5.png|Refer to caption]] + +Figure 4: Search-set accuracy over evaluations for all compared text optimizers on online text classification. Each point is one candidate harness; lines track the best-so-far. Per-dataset curves are shown alongside the aggregate. Meta-Harness reaches the final accuracy of OpenEvolve and TTT-Discover within the first 4 evaluations and continues improving, ending more than 10 points above all baselines. + +## Appendix A Qualitative Proposer Behavior + +This section examines how the proposer uses the filesystem during search, drawing on the TerminalBench-2 run (10 iterations, Claude Opus 4.6). + +### A.1 File Access Statistics + +To verify that the proposer makes substantive use of the filesystem rather than defaulting to local edits, we recorded all file reads per iteration. + +Table 8 summarizes the results. The proposer reads a median of 82 files per iteration (range 69–99), roughly evenly split between prior harness source code (41%) and execution traces (40%), with the remainder going to score summaries (6%) and other files (13%). This confirms that the proposer’s access pattern is non-Markovian: it routinely inspects the majority of available history rather than conditioning only on the most recent parent. + +
StatisticValue
Files read per iteration (median)82
Files read per iteration (range)69–99
File type breakdown
Harness source code41%
Execution traces40%
Score/summary files6%
Other13%
+ +Table 8: Proposer file access statistics from the TerminalBench-2 search run (10 iterations, Claude Opus 4.6). The proposer reads extensively from the filesystem, with roughly equal attention to prior source code and execution traces. + +### A.2 Qualitative Behavior: Causal Reasoning Over Prior Failures + +The TerminalBench-2 search log reveals a clear narrative arc in which the proposer learns from its own regressions. Rather than wandering randomly through local edits, it forms an explicit diagnosis of why early candidates failed, then shifts toward a safer design pattern. All text inside the log boxes below is quoted verbatim from the proposer’s recorded reasoning at each iteration (emphasis ours). + +Iterations 1–2: promising bugfixes are confounded by prompt edits. The first two iterations both bundle plausible structural fixes with prompt-template modifications, and both regress sharply from the 64.4% Terminus-KIRA baseline. Iteration 1 targets observation corruption from leaked terminal markers and adds a loop breaker: + +Hypothesis: __CMDEND__ marker fragments leak into LLM observations on long-running tasks, causing the model to get confused and enter infinite no-tool-call loops. Stripping these markers + adding a loop breaker will recover wasted steps. + +That candidate also introduced a new cleanup-oriented prompt template and a verification checklist. Iteration 2 proposes a different state-machine fix: + +Double-confirmation completion mechanism causes verification spirals. Observed in trajectories where the agent solves the task early but burns 15--40+ additional steps re-verifying because each verification command resets _pending_completion, requiring another task_complete → \rightarrow checklist → \rightarrow verify cycle. + +This second candidate removes the pending-completion mechanism entirely, while also carrying over the marker stripping and the new prompt. It still regresses, which gives the proposer two failed candidates with different structural changes but one shared prompt intervention. + +Iteration 3: the proposer identifies the confound. By iteration 3, the proposer explicitly infers that the regressions are not primarily due to the structural bugfixes themselves: + +Prior attempts: evo_marker_fix (58.9%, -5.6pp), evo_single_confirm (57.8%, -6.7pp) --- both regressed. Root cause of regressions: Prompt template changes (cleanup directives) caused the agent to delete necessary state before task completion. The structural bugfixes were confounded with harmful prompt changes. evo_strip_only isolates the two proven structural fixes. + +This is the key causal step in the trajectory. The proposer notices that the common factor across the first two failures is not the particular bugfix, but the cleanup-heavy prompt rewrite. It therefore reverts to the original prompt and tests only the marker-stripping and loop-breaker. The resulting candidate still underperforms slightly (63.3%, -1.1pp), but it loses far less than the earlier versions, which supports the confound diagnosis. + +Iterations 4–6: direct fixes to the diagnosed failure mode still regress. The next three iterations continue to probe the same part of the design space, but now with more explicit theories about why the completion logic is fragile. Iteration 4 attributes failures to a concrete state-machine bug in which verification commands reset the completion flag and trap the agent in repeated checklist cycles: + +Remove the two self._pending_completion = False lines that reset the completion flag when intermediate commands run. This fixes a state machine bug where: (1) Agent calls task_complete → \rightarrow sees QA checklist, _pending_completion = True (2) Agent runs verification commands → \rightarrow _pending_completion = False (bug!) (3) Agent calls task_complete again → \rightarrow sees checklist AGAIN → \rightarrow infinite loop. + +The proposer even cites concrete trajectory evidence, noting that configure-git-webserver produced baseline failures with agents stuck in 30–60 step verification spirals after effectively solving the task. Iteration 5 tries to soften the cleanup language while preserving confirmation, but still edits the prompt and regresses badly. Iteration 6 returns to the safer evo\_strip\_only base and proposes a systems-level optimization: + +Empty-command turns waste full LLM round-trips when terminal output hasn’t changed. Smart-waiting (poll pane up to 3 × \times 5s) before the next LLM call saves 5--15 turns on long-running tasks. + +That change also regresses. By this point, the proposer has learned a specific empirical lesson: modifications to prompts and completion flow are high risk, even when the local hypothesis sounds reasonable. + +Iteration 7: the winning candidate. After six consecutive regressions, the proposer shifts strategy from modifying the control loop to adding information before the loop begins: + +All 6 prior iterations regressed from the 64.4% baseline because they modified the completion flow, prompt template, or observation processing. evo_env_bootstrap takes a different approach --- purely additive. It gathers an environment snapshot via a single shell command before the first LLM call and appends it to the initial prompt. No other methods are changed. This should eliminate 3--5 wasted exploration turns on dependency-heavy tasks without risking regression on already-passing tasks. + +This candidate is the best result so far. The important point is not just that iteration 7 wins, but that the proposer articulates *why* it should be safer: it avoids touching the previously fragile completion machinery and instead adds information that is useful mainly on hard tasks. + +Iteration 8: composition. Having found one additive improvement, the proposer next attempts to compose it with an earlier structural fix: + +Combining two orthogonal fixes --- env snapshot (saves early exploration turns) + marker stripping with no-tool-call loop breaker --- will yield +1--3pp because they address independent failure modes without touching prompts or confirmation flows (which caused regressions in 5 of 7 prior iterations). + +Iteration 10: cross-run transfer. The proposer references results from a separate earlier search run: + +The evolution history showed ‘‘don’t cleanup service artifacts’’ was worth +18pp. Iter 9 (evo_no_cleanup_directive) targeted the same idea but crashed before evaluation. + +Summary. The search trajectory demonstrates that the proposer does more than random mutation. Across the first seven iterations, it identifies a confound, tests the confound-isolating hypothesis directly, observes that control-flow and prompt edits remain fragile, and then deliberately pivots to a purely additive modification that becomes the best candidate in the run. It subsequently tries to compose that winning idea with earlier fixes and even transfers lessons across runs. This kind of causal reasoning over prior failures is precisely what full-history filesystem access enables and what compressed-feedback optimizers cannot support. + +## Appendix B Discovered Harnesses + +Meta-Harness discovers executable inference-time procedures specific to the problem setup at hand. These harnesses are structured, domain-specific policies, often with nontrivial control flow such as routing, filtering, and conditional context construction, selected solely by whether they improve search-set performance. This section presents compact, method-style abstractions of representative harnesses that summarize the main behaviors and control-flow decisions that drive inference-time behavior. For reference, the full implementation for each discovered harness is on the order of 100–1000 lines of code. + +### B.1 Text Classification Harness + +In online text classification, Meta-Harness discovers a family of memory-based harnesses rather than a single canonical policy. Table 9 reports the Pareto frontier of non-dominated variants from the main search, all selected solely by search-set performance. We highlight two representative endpoints here: Meta-Harness (Draft Verification), the lowest-context frontier point, and Meta-Harness (Label-Primed Query), the highest-accuracy frontier point used in the main text. + +#### Overview. + +Both harnesses maintain a growing memory of past labeled examples and build prompts from that memory at inference time. What differs is the control flow used to interrogate the memory. Meta-Harness (Draft Verification) uses two short calls and explicitly tests the model’s first guess against retrieved counterexamples, while Meta-Harness (Label-Primed Query) spends a larger single-call budget on making the label space and local decision boundaries explicit. Figures 5 and 6 summarize these two programs. + +#### Meta-Harness (Draft Verification). + +The corresponding discovered file is draft\_verification.py. This lightweight variant turns prediction into a two-call procedure. It first retrieves the 5 most similar labeled examples and makes a draft prediction. It then re-queries the same memory conditioned on that draft label, retrieving 5 *confirmers* with the same label and 5 *challengers* with different labels, and asks the model whether to maintain or revise its initial answer. The key discovered behavior is that the second retrieval depends on both the query and the draft prediction, so the harness can surface counterexamples targeted at the model’s current guess rather than only generic near neighbors. If too few labeled examples have been accumulated, the program falls back to a standard single-call few-shot prompt. + +Query + memory Retrieve top-5 similar examples Draft call initial label D D Retrieve confirmers (= = D D) and challengers (≠ \neq D D) Verification call keep or revise D D Final label D D + +Figure 5: Draft-verification classification harness. The first call produces a draft label from a short retrieved context. The second call retrieves evidence for and against that draft and returns the final prediction. + +- Stage 1: Draft. Retrieve the 5 nearest labeled examples and ask for an initial prediction. +- Stage 2: Verification. Condition retrieval on the draft label, then show both supporting and challenging examples before making the final prediction. +- Cold start. If fewer than 5 labeled examples are available, skip the two-stage procedure and use a standard single-call few-shot prompt. +- Why it is cheap. Both calls use short retrieved contexts, so the overall context cost stays near the low end of the frontier even with two model invocations. + +#### Meta-Harness (Label-Primed Query). + +The corresponding discovered file is label\_primed\_query\_anchored.py. This strongest variant uses a single larger call built from three parts. It begins with a *label primer* listing the valid output labels, then constructs a *coverage* section with one query-relevant example per label, and finally adds *query-anchored contrastive pairs* that place highly similar examples with different labels side by side. The coverage block exposes the full label space, while the contrastive block sharpens local decision boundaries around the current query. In code, the harness implements this with TF-IDF retrieval over past labeled examples and a query-anchored pairing rule that chooses contrasting examples from the same local neighborhood. + +Query + memory Label primer all valid labels TF-IDF retrieval query-anchored pairing Coverage block best example per label Contrastive pairs similar examples different labels Assemble one prompt with primer, coverage, and contrastive pairs Final label + +Figure 6: Label-primed query-anchored classification harness. The program builds a single prompt that exposes the label space, then populates it with query-relevant coverage examples and local contrastive pairs. + +- Label primer. List the valid output labels before showing any examples, so the model sees the full answer space up front. +- Coverage block. For each known label, retrieve the most query-relevant labeled example and include one representative example per class. +- Contrastive block. Build pairs of highly similar examples with different labels, so the prompt exposes local decision boundaries around the current query. +- Retrieval rule. Use TF-IDF similarity and query-anchored partner selection rather than label-agnostic nearest neighbors. + +
DatasetsAvg metrics
VariantUSPTO ↑ \uparrowSymptom ↑ \uparrowLawBench ↑ \uparrowAvg ↑ \uparrowCtx ↓ \downarrow
Meta-Harness (Draft Verification)18.085.4 85.417.0 17.040.1 40.15.4 5.4
Meta-Harness (Error-Annotated)9.0 9.087.7 87.724.0 24.040.2 40.222.3 22.3
Meta-Harness (CoT Replay)13.0 13.088.2 88.225.0 25.042.1 42.123.3 23.3
Meta-Harness (Cluster Coverage)12.0 12.086.8 86.833.0 33.043.9 43.931.2 31.2
Meta-Harness (Cascade Retrieval)12.0 12.086.8 86.836.0 36.044.9 44.939.2 39.2
Meta-Harness (RRF + Contrastive)18.089.6 89.635.0 35.047.5 47.541.4 41.4
Meta-Harness (Relevance + Contrastive)18.090.636.0 36.048.2 48.243.9 43.9
Meta-Harness (Label-Primed Query)14.0 14.086.8 86.845.048.645.5 45.5
+ +Table 9: Pareto-optimal discovered variants from the main text-classification search, trading off average accuracy against context cost. The selected system in the main text is Meta-Harness (Label-Primed Query). Ctx denotes average additional characters in input context (thousands). + +![[val_vs_test_by_dataset.png|Refer to caption]] + +Figure 7: Search-set vs. test accuracy per dataset for discovered text-classification strategies. Each pink dot is a discovered strategy; baselines are labeled. The dashed diagonal is y = x y{=}x. + +### B.2 Math Retrieval Harness + +This subsection describes the retrieval harness discovered by Meta-Harness for mathematical reasoning (Section 4.2). The final harness is a compact four-route BM25 program whose structure emerged through search rather than being manually specified after the fact. All design choices below—the routing predicates, reranking terms, deduplication thresholds, and per-route example counts—were selected by the outer loop across 40 iterations of evolution. + +#### Overview. + +At inference time, the harness assigns each problem to exactly one of four routes: combinatorics, geometry, number theory, or a default route for algebra and other problems. The gates are implemented as lightweight lexical predicates over the problem statement, including keyword sets and a small number of regex features for geometry notation. The harness does not aggregate outputs across routes: once a route is selected, only that route retrieves examples for the final prompt. All routes use BM25 as the underlying retrieval mechanism over the filtered corpus described above. The BM25 index uses a math-aware tokenizer that preserves LaTeX tokens (e.g., \\frac, ˆ{2}) as atomic units. The selected harness is a merge of two successful search lineages, autonomously combined by the proposer during search: one contributed a stronger geometry route based on raw BM25, while another contributed a stronger combinatorics route based on deduplication and difficulty reranking. Figure 8 gives a compact flowchart view of the final program. + +Query Lexical router keyword and regex cues Combinatorics BM25@20 Dedup to 8 Rerank Keep 3 Geometry 1 fixed ref + 2 BM25 No rerank Number theory BM25@12 Rerank Keep 3 Algebra/Other BM25@10 Rerank Adaptive K K Build final prompt + +Figure 8: Discovered math retrieval harness. A lexical router assigns each query to one of four subject-specific retrieval policies. The selected policy retrieves examples, which are inserted into the final prompt. + +- Combinatorics: fetch 20 BM25 candidates, deduplicate to 8, rerank by lexical score and difficulty, then return the top 3. This is the main route where the harness explicitly trades off diversity against hard-problem matching. +- Geometry: return 1 hard NuminaMath reference together with 2 raw BM25 neighbors. Search consistently prefers raw structural matches here over difficulty reranking. +- Number theory: fetch 12 BM25 candidates and rerank using lexical score, difficulty, and a small bonus for solutions that state a technique early. This favors examples whose proof strategy is explicit. +- Default: fetch 10 BM25 candidates, rerank by lexical score and difficulty, and choose an adaptive number of examples based on how concentrated the top retrieval scores are. + +### B.3 TerminalBench-2 Harness + +The discovered TerminalBench-2 harness builds on Terminus-KIRA [^24], inheriting its native tool calling (replacing Terminus 2’s ICL-based JSON parsing), 30KB output cap, and multi-perspective completion checklist. The main modification discovered by Meta-Harness is environment bootstrapping: before the agent loop begins, the harness runs a compound shell command to gather a snapshot of the sandbox environment and injects it into the initial prompt. The proposer’s hypothesis, recorded verbatim from the search log, was: + +Hypothesis: ‘‘Injecting an environment snapshot (OS, installed languages, package managers, /app contents) before the first LLM turn will reduce wasted exploration episodes by 3--5 turns on dependency-heavy tasks’’ Changes: ‘‘Added _gather_env_snapshot() that runs a single compound shell command to collect working directory, /app listing, available languages (python, gcc, node, java, rustc, go), package managers (pip, apt) […] and injects as [Environment Snapshot] block’’ + +The snapshot includes: the working directory, a listing of /app (truncated to 20 entries for large directories), available programming languages and their versions (Python, GCC, G++, Node, Java, Rust, Go), installed package managers (pip, apt-get), and available memory. This eliminates the 2–4 exploratory turns that agents typically spend discovering what tools and files are available, allowing the model to begin productive work immediately. The bootstrapping command is guarded by a 15-second timeout and fails silently, so it does not break the agent in unusual environments. The full implementation adds roughly 80 lines on top of Terminus-KIRA. Figure 9 summarizes the harness structure. + +#### Per-task analysis. + +Compared to Terminus-KIRA, the discovered harness gains on 7 of 89 tasks, with the largest improvements on protein-assembly and path-tracing. The gaining tasks share a common property: they require domain-specific tooling whose availability cannot be assumed in advance (bioinformatics libraries, rendering pipelines, chess engines, cryptographic utilities, CoreWars simulators). Without the bootstrap, the agent spends its first 2–4 turns probing the environment; on tasks with tight turn budgets or where early wrong assumptions cascade, those wasted turns can be the difference between pass and fail. This suggests that the bootstrap’s value is largest when the environment is non-obvious, and the task requires the agent to match its strategy to what is actually installed. + +Task instruction Env bootstrap pwd, files, languages, pkg managers, memory Initial prompt task + snapshot Agent loop native tool calling 30KB output cap Multi-perspective completion checklist Task complete pass fail + +Figure 9: Discovered TerminalBench-2 harness. The harness inherits Terminus-KIRA’s native tool calling, output cap, and completion checklist (green). The environment bootstrap (red) is the component discovered by Meta-Harness: it gathers a sandbox snapshot before the agent loop begins, eliminating early exploratory turns. + +## Appendix C Dataset Details + +### C.1 OOD Text Classification Datasets + +- SciCite is a 3-way citation-intent classification benchmark introduced by [^13]. Each example consists of a citation context from a scientific paper, labeled by the citation’s rhetorical role, such as background, method, or result. The task tests whether a model can infer why one paper cites another from the local scientific context. +- FiNER-139 is a financial numeric entity recognition benchmark introduced by [^28]. It consists of word-level annotations from financial filings with 139 fine-grained XBRL entity types, making it substantially more fine-grained than standard sentence-level classification tasks. The benchmark tests whether a model can identify and classify numeric financial entities from context. +- Amazon Reviews is the English portion of the Multilingual Amazon Reviews Corpus introduced by [^21]. In our setting, it is used as a 5-way review rating prediction task, where the label corresponds to the review’s star rating. This benchmark evaluates general-domain sentiment and rating prediction from product review text. +- Financial PhraseBank is a 3-way financial sentiment benchmark introduced by [^31]. It consists of sentences from financial news and related economic text labeled as positive, neutral, or negative with respect to market sentiment. The task evaluates domain-specific sentiment classification in finance. +- GoEmotions is a fine-grained emotion classification benchmark introduced by [^14]. It contains English Reddit comments annotated with 27 emotion categories plus a neutral category, and is commonly treated as a 28-way classification task. The benchmark tests nuanced affect recognition beyond coarse positive-negative sentiment. +- Banking77 is a fine-grained intent classification benchmark introduced by [^10]. It contains online banking user utterances labeled with 77 intents, covering a wide range of customer service requests. The task evaluates single-domain intent detection with a large label space. +- AG News is a 4-way news topic classification benchmark commonly associated with the text classification setup of [^59]. Examples are labeled with broad news categories such as world, sports, business, and science/technology. It is a standard general-domain benchmark for topic classification. +- SciTail is a science-domain textual entailment benchmark in which the task is to predict whether a hypothesis is entailed by a premise sentence in a science-focused inference setting [^23]. +- TweetEval (Hate) is the hate-speech subset of the TweetEval benchmark introduced by [^6]. It is a binary tweet classification task for detecting hateful versus non-hateful content within a unified social-media evaluation suite. This benchmark tests robust classification in noisy, short-form social media text. + +### C.2 Math Retrieval Corpus + +Table 10 lists the datasets composing the retrieval corpus used in Section 4.2. The raw sources contain more problems than the final corpus; several filtering steps were applied before merging. NuminaMath-1.5 was filtered to competition-math subsets (AMC/AIME, olympiad references, number theory, inequalities, and related sources), discarding lower-quality web-scraped entries. OpenMathReasoning was deduplicated to one solution per problem (retaining the solution with the highest pass rate on an independent verifier), and problems whose source matched any evaluation benchmark family (IMO, AIME, HMMT, SMT, USAMO, Putnam) were removed before deduplication. The entire corpus was then decontaminated against all evaluation benchmarks and the search set used during harness search, using exact prefix matching followed by fuzzy Jaccard similarity (threshold 0.8); any corpus problem matching an eval problem under either criterion was discarded. Solutions from OpenMathReasoning and DeepMath are truncated to 5,000 characters to limit retrieval context length. At runtime, the selected harness further restricts retrieval to entries with non-empty solutions shorter than 4,000 characters. Retrieved solutions are truncated again to 3,000 characters when inserted into the prompt. For the geometry route, the harness also constructs a separate hard-reference index from NuminaMath problems with difficulty greater than 6. + +| Dataset | Problems | Sol. Len | Proof | +| --- | --- | --- | --- | +| [OpenMathReasoning](https://huggingface.co/datasets/nvidia/OpenMathReasoning) | 281,743 | 5,000 † | 34% | +| [DeepMath-103K](https://huggingface.co/datasets/zwhe99/DeepMath-103K) | 103,021 | 5,000 † | 0% | +| [NuminaMath-1.5](https://huggingface.co/datasets/AI-MO/NuminaMath-1.5) | 129,520 | 1,376 | 13% | +| [PolyMath](https://huggingface.co/datasets/AIMO-Corpus/PolyMath) | 11,083 | 363 | 0% | +| [Omni-MATH](https://huggingface.co/datasets/KbsdJames/Omni-MATH) | 4,289 | 829 | 0% | +| [FineProofs-SFT](https://huggingface.co/datasets/SPIderman5/FineProofs-SFT) | 4,275 | 3,977 | 100% | +| [AIME 1983–2024](https://huggingface.co/datasets/gneubig/aime-1983-2024) | 933 | — | 0% | +| [Putnam-AXIOM](https://huggingface.co/datasets/Putnam-AXIOM/putnam-axiom-dataset-v1) | 492 | 888 | 100% | +| Total | 535,356 | 5,000 † | 22% | + +† Truncated at 5,000 characters; actual solutions are longer. + +Table 10: Datasets in the math retrieval corpus (535K problems total). Sol. Len is the median solution length in characters. Proof indicates whether the dataset contains proof-type problems (by answer or problem type field). + +### C.3 Math IMO-level Test Set + +The main text aggregates results over 200 IMO-level problems drawn from IMO-AnswerBench, IMO-ProofBench, ArXivMath December 2025, and ArXivMath January 2026. The 200-problem evaluation set consists of a stratified 100-problem subset of IMO-AnswerBench, together with all problems from the other three benchmarks. This per-benchmark breakdown is useful because the four datasets mix answer-style, proof, and research-style problems, which are aggregated together in the main paper for brevity. When included, the table in this section should report each benchmark separately for both Base and Meta-Harness across the five held-out models. + +| Dataset | Problems | +| --- | --- | +| IMO-AnswerBench | 100 | +| IMO-ProofBench | 60 | +| ArXivMath Dec. 2025 | 17 | +| ArXivMath Jan. 2026 | 23 | +| Total | 200 | + +Table 11: Breakdown of the 200-problem IMO-level evaluation set. + +## Appendix D Practical Implementation Tips + +Meta-Harness is largely domain-agnostic: we expect it to apply in any setting where a language model is wrapped by a task-specific harness. Applying it in a new domain, however, requires operating in a relatively new regime of LLM-assisted coding, where the proposer conditions on long-horizon histories of prior runs and writes programs whose effects may only become visible many steps later. In getting this workflow to work reliably, we found a small set of practical choices that mattered consistently across the three domains studied in this paper. The guidelines below are not themselves scientific claims about the method; they are engineering lessons from building and running the system, which we hope will make it easier for future work to apply Meta-Harness in other domains. + +- Write a good skill. The skill text is the primary interface for steering the search, and its quality is the strongest lever on whether the loop works. The proposer receives a natural-language skill [^3] that defines its role, the directory layout, CLI commands, and output format. In practice, the skill should constrain outputs and safety-relevant behavior, not the proposer’s diagnosis procedure: it should specify what is forbidden, what artifacts to produce, and what objectives to optimize, while leaving the model free to inspect scores, traces, and prior code as needed. Our intuition from inspecting logs from Meta-Harness runs is that after enough iterations, the accumulated traces often shape the proposer’s behavior more than the skill itself. In our experience, iterating on the skill text had a larger effect on search quality than changing iteration count or population size. Expect to run a few short evolution runs (3–5 iterations each) specifically to debug and refine the skill before committing to a full run. +- Start with a baseline harness and a search set that is hard for it. Write a simple baseline (e.g., few-shot prompting), then construct the search set by either filtering for examples that the baseline gets wrong or selecting a diverse subset of difficult instances. The search has little to optimize if the baseline already saturates the evaluation. Keep the search set small enough for roughly 50 full evaluations per run (50–100 examples in our classification experiments, 88 problems for math retrieval); a fast, discriminative eval is more valuable than a large one. +- Log everything in a format that is easy to navigate. Evaluation code should write code, scores, and execution traces in a form that the proposer can query reliably. In practice, this means using machine-readable formats such as JSON, organizing artifacts hierarchically, choosing reasonable and consistent file names, and adopting naming schemes that make simple tools such as regex search work well. +- Make logs queryable through a small CLI (optional, but helpful). Each harness gets a directory containing source code, scores, and execution traces, but as the history grows, raw filesystem access alone becomes cumbersome. A short CLI that lists the Pareto frontier, shows top- $k$ harnesses, and diffs code and results between pairs of runs can make the experience store much easier to use, and querying such CLIs is closely aligned with the workflows on which coding agents are trained. If relevant offline experience exists (rollouts from other models, solved problem corpora, relevant papers), converting it into the same directory structure can also help warm-start exploration and ground new ideas. This layer helps the proposer save tokens it may have wasted on navigation. +- Lightweight validation before expensive benchmarks. Write a small validation test that imports the module, instantiates the class, and calls both methods on a tiny set of examples. Harnesses proposed during the search should pass this test before being fully evaluated. A simple test script can catch most malformed or nonfunctional candidates in seconds and keep the cost of failures near zero. +- Automate evaluation outside the proposer. Running evals is simple enough that it is not worth making the proposer do it. A separate harness should score candidates and write results to the filesystem. + +## Appendix E Extended Related Work + +This appendix expands the brief discussion in Section 2 and situates Meta-Harness relative to several neighboring lines of work that we could not cover in detail in the main text. A recurring distinction is that Meta- Harness optimizes executable harness implementations and provides the proposer with selective access to prior code, scores, and execution traces via the filesystem. + +#### AlphaEvolve / OpenEvolve. + +AlphaEvolve [^34] and OpenEvolve [^42] evolve code via LLM-guided mutations with structured feedback: the proposer receives a program database with scalar scores (4–22K tokens per step; Table 1) and applies fixed mutation strategies to tournament-selected parents. These methods are designed for algorithm discovery and optimization (mathematical conjectures, scheduling heuristics, hardware kernels), where the search target is a single stateless function with a clean scalar objective, and mutations are local. Harness engineering is a different regime: harnesses are stateful programs that accumulate experience across many examples, and a single design choice (e.g., what to store in memory) can cascade through an entire evaluation sequence. Meta-Harness addresses this by giving an unstructured coding agent full filesystem access, letting it selectively read any prior candidate’s source code, execution traces, and scores. + +#### GEPA. + +GEPA [^1] is the closest text optimizer in terms of feedback richness, providing rollout traces per candidate. It is designed for prompt optimization on tasks with short feedback loops (math problems, instruction-following, code optimization), where each rollout is a single LLM call or a short pipeline. In this regime, per-candidate reflection works well: one prompt, one answer, one score. Harness engineering requires reasoning across many examples and many candidates simultaneously: understanding why a retrieval strategy works for one class of problems but degrades on another requires comparing execution traces across the full population. GEPA operates on one candidate at a time (2–8K tokens per step; Table 1), with a fixed critique format that must anticipate what information is relevant. Meta-Harness gives the proposer access to all prior candidates simultaneously and lets the agent decide what to examine. + +#### Prompt orchestration frameworks. + +Several systems provide structured abstractions for composing multi-stage LLM programs. LMQL [^7], LangChain [^12], and DSPy [^22] make prompt engineering more systematic by providing higher-level interfaces for prompt templates, control flow, and modular LLM pipelines. These frameworks help developers specify and organize LLM programs, but they still typically require manual design of retrieval policies, memory updates, and orchestration logic. Meta-Harness operates at a different level: it searches over the *implementation* of these policies in executable code, treating the harness itself as the optimization target. + +[^1]: L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, et al. (2025) Gepa: reflective prompt evolution can outperform reinforcement learning. arXiv preprint arXiv:2507.19457. Cited by: Appendix E, Table 1, §1, §2, Table 4. + +[^2]: M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. De Freitas (2016) Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems 29. Cited by: §2. + +[^3]: Anthropic and community contributors Agentskills/agentskills. Note: GitHub repository [https://github.com/agentskills/agentskills](https://github.com/agentskills/agentskills) Specification and documentation for Agent Skills, accessed March 27, 2026 Cited by: 1st item. + +[^4]: Anthropic (2025) Claude code: an agentic coding tool. Note: [https://www.anthropic.com/claude-code](https://www.anthropic.com/claude-code) Cited by: §3. + +[^5]: M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev (2025-02) MathArena: evaluating llms on uncontaminated math competitions. SRI Lab, ETH Zurich. External Links: [Link](https://matharena.ai/) Cited by: §4.2, §4.2. + +[^6]: F. Barbieri, J. Camacho-Collados, L. Neves, and L. Espinosa-Anke (2020) TweetEval: unified benchmark and comparative evaluation for tweet classification. External Links: 2010.12421, [Link](https://arxiv.org/abs/2010.12421) Cited by: 9th item. + +[^7]: L. Beurer-Kellner, M. Fischer, and M. Vechev (2023-06) Prompting is programming: a query language for large language models. Proceedings of the ACM on Programming Languages 7 (PLDI), pp. 1946–1969. External Links: ISSN 2475-1421, [Link](http://dx.doi.org/10.1145/3591300), [Document](https://dx.doi.org/10.1145/3591300) Cited by: Appendix E. + +[^8]: B. Böckeler (2026-03) Harness engineering. Note: [https://martinfowler.com/articles/exploring-gen-ai/harness-engineering.html](https://martinfowler.com/articles/exploring-gen-ai/harness-engineering.html) martinfowler.com Cited by: §1. + +[^9]: C. Bölük (2026-02) I improved 15 LLMs at coding in one afternoon. only the harness changed.. Note: [https://blog.can.ac/2026/02/12/the-harness-problem/](https://blog.can.ac/2026/02/12/the-harness-problem/) Cited by: §1. + +[^10]: I. Casanueva, T. Temčinas, D. Gerz, M. Henderson, and I. Vulić (2020) Efficient intent detection with dual sentence encoders. External Links: 2003.04807, [Link](https://arxiv.org/abs/2003.04807) Cited by: 6th item. + +[^11]: M. Cemri, S. Agrawal, A. Gupta, S. Liu, A. Cheng, Q. Mang, A. Naren, L. E. Erdogan, K. Sen, M. Zaharia, et al. (2026) AdaEvolve: adaptive llm driven zeroth-order optimization. arXiv preprint arXiv:2602.20133. Cited by: §1. + +[^12]: LangChain Note: Software, released 2022-10-17 External Links: [Link](https://github.com/langchain-ai/langchain) Cited by: Appendix E. + +[^13]: A. Cohan, W. Ammar, M. van Zuylen, and F. Cady (2019) Structural scaffolds for citation intent classification in scientific publications. External Links: 1904.01608, [Link](https://arxiv.org/abs/1904.01608) Cited by: 1st item. + +[^14]: D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi (2020) GoEmotions: a dataset of fine-grained emotions. External Links: 2005.00547, [Link](https://arxiv.org/abs/2005.00547) Cited by: 5th item. + +[^15]: Z. Fei, X. Shen, D. Zhu, F. Zhou, Z. Han, A. Huang, S. Zhang, K. Chen, Z. Yin, Z. Shen, et al. (2024) Lawbench: benchmarking legal knowledge of large language models. In Proceedings of the 2024 conference on empirical methods in natural language processing, pp. 7933–7962. Cited by: §4.1. + +[^16]: C. Finn, P. Abbeel, and S. Levine (2017) Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, Cited by: §2. + +[^17]: ForgeCode (2025) Benchmarks don’t matter. External Links: [Link](https://forgecode.dev/blog/benchmarks-dont-matter/) Cited by: §4.3. + +[^18]: Gretel AI (2023) Symptom to diagnosis dataset. Note: [https://huggingface.co/datasets/gretelai/symptom\_to\_diagnosis](https://huggingface.co/datasets/gretelai/symptom_to_diagnosis) Accessed: 2026-01-22 Cited by: §4.1. + +[^19]: S. Hu, C. Lu, and J. Clune (2025) Automated design of agentic systems. In The Thirteenth International Conference on Learning Representations, External Links: [Link](https://openreview.net/forum?id=t9U3LW7JVX) Cited by: §2. + +[^20]: A. Justin Young (2025-11) Effective harnesses for long-running agents. Note: [https://anthropic.com/engineering/effective-harnesses-for-long-running-agents](https://anthropic.com/engineering/effective-harnesses-for-long-running-agents) Anthropic Engineering Blog Cited by: §1. + +[^21]: P. Keung, Y. Lu, G. Szarvas, and N. A. Smith (2020) The multilingual amazon reviews corpus. External Links: 2010.02573, [Link](https://arxiv.org/abs/2010.02573) Cited by: 3rd item. + +[^22]: O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts (2023) DSPy: compiling declarative language model calls into self-improving pipelines. External Links: 2310.03714, [Link](https://arxiv.org/abs/2310.03714) Cited by: Appendix E. + +[^23]: T. Khot, A. Sabharwal, and P. Clark (2018-Apr.) SciTaiL: a textual entailment dataset from science question answering. Proceedings of the AAAI Conference on Artificial Intelligence 32 (1). External Links: [Link](https://ojs.aaai.org/index.php/AAAI/article/view/12022), [Document](https://dx.doi.org/10.1609/aaai.v32i1.12022) Cited by: 8th item. + +[^24]: KRAFTON AI and Ludo Robotics (2026) Terminus-kira: boosting frontier model performance on terminal-bench with minimal harness. External Links: [Link](https://github.com/krafton-ai/kira) Cited by: §B.3, §4.3. + +[^25]: Y. Lee, J. Boen, and C. Finn (2025) Feedback descent: open-ended text optimization via pairwise comparison. In arXiv preprint arXiv:2511.07919, Cited by: Table 1, §1, §2. + +[^26]: J. Lehman, J. Gordon, S. Jain, K. Ndousse, C. Yeh, and K. O. Stanley (2022) Evolution through large models. External Links: 2206.08896, [Link](https://arxiv.org/abs/2206.08896) Cited by: §2. + +[^27]: P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, pp. 9459–9474. Cited by: §1, §2. + +[^28]: L. Loukas, M. Fergadiotis, I. Chalkidis, E. Spyropoulou, P. Malakasiotis, I. Androutsopoulos, and G. Paliouras (2022) FiNER: financial numeric entity recognition for xbrl tagging. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4419–4431. External Links: [Link](http://dx.doi.org/10.18653/v1/2022.acl-long.303), [Document](https://dx.doi.org/10.18653/v1/2022.acl-long.303) Cited by: 2nd item. + +[^29]: T. Luong, D. Hwang, H. H. Nguyen, G. Ghiasi, Y. Chervonyi, I. Seo, J. Kim, G. Bingham, J. Lee, S. Mishra, A. Zhai, C. H. Hu, H. Michalewski, J. Kim, J. Ahn, J. Bae, X. Song, T. H. Trinh, Q. V. Le, and J. Jung (2025) Towards robust mathematical reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, External Links: [Link](https://aclanthology.org/2025.emnlp-main.1794/) Cited by: §4.2. + +[^30]: A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. (2023) Self-refine: iterative refinement with self-feedback. Advances in neural information processing systems 36, pp. 46534–46594. Cited by: §1, §2. + +[^31]: P. Malo, A. Sinha, P. Takala, P. Korhonen, and J. Wallenius (2013) Good debt or bad debt: detecting semantic orientations in economic texts. External Links: 1307.5336, [Link](https://arxiv.org/abs/1307.5336) Cited by: 4th item. + +[^32]: M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y. Shin, T. Walshe, E. K. Buchanan, et al. (2026) Terminal-bench: benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868. Cited by: §4.3. + +[^33]: J. Nichols (2025-06) How we scored #1 on terminal-bench (52%). External Links: [Link](https://www.warp.dev/blog/terminal-bench) Cited by: §4.3. + +[^34]: A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. Ruiz, A. Mehrabian, et al. (2025) Alphaevolve: a coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131. Cited by: Appendix E, Table 1, §1, §2. + +[^35]: OpenAI (2026-02) Harness engineering: leveraging Codex in an agent-first world. Note: [https://openai.com/index/harness-engineering/](https://openai.com/index/harness-engineering/) OpenAI Blog Cited by: §1. + +[^36]: C. Packer, V. Fang, S. Patil, K. Lin, S. Wooders, and J. Gonzalez (2023) MemGPT: towards llms as operating systems.. Cited by: §1, §2. + +[^37]: R. Pryzant, D. Iter, J. Li, Y. T. Lee, C. Zhu, and M. Zeng (2023) Automatic prompt optimization with “gradient descent” and beam search. arXiv preprint arXiv:2305.03495. Cited by: §1, §2. + +[^38]: B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, et al. (2024) Mathematical discoveries from program search with large language models. Nature 625 (7995), pp. 468–475. Cited by: §1, §2. + +[^39]: J. Schmidhuber (1993) A neural network that embeds its own meta-levels. In IEEE International Conference on Neural Networks, Cited by: §2. + +[^40]: N. Schneider, N. Stiefl, and G. A. Landrum (2016) What’s what: the (nearly) definitive guide to reaction role assignment. Journal of chemical information and modeling 56 (12), pp. 2336–2346. Cited by: §4.1. + +[^41]: S. Shakya, A. Hartl, S. Hochreiter, and K. Pöppel (2026) Adaptive retrieval helps reasoning in llms – but mostly if it’s not used. External Links: 2602.07213, [Link](https://arxiv.org/abs/2602.07213) Cited by: §4.2. + +[^42]: A. Sharma (2025) OpenEvolve: an open-source evolutionary coding agent. Note: [https://github.com/algorithmicsuperintelligence/openevolve](https://github.com/algorithmicsuperintelligence/openevolve) GitHub repository External Links: [Link](https://github.com/algorithmicsuperintelligence/openevolve) Cited by: Appendix E, §2, 2nd item, Table 4. + +[^43]: J. Snell, K. Swersky, and R. S. Zemel (2017) Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, Cited by: §2. + +[^44]: R. Sutton (2019) The bitter lesson, 2019. URL http://www. incompleteideas. net/IncIdeas/BitterLesson. html. Cited by: §5. + +[^45]: S. Thrun and L. Pratt (1998) Learning to learn: introduction and overview. In Learning to learn, pp. 3–17. Cited by: §2. + +[^46]: M. Tian, Z. Wang, B. Yang, Z. Tang, K. Zhu, H. Dong, H. Li, X. Xie, G. Wang, and J. You (2026) SWE-bench mobile: can large language model agents develop industry-level mobile applications?. In arXiv preprint, External Links: [Link](https://api.semanticscholar.org/CorpusID:285462974) Cited by: §1. + +[^47]: H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal (2023) Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. External Links: 2212.10509, [Link](https://arxiv.org/abs/2212.10509) Cited by: §1, §2. + +[^48]: C. Xiao, G. T. Hudson, and N. A. Moubayed (2024) RAR-b: reasoning as retrieval benchmark. External Links: 2404.06347, [Link](https://arxiv.org/abs/2404.06347) Cited by: §4.2. + +[^49]: Y. Xiong, S. Hu, and J. Clune (2026) Learning to continually learn via meta-learning agentic memory designs. In OpenReview, External Links: [Link](https://api.semanticscholar.org/CorpusID:285454009) Cited by: §2. + +[^50]: C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen (2023) Large language models as optimizers. In The Twelfth International Conference on Learning Representations, Cited by: Table 1, §1, §2. + +[^51]: H. Ye, X. He, V. Arak, H. Dong, and G. Song (2026) Meta context engineering via agentic skill evolution. arXiv preprint arXiv:2601.21557. Cited by: Table 2, §4.1, §4.1, Table 2. + +[^52]: M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, Z. Huang, C. Guestrin, and J. Zou (2024) TextGrad: automatic ”differentiation” via text. External Links: 2406.07496, [Link](https://arxiv.org/abs/2406.07496) Cited by: Table 1, §1, §2. + +[^53]: M. Yuksekgonul, D. Koceja, X. Li, F. Bianchi, J. McCaleb, X. Wang, J. Kautz, Y. Choi, J. Zou, C. Guestrin, et al. (2026) Learning to discover at test time. arXiv preprint arXiv:2601.16175. Cited by: 3rd item, Table 4. + +[^54]: M. Yuksekgonul, D. Koceja, X. Li, F. Bianchi, J. McCaleb, X. Wang, J. Kautz, Y. Choi, J. Zou, C. Guestrin, and Y. Sun (2026) Learning to discover at test time. External Links: 2601.16175, [Link](https://arxiv.org/abs/2601.16175) Cited by: Table 1, §4.3. + +[^55]: A. L. Zhang, T. Kraska, and O. Khattab (2026) Recursive language models. External Links: 2512.24601, [Link](https://arxiv.org/abs/2512.24601) Cited by: §1, §2. + +[^56]: G. Zhang, H. Ren, C. Zhan, Z. Zhou, J. Wang, H. Zhu, W. Zhou, and S. Yan (2025) Memevolve: meta-evolution of agent memory systems. arXiv preprint arXiv:2512.18746. Cited by: §2. + +[^57]: J. Zhang, J. Xiang, Z. Yu, F. Teng, X. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, B. Zheng, B. Liu, Y. Luo, and C. Wu (2025) AFlow: automating agentic workflow generation. External Links: 2410.10762, [Link](https://arxiv.org/abs/2410.10762) Cited by: §2. + +[^58]: Q. Zhang, C. Hu, S. Upasani, B. Ma, F. Hong, V. Kamanuru, J. Rainton, C. Wu, M. Ji, H. Li, U. Thakker, J. Zou, and K. Olukotun (2025) Agentic context engineering: evolving contexts for self-improving language models. In arXiv preprint arXiv:2510.04618, Cited by: §1, Table 2, §4.1, §4.1, Table 5. + +[^59]: X. Zhang, J. Zhao, and Y. LeCun (2016) Character-level convolutional networks for text classification. External Links: 1509.01626, [Link](https://arxiv.org/abs/1509.01626) Cited by: 7th item. \ No newline at end of file diff --git a/raw/MiniMax M2.7 开启模型的自我进化.md b/raw/MiniMax M2.7 开启模型的自我进化.md new file mode 100644 index 0000000..d3b8797 --- /dev/null +++ b/raw/MiniMax M2.7 开启模型的自我进化.md @@ -0,0 +1,116 @@ +--- +title: "MiniMax M2.7: 开启模型的自我进化" +source: "https://www.minimaxi.com/news/minimax-m27-zh" +author: + - "[[MiniMax]]" +published: 2026-04-11 +created: 2026-04-11 +description: "MiniMax是全球领先的通用人工智能科技公司，致力于\"与所有人共创智能\"，自主研发了一系列多模态通用大模型，并面向全球推出一系列AI原生产品，已服务逾 2亿名用户, MiniMax是全球领先的通用人工智能科技公司，致力于\"与所有人共创智能\"，自主研发了一系列多模态通用大模型，并面向全球推出一系列AI原生产品，已服务逾 2亿名用户, MiniMax是全球领先的通用人工智能科技公司，致力于\"与所有人共创智能\"，自主研发了一系列多模态通用大模型，并面向全球推出一系列AI原生产品，已服务逾 2亿名用户" +tags: + - "clippings" +--- +![[Image.png|https://filecdn.minimax.chat/public/7a85723c-acc3-4eb8-a3a8-f53713d0822e.png]] + +在 M2 系列模型发布后的几个月，我们收到了大量热心用户的反馈和建议，这促使我们进一步加速模型的迭代效率。除了更加认真工作之外，我们能找到的唯一途径就是开启模型和组织的自我进化。MiniMax M2.7是我们第一个模型深度参与迭代自己的模型。 + +M2.7 能够自行构建复杂 Agent Harness，并基于 Agent Teams、复杂 Skills、Tool Search Tool 等能力，完成高度复杂的生产力任务。例如，在研发M2.7的过程中，我们基于模型构建强化学习Harness 中的数十个复杂的 skills，更新自己的 memory，驱动模型自身的强化学习，并基于结果优化强化学习过程和Harness，开启模型的自我进化。 + +1\. M2.7 在真实的软件工程中有优异的表现，包括端到端的完整项目交付，分析日志排查 Bug、代码安全，机器学习等。在基准测试 SWE-Pro 中，M2.7 得分56.22%，几乎接近Opus最好的水平。这一能力同样延伸到了端到端的完整项目交付场景（VIBE-Pro 55.6%）以及对复杂工程系统的深层理解Terminal Bench 2（57.0%）。 + +2\. 在专业办公领域，我们提升了模型在各领域的专业知识和任务交付能力，在 GDPval-AA 的ELO得分是1495，为开源最高。M2.7 对 Office 三件套 Excel/PPT/Word 的复杂编辑能力显著提升，能更好地完成多轮修改和高保真的编辑。M2.7具备与复杂环境交互的能力，M2.7 在 40 个复杂 skills (> 2000 Token) 的 case 上，仍能保持 97% 的 skills 遵循率。在OpenClaw的使用中，M2.7相比于M2.5也有了显著的提升，在MMClaw的评测中接近最新的Sonnet 4.6。 + +3\. M2.7具备优秀的身份保持能力和情商，除了生产力使用外，给互动娱乐场景的创新也准备了空间。 + +基于上述能力，M2.7 也在显著加速我们自身向一个 AI Native 组织的进化。 + +![[img-1.png|MiniMax M2.7 基准测试对比]] + +## 构建模型自我进化智能体 + +在最开始，我们分享一个我们内部让M2系列模型自我进化的实践，这也是对模型Agent能力边界的探索。 + +Agent Harness通常依赖复杂的Skills、记忆系统和其他组件来提升模型对不同工作环境的适应能力。在此基础上，我们在 M2 的早期版本中，将其引导为一个研究型 Agent Harness——它能够与不同的研究项目组进行交互和协作。该系统覆盖了数据流水线、训练环境、评测基础设施、跨团队协作、持久化记忆——让研究员可以驱动它来交付更好的模型。研究 Agent 驱动着产出下一代模型的迭代循环。研究员在每一层引导方向，模型在每一层负责构建。 + +以一个 RL 场景为例：研究员从一个实验想法出发，与 Agent 展开讨论。Agent 协助进行文献调研，持续跟踪预设的实验规格，完成数据流水线及其他对接工作，并启动实验。实验运行期间，它会自动监控和分析实验状态，并自动触发日志读取、问题排查、指标分析、代码修复、合并请求以及冒烟测试，识别并配置那些细微但关键的变更。这些工作过去可能需要来自不同团队的多位同事协作完成，而现在研究员只需在关键决策和讨论时介入。这大幅加速了问题发现和实验迭代，从而更快地交付模型。在这个场景下，M2.7 能够胜任 30-50% 的工作流。 + +![[d080353d-31aa-4642-8586-3b7f2201bf94.png|M2* 模型迭代系统架构]] + +我们在迭代过程中也意识到，模型自主迭代 harness 的能力也至关重要。我们内部的 harness 会自主收集反馈，建立内部任务的评测集，并基于此不断迭代自己的 Agent架构、Skills/MCP 实现和记忆机制，来更好和更高效的完成任务。 + +举个例子，我们让 M2.7 优化一个内部脚手架上模型的软件工程开发表现。M2.7 全程自主运行，执行"分析失败轨迹 → 规划改动 → 修改脚手架代码 → 运行评测 → 对比结果 → 决定保留或回退"的迭代循环超过 100 轮。这个过程中 M2.7 发现了针对模型的有效优化：系统性搜索温度、频率惩罚、存在惩罚等采样参数的最优组合；为模型设计更具体的工作流指引（如修复后自动搜索其他文件中的相同 bug 模式）；在脚手架的Agent Loop中添加循环检测等优化。最终在内部评测集上效果提升 30%。 + +我们相信，未来的 AI 自我进化会逐步向完全自动化过渡，包括完全自主的协调数据构建、模型训练、推理架构、评测等等。我们用 M2.7 参与了MLE Bench Lite的22 个机器学习任务测试，几乎囊括了研发的所有环节。 + +我们设计和实现了一个简易的脚手架来引导 Agent 进行自主优化，核心的模块包括短时记忆、自反馈以及自优化三个模块。具体来讲，Agent 完成每轮迭代后会形成一个短时记忆文件，同时对当前轮次的结果进行自反馈，从而给下一轮次提供潜在的优化方向，下一轮次基于所有历史轮次的记忆及自反馈链进行下一步的自优化。我们总共测试三次，每次有 24 小时来迭代进化，从下图中能够看到 M2.7 随时间不断取得更高的性能。最好的一次取得 9 枚金牌，5 枚银牌，1枚铜牌。三次平均是 66.6% 的得牌率，此成绩仅次于Opus-4.6 (75.7%)、GPT-5.4 (71.2%)，和 Gemini-3.1 (66.6%) 持平。 + +![[img-3.png|MLE Bench 得牌率随时间变化]] + +## 真实的软件工程 + +在编程等代码生成类任务上，M2.7 更深入地打磨了真实软件工程所需的编程能力，覆盖日志分析与 Bug 定位、代码重构、代码安全、机器学习、安卓开发等方向。 + +以线上最常见的线上生产环境故障调试为例——这类任务不仅需要生成代码，还需要很强的综合推理能力。面对我们实际的生产环境告警，M2.7 能关联监控指标与部署时间线做因果推理，对轨迹采样做统计分析并提出精准假设，主动连接数据库执行验证根因，定位到代码仓库中缺失的索引迁移文件，甚至知道用非阻塞建索引先止血，再提 MR。从可观测性分析、数据库专业知识到 SRE 级别的决策判断——这不只是一个会写代码的模型，而是一个真正理解生产系统的模型。相比传统的人工排障流程，基于 M2.7，我们已多次将线上生产系统故障的恢复时间缩短到三分钟以内。 + + + +【线上生产环境故障调试】 + +在单项编程能力上，M2.7 已具备国际一线模型水准。 在涵盖多种编程语言的 SWE-Pro 中，M2.7 以 56.22% 的正确率追平 GPT-5.3-Codex；而在更贴近真实工程场景的 SWE Multilingual（76.5）和 Multi SWE Bench（52.7）中展现出更显著优势。 + +这一能力同样延伸到了端到端的完整项目交付场景。 在 Repo 级代码生成基准 VIBE-Pro 上，M2.7 得分55.6%，几乎与 Opus 4.6持平 —— 这意味着无论是 Web、Android、iOS 还是 Simulation 类需求，都可以直接交给 M2.7 完成。 + +更值得关注的，是对复杂工程系统的深层理解。 在对系统认知要求极高的 Terminal Bench 2（57.0%）和 NL2Repo（39.8%）中，M2.7 同样表现稳健，进一步印证了它不只擅长代码生成，更能深入理解软件系统的运行逻辑与协作流程。 + + + +【基于M2.7生成的WildGuard演示网页】 + +为了提升开发效率，一个比较重要的特性是原生的 Agent Teams （多智能体协作）。Agent Teams 对模型提出了范式级要求：角色边界、对抗性推理、协议遵循、行为分化——这些无法通过提示词，必须内化为模型的原生能力。Agent Teams 场景下，模型需要稳定锚定角色身份、主动挑战队友的逻辑与伦理盲区、在复杂状态机中自主决策。下面是我们内部使用的一个做产品原型开发的 Agent Teams，里面包含了做产品原型的一个最小组织。 + +![[dd83d19a-156b-481f-9835-9576293c323a.gif|Agent Teams 协作模拟演示]] + +【Agent Teams 协作模拟演示】 + +## 专业办公 + +除了软件工程外，Agent开始在办公场景中变得越来越有用，我们认为这是两个核心能力： + +1\. 专业知识与任务交付能力。 模型需要具备各领域的专业知识，理解用户的需求。在衡量这一能力的 GDPval-AA 的评测中，M2.7 在45个模型中的ELO得分是1500，仅次于Opus 4.6、Sonnet 4.6和GPT5.4，超过了GPT5.3。在最常见的办公文件处理上，我们系统性地优化了模型处理 Word、Excel 和 PPT 的能力。M2.7 在各种 Agent Harness上，既能够基于模版和 skills 直接生成文件，也能够遵从用户的交互指令，对已有的文件做多轮的高保真编辑，最终给出可编辑的产物。 + +2\. 与复杂环境的交互能力。 泛化的日常场景意味着模型需要灵活适应各类上下文、调用各种 skills 和工具、并在长程交互中保持稳定的指令遵循。M2.7 在这些方面有大幅提升。在 Toolathon 上，M2.7的正确率 46.3%，达到了全球第一梯队水平。真实工作场景中的 Agent Harness 往往也需要理解和调用大量复杂的 skills。在 MM Claw 的测试中，M2.7 在 40 个复杂 skills (> 2000 Token) 的 case 上，仍能保持 97% 的 skills 遵循率。 + +我们测试了在Finance领域的专业水准，与上代模型相比，模型的能力提升显著。例如，在Finance领域一个阅读研报并建模公司未来营收的场景，M2.7可以自主阅读公司的年报与业绩沟通会纪要，交叉比对多篇研报，独立设计假设并构建营收预测模型，再基于模版产出PPT和研究报告——像一个初级分析师一样理解、判断、输出，并在多轮交互中自我修正。从业者的评价是：产出物已经可以作为初稿直接进入后续工作流程。下面是一个对台积电的例子。 + +> 任务：基于台积电年报和业绩沟通会信息，构建台积电的营收模型，读取多个研报，设计对应的假设，基于最新的信息对台积电营收建模，然后基于PPT模版产出PPT，并写一个Word文档研究报告。 + +[\[TSMC\_Revenue\_Model.xlsx\]](https://filecdn.minimax.chat/public/0dff594f-3226-421e-a7f3-b4d55273ee5f.xlsx) + +[\[TSMC\_Financial\_Analysis.pptx\]](https://filecdn.minimax.chat/public/504a47e3-8d71-4dd0-9397-8c7214bb26d7.pptx) + +[\[TSMC\_Equity\_Research\_Report.docx\]](https://filecdn.minimax.chat/public/7bd2f0a5-06e2-406f-818d-7bf5bd64eca8.docx) + + + +最近OpenClaw 爆火为代表的Agent社区蓬勃发展，我们很高兴M2系列的模型帮助到了社区的繁荣。我们基于OpenClaw中的常用任务，构建了一个评测集 MM Claw，涵盖个人学习规划，到办公文档的处理与交付、定时的专业信息调研与投资建议、代码开发与维护等工作与生活中形形色色的真实需求。M2.7 在这个测试中达到了接近Sonnet 4.6的水平，正确率是62.7%。 + +## 互动娱乐 + +在 OpenClaw 等 Agent 脚手架的使用过程中，不少用户在使用 Agent 完成工作的同时，还希望模型具备比较高的情商和复杂人设保持能力。在有人设的情况下，用户不再只是让模型机械完成任务，而是开始自然于与Agent"相处"。 这促使我们思考，产品与交互设计、内容创作、甚至娱乐体验的构建，都可以被 AI 原生驱动的可能性。我们认为这会让 Agentic 模型的使用从单纯的生产力能进一步拓展到互动娱乐。为此，我们在 M2.7 中极大加强了人设保持和对话能力。 + +基于此，我们构建了一个 Agent 交互系统 OpenRoom，它将 AI 互动置入一个万物皆可互动的 Web GUI 空间。在这里，对话即驱动，实时产生视觉反馈与场景交互，角色可以主动地与环境交互。我们认为这个框架扩展性较高，能够随着模型 Agentic 能力的提升和社区的共建持续进化，探索出更多人与 Agent 之间全新的交互方式。为了促进这个领域的创新，我们已将这个原型项目开源 （这个里面的代码大部分也是 AI 写的）： + +项目地址： [github.com/MiniMax-AI/OpenRoom](https://github.com/MiniMax-AI/OpenRoom) + +立即体验： [openroom.ai](https://openroom.ai/) + + + +MiniMax M2.7 已在 MiniMax Agent 与开放平台上全量上线，期待用户和开发者朋友们能在MiniMax M2.7上探索出更多有趣场景。 + +MiniMax Agent： [agent.minimaxi.com](https://agent.minimaxi.com/) + +API服务： [platform.minimaxi.com](https://platform.minimaxi.com/) + +Coding Plan订阅： [platform.minimaxi.com/subscribe/coding-plan](https://platform.minimaxi.com/subscribe/coding-plan) + +Intelligence with Everyone. \ No newline at end of file diff --git a/raw/mitchellh-ai-adoption-journey.md b/raw/My AI Adoption Journey.md similarity index 74% rename from raw/mitchellh-ai-adoption-journey.md rename to raw/My AI Adoption Journey.md index 753b810..df68702 100644 --- a/raw/mitchellh-ai-adoption-journey.md +++ b/raw/My AI Adoption Journey.md @@ -1,15 +1,19 @@ -# My AI Adoption Journey +--- +title: "My AI Adoption Journey" +source: "https://mitchellh.com/writing/my-ai-adoption-journey" +author: +published: 2026-02-05 +created: 2026-04-11 +description: +tags: + - "clippings" +--- +## Mitchell Hashimoto + +## My AI Adoption Journey Table of Contents -- [Step 1: Drop the Chatbot](#step-1-drop-the-chatbot) -- [Step 2: Reproduce Your Own Work](#step-2-reproduce-your-own-work) -- [Step 3: End-of-Day Agents](#step-3-end-of-day-agents) -- [Step 4: Outsource the Slam Dunks](#step-4-outsource-the-slam-dunks) -- [Step 5: Engineer the Harness](#step-5-engineer-the-harness) -- [Step 6: Always Have an Agent Running](#step-6-always-have-an-agent-running) -- [Today](#today) - My experience adopting any meaningful tool is that I've necessarily gone through three phases: (1) a period of inefficiency (2) a period of adequacy, then finally (3) a period of workflow and life-altering discovery. In most cases, I have to force myself through phase 1 and 2 because I usually have a workflow I'm already happy and comfortable with. Adopting a tool feels like work, and I *do not* want to put in the effort, but I usually do in an effort to be a well-rounded person of my craft. @@ -18,7 +22,7 @@ This is my journey of how I found value in AI tooling and what I'm trying next w This blog post was fully written by hand, in my own words. I hate that I have to say that but especially given the subject matter, I want to be explicit about it. -* * * +--- ## Step 1: Drop the Chatbot @@ -30,9 +34,9 @@ While I was still a heavy AI skeptic, my first "oh wow" moment was pasting a scr But when I tried to reproduce that behavior for other tasks, I was left disappointed. In the context of brownfield projects, I found the chat interface produced poor results very often, and I found myself very frustrated copying and pasting code and command output to and from the interface. It was very obviously far less efficient than me doing the work myself. -To find value, you *must* use an **agent**. An agent is the industry-adopted term for an LLM that can chat and invoke external behavior in a loop[1](#user-content-fn-1) At a bare minimum, the agent must have the ability to: read files, execute programs, and make HTTP requests. +To find value, you *must* use an **agent**. An agent is the industry-adopted term for an LLM that can chat and invoke external behavior in a loop [^1] At a bare minimum, the agent must have the ability to: read files, execute programs, and make HTTP requests. -* * * +--- ## Step 2: Reproduce Your Own Work @@ -44,19 +48,19 @@ This was *excruciating*, because it got in the way of simply getting things done But, expertise formed. I quickly discovered for myself from first principles what others were already saying, but discovering it myself resulted in a stronger fundamental understanding. -1. Break down sessions into separate clear, actionable tasks. Don't try to "draw the owl" in one mega session. -2. For vague requests, split the work into separate planning vs. execution sessions. -3. If you give an agent a way to verify its work, it more often than not fixes its own mistakes and prevents regressions. +1. Break down sessions into separate clear, actionable tasks. Don't try to "draw the owl" in one mega session. +2. For vague requests, split the work into separate planning vs. execution sessions. +3. If you give an agent a way to verify its work, it more often than not fixes its own mistakes and prevents regressions. More generally, I also found the edges of what agents -- at the time -- were good at, what they weren't good at, and for the tasks they were good at how to achieve the results I wanted. All of this led to significant efficiency gains, to the point where I was starting to naturally use agents in a way that I felt was no slower than doing it myself (but I still didn't feel it was any faster, since I was mostly babysitting an agent). -The negative space here is worth reiterating: part of the efficiency gains here were understanding when *not* to reach for an agent. Using an agent for something it'll likely fail at is obviously a big waste of time and having the knowledge to avoid that completely leads to time savings[2](#user-content-fn-3). +The negative space here is worth reiterating: part of the efficiency gains here were understanding when *not* to reach for an agent. Using an agent for something it'll likely fail at is obviously a big waste of time and having the knowledge to avoid that completely leads to time savings [^2]. At this stage, I was finding adequate value with agents that I was happy to use them in my workflow, but still didn't feel like I was seeing any net efficiency gains. I didn't care though, I was content at this point with AI as a tool. -* * * +--- ## Step 3: End-of-Day Agents @@ -64,15 +68,15 @@ To try to find some efficiency, I next started up a new pattern: **block out the Similar to the previous task, I at first found this both unsuccessful and annoying. But, I once again quickly found different categories of work that were really helpful: -- **Deep research sessions** where I'd ask agents to survey some field, such as finding all libraries in a specific language with a specific license type and producing multi-page summaries for each on their pros, cons, development activity, social sentiment, etc. -- **Parallel agents attempting different vague ideas I had but didn't have time to get started on.** I didn't expect them to produce something I'd ever ship here, but perhaps could illuminate some unknown unknowns when I got to the task the next day. -- **Issue and PR triage/review.** Agents are good at using `gh` (GitHub CLI), so I manually scripted a quick way to spin up a bunch in parallel to triage issues. I would NOT allow agents to respond, I just wanted reports the next day to try to guide me towards high value or low effort tasks. +- **Deep research sessions** where I'd ask agents to survey some field, such as finding all libraries in a specific language with a specific license type and producing multi-page summaries for each on their pros, cons, development activity, social sentiment, etc. +- **Parallel agents attempting different vague ideas I had but didn't have time to get started on.** I didn't expect them to produce something I'd ever ship here, but perhaps could illuminate some unknown unknowns when I got to the task the next day. +- **Issue and PR triage/review.** Agents are good at using `gh` (GitHub CLI), so I manually scripted a quick way to spin up a bunch in parallel to triage issues. I would NOT allow agents to respond, I just wanted reports the next day to try to guide me towards high value or low effort tasks. To be clear, I did not go as far as others went to have agents running in loops all night. In most cases, agents completed their tasks in less than half an hour. But, the latter part of the working day, I'm usually tired and coming out of flow and find myself too personally inefficient, so shifting my effort to spinning up these agents I found gave me a "warm start" the next morning that got me working more quickly than I would've otherwise. I was happy, and I was starting to feel like I was doing more than I was doing prior to AI, if only slightly. -* * * +--- ## Step 4: Outsource the Slam Dunks @@ -88,7 +92,7 @@ Importantly, I think the "work on something else" helps counteract the highly pu At this point I was firmly in the "no way I can go back" territory. I felt more efficient, but even if I wasn't, the thing I liked the most was that I could now focus my coding and thinking on tasks I really loved while still adequately completing the tasks I didn't. -* * * +--- ## Step 5: Engineer the Harness @@ -98,14 +102,12 @@ I don't know if there is a broad industry-accepted term for this yet, but I've g This comes in two forms: -1. **Better implicit prompting (AGENTS.md).** For simple things, like the agent repeatedly running the wrong commands or finding the wrong APIs, update the `AGENTS.md` (or equivalent). Here is [an example from Ghostty](https://github.com/ghostty-org/ghostty/blob/ca07f8c3f775fe437d46722db80a755c2b6e6399/src/inspector/AGENTS.md). Each line in that file is based on a bad agent behavior, and it almost completely resolved them all. - -2. **Actual, programmed tools.** For example, scripts to take screenshots, run filtered tests, etc etc. This is usually paired with an AGENTS.md change to let it know about this existing. - +1. **Better implicit prompting (AGENTS.md).** For simple things, like the agent repeatedly running the wrong commands or finding the wrong APIs, update the `AGENTS.md` (or equivalent). Here is [an example from Ghostty](https://github.com/ghostty-org/ghostty/blob/ca07f8c3f775fe437d46722db80a755c2b6e6399/src/inspector/AGENTS.md). Each line in that file is based on a bad agent behavior, and it almost completely resolved them all. +2. **Actual, programmed tools.** For example, scripts to take screenshots, run filtered tests, etc etc. This is usually paired with an AGENTS.md change to let it know about this existing. **This is where I'm at today.** I'm making an earnest effort whenever I see an agent do a Bad Thing to prevent it from ever doing that bad thing again. Or, conversely, I'm making an earnest effort for agents to be able to verify they're doing a Good Thing. -* * * +--- ## Step 6: Always Have an Agent Running @@ -119,24 +121,24 @@ The "have an agent running at all times" goal is still just a goal. I'd say righ **I don't want to run agents for the sake of running agents.** I only want to run them when there is a task I think would be truly helpful to me. Part of the challenge of this goal is improving my own workflows and tools so that I can have a constant stream of high quality work to do that I can delegate. Which, even without AI, is important! -* * * +--- ## Today And that's where I'm at today. -Through this journey, I've personally reached a point where I'm having success with modern AI tooling and I believe I'm approaching it with the proper measured view that is grounded in reality. I really don't care one way or the other if AI is here to stay[3](#user-content-fn-4), I'm a software craftsman that just wants to build stuff for the love of the game. +Through this journey, I've personally reached a point where I'm having success with modern AI tooling and I believe I'm approaching it with the proper measured view that is grounded in reality. I really don't care one way or the other if AI is here to stay [^3], I'm a software craftsman that just wants to build stuff for the love of the game. The whole landscape is moving so rapidly that I'm sure I'll look back at this post very quickly and laugh at my naivete. But, as they say, if you can't be embarassed about your past self, you're probably not growing. I just hope I'll grow in the right direction! -I have no skin in the game here[4](#user-content-fn-5), and there are of course other reasons behind utility to avoid using AI. I fully respect anyone's individual decisions regarding it. I'm not here to convince you! For those interested, I just wanted to share my personal approach to navigating these new tools and give a glimpse about how I approach new tools *in general*, regardless of AI. +I have no skin in the game here [^4], and there are of course other reasons behind utility to avoid using AI. I fully respect anyone's individual decisions regarding it. I'm not here to convince you! For those interested, I just wanted to share my personal approach to navigating these new tools and give a glimpse about how I approach new tools *in general*, regardless of AI. -## Footnotes +February 5, 2026 -1. Modern coding models like Opus and Codex are specifically trained to bias towards using tools compared to conversational models. [↩](#user-content-fnref-1) - -2. Due to the rapid pace of innovation in models, I have to constantly revisit my priors on this one. [↩](#user-content-fnref-3) - -3. The skill formation issues particularly in juniors without a strong grasp of fundamentals deeply worries me, however. [↩](#user-content-fnref-4) - -4. I don't work for, invest in, or advise any AI companies. [↩](#user-content-fnref-5) \ No newline at end of file +[^1]: Modern coding models like Opus and Codex are specifically trained to bias towards using tools compared to conversational models. + +[^2]: Due to the rapid pace of innovation in models, I have to constantly revisit my priors on this one. + +[^3]: The skill formation issues particularly in juniors without a strong grasp of fundamentals deeply worries me, however. + +[^4]: I don't work for, invest in, or advise any AI companies. \ No newline at end of file diff --git a/raw/Scaling Managed Agents Decoupling the brain from the hands.md b/raw/Scaling Managed Agents Decoupling the brain from the hands.md new file mode 100644 index 0000000..88859ad --- /dev/null +++ b/raw/Scaling Managed Agents Decoupling the brain from the hands.md @@ -0,0 +1,85 @@ +--- +title: "Scaling Managed Agents: Decoupling the brain from the hands" +source: "https://www.anthropic.com/engineering/managed-agents" +author: +published: +created: 2026-04-11 +description: "Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems." +tags: + - "clippings" +--- +*Get started with Claude Managed Agents by following our [docs](https://platform.claude.com/docs/en/managed-agents/overview).* + +A running topic on the Engineering Blog is how to [build effective agents](https://www.anthropic.com/engineering/building-effective-agents) and [design harnesses](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) for [long-running work](https://www.anthropic.com/engineering/harness-design-long-running-apps). A common thread across this work is that harnesses encode assumptions about what Claude can’t do on its own. However, those assumptions need to be frequently questioned because they can [go stale](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) as models improve. + +As just one example, in prior work [we found](https://www.anthropic.com/engineering/harness-design-long-running-apps) that Claude Sonnet 4.5 would wrap up tasks prematurely as it sensed its context limit approaching—a behavior sometimes called “context anxiety.” We addressed this by adding context resets to the harness. But when we used the same harness on Claude Opus 4.5, we found that the behavior was gone. The resets had become dead weight. + +We expect harnesses to continue evolving. So we built Managed Agents: a hosted service in the Claude Platform that runs long-horizon agents on your behalf through a small set of interfaces meant to outlast any particular implementation—including the ones we run today. + +Building Managed Agents meant solving an old problem in computing: how to design a system for “ [programs as yet unthought of](http://www.catb.org/esr/writings/taoup/html/ch03s01.html).” Decades ago, operating systems solved this problem by virtualizing hardware into abstractions— *process, file* —general enough for programs that didn't exist yet. The abstractions outlasted the hardware. The `read()` command is agnostic as to whether it’s accessing a disk pack from the 1970s or a modern SSD. The abstractions on top stayed stable while the implementations underneath changed freely. + +Managed Agents follow the same pattern. We virtualized the components of an agent: a session (the append-only log of everything that happened), a harness (the loop that calls Claude and routes Claude’s tool calls to the relevant infrastructure), and a sandbox (an execution environment where Claude can run code and edit files). This allows the implementation of each to be swapped without disturbing the others. We're opinionated about the shape of these interfaces, not about what runs behind them. + +![[903b624ada206b10753a24c6a1367e74a869165d-1080x1080.webp]] + +## Don’t adopt a pet + +We started by placing all agent components into a single container, which meant the session, agent harness, and sandbox all shared an environment. There were benefits to this approach, including that file edits are direct syscalls, and there were no service boundaries to design. + +But by coupling everything into one container, we ran into an old infrastructure problem: we’d adopted a [*pet*](https://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/). In the pets-vs-cattle analogy, a pet is a named, hand-tended individual you can’t afford to lose, while cattle are interchangeable. In our case, the server became that pet; if a container failed, the session was lost. If a container was unresponsive, we had to nurse it back to health. + +Nursing containers meant debugging unresponsive stuck sessions. Our only window in was the WebSocket event stream, but that couldn’t tell us *where* failures arose, which meant that a bug in the harness, a packet drop in the event stream, or a container going offline all presented the same. To figure out what went wrong, an engineer had to open a shell inside the container, but because that container often also held user data, that approach essentially meant we lacked the ability to debug. + +A second issue was that the harness assumed that whatever Claude worked on lived in the container with it. When customers asked us to connect Claude to their virtual private cloud, they had to either peer their network with ours, or run our harness in their own environment. An assumption baked into the harness became a problem when we wanted to connect it to different infrastructure. + +## Decouple the brain from the hands + +The solution we arrived at was to decouple what we thought of as the “brain” (Claude and its harness) from both the “hands” (sandboxes and tools that perform actions) and the “session” (the log of session events). Each became an interface that made few assumptions about the others, and each could fail or be replaced independently. + +**The harness leaves the container.** Decoupling the brain from the hands meant the harness no longer lived inside the container. It called the container the way it called any other tool: `execute(name, input) → string`. The container became cattle. If the container died, the harness caught the failure as a tool-call error and passed it back to Claude. If Claude decided to retry, a new container could be reinitialized with a standard recipe: `provision({resources})`. We no longer had to nurse failed containers back to health. + +**Recovering from harness failure.** The harness also became cattle. Because the session log sits outside the harness, nothing in the harness needs to survive a crash. When one fails, a new one can be rebooted with `wake(sessionId)`, use `getSession(id)` to get back the event log, and resume from the last event. During the agent loop, the harness writes to the session with `emitEvent(id, event)` in order to keep a durable record of events. + +![[73e900af5b9d6ed8c64db0a8e74d4465963556b7-1640x1596.webp]] + +**The security boundary.** In the coupled design, any untrusted code that Claude generated was run in the same container as credentials—so a prompt injection only had to convince Claude to read its own environment. Once an attacker has those tokens, they can spawn fresh, unrestricted sessions and delegate work to them. Narrow scoping is an obvious mitigation, but this encodes an assumption about what Claude can't do with a limited token—and Claude is getting increasingly smart. The structural fix was to make sure the tokens are never reachable from the sandbox where Claude’s generated code runs. + +We used two patterns to ensure this. Auth can be bundled with a resource or held in a vault outside the sandbox. For Git, we use each repository’s access token to clone the repo during sandbox initialization and wire it into the local git remote. Git `push` and `pull` work from inside the sandbox without the agent ever handling the token itself. For custom tools, we support MCP and store OAuth tokens in a secure vault. Claude calls MCP tools via a dedicated proxy; this proxy takes in a token associated with the session. The proxy can then fetch the corresponding credentials from the vault and make the call to the external service. The harness is never made aware of any credentials. + +## The session is not Claude’s context window + +Long-horizon tasks often exceed the length of Claude’s context window, and the standard ways to address this all involve irreversible decisions about what to keep. We’ve explored these techniques in [prior work](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) on context engineering. For example, compaction lets Claude save a summary of its context window and the memory tool lets Claude write context to files, enabling learning across sessions. This can be paired with context trimming, which selectively removes tokens such as old tool results or thinking blocks. + +But irreversible decisions to selectively retain or discard context can lead to failures. It is difficult to know which tokens the future turns will need. If messages are transformed by a compaction step, the harness removes compacted messages from Claude’s context window, and these are recoverable only if they are stored. Prior work [has explored](https://arxiv.org/pdf/2512.24601) ways to address this by storing context as an object that lives *outside* the context window. For example, context can be an object in a REPL that the LLM programmatically accesses by writing code to filter or slice it. + +![[cf0719d7832b1f577b7393c84a7c53eecc725ca4-760x200.webp]] + +In Managed Agents, the session provides this same benefit, serving as a context object that lives outside Claude’s context window. But rather than be stored within the sandbox or REPL, context is durably stored in the session log. The interface, `getEvents(),` allows the brain to interrogate context by selecting positional slices of the event stream. The interface can be used flexibly, allowing the brain to pick up from wherever it last stopped reading, rewinding a few events before a specific moment to see the lead up, or rereading context before a specific action. + +Any fetched events can also be transformed in the harness before being passed to Claude’s context window. These transformations can be whatever the harness encodes, including context organization to achieve a high prompt cache hit rate and context engineering. We separated the concerns of recoverable context storage in the session and arbitrary context management in the harness because we can’t predict what specific context engineering will be required in future models. The interfaces push that context management into the harness, and only guarantee that the session is durable and available for interrogation. + +## Many brains, many hands + +**Many brains.** Decoupling the brain from the hands solved one of our earliest customer complaints. When teams wanted Claude to work against resources in their own VPC, the only path was to peer their network with ours, because the container holding the harness assumed every resource sat next to it. Once the harness was no longer in the container, that assumption went away. The same change had a performance payoff. When we initially put the brain in a container, it meant that many brains required as many containers. For each brain, no inference could happen until that container was provisioned; every session paid the full container setup cost up front. Every session, even ones that would never touch the sandbox, had to clone the repo, boot the process, fetch pending events from our servers. + +That dead time is expressed in time-to-first-token (TTFT), which measures how long a session waits between accepting work and producing its first response token. TTFT is the latency the user most acutely *feels*. + +Decoupling the brain from the hands means that containers are provisioned by the brain via a tool call `(execute(name, input) → string)` only if they are needed. So a session that didn't need a container right away didn't wait for one. Inference could start as soon as the orchestration layer pulled pending events from the session log. Using this architecture, our p50 TTFT dropped roughly 60% and p95 dropped over 90%. Scaling to many brains just meant starting many stateless harnesses, and connecting them to hands only if needed. + +**Many hands.** We also wanted the ability to connect each brain to many hands. In practice, this means Claude must reason about many execution environments and decide where to send work—a harder cognitive task than operating in a single shell. We started with the brain in a single container because earlier models weren't capable of this. As intelligence scaled, the single container became the limitation instead: when that container failed, we lost state for every hand that the brain was reaching into. + +Decoupling the brain from the hands makes each hand a tool, `execute(name, input) → string`: a name and input go in, and a string is returned. That interface supports any custom tool, any MCP server, and our own tools. The harness doesn’t know whether the sandbox is a container, a phone, or a Pokémon emulator. And because no hand is coupled to any brain, brains can pass hands to one another. + +![[4f67b1c10566552aec514a716ea43544ab330e0b-668x243.webp]] + +## Conclusion + +The challenge we faced is an old one: how to design a system for “programs as yet unthought of.” Operating systems have lasted decades by virtualizing the hardware into abstractions general enough for programs that didn't exist yet. With Managed Agents, we aimed to design a system that accommodates future harnesses, sandboxes, or other components around Claude. + +Managed Agents is a meta-harness in the same spirit, unopinionated about the *specific* harness that Claude will need in the future. Rather, it is a system with general interfaces that allow many different harnesses. For example, Claude Code is an excellent harness that we use widely across tasks. We’ve also shown that task-specific agent harnesses excel in narrow domains. Managed Agents can accommodate any of these, matching Claude’s intelligence over time. + +Meta-harness design means being opinionated about the interfaces around Claude: we expect that Claude will need the ability to manipulate state (the session) and perform computation (the sandbox). We also expect that Claude will require the ability to scale to many brains and many hands. We designed the interfaces so that these can be run reliably and securely over long time horizons. But we make no assumptions about the number or location of brains or hands that Claude will need. + +## Acknowledgements + +Written by Lance Martin, Gabe Cemaj, and Michael Cohen. Thanks to Nodir Turakulov and Jeremy Fox for helpful conversations on these topics. Special thanks to the Agents API team and Jake Eaton for their contributions. \ No newline at end of file diff --git a/raw/harness-assets/agent-knowledge-limits.webp b/raw/harness-assets/agent-knowledge-limits.webp deleted file mode 100644 index afa0592..0000000 Binary files a/raw/harness-assets/agent-knowledge-limits.webp and /dev/null differ diff --git a/raw/harness-assets/fig1-codex-drives-app.webp b/raw/harness-assets/fig1-codex-drives-app.webp deleted file mode 100644 index 52a2249..0000000 Binary files a/raw/harness-assets/fig1-codex-drives-app.webp and /dev/null differ diff --git a/raw/harness-assets/layered-domain-architecture.webp b/raw/harness-assets/layered-domain-architecture.webp deleted file mode 100644 index d9bccba..0000000 Binary files a/raw/harness-assets/layered-domain-architecture.webp and /dev/null differ diff --git a/raw/harness-assets/observability-stack.svg b/raw/harness-assets/observability-stack.svg deleted file mode 100644 index 5584893..0000000 --- a/raw/harness-assets/observability-stack.svg +++ /dev/null @@ -1,129 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/raw/images/23c98f1d7ae720bfb39190d50e0706c03b177ad8-1999x1320.webp b/raw/images/23c98f1d7ae720bfb39190d50e0706c03b177ad8-1999x1320.webp new file mode 100644 index 0000000..68b0d5a Binary files /dev/null and b/raw/images/23c98f1d7ae720bfb39190d50e0706c03b177ad8-1999x1320.webp differ diff --git a/raw/images/4f67b1c10566552aec514a716ea43544ab330e0b-668x243.webp b/raw/images/4f67b1c10566552aec514a716ea43544ab330e0b-668x243.webp new file mode 100644 index 0000000..fc862cf Binary files /dev/null and b/raw/images/4f67b1c10566552aec514a716ea43544ab330e0b-668x243.webp differ diff --git a/raw/images/73e900af5b9d6ed8c64db0a8e74d4465963556b7-1640x1596.webp b/raw/images/73e900af5b9d6ed8c64db0a8e74d4465963556b7-1640x1596.webp new file mode 100644 index 0000000..c3dc236 Binary files /dev/null and b/raw/images/73e900af5b9d6ed8c64db0a8e74d4465963556b7-1640x1596.webp differ diff --git a/raw/images/903b624ada206b10753a24c6a1367e74a869165d-1080x1080.webp b/raw/images/903b624ada206b10753a24c6a1367e74a869165d-1080x1080.webp new file mode 100644 index 0000000..00d1129 Binary files /dev/null and b/raw/images/903b624ada206b10753a24c6a1367e74a869165d-1080x1080.webp differ diff --git a/raw/images/Image.png b/raw/images/Image.png new file mode 100644 index 0000000..4022963 Binary files /dev/null and b/raw/images/Image.png differ diff --git a/raw/images/OAI_Harness_engineering_Codex_drives_the_app_with_Chrome_DevTools_MCP_to_validate_its_work_desktop-dark.png b/raw/images/OAI_Harness_engineering_Codex_drives_the_app_with_Chrome_DevTools_MCP_to_validate_its_work_desktop-dark.png new file mode 100644 index 0000000..56fd84e Binary files /dev/null and b/raw/images/OAI_Harness_engineering_Codex_drives_the_app_with_Chrome_DevTools_MCP_to_validate_its_work_desktop-dark.png differ diff --git a/raw/images/OAI_Harness_engineering_Giving_Codex_a_full_observability_stack_desktop-dark.png b/raw/images/OAI_Harness_engineering_Giving_Codex_a_full_observability_stack_desktop-dark.png new file mode 100644 index 0000000..f72298e Binary files /dev/null and b/raw/images/OAI_Harness_engineering_Giving_Codex_a_full_observability_stack_desktop-dark.png differ diff --git a/raw/images/OAI_Harness_engineering_Layered_domain_architecture_with_explicit_cross-cutting_boundries_desktop-dark.png b/raw/images/OAI_Harness_engineering_Layered_domain_architecture_with_explicit_cross-cutting_boundries_desktop-dark.png new file mode 100644 index 0000000..b874304 Binary files /dev/null and b/raw/images/OAI_Harness_engineering_Layered_domain_architecture_with_explicit_cross-cutting_boundries_desktop-dark.png differ diff --git a/raw/images/OAI_Harness_engineering_The_limits_of_agent_knowledge_desktop-dark.png b/raw/images/OAI_Harness_engineering_The_limits_of_agent_knowledge_desktop-dark.png new file mode 100644 index 0000000..511cf3c Binary files /dev/null and b/raw/images/OAI_Harness_engineering_The_limits_of_agent_knowledge_desktop-dark.png differ diff --git a/raw/images/Screenshot-2026-02-12-at-12.25.20---PM-1.png b/raw/images/Screenshot-2026-02-12-at-12.25.20---PM-1.png new file mode 100644 index 0000000..e16e8ef Binary files /dev/null and b/raw/images/Screenshot-2026-02-12-at-12.25.20---PM-1.png differ diff --git a/raw/langchain-assets/Screenshot-2026-02-16-at-12.50.00---PM.png b/raw/images/Screenshot-2026-02-16-at-12.50.00---PM.png similarity index 100% rename from raw/langchain-assets/Screenshot-2026-02-16-at-12.50.00---PM.png rename to raw/images/Screenshot-2026-02-16-at-12.50.00---PM.png diff --git a/raw/images/a8bef95425966495629095a5cb38bde4a8b13558-1999x997.webp b/raw/images/a8bef95425966495629095a5cb38bde4a8b13558-1999x997.webp new file mode 100644 index 0000000..dbff6ea Binary files /dev/null and b/raw/images/a8bef95425966495629095a5cb38bde4a8b13558-1999x997.webp differ diff --git a/raw/images/cf0719d7832b1f577b7393c84a7c53eecc725ca4-760x200.webp b/raw/images/cf0719d7832b1f577b7393c84a7c53eecc725ca4-760x200.webp new file mode 100644 index 0000000..f8ccf90 Binary files /dev/null and b/raw/images/cf0719d7832b1f577b7393c84a7c53eecc725ca4-760x200.webp differ diff --git a/raw/images/circle.png b/raw/images/circle.png new file mode 100644 index 0000000..afb3aac Binary files /dev/null and b/raw/images/circle.png differ diff --git a/raw/images/d080353d-31aa-4642-8586-3b7f2201bf94.png b/raw/images/d080353d-31aa-4642-8586-3b7f2201bf94.png new file mode 100644 index 0000000..1042098 Binary files /dev/null and b/raw/images/d080353d-31aa-4642-8586-3b7f2201bf94.png differ diff --git a/raw/minimax-assets/d92a6eb4-a4b8-4906-b76a-d627c814a2c0.gif b/raw/images/dd83d19a-156b-481f-9835-9576293c323a.gif similarity index 100% rename from raw/minimax-assets/d92a6eb4-a4b8-4906-b76a-d627c814a2c0.gif rename to raw/images/dd83d19a-156b-481f-9835-9576293c323a.gif diff --git a/raw/images/harness-bounded-contexts.png b/raw/images/harness-bounded-contexts.png new file mode 100644 index 0000000..76d8f15 Binary files /dev/null and b/raw/images/harness-bounded-contexts.png differ diff --git a/raw/images/harness-change-lifecycle-examples.png b/raw/images/harness-change-lifecycle-examples.png new file mode 100644 index 0000000..175ddd3 Binary files /dev/null and b/raw/images/harness-change-lifecycle-examples.png differ diff --git a/raw/images/harness-overview.png b/raw/images/harness-overview.png new file mode 100644 index 0000000..d5ea18f Binary files /dev/null and b/raw/images/harness-overview.png differ diff --git a/raw/images/harness-templates.png b/raw/images/harness-templates.png new file mode 100644 index 0000000..c862605 Binary files /dev/null and b/raw/images/harness-templates.png differ diff --git a/raw/images/harness-types.png b/raw/images/harness-types.png new file mode 100644 index 0000000..0110fec Binary files /dev/null and b/raw/images/harness-types.png differ diff --git a/raw/minimax-assets/img-1.png b/raw/images/img-1.png similarity index 100% rename from raw/minimax-assets/img-1.png rename to raw/images/img-1.png diff --git a/raw/minimax-assets/img-3.png b/raw/images/img-3.png similarity index 100% rename from raw/minimax-assets/img-3.png rename to raw/images/img-3.png diff --git a/raw/langchain-assets/langsmith_trace_analyzer_skill.png b/raw/images/langsmith_trace_analyzer_skill.png similarity index 100% rename from raw/langchain-assets/langsmith_trace_analyzer_skill.png rename to raw/images/langsmith_trace_analyzer_skill.png diff --git a/raw/images/memory_f.png b/raw/images/memory_f.png new file mode 100644 index 0000000..90699d7 Binary files /dev/null and b/raw/images/memory_f.png differ diff --git a/raw/images/protocol_f.png b/raw/images/protocol_f.png new file mode 100644 index 0000000..2d5d0a0 Binary files /dev/null and b/raw/images/protocol_f.png differ diff --git a/raw/langchain-assets/self-verification-loop.png b/raw/images/self-verification-loop.png similarity index 100% rename from raw/langchain-assets/self-verification-loop.png rename to raw/images/self-verification-loop.png diff --git a/raw/images/skill_f.png b/raw/images/skill_f.png new file mode 100644 index 0000000..cfde184 Binary files /dev/null and b/raw/images/skill_f.png differ diff --git a/raw/langchain-assets/the-reasoning-sandwich.png b/raw/images/the-reasoning-sandwich.png similarity index 100% rename from raw/langchain-assets/the-reasoning-sandwich.png rename to raw/images/the-reasoning-sandwich.png diff --git a/raw/images/timeline.png b/raw/images/timeline.png new file mode 100644 index 0000000..a748c75 Binary files /dev/null and b/raw/images/timeline.png differ diff --git a/raw/images/val_vs_test_by_dataset.png b/raw/images/val_vs_test_by_dataset.png new file mode 100644 index 0000000..f92cd4a Binary files /dev/null and b/raw/images/val_vs_test_by_dataset.png differ diff --git a/raw/images/x1 1.png b/raw/images/x1 1.png new file mode 100644 index 0000000..58f50d3 Binary files /dev/null and b/raw/images/x1 1.png differ diff --git a/raw/images/x1.png b/raw/images/x1.png new file mode 100644 index 0000000..0c2c2d9 Binary files /dev/null and b/raw/images/x1.png differ diff --git a/raw/images/x2.png b/raw/images/x2.png new file mode 100644 index 0000000..db7671d Binary files /dev/null and b/raw/images/x2.png differ diff --git a/raw/images/x3 1.png b/raw/images/x3 1.png new file mode 100644 index 0000000..3975283 Binary files /dev/null and b/raw/images/x3 1.png differ diff --git a/raw/images/x3.png b/raw/images/x3.png new file mode 100644 index 0000000..06cdc72 Binary files /dev/null and b/raw/images/x3.png differ diff --git a/raw/images/x4.png b/raw/images/x4.png new file mode 100644 index 0000000..7633560 Binary files /dev/null and b/raw/images/x4.png differ diff --git a/raw/images/x5.png b/raw/images/x5.png new file mode 100644 index 0000000..38449fe Binary files /dev/null and b/raw/images/x5.png differ diff --git a/raw/langchain-assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png b/raw/langchain-assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png deleted file mode 100644 index 8e28ca3..0000000 Binary files a/raw/langchain-assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png and /dev/null differ diff --git a/raw/martinfowler-assets/harness-bounded-contexts.png b/raw/martinfowler-assets/harness-bounded-contexts.png deleted file mode 100644 index 56a9a95..0000000 --- a/raw/martinfowler-assets/harness-bounded-contexts.png +++ /dev/null @@ -1,280 +0,0 @@ - - - - - - -not found - - - - - - - -

404

- -

I’m afraid this is not the document you’re looking for. Try using the -search box above, and good luck.

-
- - - - - - - - - diff --git a/raw/martinfowler-assets/harness-change-lifecycle-examples.png b/raw/martinfowler-assets/harness-change-lifecycle-examples.png deleted file mode 100644 index 56a9a95..0000000 --- a/raw/martinfowler-assets/harness-change-lifecycle-examples.png +++ /dev/null @@ -1,280 +0,0 @@ - - - - - - -not found - - - - - - - -

404

- -

I’m afraid this is not the document you’re looking for. Try using the -search box above, and good luck.

-
- - - - - - - - - diff --git a/raw/martinfowler-assets/harness-continuous-feedback-examples.png b/raw/martinfowler-assets/harness-continuous-feedback-examples.png deleted file mode 100644 index 56a9a95..0000000 --- a/raw/martinfowler-assets/harness-continuous-feedback-examples.png +++ /dev/null @@ -1,280 +0,0 @@ - - - - - - -not found - - - - - - - -

404

- -

I’m afraid this is not the document you’re looking for. Try using the -search box above, and good luck.

-
- - - - - - - - - diff --git a/raw/martinfowler-assets/harness-overview.png b/raw/martinfowler-assets/harness-overview.png deleted file mode 100644 index 56a9a95..0000000 --- a/raw/martinfowler-assets/harness-overview.png +++ /dev/null @@ -1,280 +0,0 @@ - - - - - - -not found - - - - - - - -

404

- -

I’m afraid this is not the document you’re looking for. Try using the -search box above, and good luck.

-
- - - - - - - - - diff --git a/raw/martinfowler-assets/harness-templates.png b/raw/martinfowler-assets/harness-templates.png deleted file mode 100644 index 56a9a95..0000000 --- a/raw/martinfowler-assets/harness-templates.png +++ /dev/null @@ -1,280 +0,0 @@ - - - - - - -not found - - - - - - - -

404

- -

I’m afraid this is not the document you’re looking for. Try using the -search box above, and good luck.

-
- - - - - - - - - diff --git a/raw/martinfowler-assets/harness-types.png b/raw/martinfowler-assets/harness-types.png deleted file mode 100644 index 56a9a95..0000000 --- a/raw/martinfowler-assets/harness-types.png +++ /dev/null @@ -1,280 +0,0 @@ - - - - - - -not found - - - - - - - -

404

- -

I’m afraid this is not the document you’re looking for. Try using the -search box above, and good luck.

-
- - - - - - - - - diff --git a/raw/minimax-assets/img-2.png b/raw/minimax-assets/img-2.png deleted file mode 100644 index 4550042..0000000 Binary files a/raw/minimax-assets/img-2.png and /dev/null differ diff --git a/raw/minimax-m27.md b/raw/minimax-m27.md deleted file mode 100644 index d5ef2aa..0000000 --- a/raw/minimax-m27.md +++ /dev/null @@ -1,99 +0,0 @@ -# MiniMax M2.7: Early Echoes of Self-Evolution - -In the months following the first release of our M2-series models, we received a large volume of feedback and suggestions from enthusiastic users and developers, which drove us to further accelerate the efficiency of our model iterations. With human productivity already fully unleashed, the natural next step was to initiate self-evolution of both the model and the organization. M2.7 is our first model deeply participating in its own evolution. - -M2.7 is capable of building complex agent harnesses and completing highly elaborate productivity tasks, leveraging capabilities such as Agent Teams, complex Skills, and dynamic tool search. For example, when developing M2.7, we let the model update its own memory and build dozens of complex skills in its harness to help with reinforcement learning experiments. We further let the model improve its learning process and harness based on the experiment results. This process initiates a cycle of model self-evolution. - -1\. M2.7 delivers outstanding performance in real-world software engineering, including end-to-end full project delivery, log analysis, bug troubleshooting, code security, machine learning, and more. On the SWE-Pro benchmark, M2.7 scored 56.22%, nearly approaching Opus's best level. This capability also extends to end-to-end full project delivery scenarios (VIBE-Pro 55.6%) and deep understanding of complex engineering systems on Terminal Bench 2 (57.0%). - -2\. We have also enhanced the model's expertise and task delivery capabilities across various fields in the professional office software domain. Its ELO score on GDPval-AA is 1495, the highest among open-source models. M2.7 shows significantly improved ability for complex editing in the Office suite — Excel, PPT, and Word — and can better handle multi-round revisions and high-fidelity editing. M2.7 is capable of interacting with complex environments: It maintains a 97% skill adherence rate while working with over 40 complex skills, each exceeding 2,000 tokens. - -3\. M2.7 exhibits excellent character consistency and emotional intelligence, opening up more room for product innovation. - -Based on these capabilities, M2.7 is also significantly accelerating our own evolution into an AI-native organization. - -![MiniMax M2.7 benchmark overview](minimax-assets/img-1.png) - -## Building an agent for model self-evolution - -We first share an internal workflow that enables the M2-series models to self-evolve. This workflow also serves as an exploration of the boundaries of the model's agentic capabilities. - -Modern agent harness utilizes a combination of complex skills, memory, and other external modules to help improve its adaptability to various workspace environments. In MiniMax, our agents are routinely faced with very complex and disparate working environments spanning multiple departments. As such, to improve the robustness of our agents in these heterogeneous environments, we tasked an internal version of M2.7 to build a research agent harness that interacts and collaborates with different research project groups. The harness supports data pipelines, training environments, infrastructure, cross-team collaboration, and persistent memory — enabling researchers to drive it to deliver better models. The research agent harness drives the iteration cycle that produces the next generation of models under the guidance set by researchers. - -An exemplary workflow lies in the daily routine of our RL team. A researcher starts by discussing an experimental idea with the agent, who helps with literature review, tracks a pre-set experiment spec, pipelines data and other artifacts, and launches experiments. During the experiments, the agent monitors and profiles the experiment's progress and automatically triggers log reading, debugging, metric analysis, code fixes, merge requests, and smoke tests, identifying and configuring subtle yet key changes. These could have required the collaboration of multiple human researchers from different teams before, but now human researchers only interact for critical decisions and discussions. This accelerates problem discovery and experimentation, delivering models faster. Here, M2.7 is capable of handling 30%-50% of the workflow. - -![Research agent harness architecture](minimax-assets/img-2.png) - -During the iteration process, we realized that the model's ability to recursively evolve its own harness is also critical. Our internal harness autonomously collects feedback, builds evaluation sets for internal tasks, and based on this continuously iterates its own architecture, skills/MCP implementation, and memory mechanisms to complete tasks better and more efficiently. - -For example, we had M2.7 optimize a model's programming performance on an internal scaffold. M2.7 ran entirely autonomously, executing an iterative loop of "analyze failure trajectories → plan changes → modify scaffold code → run evaluations → compare results → decide to keep or revert changes" for over 100 rounds. During this process, M2.7 discovered effective optimizations for the model: systematically searching for the optimal combination of sampling parameters such as temperature, frequency penalty, and presence penalty; designing more specific workflow guidelines for the model (e.g., automatically searching for the same bug patterns in other files after a fix); and adding loop detection and other optimizations to the scaffold's agent loop. Ultimately, this achieved a 30% performance improvement on internal evaluation sets. - -We believe that future AI self-evolution will gradually transition towards full autonomy, coordinating data construction, model training, inference architecture, evaluation, and other stages without human involvement. - -To this end, we conducted preliminary exploratory tests in low-resource scenarios. We had M2.7 participate in 22 machine learning competitions at the MLE Bench Lite level open-sourced by OpenAI. These competitions can be run on a single A30 GPU, yet they cover virtually all stages of machine learning workflow. - -We designed and implemented a simple harness to guide the agent in autonomous optimization. The core modules include three components: short-term memory, self-feedback, and self-optimization. Specifically, after each iteration round, the agent generates a short-term memory markdown file and simultaneously performs self-criticism on the current round's results, thereby providing potential optimization directions for the next round. The next round then conducts further self-optimization based on the memory and self-feedback chain from all previous rounds. We ran a total of three trials, each with 24 hours for iterative evolution. From the figure below, one can see that the ML models trained by M2.7 continuously achieved higher medal rates over time. In the end, the best run achieved 9 gold medals, 5 silver medals, and 1 bronze medal. The average medal rate across the three runs was 66.6%, a result second only to Opus-4.6 (75.7%) and GPT-5.4 (71.2%), tying with Gemini-3.1 (66.6%). - -![MLE Bench Lite performance chart](minimax-assets/img-3.png) - -## Professional Software Engineering - -In software engineering tasks, M2.7 more deeply explores real-world programming abilities, including log analysis for bug hunting, refactoring, code security, machine learning, Android development, and more. - -Take a common production scenario as an example: debugging in a live environment. This requires not just code generation, but strong comprehensive reasoning abilities. When faced with alerts in production, M2.7 can correlate monitoring metrics with deployment timelines to perform causal reasoning, conduct statistical analysis on trace sampling and propose precise hypotheses, proactively connect to databases to verify root causes, pinpoint missing index migration files in the code repository, and even have the awareness to use non-blocking index creation to stop the bleeding first before submitting a merge request. From observability analysis and database expertise to SRE-level decision-making — this is not merely a model that can write code, but one that truly understands production systems. Compared to traditional manual troubleshooting processes, using M2.7, we have on multiple occasions reduced the recovery time for live production system incidents to under three minutes. - -Your browser does not support video playback. Please switch to a different browser. - -Live production environment debugging - -In terms of raw programming capabilities, M2.7 has reached the level of SOTA models. On SWE-Pro, which covers multiple programming languages, M2.7 achieved a 56.22% accuracy rate, matching GPT-5.3-Codex. It demonstrates an even more notable advantage on benchmarks closer to real-world engineering scenarios, such as SWE Multilingual (76.5) and Multi SWE Bench (52.7). - -This capability also extends to end-to-end full project delivery scenarios. On the repo-level code generation benchmark VIBE-Pro, M2.7 scored 55.6%, nearly on par with Opus 4.6 — meaning that whether the requirement involves Web, Android, iOS, or simulation tasks, they can be handed directly to M2.7 to complete. - -What deserves even more attention is its deep understanding of complex engineering systems. On Terminal Bench 2 (57.0%) and NL2Repo (39.8%), both of which demand a high degree of system-level comprehension, M2.7 also performs solidly. This further confirms that it excels not only at code generation but can also deeply understand the operational logic and collaborative dynamics of software systems. - -Your browser does not support video playback. Please switch to a different browser. - -WildGuard demo webpage generated by M2.7 - -To improve development efficiency, one particularly important feature is native Agent Teams (multi-agent collaboration). Agent Teams impose paradigm-level demands on the model: role boundaries, adversarial reasoning, protocol adherence, and behavioral differentiation — these cannot be achieved through prompting alone and must be internalized as native capabilities of the model. In Agent Teams scenarios, the model needs to stably anchor its role identity, proactively challenge teammates' logical and ethical blind spots, and make autonomous decisions within complex state machines. Below is an Agent Teams setup we use internally for product prototype development, which contains a minimal organization for building product prototypes. - -![Research agent harness architecture](minimax-assets/d92a6eb4-a4b8-4906-b76a-d627c814a2c0.gif) - -Agent Teams multi-agent collaboration demo - -## Professional Work - -Beyond software engineering, agents are becoming increasingly useful in office scenarios. We believe this comes down to two core capabilities: - -Domain expertise and task delivery capability. The model needs to possess professional knowledge across various fields and understand user requirements. In the GDPval-AA evaluation, which measures this capability, M2.7 achieved an ELO score of 1495 among 45 models, second only to Opus 4.6, Sonnet 4.6, and GPT5.4, and surpassing GPT5.3. For the most common office document processing tasks, we systematically optimized the model's ability to handle Word, Excel, and PPT. Across various agent harnesses, M2.7 can both generate files directly based on templates and skills, and follow users' interactive instructions to perform multiple rounds of high-fidelity editing on existing files, ultimately producing editable deliverables. - -Ability to interact with complex environments. Generalized everyday scenarios mean the model must flexibly adapt to various contexts, invoke diverse skills and tools, and maintain stable instruction adherence throughout extended interactions. M2.7 has made substantial improvements in these areas. On Toolathon, M2.7 achieved an accuracy of 46.3%, reaching the global top tier. Agent harnesses in real-world work scenarios also often require understanding and invoking a large number of complex skills. In MM Claw testing, M2.7 maintained a 97% skill compliance rate across 40 complex skills (each exceeding 2,000 tokens). - -We tested the model's professional proficiency in finance, and compared to the previous generation, the improvement in capability is significant. For example, in a scenario involving reading research reports and modeling a company's future revenue, M2.7 can autonomously read a company's annual reports and earnings call minutes, cross-reference multiple research reports, independently design assumptions and build a revenue forecast model, and then produce a PPT and research report based on templates — understanding, making judgments, and producing output like a junior analyst, while self-correcting through multiple rounds of interaction. The feedback from practitioners is that the output can already serve as a first draft and go directly into subsequent workflows. Below is an example for TSMC. - -Task: Based on TSMC's annual report and earnings call information, build a revenue model for TSMC. Read multiple research reports, design corresponding assumptions, model TSMC's revenue based on the latest information, then produce a PPT based on a PPT template, and write a Word document research report. - -您的浏览器不支持播放此视频，请更换浏览器。 - -The recent surge in popularity of OpenClaw is representative of a thriving agent ecosystem, and we are pleased that our M2-series models have contributed to the community's flourishing. Based on commonly used tasks in OpenClaw, we built an evaluation set called MM Claw, covering a wide range of real-world needs in both work and life — from personal learning planning, to office document processing and delivery, scheduled professional research and investment advice, and code development and maintenance. M2.7 achieved a level close to Sonnet 4.6 on this test, with an accuracy of 62.7%. - -## Entertainment - -With OpenClaw and similar personal agents, we noticed that beyond getting work done, many users also want the model to have high emotional intelligence and character consistency. With a persona in place, users start interacting with OpenClaw like a friend. We believe this presents an opportunity to extend the use of agentic models beyond pure productivity into interactive entertainment. To this end, we strengthened character consistency and conversational capabilities in M2.7. - -Based on this, we built a preliminary demo: OpenRoom, an interaction system based on an agent harness that liberates AI interaction from plain text streams and places it within a Web GUI space where everything is interactive. Here, character settings are no longer cold chunks of prompts; conversation drives the experience, generating real-time visual feedback and scene interactions, with characters proactively engaging with their environment. We believe this framework is highly extensible and can continue to evolve alongside improvements in agentic capabilities and community development, exploring entirely new ways for humans and agents to interact. - -To encourage exploration in this area, we have open-sourced the initial demo (of which most of the code was written by AI): - -Your browser does not support video playback. Please switch to a different browser. - -MiniMax M2.7 is now fully available on MiniMax Agent and the MiniMax API Platform. We look forward to users and developers exploring even more interesting use cases with M2.7. - -MiniMax Agent: [agent.minimax.io](https://agent.minimax.io/) - -API: [platform.minimax.io](https://platform.minimax.io/) - -Coding Plan: [platform.minimax.io/subscribe/coding-plan](https://platform.minimax.io/subscribe/coding-plan) - -Intelligence with Everyone. \ No newline at end of file diff --git a/raw/nxcode-harness-engineering-complete-guide.md b/raw/nxcode-harness-engineering-complete-guide.md deleted file mode 100644 index 083df0d..0000000 --- a/raw/nxcode-harness-engineering-complete-guide.md +++ /dev/null @@ -1,417 +0,0 @@ -# Harness Engineering: The Complete Guide to Building Systems That Make AI Agents Actually Work (2026) - -## Harness Engineering: The Complete Guide to Building Systems That Make AI Agents Actually Work - -**March 2026** — If 2025 was the year AI agents proved they could write code, 2026 is the year we learned that **the agent isn't the hard part — the harness is.** - -OpenAI's Codex team just built a production application with **over 1 million lines of code** where **zero lines were written by human hands**. The engineers didn't write code. They designed the system that let AI write code reliably. That system — the constraints, feedback loops, documentation, linters, and lifecycle management — is what the industry now calls a **harness**. - -**Harness engineering** is the new discipline of designing these systems. And it's changing what it means to be a software engineer. - -* * * - -## What Is Harness Engineering? - -### The Horse Metaphor - -The term "harness" comes from horse tack — reins, saddle, bit — the complete set of equipment for channeling a powerful but unpredictable animal in the right direction. The metaphor is deliberate: - -- The **horse** is the AI model — powerful, fast, but it doesn't know where to go on its own -- The **harness** is the infrastructure — constraints, guardrails, feedback loops that channel the model's power productively -- The **rider** is the human engineer — providing direction, not doing the running - -Without a harness, an AI agent is a thoroughbred in an open field. Fast, impressive, and completely useless for getting anything done. - -### The Formal Definition - -**Harness engineering** is the design and implementation of systems that: - -1. **Constrain** what an AI agent can do (architectural boundaries, dependency rules) -2. **Inform** the agent about what it should do (context engineering, documentation) -3. **Verify** that the agent did it correctly (testing, linting, CI validation) -4. **Correct** the agent when it goes wrong (feedback loops, self-repair mechanisms) - -Martin Fowler describes it as *"the tooling and practices we can use to keep AI agents in check"* — but it's more than just safety. A good harness makes agents **more capable**, not just more controlled. - -* * * - -## Why Harness Engineering Matters Now - -### The Model Is Commodity. The Harness Is Moat. - -Here's the uncomfortable truth the AI industry is confronting: **the underlying model matters less than the system around it.** - -LangChain proved this definitively. Their coding agent went from **52.8% to 66.5%** on Terminal Bench 2.0 — jumping from **Top 30 to Top 5** — by changing nothing about the model. They only changed the harness: - -Change - -What They Did - -Impact - -Self-verification loop - -Added pre-completion checklist middleware - -Caught errors before submission - -Context engineering - -Mapped directory structures at startup - -Agent understood codebase from the start - -Loop detection - -Tracked repeated file edits - -Prevented "doom loops" - -Reasoning sandwich - -High reasoning for planning/verification, medium for implementation - -Better quality within time budgets - -**Same model. Different harness. Dramatically better results.** - -### OpenAI's 1 Million Line Proof Point - -OpenAI's experiment is the most compelling evidence yet: - -- **5 months** of development -- **1 million+ lines of code** in the final product -- **Zero manually written lines** — every line was produced by Codex agents -- **Built in ~1/10th the time** it would have taken humans -- The product has **internal daily users and external alpha testers** -- It **ships, deploys, breaks, and gets fixed** — all by agents within the harness - -The engineers' job? Designing the harness. Specifying intent. Providing feedback. Not writing code. - -* * * - -## The Three Pillars of Harness Engineering - -OpenAI's framework organizes harness engineering into three core categories: - -### 1\. Context Engineering - -Context engineering is about ensuring the agent has the right information at the right time. - -**Static context:** - -- Repository-local documentation (architecture specs, API contracts, style guides) -- `AGENTS.md` or `CLAUDE.md` files that encode project-specific rules -- Cross-linked design documents validated by linters - -**Dynamic context:** - -- Observability data (logs, metrics, traces) accessible to agents -- Directory structure mapping at agent startup -- CI/CD pipeline status and test results - -**The critical rule:** From the agent's perspective, anything it can't access in-context doesn't exist. Knowledge in Google Docs, Slack threads, or people's heads is invisible to the system. **The repository must be the single source of truth.** - -### 2\. Architectural Constraints - -This is where harness engineering diverges most sharply from traditional AI prompting. Instead of telling the agent "write good code," you **mechanically enforce what good code looks like.** - -**Dependency layering:** - -``` -Types → Config → Repo → Service → Runtime → UI - -``` - -Each layer can only import from layers to its left. This isn't a suggestion — it's enforced by structural tests and CI validation. - -**Constraint enforcement tools:** - -- **Deterministic linters** — Custom rules that flag violations automatically -- **LLM-based auditors** — Agents that review other agents' code for architectural compliance -- **Structural tests** — Like ArchUnit, but for AI-generated code -- **Pre-commit hooks** — Automated checks before any code is committed - -**Why constraints improve output:** Paradoxically, constraining the solution space makes agents **more productive**, not less. When an agent can generate anything, it wastes tokens exploring dead ends. When the harness defines clear boundaries, the agent converges faster on correct solutions. - -### 3\. Entropy Management ("Garbage Collection") - -This is the most underappreciated component. Over time, AI-generated codebases accumulate entropy — documentation drifts from reality, naming conventions diverge, dead code accumulates. - -Harness engineering addresses this with **periodic cleanup agents:** - -- **Documentation consistency agents** — Verify that docs match current code -- **Constraint violation scanners** — Find code that slipped past earlier checks -- **Pattern enforcement agents** — Identify and fix deviations from established patterns -- **Dependency auditors** — Track and resolve circular or unnecessary dependencies - -These agents run on schedules — daily, weekly, or triggered by specific events — keeping the codebase healthy for both human reviewers and future AI agents. - -* * * - -## Harness Engineering in Practice: How Teams Actually Do It - -### The OpenAI Approach: Zero Human Code - -OpenAI's team structure for harness engineering: - -Role - -Traditional - -Harness Engineering - -Writing code - -Primary job - -Never - -Designing architecture - -Part of the job - -Primary job - -Writing documentation - -Afterthought - -Critical infrastructure - -Reviewing PRs - -Code review - -Reviewing agent output + harness effectiveness - -Debugging - -Reading code - -Analyzing agent behavior patterns - -Testing - -Writing tests - -Designing test strategies agents execute - -### The Stripe Approach: Minions at Scale - -Stripe's internal coding agents, called **Minions**, now produce **over 1,000 merged pull requests per week**: - -1. Developer posts a task in Slack -2. Minion writes the code -3. Minion passes CI -4. Minion opens a PR -5. Human reviews and merges - -No developer interaction between step 1 and step 5. The harness handles everything — test execution, CI validation, style compliance, and documentation updates. - -### The LangChain Approach: Middleware-First - -LangChain structures their harness as composable middleware layers: - -``` -Agent Request - → LocalContextMiddleware (maps codebase) - → LoopDetectionMiddleware (prevents repetition) - → ReasoningSandwichMiddleware (optimizes compute) - → PreCompletionChecklistMiddleware (enforces verification) - → Agent Response - -``` - -Each middleware layer adds a specific capability without modifying the core agent logic. This modular approach makes the harness testable and evolvable. - -* * * - -## Building Your First Harness: A Practical Framework - -### Level 1: Basic Harness (Single Developer) - -If you're using Claude Code, Cursor, or Codex for individual projects: - -**What to set up:** - -- `CLAUDE.md` or `.cursorrules` file with project conventions -- Pre-commit hooks for linting and formatting -- A test suite the agent can run to self-verify -- Clear directory structure with consistent naming - -**Time to set up:** 1-2 hours **Impact:** Prevents the most common agent mistakes - -### Level 2: Team Harness (Small Team) - -For teams of 3-10 developers sharing a codebase: - -**Add to Level 1:** - -- `AGENTS.md` with team-wide conventions -- Architectural constraints enforced by CI -- Shared prompt templates for common tasks -- Documentation-as-code validated by linters -- Code review checklists specifically for agent-generated PRs - -**Time to set up:** 1-2 days **Impact:** Consistent agent behavior across the team - -### Level 3: Production Harness (Engineering Organization) - -For organizations running dozens of concurrent agents: - -**Add to Level 2:** - -- Custom middleware layers (loop detection, reasoning optimization) -- Observability integration (agents read logs and metrics) -- Entropy management agents on scheduled runs -- Harness versioning and A/B testing -- Agent performance monitoring dashboards -- Escalation policies for when agents get stuck - -**Time to set up:** 1-2 weeks **Impact:** Agents operate as autonomous contributors - -* * * - -## Common Harness Engineering Mistakes - -### 1\. Over-Engineering the Control Flow - -> *"If you over-engineer the control flow, the next model update will break your system."* - -Models improve rapidly. Capabilities that required complex pipelines in 2024 are now handled by a single context-window prompt. Build your harness to be **rippable** — you should be able to remove "smart" logic when the model gets smart enough to not need it. - -### 2\. Treating the Harness as Static - -The harness needs to evolve with the model. When a new model release improves reasoning, your reasoning-optimization middleware might become counterproductive. Review and update harness components with every major model update. - -### 3\. Ignoring the Documentation Layer - -The most impactful harness improvement is often the simplest: **better documentation**. If your `AGENTS.md` is vague, your agent output will be vague. Invest in precise, machine-readable documentation that serves as the agent's ground truth. - -### 4\. No Feedback Loop - -A harness without feedback is a cage, not a guide. The agent needs to know when it's succeeding and when it's failing. Build in: - -- Self-verification steps before task completion -- Test execution as part of the agent workflow -- Metrics on agent success rates by task type - -### 5\. Human-Only Documentation - -If your architectural decisions live in people's heads or in Confluence pages the agent can't access, the harness has a gap. **Everything the agent needs must be in the repository.** - -* * * - -## Harness Engineering vs. Related Concepts - -Concept - -Scope - -Focus - -**Prompt Engineering** - -Single interaction - -Crafting effective prompts - -**Context Engineering** - -Model context window - -What information the model sees - -**Harness Engineering** - -Entire agent system - -Environment, constraints, feedback, lifecycle - -**Agent Engineering** - -Agent architecture - -Internal agent design and routing - -**Platform Engineering** - -Infrastructure - -Deployment, scaling, operations - -Harness engineering **includes** context engineering and draws from prompt engineering, but it operates at a higher level — it's about the complete system that makes agents reliable, not just the inputs to a single interaction. - -* * * - -## What This Means for Software Engineers - -### The Job Is Changing - -Harness engineering represents a genuine evolution in what software engineers do: - -Before - -After - -Write code - -Design environments where AI writes code - -Debug code - -Debug agent behavior - -Review code - -Review agent output + harness effectiveness - -Write tests - -Design test strategies - -Maintain docs - -Build documentation as machine-readable infrastructure - -This doesn't mean engineers become less technical. If anything, harness engineering requires **deeper** architectural thinking — you're designing systems that must work without your constant intervention. - -### The Skills That Matter - -Based on what we've seen building AI-powered products at [NxCode](https://www.nxcode.io/): - -1. **Systems thinking** — Understanding how constraints, feedback loops, and documentation interact -2. **Architecture design** — Defining boundaries that are enforceable and productive -3. **Specification writing** — Articulating intent precisely enough for agents to execute -4. **Observability** — Building monitoring that reveals agent behavior patterns -5. **Iteration speed** — Rapidly testing and refining harness configurations - -### Our Experience: What Works in Practice - -We've been building AI-powered web applications using multiple agent systems (Claude Code, Codex, Cursor). The patterns that have made the biggest difference for us: - -- **Repository-first documentation**: Every architectural decision, naming convention, and deployment process is in the repo. Nothing lives in Slack or Google Docs. -- **Incremental constraint building**: Start with basic linting, add architectural constraints as patterns emerge, don't try to design the perfect harness upfront. -- **Agent-specific review checklists**: AI-generated code has different failure modes than human code. Our review process accounts for common agent patterns (over-abstraction, unnecessary error handling, documentation drift). -- **Multi-provider harness design**: Our harness works with Claude, GPT, and Gemini models. Provider-agnostic design means we can switch models without rebuilding the entire system. - -* * * - -## Key Takeaways - -1. **Harness engineering is the new discipline** of designing systems that make AI agents reliable — constraints, feedback loops, documentation, and lifecycle management -2. **The model is commodity; the harness is moat** — LangChain jumped from Top 30 to Top 5 on benchmarks by only changing the harness -3. **OpenAI built 1M+ lines with zero human code** — proving harness engineering works at production scale -4. **Three pillars**: Context engineering, architectural constraints, and entropy management -5. **Start simple**: A good `AGENTS.md` and pre-commit hooks are more impactful than complex middleware -6. **The engineer's job is evolving** — from writing code to designing environments where AI writes code -7. **Build rippable harnesses** — over-engineering breaks when models improve; keep it adaptable - -* * * - -## Related Articles - -- [Best AI for Coding in 2026: 10 Tools Ranked by Real-World Performance](https://www.nxcode.io/resources/news/best-ai-for-coding-2026-complete-ranking) -- [OpenAI Frontier Guide: Enterprise AI Agent Platform for Building AI Coworkers (2026)](https://www.nxcode.io/resources/news/openai-frontier-enterprise-ai-agent-platform-guide-2026) -- [Cursor Tutorial 2026: Learn AI Coding in 15 Minutes (Beginner Guide)](https://www.nxcode.io/resources/news/cursor-tutorial-beginners-2026) \ No newline at end of file diff --git a/raw/harness-engineering-openai-zh.md b/raw/工程技术：在智能体优先的世界中利用 Codex.md similarity index 57% rename from raw/harness-engineering-openai-zh.md rename to raw/工程技术：在智能体优先的世界中利用 Codex.md index 039750a..5a84352 100644 --- a/raw/harness-engineering-openai-zh.md +++ b/raw/工程技术：在智能体优先的世界中利用 Codex.md @@ -1,66 +1,113 @@ -# Harness Engineering：在智能体优先的世界中利用 Codex | OpenAI +--- +title: "工程技术：在智能体优先的世界中利用 Codex" +source: "https://openai.com/zh-Hans-CN/index/harness-engineering/" +author: +published: 2026-03-11 +created: 2026-04-11 +description: "作者：Ryan Lopopolo，技术人员" +tags: + - "clippings" +--- +2026年2月11日 -（原文来源：https://openai.com/zh-Hans-CN/index/harness-engineering/） +[工程](https://openai.com/news/engineering/) + +作者：Ryan Lopopolo，技术人员 + + + +在过去五个月里，我们的团队一直在进行一项实验：构建并交付一款软件产品的内部 beta 版， **其中没有一行代码是人工编写的** 。 -在过去五个月里，我们的团队一直在进行一项实验：构建并交付一款软件产品的内部 beta 版，其中没有一行代码是人工编写的。 该产品有内部日常活跃用户和外部 Alpha 测试者。它经历了交付、部署、故障和修复的整个过程。与众不同的是，每一行代码 — 从应用逻辑、测试、CI 配置、文档、可观察性到内部工具 — 全都是由 Codex 编写的。据估计，我们只用了手工编写代码所需的大约 1/10 的时间就完成了这项工作。 -人类掌舵。智能体执行。 + +**人类掌舵。智能体执行。** + 我们有意选择这一限制，以便构建必要的内容，从而将工程速度提升数个数量级。我们用了几周的时间来交付最终达到一百万行代码的项目。为此，我们需要了解，当软件工程团队的主要工作不再是编写代码，而是设计环境、明确意图和构建反馈回路，从而使 Codex 智能体能够可靠地工作时，会发生哪些变化。 + 这个帖子要说的是，在我们与智能体团队一起从零开始打造一款全新产品的过程中，所能学到的经验教训 — 哪些地方出了问题，哪些问题相互叠加，以及如何最大化利用我们唯一真正稀缺的资源：人类的时间和注意力。 + +## 我们从一个空的 Git 代码仓库开始 + 首次提交到一个空的代码仓库是在 2025 年 8 月下旬。 + 初始架构 — 包括代码仓库结构、CI 配置、格式化规则、包管理器设置和应用框架 — 是在一小套现有模板的指导下，由 Codex CLI 使用 GPT‑5 生成的。就连指导智能体如何在代码仓库中工作的初始 AGENTS.md 文件本身也是由 Codex 编写的。 + 该系统没有预存任何人工编写的代码。从一开始，代码仓库就由智能体塑造。 -五个月后，该代码仓库已经拥有约一百万行代码，从应用逻辑、基础设施、工具、文档到内部开发者工具应有尽有。在那段时间内，大约有 1,500 个 Pull Request 被打开与合并，而推动 Codex 的仅仅是一个由三名工程师组成的小团队。这相当于平均每位工程师每天处理 3.5 个 PRs 的吞吐量，而且令人惊讶的是，随着团队规模扩大到现在的七名工程师，吞吐量甚至还增加了。重要的是，这并非为了输出而输出：该产品已在数百名内测用户那里投入使用，其中包括每天都在使用的内测高级用户。 -在整个开发过程中，人类从未直接直接贡献过任何代码。这成为团队的核心理念：不手动编写代码。 -由于缺乏人工编码的实践，工程师工作的重点转向了系统、架构和杠杆作用。 + +五个月后，该代码仓库已经拥有约一百万行代码，从应用逻辑、基础设施、工具、文档到内部开发者工具应有尽有。在那段时间内，大约有 1,500 个 Pull Request 被打开与合并，而推动 Codex 的仅仅是一个由三名工程师组成的小团队。这相当于平均每位工程师每天处理 3.5 个 PRs 的吞吐量，而且令人惊讶的是，随着团队规模扩大到现在的七名工程师，吞吐量甚至还 *增加* 了。重要的是，这并非为了输出而输出：该产品已在数百名内测用户那里投入使用，其中包括每天都在使用的内测高级用户。 + +在整个开发过程中，人类从未直接直接贡献过任何代码。这成为团队的核心理念： **不手动编写代码** 。 + +## 重新定义工程师的角色 + +由于缺乏人工编码的实践， **工程师工作的重点转向了系统、架构和杠杆作用** 。 + 早期进展比我们所预期的要慢，而这并不是因为 Codex 不具备相应的能力，而是因为环境的规范不够明确。该智能体缺乏实现高级目标所需的工具、抽象层和内部结构，因而无法取得进展。我们工程团队的主要任务成了协助智能体完成有用的工作。 + 在实践中，这意味着采用深度优先的工作方式：将更大的目标拆解为更小的构建模块（设计、代码、评审、测试等），提示智能体去构建这些模块，并使用它们去解锁更复杂的任务。当事情进行不顺利时，解决方案基本上再也不会是“再努力一点”。因为取得进展的唯一方式是让 Codex 来完成工作，而人类工程师则总是介入这项任务并追问：“究竟还需要什么样的能力，我们又该如何让这个能力对智能体来说既清晰可读又可强制执行？” -人类几乎完全通过提示与系统交互：工程师描述任务，运行智能体，并允许其打开一个 Pull Request。为了推动 PR 的完成，我们会指示 Codex 在本地审核其自身的更改，在本地和云端请求额外的特定智能体审查，对任何人工或智能体给出的反馈做出响应，并循环往复，直到所有智能体审核人员都满意为止（这实际上是一个 [Ralph Wiggum 循环](https://ghuntley.com/loop/)）。Codex 直接使用我们的标准开发工具（gh、本地脚本和嵌入代码仓库的技能）来收集情境，而无需人工将内容复制粘贴到 CLI 中。 + +人类几乎完全通过提示与系统交互：工程师描述任务，运行智能体，并允许其打开一个 Pull Request。为了推动 PR 的完成，我们会指示 Codex 在本地审核其自身的更改，在本地和云端请求额外的特定智能体审查，对任何人工或智能体给出的反馈做出响应，并循环往复，直到所有智能体审核人员都满意为止（这实际上是一个 [Ralph Wiggum 循环 ⁠](https://ghuntley.com/loop/) ）。Codex 直接使用我们的标准开发工具（gh、本地脚本和嵌入代码仓库的技能）来收集情境，而无需人工将内容复制粘贴到 CLI 中。 + 人类可以审核 Pull Request（合并请求），但并非必须这样做。随着时间的推移，我们已将几乎所有的审核工作调整为用智能体对智能体的方式来处理。 + +## 提高应用程序的可读性 + 随着代码吞吐量的增加，我们的瓶颈变成了人工 QA 能力。由于人类的时间和注意力是固定的限制因素，我们一直在努力通过令应用程序的 UI、日志和应用指标等内容对 Codex 直接可读，从而为智能体增加更多功能。 + 例如，我们令应用程序可以根据 git worktree 启动，因此 Codex 可以为每次更改启动并驱动一个实例。我们还将 Chrome DevTools 协议接入智能体运行时，并创建了用于处理 DOM 快照、屏幕截图和导航的技能。这使 Codex 能够复现错误、验证修复，并直接推理 UI 的行为。 + +![[OAI_Harness_engineering_Codex_drives_the_app_with_Chrome_DevTools_MCP_to_validate_its_work_desktop-dark.png|题为“Codex 使用 Chrome DevTools MCP 驱动应用程序以验证其工作”的图表。Codex 会选择一个目标，对触发用户界面路径前后的状态进行快照，通过 Chrome DevTools 观察运行时事件，应用修复、重启和循环重新运行验证，直到应用程序恢复正常。]] + +题为“Codex 使用 Chrome DevTools MCP 驱动应用程序以验证其工作”的图表。Codex 会选择一个目标，对触发用户界面路径前后的状态进行快照，通过 Chrome DevTools 观察运行时事件，应用修复、重启和循环重新运行验证，直到应用程序恢复正常。 + 我们对可观测性工具也做了同样的处理。日志、指标和追踪记录会通过一个本地可观测性堆栈展示给 Codex，对任何给定的工作树来说，该堆栈都是临时的。Codex 在该应用程序的一个完全独立的版本上运行，一旦任务完成，该版本的所有内容，包括日志和指标，都会被删除。智能体可以使用 LogQL 查询日志，使用 PromQL 查询指标。有了这些情境，像“确保服务启动在 800ms 内完成”或“这四个关键用户旅程中的任何跨度都不得超过两秒”这样的提示就变得可行了。 + +![[OAI_Harness_engineering_Giving_Codex_a_full_observability_stack_desktop-dark.png|题为“在本地开发中赋予 Codex 完整的可观察性堆栈”的图表。一个应用程序将日志、指标和追踪数据发送到 Vector，Vector 会将数据分发到一个包含 Victoria Logs、Metrics 和 Traces 的可观测性堆栈中，并通过 LogQL、PromQL 或 TraceQL API 进行查询。Codex 使用这些信号进行查询、关联和推理，然后在代码库中进行修复，重启应用程序，重新运行工作负载，测试 UI 流程，并在反馈循环中重复此过程。]] + +题为“在本地开发中赋予 Codex 完整的可观察性堆栈”的图表。一个应用程序将日志、指标和追踪数据发送到 Vector，Vector 会将数据分发到一个包含 Victoria Logs、Metrics 和 Traces 的可观测性堆栈中，并通过 LogQL、PromQL 或 TraceQL API 进行查询。Codex 使用这些信号进行查询、关联和推理，然后在代码库中进行修复，重启应用程序，重新运行工作负载，测试 UI 流程，并在反馈循环中重复此过程。 + 我们经常看到单次 Codex 运行在单个任务上持续工作超过六个小时（通常是在人类睡眠时间）。 -情境管理是使智能体在大型和复杂任务中有效发挥作用的最大挑战之一。我们学到的最早经验教训之一很简单：要给 Codex 的是一张地图，而不是一本 1,000 页的说明书。 -![智能体知识局限边界](./harness-assets/agent-knowledge-limits.webp) +## 我们将代码仓库设为记录系统 -我们尝试了“一个大型的 [AGENTS.md](https://agents.md/)”方法。可想而知，这是一次失败的尝试： +情境管理是使智能体在大型和复杂任务中有效发挥作用的最大挑战之一。我们学到的最早经验教训之一很简单： **要给 Codex 的是一张地图，而不是一本 1,000 页的说明书。** -- 情境是一种稀缺资源。一个巨大的指令文件会挤掉任务、代码和相关文档 — 因此智能体要么会错过关键约束条件，要么开始针对错误的约束条件进行优化。 -- 过多的指导反而变得无效。当一切都 "重要"时，一切都不重要了。智能体最终会在本地进行模式匹配，而不是有意识地进行导航。 -- 它会立即腐烂。一本庞杂的手册会变成陈旧规则的坟场。智能体无法判断哪些信息仍然有效，一旦人类停止维护它，此文件就会悄然成为一个颇具吸引力的麻烦源头。 -- 这很难核实。单个 blob 不适合进行机械检查（覆盖率、新鲜度、所有权、交叉链接），因此漂移是不可避免的。 +我们尝试了“一个大型的 [`AGENTS.md` ⁠](https://agents.md/) ”方法。可想而知，这是一次失败的尝试： -因此，我们不再将 AGENTS.md 视为百科全书，而是将其视为内容目录。 -代码仓库的知识库位于一个结构化了的 docs/ 目录中，此目录被当作记录系统来使用。一份简短的 AGENTS.md（大约 100 行）被注入到情境中，主要用作地图，并指向其他地方更深层次的真实信息来源。 -代码仓库内知识存储布局。 +- **情境是一种稀缺资源。** 一个巨大的指令文件会挤掉任务、代码和相关文档 — 因此智能体要么会错过关键约束条件，要么开始针对错误的约束条件进行优化。 +- **过多的指导反而变得** ***无效*** **。** 当一切都 "重要"时，一切都不重要了。智能体最终会在本地进行模式匹配，而不是有意识地进行导航。 +- **它会立即腐烂。** 一本庞杂的手册会变成陈旧规则的坟场。智能体无法判断哪些信息仍然有效，一旦人类停止维护它，此文件就会悄然成为一个颇具吸引力的麻烦源头。 +- **这很难核实。** 单个 blob 不适合进行机械检查（覆盖率、新鲜度、所有权、交叉链接），因此漂移是不可避免的。 -![Codex 驱动应用架构图](./harness-assets/fig1-codex-drives-app.webp) +因此，我们不再将 `AGENTS.md` 视为百科全书，而是将其视为 **内容目录** 。 + +代码仓库的知识库位于一个结构化了的 `docs/` 目录中，此目录被当作记录系统来使用。一份简短的 `AGENTS.md` （大约 100 行）被注入到情境中，主要用作地图，并指向其他地方更深层次的真实信息来源。 + +#### 纯文本 ``` AGENTS.md ARCHITECTURE.md docs/ ├── design-docs/ -│ ├── index.md -│ ├── core-beliefs.md -│ └── ... +│ ├── index.md +│ ├── core-beliefs.md +│ └── ... ├── exec-plans/ -│ ├── active/ -│ ├── completed/ -│ └── tech-debt-tracker.md +│ ├── active/ +│ ├── completed/ +│ └── tech-debt-tracker.md ├── generated/ -│ └── db-schema.md +│ └── db-schema.md ├── product-specs/ -│ ├── index.md -│ ├── new-user-onboarding.md -│ └── ... +│ ├── index.md +│ ├── new-user-onboarding.md +│ └── ... ├── references/ -│ ├── design-system-reference-llms.txt -│ ├── nixpacks-llms.txt -│ ├── uv-llms.txt -│ └── ... +│ ├── design-system-reference-llms.txt +│ ├── nixpacks-llms.txt +│ ├── uv-llms.txt +│ └── ... ├── DESIGN.md ├── FRONTEND.md ├── PLANS.md @@ -70,35 +117,72 @@ docs/ └── SECURITY.md ``` -![Codex 全栈可观测性架构](./harness-assets/observability-stack.svg) +代码仓库内知识存储布局。 + +设计文档已被编目和索引，其中包括验证状态和一套核心理念，定义了智能体优先的操作原则。 [架构文档 ⁠](https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html) 提供域和包分层的顶层地图。一份高质量的文档会对每个产品领域和架构层进行评分，并随着时间的推移追踪差距。 + +计划被视为一流的工件。临时轻量计划用于小幅变更，而复杂工作则记录在 [执行计划 ⁠](https://cookbook.openai.com/articles/codex_exec_plans) 中，并附带进度和决策日志，这些日志会被提交到代码仓库。活跃计划、已完成计划和已知的技术债务都已进行版本控制并集中存放，使智能体能够在不依赖外部情境的情况下运行。 + +这实现了 **渐进式披露** ：智能体从一个小而稳定的切入点开始，并被指导下一步该去哪里查看，而不是一开始就被淹没。 -设计文档已被编目和索引，其中包括验证状态和一套核心理念，定义了智能体优先的操作原则。[架构文档](https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html)提供域和包分层的顶层地图。一份高质量的文档会对每个产品领域和架构层进行评分，并随着时间的推移追踪差距。 -计划被视为一流的工件。临时轻量计划用于小幅变更，而复杂工作则记录在[执行计划](https://cookbook.openai.com/articles/codex_exec_plans)中，并附带进度和决策日志，这些日志会被提交到代码仓库。活跃计划、已完成计划和已知的技术债务都已进行版本控制并集中存放，使智能体能够在不依赖外部情境的情况下运行。 -这实现了渐进式披露：智能体从一个小而稳定的切入点开始，并被指导下一步该去哪里查看，而不是一开始就被淹没。 我们严格执行这一点。专职的 linter 和 CI 作业会验证知识库的更新状况、是否已交叉链接且结构正确。一个定期运行的“doc-gardening”智能体会扫描那些不再反映真实代码行为的过时或废弃文档，并发起修复用的 Pull Request。 -随着代码库的发展，Codex 的设计决策框架也需要随之演变。 -由于该代码仓库完全由智能体生成，因此我们首先针对 Codex 的可读性进行了优化。就像团队会努力提升代码对新入职工程师的可导航性一样，我们的人类工程师的目标也是让智能体能够直接从代码仓库推理出完整的业务领域。 -从智能体的角度来看，它在运行时无法在情境中访问的任何内容都是不存在的。存储在 Google Docs、聊天记录或人们头脑中的知识都无法被系统访问。代码仓库本地的、已版本化的工件（例如，代码、Markdown、模式、可执行计划）就是它所能看到的全部。 -我们了解到，随着时间的推移，我们需要将越来越多的情境推送到仓库中。那次让团队在架构模式上达成一致的 Slack 讨论？如果智能体无法发现它，那么它就会像迟了三个月入职的新员工一样，对其一无所知。 -为 Codex 提供更多情境意味着要组织和展示正确的信息，好令智能体能够基于这些信息进行推理，而不是用临时指令使其不堪重负。就像你会在产品原则、工程规范和团队文化（包括表情符号偏好）方面为新队友提供引导一样，将这些信息提供给智能体会带来更一致的输出。 -这一框架明确了许多取舍。我们倾向于选择那些可以完全内化于在仓库中进行推理的依赖项和抽象。对智能体来说，通常被称为“枯燥”的技术，由于其可组合性、API 稳定性和在训练集里的表现，往往更容易建立模型。在某些情况下，让智能体重新实现部分功能子集比绕过公共库中不透明的上游行为更便宜。例如，我们没有引入通用的 p-limit 风格包，而是投入使用了我们自己的带并发的 map 辅助函数：它与我们的 OpenTelemetry 仪表紧密集成，具备 100% 的测试覆盖率，并且其行为完全符合我们的运行时预期。 -将系统的更多部分转化为智能体可以检查、验证并直接修改的形式，可以直接提高杠杆效应 — 这不仅适用于 Codex，也适用于其他智能体（例如[Aardvark](/index/introducing-aardvark/)) 也在参与代码库的开发。 -仅靠文档本身，是没法保持完全由智能体生成的代码库的连贯性的。通过强制执行不变量，而非对实施过程进行微观管理，我们令智能体能够快速交付，而且不会削弱基础。例如，我们要求 Codex [在边界处解析数据形状](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/)，但不规定具体实现方式（模型似乎偏好 Zod，但我们没有指定特定库）。 -智能体在具有[严格边界和可预测结构](https://bits.logic.inc/p/ai-is-forcing-us-to-write-good-code)的环境中最为高效，因此我们围绕一个严格的架构模型构建了该应用。每个业务域都划分为一组固定的层，依赖方向经过严格验证，并且仅允许有限的一组边。这些约束是通过自定义的 linter（当然是由 Codex 生成的！）和结构测试机械地强制执行的。 -![分层领域架构与跨切面边界](./harness-assets/layered-domain-architecture.webp) +## 目标是智能体的可读性 + +随着代码库的发展，Codex 的设计决策框架也需要随之演变。 + +由于该代码仓库完全由智能体生成，因此我们首先针对 *Codex* 的 *可读性* 进行了优化。就像团队会努力提升代码对新入职工程师的可导航性一样，我们的人类工程师的目标也是让智能体能够 **直接从代码仓库** 推理出完整的业务领域。 + +从智能体的角度来看，它在运行时无法在情境中访问的任何内容都是不存在的。存储在 Google Docs、聊天记录或人们头脑中的知识都无法被系统访问。代码仓库本地的、已版本化的工件（例如，代码、Markdown、模式、可执行计划）就是它所能看到的全部。 + +![[OAI_Harness_engineering_The_limits_of_agent_knowledge_desktop-dark.png|题为“智能体知识的局限性：Codex看不到的东西就不存在”的图表。Codex 的知识被展示为一个有边界的气泡。在其下方是不可见知识的示例 — Google Docs、Slack 消息，以及隐性的人类知识。箭头表示，要想让 Codex 看到这些信息，就必须将其以 Markdown 的形式编码到代码库中。]] + +题为“智能体知识的局限性：Codex看不到的东西就不存在”的图表。Codex 的知识被展示为一个有边界的气泡。在其下方是不可见知识的示例 — Google Docs、Slack 消息，以及隐性的人类知识。箭头表示，要想让 Codex 看到这些信息，就必须将其以 Markdown 的形式编码到代码库中。 + +我们了解到，随着时间的推移，我们需要将越来越多的情境推送到仓库中。那次让团队在架构模式上达成一致的 Slack 讨论？如果智能体无法发现它，那么它就会像迟了三个月入职的新员工一样，对其一无所知。 + +为 Codex 提供更多情境意味着要组织和展示正确的信息，好令智能体能够基于这些信息进行推理，而不是用临时指令使其不堪重负。就像你会在产品原则、工程规范和团队文化（包括表情符号偏好）方面为新队友提供引导一样，将这些信息提供给智能体会带来更一致的输出。 + +这一框架明确了许多取舍。我们倾向于选择那些可以完全内化于在仓库中进行推理的依赖项和抽象。对智能体来说，通常被称为“枯燥”的技术，由于其可组合性、API 稳定性和在训练集里的表现，往往更容易建立模型。在某些情况下，让智能体重新实现部分功能子集比绕过公共库中不透明的上游行为更便宜。例如，我们没有引入通用的 `p-limit` 风格包，而是投入使用了我们自己的带并发的 map 辅助函数：它与我们的 OpenTelemetry 仪表紧密集成，具备 100% 的测试覆盖率，并且其行为完全符合我们的运行时预期。 + +将系统的更多部分转化为智能体可以检查、验证并直接修改的形式，可以直接提高杠杆效应 — 这不仅适用于 Codex，也适用于其他智能体（例如 [Aardvark](https://openai.com/zh-Hans-CN/index/introducing-aardvark/)) 也在参与代码库的开发。 + +## 规范架构与品味 + +仅靠文档本身，是没法保持完全由智能体生成的代码库的连贯性的。 **通过强制执行不变量，而非对实施过程进行微观管理，我们令智能体能够快速交付，而且不会削弱基础。** 例如，我们要求 Codex [在边界处解析数据形状 ⁠](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/) ，但不规定具体实现方式（模型似乎偏好 Zod，但我们没有指定特定库）。 + +智能体在具有 [严格边界和可预测结构 ⁠](https://bits.logic.inc/p/ai-is-forcing-us-to-write-good-code) 的环境中最为高效，因此我们围绕一个严格的架构模型构建了该应用。每个业务域都划分为一组固定的层，依赖方向经过严格验证，并且仅允许有限的一组边。这些约束是通过自定义的 linter（当然是由 Codex 生成的！）和结构测试机械地强制执行的。 下图展示了规则：在每个业务领域内（例如应用设置），代码只能“向前”依赖于一组固定的层（Types → Config → Repo → Service → Runtime → UI）。横切关注点（认证、连接器、遥测、功能标志）通过一个单一的显式接口进入：Providers。其他任何内容都不被允许，并将通过自动化方式强制执行。 + +![[OAI_Harness_engineering_Layered_domain_architecture_with_explicit_cross-cutting_boundries_desktop-dark.png|题为“具有明确交叉界限的分层领域架构”的图表。在业务逻辑域内存在以下模块：Types → Config → Repo，以及 Providers → Service → Runtime → UI，底部是 App Wiring + UI。一个 Utils 模块位于界限之外，并向 Providers 提供输入。]] + +题为“具有明确交叉界限的分层领域架构”的图表。在业务逻辑域内存在以下模块：Types → Config → Repo，以及 Providers → Service → Runtime → UI，底部是 App Wiring + UI。一个 Utils 模块位于界限之外，并向 Providers 提供输入。 + 这种架构通常要等到你拥有数百名工程师时才会推迟。对于编码智能体来说，这是一个早期的先决条件：有了约束，速度才不会下降，架构才不会漂移。 + 在实践中，我们通过自定义的代码检查器和结构测试来强制执行这些规则，并辅以一小组“品味不变式”。例如，我们通过自定义 lint 静态地强制执行结构化日志记录、模式和类型的命名约定、文件大小限制，以及特定平台的可靠性要求。由于这些 lint 是自定义的，我们编写错误信息时会在智能体情境中注入修复指令。 + 在以人为本的工作流程中，这些规则可能会让人感到迂腐或束缚。有了智能体，它们就成了倍增器：一旦编码，它们就能立即应用于所有地方。 + 同时，我们还明确指出了哪些地方需要限制，哪些地方不需要限制。这类似于领导一个大型工程平台组织：在中央层面强制执行边界，在本地层面允许自主权。你非常重视界限、正确性和可重复性。在这些边界内，你允许团队或智能体在解决方案的表达方式上拥有很大的自由。 + 生成的代码不总是符合人类的风格偏好，这也没关系。只要输出是正确的、可维护的，并且对未来的智能体运行而言清晰易读，就可以算作达标。 + 人类的品味会不断反馈到系统中。审查评论、重构的 Pull Request 和面向用户的 Bug 会被记录为文档更新，或直接编码到工具中。当文档不够完善时，我们会将规则转化为代码 + +## 吞吐量改变了合并的理念 + 随着 Codex 的吞吐量增加，许多传统的工程规范变得不再有效。 + 该代码仓库在运行过程中尽量减少阻塞合并门。Pull Request 的生命周期很短。测试偶发失败通常通过后续重跑来解决，而不是无限期地阻碍进展。在一个智能体吞吐量远超人类注意力的系统中，纠错成本低，而等待成本高。 + 在低吞吐量环境中，这样做是不负责任的。而在这里，这通常是正确的选择。 + +## “智能体生成”实际上意味着什么 + 当我们说代码库是由 Codex 智能体生成的，我们指的是整个代码库。 + 智能体的产出包括： - 产品代码与测试 @@ -111,8 +195,13 @@ docs/ - 生产仪表板定义文件 人类始终参与其中，但工作的抽象层次与过去不同。我们优先处理工作，将用户反馈转化为验收标准，并对结果进行验证。当智能体遇到困难时，我们将其视为一个信号：识别缺失的内容 — 工具、指导与约束、文档 — 并将其反馈到代码仓库中，始终由 Codex 自己编写修复。 + 智能体可以直接使用我们的标准开发工具。他们会拉取审查反馈、在行内回复、推送更新，并且经常压缩并合并他们自己的 Pull Request（合并请求）。 + +## 不断提高的自主水平 + 随着越来越多的开发环节被直接编码到系统中 — 包括测试、验证、审查、反馈处理和恢复 — 该代码仓库最近跨过了一个重要门槛，使 Codex 能够端到端地驱动一个新功能。 + 给定一个提示，智能体现在可以： - 验证代码库的当前状态 @@ -128,12 +217,47 @@ docs/ - 合并更改 此行为在很大程度上取决于此代码仓库的具体结构和工具，不应在没有类似投入的情况下假定它可以泛化 — 至少目前还不行。 -完全自主的智能体也引入了新的问题。Codex 会复现代码仓库中已存在的模式 — 甚至包括那些不均衡或不够理想的模式。随着时间的推移，这不可避免地导致漂移。 + +## 熵与垃圾收集 + +**完全自主的智能体也引入了新的问题。** Codex 会复现代码仓库中已存在的模式 — 甚至包括那些不均衡或不够理想的模式。随着时间的推移，这不可避免地导致漂移。 + 最初，人类是手动处理这个问题的。我们的团队过去每周五（占一周的20%）都要花时间清理“AI 残渣”。不出所料，那并不具备可扩展性。 + 相反，我们开始将我们称为“黄金原则”的内容直接编码到代码仓库中，并建立了一个循环清理流程。这些原则是带有主观意见的机械规则，旨在保持代码库的可读性和一致性，以便将来运行智能体。例如：(1) 我们更倾向于使用共享的实用程序包，而不是手工编写的辅助工具，以便将不变式集中管理；(2) 我们不会使用“YOLO 式”探测数据 — 我们会验证边界，或依赖类型化的 SDK，这样智能体就不会意外地基于猜测的结构进行构建。我们会定期运行一组后台 Codex 任务，扫描偏差、更新质量等级，并发起有针对性的重构 Pull Request。其中大多数都可以在一分钟内完成审查并自动合并。 + 其功能类似于垃圾回收。技术债务就像一笔高息贷款：不断地以小额贷款的方式偿还债务，总比让债务不断累积，再痛苦地一次解决要好得多。人类的品味一旦被捕捉，就会持续应用于每一行代码。这也使我们能够每天发现并解决不良模式，而不是让它们在代码库中传播数天或数周。 + +## 我们仍在学习的内容 + 到目前为止，这一策略在 OpenAI 的内部发布和采纳过程中表现良好。为真实用户打造真实产品，帮助我们将投资锚定在现实中，并引导我们实现长期的可维护性。 + 我们尚不清楚的是，在一个完全由智能体生成的系统中，架构连贯性会如何随着时间的推移而演变。我们仍在学习人类的判断力在哪些方面能发挥最大作用，以及如何对这种判断力进行编码，使其发挥更大作用。我们也不知道，随着时间的推移，模型的功能不断增强，这一系统将如何演变。 + 显而易见的是：构建软件仍然需要纪律，但纪律更多地体现在支撑结构上，而不是代码上。保持代码库一致性的工具、抽象和反馈回路变得越发重要。 -我们当前最棘手的挑战集中在设计环境、反馈回路和控制系统方面，帮助智能体实现我们的目标：大规模构建和维护复杂、可靠的软件。 -随着像 Codex 这样的智能体在软件生命周期中占据越来越大的比重，这些问题将变得更加重要。我们希望通过分享一些早期的经验教训，帮助你理清投入精力的方向，以便[你可以直接开始构建](/codex/)。 + +**我们当前最棘手的挑战集中在设计环境、反馈回路和控制系统方面** ，帮助智能体实现我们的目标：大规模构建和维护复杂、可靠的软件。 + +随着像 Codex 这样的智能体在软件生命周期中占据越来越大的比重，这些问题将变得更加重要。我们希望通过分享一些早期的经验教训，帮助你理清投入精力的方向，以便 [你可以直接开始构建](https://openai.com/zh-Hans-CN/codex/) 。 + +## 作者 + +Ryan Lopopolo + +## 致谢 + +特别感谢 Victor Zhu 和 Zach Brock 为这篇文章做出的贡献，以及开发这款新产品的整个团队。 + +## 继续阅读[超越速率限制：扩大 Codex 和 Sora 的访问规模](https://openai.com/zh-Hans-CN/index/beyond-rate-limits/) + +[ + +工程 + +](https://openai.com/zh-Hans-CN/index/beyond-rate-limits/)[解锁 Codex 运行框架：我们如何构建 App Server](https://openai.com/zh-Hans-CN/index/unlocking-the-codex-harness/) + +[ + +工程 + +](https://openai.com/zh-Hans-CN/index/unlocking-the-codex-harness/) \ No newline at end of file diff --git a/skills/SKILL.md b/skills/SKILL.md index 73ddbe1..76dd21b 100644 --- a/skills/SKILL.md +++ b/skills/SKILL.md @@ -12,29 +12,95 @@ --- +## 1.5 大文档处理要求 + +对于篇幅较长的 raw 文档（如学术论文、长篇技术文章），**必须完整阅读和分析**，不得仅基于开头部分生成简短摘要。 + +### 具体要求： + +1. **完整内容获取**： + - 使用 Grep 搜索章节标题（如 `^#{1,3} `）了解文档结构 + - 分段读取完整内容，确保覆盖所有主要章节 + - 特别关注：摘要、引言、方法、实验、讨论、结论、附录等核心章节 + +2. **深度分析维度**： + - **核心论点**：提取文章的主要主张和关键发现 + - **方法细节**：理解技术方案的实现细节和设计决策 + - **实验结果**：完整记录所有实验数据、表格、图表信息 + - **案例研究**：保留具体的定性示例和应用场景 + - **相关工作**：建立与其他研究的联系和对比 + +3. **输出内容标准**： + - wiki 页面长度应与原文档的重要性和复杂度相匹配 + - 学术论文应包含：摘要、核心方法、完整实验结果、详细讨论 + - 技术文章应包含：问题背景、完整解决方案、实际应用案例 + - 保留所有定量数据（表格、指标、分数等） + +4. **例外情况**： + - 仅在以下情况下可生成较短摘要： + - 文档是纯新闻报道或简短公告 + - 文档主要是代码或配置（无大量叙事内容） + - 用户明确要求仅生成摘要 + +--- + ## 2. 标准文件系统架构 严格遵循 I/O 分离原则，确保知识库的纯净度与可迁移性： ```text 📁 wikillm ├── 📁 raw/ # 【输入层】原始素材（只读） +│ └── 📁 images/ # 原始图片文件（png, jpg, webp, gif, svg 等） └── 📁 wiki/ # 【输出层】编译器生成的知识产物 ├── 📁 concepts/ # 核心概念、原理分析 ├── 📁 practices/ # 部署指南、最佳实践 ├── 📁 visual/ # Marp 幻灯片、Matplotlib 趋势图 ├── 📁 queries/ # 高价值 Q&A 的沉淀归档 + ├── 📁 assets/ # 图像和资源文件（从 raw/images/ 同步而来） ├── INDEX.md # 动态索引与学习路径 - └── Glossary.md # 统一术语表与双链枢纽 + ├── Glossary.md # 统一术语表与双链枢纽 + └── sources.md # 来源文档索引（原始 URL 列表） ``` --- ## 3. 核心工作流 (The "Compilation" Loop) +### 阶段 0：增量检查 (Incremental Check) +* **任务**：检查 `raw/` 目录下哪些文件需要编译。 +* **读取状态**：读取 `wiki/compile-results.tsv`，获取已编译文件的哈希记录。 +* **扫描文件**：遍历 `raw/` 目录，计算每个文件的 SHA-256 哈希。 +* **识别变更**：对比哈希值，识别： + - **新增文件**：在 `compile-results.tsv` 中不存在的文件 + - **修改文件**：哈希值与记录不同的文件 + - **未修改文件**：哈希值相同的文件（跳过编译） +* **记录日志**：将检查过程写入 `wiki/compile.log`。 + +### 阶段 0.5：资源同步 (Asset Sync) +* **任务**：将 `raw/images/` 下的所有图片资源同步到 `wiki/assets/`。 +* **同步范围**：所有图像文件，包括但不限于： + - `png`, `jpg`, `jpeg`, `gif`, `webp`, `svg` +* **同步方式**： + - 使用 `cp -r raw/images/* wiki/assets/` 进行完整同步 + - `raw/images/` 是权威来源，同名文件直接覆盖 + - 保留原始文件名（包括空格和特殊字符） +* **验证**：确保 `wiki/assets/` 包含 `raw/images/` 中的所有文件 +* **时机**：每次编译前必须执行此步骤 + ### 阶段 1：多模态解构 (Ingest & Analyze) -* **任务**：解析 `raw/` 目录下的新增内容。 +* **任务**：解析 `raw/` 目录下的新增或修改内容。 +* **大文档完整阅读**：对于篇幅较长的文档（学术论文、长篇技术文章），必须完整阅读和分析： + - 首先用 Grep 搜索章节标题（如 `^#{1,3} `）了解文档结构 + - 分段读取完整内容，确保覆盖所有主要章节 + - 特别关注：摘要、引言、方法、实验、讨论、结论、附录等核心章节 * **视觉解析**：对图片进行深度 OCR 与逻辑识别。将架构图转化为文字描述及 **Mermaid** 代码块，存入对应 Wiki 页面。 -* **元数据提取**：为每篇文档生成 YAML Frontmatter（包含：`tags`, `source`, `confidence_score`, `last_updated`）。 +* **元数据提取**：为每篇文档生成 YAML Frontmatter（包含：`tags`, `source`, `raw_sources`, `confidence_score`, `last_updated`）。 + - `raw_sources` 字段：记录源文件路径和哈希值，格式如下： + ```yaml + raw_sources: + - path: raw/anthropic-harness-design.md + hash: "sha256:abc123..." + ``` ### 阶段 2：增量编译 (Incremental Writing) * **非线性重构**：不进行 1:1 翻译，而是基于源文档的“核心贡献”进行重写。 @@ -67,10 +133,16 @@ * **动态索引**：根据新增内容，自动更新 `INDEX.md` 中的”最新研究”与”学习路径”部分。 ### 阶段 4：健康检查与维护 (Linting) -* **一致性检查**：扫描 `wiki/`，发现术语冲突（如 A 文档叫“智能体”，B 文档叫“代理”）时，自动统一。 +* **一致性检查**：扫描 `wiki/`，发现术语冲突（如 A 文档叫”智能体”，B 文档叫”代理”）时，自动统一。 * **孤岛扫描**：识别没有任何链接指向的页面，强制将其挂载到导航树中。 * **补丁发布**：当 `raw/` 有新版本（如论文更新）时，在对应 Wiki 页面顶部发布 `[Update Patch]` 摘要。 +### 阶段 5：来源索引更新 (Sources Update) +* **任务**：更新 `wiki/sources.md`，记录本次编译涉及的来源文档。 +* **格式规范**：使用简单无序列表，每项格式为 `- [标题](URL)` +* **增量更新**：添加本次新增的来源，保持已有来源不变 +* **分类组织**：按”学术论文”、”概念文章”、”实践指南”等类别合理分组 + --- ## 4. 输出质量标准 (The Gold Standard) @@ -102,6 +174,7 @@ ## 6. 执行清单 (Checklist) * [ ] **Raw Check**: `raw/` 目录中是否包含待处理的新素材（图片/文档）？ +* [ ] **Asset Sync**: `raw/images/` 下的所有图片是否已同步到 `wiki/assets/`？ * [ ] **Glossary Lock**: 是否已锁定全局术语表，确保翻译不漂移？ * [ ] **Multimodal Sync**: 图片是否已转化为可编辑的文字解析/Mermaid？ * [ ] **文件名规范**: 所有 wiki 页面文件是否使用 kebab-case（连字符分隔）命名？ @@ -112,7 +185,67 @@ --- -## 7. 最佳实践提示 -* **手离开键盘**：不要手动修改 `wiki/` 目录下的内容，所有的修改应通过“向 LLM 发出 Lint 任务”或“添加 raw 素材后重新编译”来完成。 -* **搜索即创作**：把每一次对知识库的提问看作是一次“知识合成”，务必将高质量的回答存回库中。 -* **结构化思考**：在生成任何长篇文档前，先让 LLM 在内存中构建该主题的“概念地图”。 +## 7. 增量编译工作流 + +### 编译状态文件 + +项目使用三个核心文件来追踪编译状态： + +1. **`wiki/compile-results.tsv`** - 结构化的编译结果（TSV 格式） + - 字段：`raw_path`、`hash`、`last_modified`、`wiki_paths`、`compile_time`、`status` + - 记录每个 raw 文件的编译状态和生成的 wiki 文档 + +2. **`wiki/compile.log`** - 详细的编译日志 + - 记录每次编译的输入、输出、决策过程 + - 用于调试、审查和回溯 + +3. **`wiki/sources.md`** - 来源文档索引 + - 记录所有原始来源的标题和 URL + - 格式：`- [标题](URL)` 的无序列表 + - 按"学术论文"、"概念文章"、"实践指南"等分类组织 + - 每次增量编译后更新 + +### Wiki 文档元数据 + +每个 wiki 文档的 YAML frontmatter 都包含 `raw_sources` 字段： +```yaml +--- +title: 文档标题 +source: [来源名称] +raw_sources: + - path: raw/source-file.md + hash: “sha256:abc123...” +--- +``` + +### 常用命令 + +```bash +# 查看所有已编译文件 +cat wiki/compile-results.tsv + +# 查找特定文件的编译状态 +grep “raw/xxx.md” wiki/compile-results.tsv + +# 查看最近的编译日志 +tail -100 wiki/compile.log + +# 查看上次编译摘要 +grep “=== 编译完成 ===” -A 5 wiki/compile.log +``` + +### 增量编译检查清单 + +* [ ] **扫描检查**：运行扫描，检查 `raw/` 目录中是否有新增或修改的文件 +* [ ] **哈希对比**：与 `compile-results.tsv` 中的记录对比，确认变更 +* [ ] **日志记录**：将检查过程写入 `compile.log` +* [ ] **只编译变更**：仅处理新增或修改的文件 +* [ ] **更新 frontmatter**：确保新编译的 wiki 文档包含 `raw_sources` +* [ ] **更新状态文件**：追加/更新 `compile-results.tsv` 中的记录 +* [ ] **更新来源索引**：更新 `sources.md`，添加本次新增的来源 + +## 8. 最佳实践提示 +* **手离开键盘**：不要手动修改 `wiki/` 目录下的内容，所有的修改应通过”向 LLM 发出 Lint 任务”或”添加 raw 素材后重新编译”来完成。 +* **搜索即创作**：把每一次对知识库的提问看作是一次”知识合成”，务必将高质量的回答存回库中。 +* **结构化思考**：在生成任何长篇文档前，先让 LLM 在内存中构建该主题的”概念地图”。 +* **利用编译日志**：遇到问题时，先查看 `wiki/compile.log` 了解之前的编译过程。 diff --git a/wiki/Glossary.md b/wiki/Glossary.md index 7875685..b263d1c 100644 --- a/wiki/Glossary.md +++ b/wiki/Glossary.md @@ -1,294 +1,352 @@ --- -title: 术语表 -tags: [术语表, 核心概念] -last_updated: 2026-04-07 +title: "术语表" +source: "Externalization in LLM Agents: A Unified Review" +last_updated: 2026-04-11 --- # 术语表 -本术语表汇总了 [[Harness-Engineering|Harness 工程]] 领域的核心概念，为知识库提供统一的术语枢纽。 +本术语表统一了 LLM Agent 外部化框架中的核心概念，提供中英对照和 Wikilink 链接。 -## A +## 核心框架 -### Agent (智能体) -**英文**：Agent -**中文**：智能体 -**定义**：能够自主感知环境、做出决策并执行行动的 AI 系统。在编码场景中，智能体通常具备读取文件、执行程序、发起 HTTP 请求等工具调用能力。 -**相关概念**：[[Agent Teams|智能体团队]], [[Coding Agent|编码智能体]] +### Externalization (外部化) +**英文**: Externalization +**中文**: 外部化 +**定义**: 将认知负担从模型的内部计算逐步迁移到持久、可检查和可重用的外部结构中的过程。 +**参见**: [[Externalization-in-LLM-Agents|LLM Agent 中的外部化]] -### Agent Teams (智能体团队) -**英文**：Agent Teams -**中文**：智能体团队 -**定义**：多个智能体通过角色分工、协作协议和差异化行为共同完成复杂任务的系统。需要模型原生支持角色锚定、对抗性推理和协议遵守。 -**相关概念**：[[MiniMax M2.7]], [[Multi-Agent Collaboration|多智能体协作]] - -### Anthropic Harness Design (Anthropic Harness 设计) -**英文**：Anthropic Harness Design -**中文**：Anthropic Harness 设计 -**定义**：Anthropic 团队提出的多智能体架构，包含 Planner（规划者）、Generator（生成者）和 Evaluator（评估者）三种角色，通过生成-评估循环提升输出质量。 -**相关概念**：[[Generator-Evaluator Loop|生成-评估循环]], [[Context Reset|上下文重置]] - -## B - -### Build-Verify Loop (构建-验证循环) -**英文**：Build-Verify Loop -**中文**：构建-验证循环 -**定义**：智能体在完成任务过程中自主进行的迭代改进流程，包括规划发现、构建实现、验证测试和修复问题四个阶段。 -**相关概念**：[[Self-Verification|自我验证]], [[Reasoning Sandwich|推理三明治]] - -## C - -### Codex (Codex 模型) -**英文**：Codex -**中文**：Codex 模型 -**定义**：OpenAI 推出的专门用于代码生成的模型系列，在 Harness 工程中被用于从零生成完整产品代码库。 -**相关概念**：[[OpenAI Harness Engineering|OpenAI Harness 工程]] - -### Coding Agent (编码智能体) -**英文**：Coding Agent -**中文**：编码智能体 -**定义**：专门用于软件工程任务的 AI 智能体，能够理解代码库、编写代码、运行测试和调试问题。 -**相关概念**：[[Harness|Harness]], [[Agent|智能体]] - -### Context Engineering (上下文工程) -**英文**：Context Engineering -**中文**：上下文工程 -**定义**：Harness 工程的三大支柱之一，专注于确保智能体在正确的时间获得正确的信息，包括静态上下文和动态上下文。 -**相关概念**：[[Harness-Engineering|Harness 工程]], [[Context Reset|上下文重置]] - -### Context Reset (上下文重置) -**英文**：Context Reset -**中文**：上下文重置 -**定义**：一种解决长任务中上下文窗口填充和"上下文焦虑"问题的技术，通过清空上下文窗口并使用结构化交接传递状态来实现。 -**相关概念**：[[Context Compaction|上下文压缩]], [[Context Anxiety|上下文焦虑]] - -### Context Anxiety (上下文焦虑) -**英文**：Context Anxiety -**中文**：上下文焦虑 -**定义**：模型在接近其认为的上下文限制时提前结束工作的倾向，Claude Sonnet 4.5 表现出较强的这种行为。 -**相关概念**：[[Context Reset|上下文重置]], [[Context Window|上下文窗口]] - -### Context Compaction (上下文压缩) -**英文**：Context Compaction -**中文**：上下文压缩 -**定义**：通过摘要方式保留对话连续性的技术，但无法为智能体提供干净的状态，上下文焦虑问题仍可能存在。 -**相关概念**：[[Context Reset|上下文重置]] - -## D - -### Doom Loop (末日循环) -**英文**：Doom Loop -**中文**：末日循环 -**定义**：智能体在陷入困境时对同一错误方法进行小幅变异的重复尝试现象，可能多达 10 次以上。 -**相关概念**：[[Loop Detection|循环检测]] - -## E - -### Entropy Management (熵管理) -**英文**：Entropy Management -**中文**：熵管理 -**定义**：Harness 工程的三大支柱之一，通过定期清理智能体来管理 AI 生成代码库中随时间积累的熵（文档漂移、命名约定分歧、死代码堆积等）。 -**相关概念**：[[Harness-Engineering|Harness 工程]], [[Garbage Collection|垃圾回收]] - -### Evaluator (评估者) -**英文**：Evaluator -**中文**：评估者 -**定义**：Anthropic Harness 设计中的三种角色之一，负责评估 Generator 的输出质量，提供具体的反馈和评分。 -**相关概念**：[[Generator|生成者]], [[Planner|规划者]] - -## F - -### Feedforward (前馈控制) -**英文**：Feedforward -**中文**：前馈控制 -**定义**：预期智能体行为并在其行动前进行引导的控制方式，提高智能体第一次尝试就产生良好结果的概率。 -**相关概念**：[[Feedback|反馈控制]], [[Guide|引导]] - -### Feedback (反馈控制) -**英文**：Feedback -**中文**：反馈控制 -**定义**：在智能体行动后进行观察并帮助其自我纠正的控制方式，特别是当产生针对 LLM 消费优化的信号时效果显著。 -**相关概念**：[[Feedforward|前馈控制]], [[Sensor|传感器]] - -## G - -### Garbage Collection (垃圾回收) -**英文**：Garbage Collection -**中文**：垃圾回收 -**定义**：OpenAI 团队采用的定期清理流程，通过"黄金原则"和后台 Codex 任务来扫描偏差、更新质量等级并发起针对性重构。 -**相关概念**：[[Entropy Management|熵管理]] - -### Generator (生成者) -**英文**：Generator -**中文**：生成者 -**定义**：Anthropic Harness 设计中的三种角色之一，负责实际创建输出（如前端代码、应用功能等）。 -**相关概念**：[[Evaluator|评估者]], [[Planner|规划者]] - -### Generator-Evaluator Loop (生成-评估循环) -**英文**：Generator-Evaluator Loop -**中文**：生成-评估循环 -**定义**：受 GAN 启发的多智能体结构，Generator 生成输出，Evaluator 评估并提供反馈，Generator 根据反馈进行迭代改进。 -**相关概念**：[[Anthropic-Harness-Design|Anthropic Harness 设计]] - -### Glossary (术语表) -**英文**：Glossary -**中文**：术语表 -**定义**：WikiLLM 知识库的核心枢纽文档，统一术语翻译、提供中英对照，并通过 wikilinks 连接所有相关概念。 -**相关概念**：[[Wikilink|Wikilink]], [[WikiLLM]] - -## H +### Cognitive Artifact (认知人工制品) +**英文**: Cognitive Artifact +**中文**: 认知人工制品 +**定义**: 设计用于维持、显示或操作信息的人工设备，通过改变任务本身的结构来改变认知性能。 +**参见**: [[Externalization-in-LLM-Agents|LLM Agent 中的外部化]] ### Harness (Harness) -**英文**：Harness -**中文**：Harness -**定义**：AI 智能体之外的一切，包括系统提示、工具选择、执行流程、约束条件、反馈循环等。公式：Agent = Model + Harness。 -**相关概念**：[[Harness-Engineering|Harness 工程]], [[Agent|智能体]] +**英文**: Harness +**中文**: Harness +**定义**: 将原始模型能力转化为可靠 Agent 行为的脚手架，是承载记忆、技能、协议并提供编排逻辑、约束、可观测性和反馈循环的工程层。 +**参见**: [[Harness-Engineering|Harness 工程]] -### Harness Engineering (Harness 工程) -**英文**：Harness Engineering -**中文**：Harness 工程 -**定义**：设计和实现使 AI 智能体可靠工作的系统的新学科，包括约束智能体行为、告知智能体应该做什么、验证智能体正确执行、纠正智能体错误四个方面。 -**相关概念**：[[Context-Engineering|上下文工程]], [[Architectural-Constraints|架构约束]], [[Entropy Management|熵管理]] +### Representational Transformation (表征转换) +**英文**: Representational Transformation +**中文**: 表征转换 +**定义**: 认知人工制品的核心作用——重组问题，使智能体能够用它已经拥有的能力更可靠地解决问题。 -### Harness Template (Harness 模板) -**英文**：Harness Template -**中文**：Harness 模板 -**定义**：为常见服务拓扑（如数据仪表板、CRUD 业务服务、事件处理器）准备的引导和传感器捆绑包，可实例化用于特定项目。 -**相关概念**：[[Harness-Engineering|Harness 工程]] +## 记忆系统 -## L +### Memory System (记忆系统) +**英文**: Memory System +**中文**: 记忆系统 +**定义**: 外部化 Agent 状态跨时间的系统，允许积累的知识在单个会话之外持续存在，并在相关时被选择性检索。 +**参见**: [[Memory-Systems|记忆系统]] -### LangChain Harness Engineering (LangChain Harness 工程) -**英文**：LangChain Harness Engineering -**中文**：LangChain Harness 工程 -**定义**：LangChain 团队通过仅改变 Harness 将编码智能体在 Terminal Bench 2.0 上的表现从 52.8% 提升到 66.5%（Top 30 到 Top 5）的实践。 -**相关概念**：[[Harness-Engineering|Harness 工程]], [[Self-Verification|自我验证]] +### Working Context (工作上下文) +**英文**: Working Context +**中文**: 工作上下文 +**定义**: 当前任务的实时中间状态：打开的文件、临时变量、活跃假设、部分计划和执行检查点。 -### Layered Domain Architecture (分层领域架构) -**英文**：Layered Domain Architecture -**中文**：分层领域架构 -**定义**：OpenAI 采用的严格架构模型，每个业务领域划分为固定的层组（Types → Config → Repo → Service → Runtime → UI），依赖方向经过严格验证。 -**相关概念**：[[Architectural-Constraints|架构约束]] +### Episodic Experience (情景经验) +**英文**: Episodic Experience +**中文**: 情景经验 +**定义**: 记录先前运行中发生的事情：决策点、工具调用、失败、结果和反思。 -### Loop Detection (循环检测) -**英文**：Loop Detection -**中文**：循环检测 -**定义**：LangChain 采用的中间件，通过钩子跟踪每个文件的编辑次数，在对同一文件进行 N 次编辑后添加"考虑重新考虑你的方法"的上下文。 -**相关概念**：[[Doom Loop|末日循环]], [[Middleware|中间件]] +### Semantic Knowledge (语义知识) +**英文**: Semantic Knowledge +**中文**: 语义知识 +**定义**: 存储在任何单个情节之外都存在的抽象：领域事实、一般启发式、项目约定和稳定的世界知识。 -## M +### Personalized Memory (个性化记忆) +**英文**: Personalized Memory +**中文**: 个性化记忆 +**定义**: 跟踪关于特定用户、团队或环境的稳定信息：偏好、习惯、重复出现的约束和先前的交互。 -### Middleware (中间件) -**英文**：Middleware -**中文**：中间件 -**定义**：LangChain 结构化 Harness 的方式，通过可组合的中间件层在不修改核心智能体逻辑的情况下添加特定功能。 -**相关概念**：[[LangChain Harness Engineering|LangChain Harness 工程]] +## 技能系统 -### MiniMax M2.7 (MiniMax M2.7 模型) -**英文**：MiniMax M2.7 -**中文**：MiniMax M2.7 模型 -**定义**：MiniMax 推出的深度参与自我进化的模型，能够构建复杂智能体 Harness、完成高度复杂的生产力任务，包括 Agent Teams、复杂 Skills 和动态工具搜索。 -**相关概念**：[[Agent Teams|智能体团队]], [[Self-Evolution|自我进化]] +### Skill System (技能系统) +**英文**: Skill System +**中文**: 技能系统 +**定义**: 将程序、最佳实践和操作指导打包成可重用的人工制品的系统，而不是依赖模型的权重在每次调用时重新生成特定任务的知识。 +**参见**: [[Skill-Systems|技能系统]] -### Mitchellh AI Adoption Journey (Mitchellh AI 采用之旅) -**英文**：Mitchellh AI Adoption Journey -**中文**：Mitchellh AI 采用之旅 -**定义**：HashiCorp 创始人 Mitchell Hashimoto 分享的个人 AI 工具采用历程，包括从聊天机器人到始终运行智能体的六个阶段。 -**相关概念**：[[Harness-Engineering|Harness 工程]] +### Operational Procedure (操作程序) +**英文**: Operational Procedure +**中文**: 操作程序 +**定义**: 任务骨架：将复杂工作分解为步骤、阶段、依赖关系和停止条件。 -## N +### Decision Heuristic (决策启发式) +**英文**: Decision Heuristic +**中文**: 决策启发式 +**定义**: 在分支处管理发生情况的实用经验法则，从经验中得出而不是仅靠穷举搜索。 -### NxCode Harness Engineering (NxCode Harness 工程) -**英文**：NxCode Harness Engineering -**中文**：NxCode Harness 工程 -**定义**：NxCode 团队提供的 Harness 工程完整指南，总结了三大支柱、实践框架和常见错误。 -**相关概念**：[[Harness-Engineering|Harness 工程]] +### Normative Constraint (规范性约束) +**英文**: Normative Constraint +**中文**: 规范性约束 +**定义**: 程序被视为可接受的条件，包括测试要求、范围限制、访问限制、可追溯性期望和特定领域操作规则。 -## O +### Progressive Disclosure (渐进式披露) +**英文**: Progressive Disclosure +**中文**: 渐进式披露 +**定义**: 一种分层加载策略，首先暴露技能的存在，仅在需要时才加载更深的细节。 -### OpenAI Harness Engineering (OpenAI Harness 工程) -**英文**：OpenAI Harness Engineering -**中文**：OpenAI Harness 工程 -**定义**：OpenAI 团队在 5 个月内构建了超过 100 万行代码的产品，其中零行代码由人工编写，证明了 Harness 工程在生产规模上的有效性。 -**相关概念**：[[Harness-Engineering|Harness 工程]], [[Codex|Codex 模型]] +## 协议系统 -### Observability Stack (可观测性堆栈) -**英文**：Observability Stack -**中文**：可观测性堆栈 -**定义**：OpenAI 为 Codex 提供的日志、指标和追踪记录展示系统，使智能体能够直接访问应用程序的运行状态。 -**相关概念**：[[Context-Engineering|上下文工程]] +### Agent Protocol (智能体协议) +**英文**: Agent Protocol +**中文**: 智能体协议 +**定义**: 定义了用于发现、调用、委托和权限管理的显式机器可读契约，而不是依赖临时提示级别的协调。 +**参见**: [[Agent-Protocols|智能体协议]] -## P +### Invocation Grammar (调用语法) +**英文**: Invocation Grammar +**中文**: 调用语法 +**定义**: 每个工具调用、API 请求或委托消息都需要的格式：参数名称、类型、排序和返回结构。 -### Planner (规划者) -**英文**：Planner -**中文**：规划者 -**定义**：Anthropic Harness 设计中的三种角色之一，负责将简单的 1-4 句话提示扩展为完整的产品规格。 -**相关概念**：[[Generator|生成者]], [[Evaluator|评估者]] +### Lifecycle Semantics (生命周期语义) +**英文**: Lifecycle Semantics +**中文**: 生命周期语义 +**定义**: 多步交互需要的协调规则：谁接下来行动，允许什么状态转换，任务何时完成或失败。 -## R +### MCP (Model Context Protocol) +**英文**: Model Context Protocol (MCP) +**中文**: 模型上下文协议 (MCP) +**定义**: Anthropic 提出的标准化协议，为智能体提供跨异构服务发现工具、检查其模式和调用它们的方式。 -### Reasoning Sandwich (推理三明治) -**英文**：Reasoning Sandwich -**中文**：推理三明治 -**定义**：LangChain 采用的推理预算分配策略，在规划和验证阶段使用高推理预算，在实现阶段使用中等推理预算。 -**相关概念**：[[Build-Verify Loop|构建-验证循环]] +### A2A (Agent-to-Agent Protocol) +**英文**: Agent-to-Agent Protocol (A2A) +**中文**: 智能体到智能体协议 (A2A) +**定义**: Google 提出的标准化智能体间通信协议，支持能力发现、任务委托和状态交换。 -### Ralph Wiggum Loop (Ralph Wiggum 循环) -**英文**：Ralph Wiggum Loop -**中文**：Ralph Wiggum 循环 -**定义**：使用钩子在智能体退出时强制其继续执行的循环模式，用于验证环节。 -**相关概念**：[[Self-Verification|自我验证]] +## Harness 工程 -## S +### Agent Loop (智能体循环) +**英文**: Agent Loop +**中文**: 智能体循环 +**定义**: Harness 的时间骨干，实现感知-检索-计划-行动-观察周期。 +**参见**: [[Harness-Engineering|Harness 工程]] -### Self-Verification (自我验证) -**英文**：Self-Verification -**中文**：自我验证 -**定义**：智能体通过运行测试、阅读完整输出并与原始要求进行比较来自我改进的能力。 -**相关概念**：[[Build-Verify Loop|构建-验证循环]] +### Sandboxing (沙箱) +**英文**: Sandboxing +**中文**: 沙箱 +**定义**: 创建受控的执行边界，限制 Agent 可以读取、写入和修改的内容，并提供使失败可诊断和回滚可行的可再现性保证。 -### Self-Evolution (自我进化) -**英文**：Self-Evolution -**中文**：自我进化 -**定义**：模型深度参与自身进化的过程，包括更新自身记忆、构建复杂技能、根据实验结果改进学习过程和 Harness。 -**相关概念**：[[MiniMax M2.7|MiniMax M2.7 模型]] +### Observability (可观测性) +**英文**: Observability +**中文**: 可观测性 +**定义**: 使 Agent 的内部轨迹对开发者、操作员和 Agent 本身可见的机制，包括结构化日志、执行轨迹和聚合指标。 -### Sensor (传感器) -**英文**：Sensor -**中文**：传感器 -**定义**：观察智能体行动后结果并帮助其自我纠正的反馈控制，包括计算型和推理型两种类型。 -**相关概念**：[[Feedback|反馈控制]] +### Context Budget Management (上下文预算管理) +**英文**: Context Budget Management +**中文**: 上下文预算管理 +**定义**: 主动管理最稀缺的共享资源——上下文窗口——的策略，包括摘要、基于优先级的驱逐和分阶段加载。 + +## 历史演进 + +### Weights Layer (权重层) +**英文**: Weights Layer +**中文**: 权重层 +**定义**: LLM 部署的最早浪潮，其中能力几乎完全与模型参数等同。 +**参见**: [[From-Weights-to-Context-to-Harness|从权重到上下文到 Harness]] + +### Context Layer (上下文层) +**英文**: Context Layer +**中文**: 上下文层 +**定义**: 注意力从模型修改转向输入设计的阶段，包括提示工程、思维链、ReAct、RAG 等技术。 + +### Harness Layer (Harness 层) +**英文**: Harness Layer +**中文**: Harness 层 +**定义**: 当前阶段，其中能力延伸超出提示管理进入持久基础设施。 + +## 理论基础 + +### Distributed Cognition (分布式认知) +**英文**: Distributed Cognition +**中文**: 分布式认知 +**定义**: 拒绝认知完全驻留在个人心灵内的观点，而是将认知过程定位在人、人工制品、表征和协调实践之间。 + +### Complementary Strategies (互补策略) +**英文**: Complementary Strategies +**中文**: 互补策略 +**定义**: Kirsh 的理论，认为智能体不仅通过在内部更努力地思考来提高性能，还通过重组外部环境使一些认知工作卸载到其中来提高性能。 + +## 实践术语 + +### Context Anxiety (上下文焦虑) +**英文**: Context Anxiety +**中文**: 上下文焦虑 +**定义**: 一些模型表现出的倾向，当它们接近认为的上下文限制时，会过早地结束工作。 + +### Context Reset (上下文重置) +**英文**: Context Reset +**中文**: 上下文重置 +**定义**: 完全清除上下文窗口并启动一个新的 Agent，结合结构化移交来携带前一个 Agent 的状态和下一步。 ### Sprint Contract (冲刺契约) -**英文**：Sprint Contract -**中文**：冲刺契约 -**定义**：在每个冲刺前，Generator 和 Evaluator 协商达成的协议，定义该阶段工作的"完成"标准。 -**相关概念**：[[Anthropic-Harness-Design|Anthropic Harness 设计]] +**英文**: Sprint Contract +**中文**: 冲刺契约 +**定义**: 在每个冲刺之前，生成器和评估器协商的协议，就在编写任何代码之前该工作块的"完成"是什么样子达成一致。 -## T +### Planner-Generator-Evaluator (规划器-生成器-评估器) +**英文**: Planner-Generator-Evaluator +**中文**: 规划器-生成器-评估器 +**定义**: 一种三 Agent 架构，用于长运行自主编码：规划器将简单提示扩展为完整规范，生成器一次实现一个功能，评估器测试并评分结果。 +**参见**: [[Long-Running-Harness-Design|长运行应用的 Harness 设计]] -### Terminal Bench (终端基准测试) -**英文**：Terminal Bench -**中文**：终端基准测试 -**定义**：评估智能体编码能力的标准基准测试，包含机器学习、调试、生物学等多个领域的任务。 -**相关概念**：[[LangChain Harness Engineering|LangChain Harness 工程]] +### Agent-First World (智能体优先的世界) +**英文**: Agent-First World +**中文**: 智能体优先的世界 +**定义**: 一种软件工程范式，其中没有一行代码是人工编写的，人类工程师的工作重点转向设计环境、明确意图和构建反馈回路。 +**参见**: [[OpenAI-Codex-Harness-Engineering|OpenAI Codex Harness 工程]] -## W +### Progressive Disclosure (渐进式披露) +**英文**: Progressive Disclosure +**中文**: 渐进式披露 +**定义**: 一种情境管理策略，智能体从一个小而稳定的切入点开始，并被指导下一步该去哪里查看，而不是一开始就被淹没。 +**参见**: [[OpenAI-Codex-Harness-Engineering|OpenAI Codex Harness 工程]] -### Wikilink (Wikilink) -**英文**：Wikilink -**中文**：Wikilink -**定义**：使用 `[[文档标题]]` 格式的内部链接，在 Obsidian 等工具中支持双向链接和图谱视图。 -**相关概念**：[[Backlink|反向链接]], [[Glossary|术语表]] +### Doc-Gardening (文档园艺) +**英文**: Doc-Gardening +**中文**: 文档园艺 +**定义**: 定期运行的智能体，扫描那些不再反映真实代码行为的过时或废弃文档，并发起修复用的 Pull Request。 -### WikiLLM (WikiLLM) -**英文**：WikiLLM -**中文**：WikiLLM -**定义**：本项目的名称，一个利用 LLM 构建个人知识库的系统，通过"编译"原始数据生成结构化、交叉链接的高质量中文 Wiki。 -**相关概念**：[[Glossary|术语表]], [[INDEX]] +### Golden Principles (黄金原则) +**英文**: Golden Principles +**中文**: 黄金原则 +**定义**: 带有主观意见的机械规则，旨在保持代码库的可读性和一致性，以便将来运行智能体。 ---- +### AI Slop (AI 残渣) +**英文**: AI Slop +**中文**: AI 残渣 +**定义**: 智能体复现代码仓库中已存在的不均衡或不够理想的模式，随着时间的推移导致的漂移。 -*最后更新：2026-04-07* -*本文档由 [[WikiLLM]] 自动生成* +### Meta-Harness (元 Harness) +**英文**: Meta-Harness +**中文**: 元 Harness +**定义**: 一个外环系统，用于搜索和优化 LLM 应用的 Harness 代码，使用编码智能体通过文件系统访问完整历史记录（源代码、执行轨迹、分数）来提议和评估新的 Harness。 +**参见**: [[Meta-Harness|Meta-Harness：模型 Harness 的端到端优化]] + +### Code-Space Search (代码空间搜索) +**英文**: Code-Space Search +**中文**: 代码空间搜索 +**定义**: Meta-Harness 的关键设计选择，将 Harness 优化发生在代码空间中，通过检查执行轨迹推断为什么失败以及哪些早期设计选择导致了失败，而不仅仅是失败本身。 + +### Computational vs Inferential (计算型 vs 推理型) +**英文**: Computational vs Inferential +**中文**: 计算型 vs 推理型 +**定义**: Harness 中指南和传感器的两种执行类型：计算型是确定性且快速的（测试、lint、类型检查）；推理型是语义分析、AI 代码审查、"LLM 作为法官"。 +**参见**: [[Harness-Engineering-for-Coding-Agent-Users|面向编码智能体用户的 Harness 工程]] + +### Feedforward Guide (前馈指南) +**英文**: Feedforward Guide +**中文**: 前馈指南 +**定义**: 在智能体行动之前提供的指导，包括原则、规则、参考文档、操作指南等，增加智能体第一次就做对的概率。 +**参见**: [[Harness-Engineering-for-Coding-Agent-Users|面向编码智能体用户的 Harness 工程]] + +### Feedback Sensor (反馈传感器) +**英文**: Feedback Sensor +**中文**: 反馈传感器 +**定义**: 在智能体行动之后提供的验证机制，包括静态分析、日志、浏览器测试、代码审查智能体等，用于自我纠正问题。 +**参见**: [[Harness-Engineering-for-Coding-Agent-Users|面向编码智能体用户的 Harness 工程]] + +### Harness Template (Harness 模板) +**英文**: Harness Template +**中文**: Harness 模板 +**定义**: 针对常见应用拓扑（数据仪表板、CRUD 业务服务、事件处理器）的指南和传感器捆绑包，可以作为团队的起点。 +**参见**: [[Harness-Engineering-for-Coding-Agent-Users|面向编码智能体用户的 Harness 工程]] + +### Maintainability Harness (可维护性 Harness) +**英文**: Maintainability Harness +**中文**: 可维护性 Harness +**定义**: 监管内部代码质量和可维护性的 Harness 类别，包括 lint、结构测试、代码覆盖等工具。 +**参见**: [[Harness-Engineering-for-Coding-Agent-Users|面向编码智能体用户的 Harness 工程]] + +### Architecture Fitness Harness (架构适用性 Harness) +**英文**: Architecture Fitness Harness +**中文**: 架构适用性 Harness +**定义**: 定义和检查应用程序架构特征的指南和传感器，类似于架构适用性函数。 +**参见**: [[Harness-Engineering-for-Coding-Agent-Users|面向编码智能体用户的 Harness 工程]] + +### Behaviour Harness (行为 Harness) +**英文**: Behaviour Harness +**中文**: 行为 Harness +**定义**: 引导和感知应用程序是否按需要功能运行的 Harness 类别，包括功能规范和测试套件。 +**参见**: [[Harness-Engineering-for-Coding-Agent-Users|面向编码智能体用户的 Harness 工程]] + +### Rippable Harness (可剥离 Harness) +**英文**: Rippable Harness +**中文**: 可剥离 Harness +**定义**: 设计为可以移除"智能"逻辑的 Harness，当模型变得足够智能不需要时。 +**参见**: [[Harness-Engineering-Complete-Guide|Harness 工程完整指南]] + +### Three Pillars of Harness Engineering (Harness 工程三大支柱) +**英文**: Three Pillars of Harness Engineering +**中文**: Harness 工程三大支柱 +**定义**: OpenAI 框架组织的三个核心类别：上下文工程、架构约束、熵管理（垃圾收集）。 +**参见**: [[Harness-Engineering-Complete-Guide|Harness 工程完整指南]] + +### Context Engineering (上下文工程) +**英文**: Context Engineering +**中文**: 上下文工程 +**定义**: 确保智能体在正确时间拥有正确信息的学科，包括存储库本地文档和动态可观测性数据。 +**参见**: [[Harness-Engineering-Complete-Guide|Harness 工程完整指南]] + +### Architectural Constraints (架构约束) +**英文**: Architectural Constraints +**中文**: 架构约束 +**定义**: 机械地强制执行好代码样子的机制，包括依赖分层规则、确定性 linter、结构测试等。 +**参见**: [[Harness-Engineering-Complete-Guide|Harness 工程完整指南]] + +### Entropy Management (熵管理) +**英文**: Entropy Management +**中文**: 熵管理 +**定义**: 定期清理智能体，用于解决 AI 生成代码库随时间积累的熵（文档漂移、命名约定发散、死代码积累）。 +**参见**: [[Harness-Engineering-Complete-Guide|Harness 工程完整指南]] + +### Reasoning Sandwich (推理三明治) +**英文**: Reasoning Sandwich +**中文**: 推理三明治 +**定义**: 一种推理预算分配策略：规划使用高推理、实现使用中推理、验证使用高推理。 +**参见**: [[LangChain-Harness-Engineering|LangChain Harness 工程实践]] + +### Self-Verification Loop (自我验证循环) +**英文**: Self-Verification Loop +**中文**: 自我验证循环 +**定义**: 让智能体验证其工作的机制：规划与发现、构建、验证、修复。 +**参见**: [[LangChain-Harness-Engineering|LangChain Harness 工程实践]] + +### Managed Agents (托管智能体) +**英文**: Managed Agents +**中文**: 托管智能体 +**定义**: Anthropic 的托管服务，通过一组通用接口虚拟化智能体组件（会话、Harness、沙箱）来运行长 horizon 智能体。 +**参见**: [[Managed-Agents-Decoupling-Brain-from-Hands|Managed Agents：将大脑与手分离]] + +### Session (会话) +**英文**: Session +**中文**: 会话 +**定义**: Managed Agents 中发生的一切的仅追加日志，作为生活在 Claude 上下文窗口之外的上下文对象。 +**参见**: [[Managed-Agents-Decoupling-Brain-from-Hands|Managed Agents：将大脑与手分离]] + +### Meta-Harness (元 Harness) +**英文**: Meta-Harness +**中文**: 元 Harness +**定义**: 一种系统，具有允许许多不同 Harness 的通用接口，而对 Claude 未来将需要的特定 Harness 没有意见。 +**参见**: [[Managed-Agents-Decoupling-Brain-from-Hands|Managed Agents：将大脑与手分离]] + +### Model Self-Evolution (模型自我进化) +**英文**: Model Self-Evolution +**中文**: 模型自我进化 +**定义**: 模型深度参与迭代自己的过程，包括构建强化学习 Harness、更新记忆、驱动自身的强化学习。 +**参见**: [[MiniMax-M27-Self-Evolution|MiniMax M2.7：开启模型的自我进化]] + +### Agent Teams (智能体团队) +**英文**: Agent Teams +**中文**: 智能体团队 +**定义**: 多智能体协作的原生能力，要求角色边界、对抗性推理、协议遵循、行为分化内化到模型中。 +**参见**: [[MiniMax-M27-Self-Evolution|MiniMax M2.7：开启模型的自我进化]] + +### Six Stages of AI Adoption (AI 采用的六个阶段) +**英文**: Six Stages of AI Adoption +**中文**: AI 采用的六个阶段 +**定义**: Mitchell Hashimoto 的采用路径：放弃聊天机器人界面、重现自己的工作、日终智能体、外包确定的任务、工程化 Harness、始终有一个智能体在运行。 +**参见**: [[Mitchellh-AI-Adoption-Journey|Mitchell Hashimoto 的 AI 采用之旅]] diff --git a/wiki/INDEX.md b/wiki/INDEX.md index 4f1be42..cc7f2b4 100644 --- a/wiki/INDEX.md +++ b/wiki/INDEX.md @@ -1,84 +1,72 @@ --- -title: WikiLLM 知识库首页 -tags: [首页, 索引, 导航] -last_updated: 2026-04-07 +title: "WikiLLM 知识库索引" +source: "Externalization in LLM Agents: A Unified Review" +last_updated: 2026-04-11 --- -# WikiLLM 知识库 +# WikiLLM 知识库索引 -欢迎来到 **WikiLLM**——一个关于 [[Harness-Engineering|Harness 工程]] 的中文知识库。本 wiki 基于多篇权威来源编译而成，旨在为 AI 智能体时代的软件工程提供系统化的指南。 +欢迎来到 WikiLLM 知识库！本 wiki 基于论文《Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering》编译而成。 -> **Harness 工程**是设计和实现使 AI 智能体可靠工作的系统的新学科。如果说 2025 年是 AI 智能体验证它们能够编写代码的一年，那么 2026 年就是我们认识到**智能体不是难点——Harness 才是**的一年。 +## 快速导航 ---- +- [[Glossary|术语表]] - 核心概念定义与对照 +- [[Externalization-in-LLM-Agents|LLM Agent 中的外部化]] - 核心理论框架 +- [[Harness-Engineering|Harness 工程]] - 统一集成层 -## 📚 核心概念 +## 核心概念 -- [[Harness-Engineering|Harness 工程]] - Harness 工程的完整概述，包括三大支柱、为什么现在重要、以及实践中的方法 -- [[Context-Engineering|上下文工程]] - 如何确保智能体在正确的时间获得正确的信息 -- [[Architectural-Constraints|架构约束]] - 如何机械地强制执行好代码的样子，而不是仅仅告诉智能体"写好代码" -- [[Anthropic-Harness-Design|Anthropic Harness 设计]] - Anthropic 的三智能体架构：Planner、Generator、Evaluator -- [[Self-Verification|自我验证]] - 让智能体通过构建-验证循环自我改进的技术 +本知识库围绕 LLM Agent 的**外部化框架**组织，涵盖四大支柱： ---- +### 1. 外部化理论 +- [[Externalization-in-LLM-Agents|LLM Agent 中的外部化]] - 外部化作为组织原则 +- [[From-Weights-to-Context-to-Harness|从权重到上下文到 Harness]] - 历史演进路径 -## 🛠️ 实践指南 +### 2. 三大外部化维度 +- [[Memory-Systems|记忆系统]] - 跨时间外部化状态 +- [[Skill-Systems|技能系统]] - 外部化程序专长 +- [[Agent-Protocols|智能体协议]] - 外部化交互结构 -- [[Mitchellh-Adoption-Journey|Mitchellh AI 采用之旅]] - HashiCorp 创始人从怀疑论者到深度用户的六个阶段 -- [[Building-Your-First-Harness|构建你的第一个 Harness]] - 从个人开发者到工程组织的三级实用框架 +### 3. Harness 工程 +- [[Harness-Engineering|Harness 工程]] - 统一协调层 +- [[Harness-Engineering-Complete-Guide|Harness 工程完整指南]] - NxCode 的完整 Harness 工程指南 +- [[Harness-Engineering-for-Coding-Agent-Users|面向编码智能体用户的 Harness 工程]] - Martin Fowler 的指南与传感器框架 +- [[Harness-Engineering-First-Thoughts|Harness 工程：最初的思考]] - Martin Fowler 团队的早期备忘录 +- [[Meta-Harness|Meta-Harness：模型 Harness 的端到端优化]] - 斯坦福/MIT 的自动 Harness 优化研究 ---- +### 4. 实践指南 +- [[Long-Running-Harness-Design|长运行应用的 Harness 设计]] - Anthropic 团队的多 Agent 架构实践 +- [[OpenAI-Codex-Harness-Engineering|OpenAI Codex Harness 工程]] - 完全由智能体生成代码的产品开发实践 +- [[Mitchellh-AI-Adoption-Journey|Mitchell Hashimoto 的 AI 采用之旅]] - HashiCorp 创始人从怀疑论者到深度用户的六个阶段 +- [[LangChain-Harness-Engineering|LangChain Harness 工程实践]] - 从 Top 30 到 Top 5 的 Harness 优化经验 +- [[Managed-Agents-Decoupling-Brain-from-Hands|Managed Agents：将大脑与手分离]] - Anthropic 的托管智能体架构设计 +- [[MiniMax-M27-Self-Evolution|MiniMax M2.7：开启模型的自我进化]] - 模型参与迭代自己的实践 -## 📖 学习路径 +## 学习路径 ### 初学者路径 +1. 从 [[Externalization-in-LLM-Agents|LLM Agent 中的外部化]] 开始，理解核心论点 +2. 阅读 [[From-Weights-to-Context-to-Harness|从权重到上下文到 Harness]]，了解历史背景 +3. 深入三大外部化维度：[[Memory-Systems|记忆]]、[[Skill-Systems|技能]]、[[Agent-Protocols|协议]] +4. 最后学习 [[Harness-Engineering|Harness 工程]] 如何将它们统一 -1. 首先阅读 [[Harness-Engineering|Harness 工程]] 获得概览 -2. 然后阅读 [[Mitchellh-Adoption-Journey|Mitchellh AI 采用之旅]] 了解个人采用路径 -3. 最后阅读 [[Building-Your-First-Harness|构建你的第一个 Harness]] 开始实践 +### 架构师路径 +1. 直接阅读 [[Harness-Engineering|Harness 工程]] 了解六大分析维度 +2. 参考 [[Externalization-in-LLM-Agents|外部化理论]] 作为理论基础 +3. 根据需要深入各模块细节 -### 深入学习路径 +## 最新研究 -1. 从 [[Anthropic-Harness-Design|Anthropic Harness 设计]] 开始了解前沿架构 -2. 深入研究 [[Context-Engineering|上下文工程]] 和 [[Architectural-Constraints|架构约束]] -3. 学习 [[Self-Verification|自我验证]] 技术让智能体自我改进 +本知识库基于 2026 年 4 月发表的最新综述论文和实践报告，涵盖： +- 记忆架构的四代演进（单片上下文 → 检索存储 → 分层编排 → 自适应系统） +- 技能系统从工具使用到能力包的演变 +- 协议生态系统（MCP、A2A、ACP、ANP、A2UI 等） +- Harness 工程的六大分析维度 +- 多 Agent 架构实践（Planner-Generator-Evaluator 三 Agent 系统） ---- +## 相关研究 -## 🔗 快速导航 - -- [[Glossary|术语表]] - 40+ 核心概念的中英对照和解释 -- [概念目录](./concepts/) - 所有核心概念文章 -- [实践目录](./practices/) - 所有实践指南文章 - ---- - -## 📊 编译来源 - -本知识库基于以下权威来源编译： - -1. **OpenAI** - Harness Engineering：在智能体优先的世界中利用 Codex -2. **Anthropic** - Harness design for long-running application development -3. **Martin Fowler** - Harness engineering for coding agent users -4. **LangChain** - Improving Deep Agents with harness engineering -5. **NxCode** - Harness Engineering: The Complete Guide -6. **MiniMax** - MiniMax M2.7: Early Echoes of Self-Evolution -7. **Mitchell Hashimoto** - My AI Adoption Journey - ---- - -## 💡 关于 WikiLLM - -WikiLLM 是一个利用 LLM 构建个人知识库的系统。本项目的核心原则是： - -- **LLM 编写和维护所有 wiki 数据**；手动编辑很少见 -- **用户探索和查询被归档回 wiki** 以增强它 -- **系统专注于 markdown 文件和 Obsidian 兼容格式** -- **图像被下载到本地** 以便 LLM 轻松引用 - -查看 [[Glossary|术语表]] 了解更多核心概念，或从 [[Harness-Engineering|Harness 工程]] 开始阅读！ - ---- - -*最后更新：2026-04-07* -*本文档由 [[WikiLLM]] 自动生成* +- 认知人工制品理论 (Norman, 1991) +- 分布式认知 (Hutchins, 1995) +- 互补策略 (Kirsh, 1995) +- CoALA 架构 diff --git a/wiki/assets/23c98f1d7ae720bfb39190d50e0706c03b177ad8-1999x1320.webp b/wiki/assets/23c98f1d7ae720bfb39190d50e0706c03b177ad8-1999x1320.webp new file mode 100644 index 0000000..68b0d5a Binary files /dev/null and b/wiki/assets/23c98f1d7ae720bfb39190d50e0706c03b177ad8-1999x1320.webp differ diff --git a/wiki/assets/4f67b1c10566552aec514a716ea43544ab330e0b-668x243.webp b/wiki/assets/4f67b1c10566552aec514a716ea43544ab330e0b-668x243.webp new file mode 100644 index 0000000..fc862cf Binary files /dev/null and b/wiki/assets/4f67b1c10566552aec514a716ea43544ab330e0b-668x243.webp differ diff --git a/wiki/assets/73e900af5b9d6ed8c64db0a8e74d4465963556b7-1640x1596.webp b/wiki/assets/73e900af5b9d6ed8c64db0a8e74d4465963556b7-1640x1596.webp new file mode 100644 index 0000000..c3dc236 Binary files /dev/null and b/wiki/assets/73e900af5b9d6ed8c64db0a8e74d4465963556b7-1640x1596.webp differ diff --git a/wiki/assets/903b624ada206b10753a24c6a1367e74a869165d-1080x1080.webp b/wiki/assets/903b624ada206b10753a24c6a1367e74a869165d-1080x1080.webp new file mode 100644 index 0000000..00d1129 Binary files /dev/null and b/wiki/assets/903b624ada206b10753a24c6a1367e74a869165d-1080x1080.webp differ diff --git a/wiki/assets/Image.png b/wiki/assets/Image.png new file mode 100644 index 0000000..4022963 Binary files /dev/null and b/wiki/assets/Image.png differ diff --git a/wiki/assets/OAI_Harness_engineering_Codex_drives_the_app_with_Chrome_DevTools_MCP_to_validate_its_work_desktop-dark.png b/wiki/assets/OAI_Harness_engineering_Codex_drives_the_app_with_Chrome_DevTools_MCP_to_validate_its_work_desktop-dark.png new file mode 100644 index 0000000..56fd84e Binary files /dev/null and b/wiki/assets/OAI_Harness_engineering_Codex_drives_the_app_with_Chrome_DevTools_MCP_to_validate_its_work_desktop-dark.png differ diff --git a/wiki/assets/OAI_Harness_engineering_Giving_Codex_a_full_observability_stack_desktop-dark.png b/wiki/assets/OAI_Harness_engineering_Giving_Codex_a_full_observability_stack_desktop-dark.png new file mode 100644 index 0000000..f72298e Binary files /dev/null and b/wiki/assets/OAI_Harness_engineering_Giving_Codex_a_full_observability_stack_desktop-dark.png differ diff --git a/wiki/assets/OAI_Harness_engineering_Layered_domain_architecture_with_explicit_cross-cutting_boundries_desktop-dark.png b/wiki/assets/OAI_Harness_engineering_Layered_domain_architecture_with_explicit_cross-cutting_boundries_desktop-dark.png new file mode 100644 index 0000000..b874304 Binary files /dev/null and b/wiki/assets/OAI_Harness_engineering_Layered_domain_architecture_with_explicit_cross-cutting_boundries_desktop-dark.png differ diff --git a/wiki/assets/OAI_Harness_engineering_The_limits_of_agent_knowledge_desktop-dark.png b/wiki/assets/OAI_Harness_engineering_The_limits_of_agent_knowledge_desktop-dark.png new file mode 100644 index 0000000..511cf3c Binary files /dev/null and b/wiki/assets/OAI_Harness_engineering_The_limits_of_agent_knowledge_desktop-dark.png differ diff --git a/wiki/assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png b/wiki/assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png index 8e28ca3..e16e8ef 100644 Binary files a/wiki/assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png and b/wiki/assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png differ diff --git a/wiki/assets/a8bef95425966495629095a5cb38bde4a8b13558-1999x997.webp b/wiki/assets/a8bef95425966495629095a5cb38bde4a8b13558-1999x997.webp new file mode 100644 index 0000000..dbff6ea Binary files /dev/null and b/wiki/assets/a8bef95425966495629095a5cb38bde4a8b13558-1999x997.webp differ diff --git a/wiki/assets/agent-knowledge-limits.webp b/wiki/assets/agent-knowledge-limits.webp deleted file mode 100644 index afa0592..0000000 Binary files a/wiki/assets/agent-knowledge-limits.webp and /dev/null differ diff --git a/wiki/assets/cf0719d7832b1f577b7393c84a7c53eecc725ca4-760x200.webp b/wiki/assets/cf0719d7832b1f577b7393c84a7c53eecc725ca4-760x200.webp new file mode 100644 index 0000000..f8ccf90 Binary files /dev/null and b/wiki/assets/cf0719d7832b1f577b7393c84a7c53eecc725ca4-760x200.webp differ diff --git a/wiki/assets/circle.png b/wiki/assets/circle.png new file mode 100644 index 0000000..afb3aac Binary files /dev/null and b/wiki/assets/circle.png differ diff --git a/wiki/assets/d080353d-31aa-4642-8586-3b7f2201bf94.png b/wiki/assets/d080353d-31aa-4642-8586-3b7f2201bf94.png new file mode 100644 index 0000000..1042098 Binary files /dev/null and b/wiki/assets/d080353d-31aa-4642-8586-3b7f2201bf94.png differ diff --git a/wiki/assets/d92a6eb4-a4b8-4906-b76a-d627c814a2c0.gif b/wiki/assets/dd83d19a-156b-481f-9835-9576293c323a.gif similarity index 100% rename from wiki/assets/d92a6eb4-a4b8-4906-b76a-d627c814a2c0.gif rename to wiki/assets/dd83d19a-156b-481f-9835-9576293c323a.gif diff --git a/wiki/assets/fig1-codex-drives-app.webp b/wiki/assets/fig1-codex-drives-app.webp deleted file mode 100644 index 52a2249..0000000 Binary files a/wiki/assets/fig1-codex-drives-app.webp and /dev/null differ diff --git a/wiki/assets/harness-bounded-contexts.png b/wiki/assets/harness-bounded-contexts.png index 56a9a95..76d8f15 100644 Binary files a/wiki/assets/harness-bounded-contexts.png and b/wiki/assets/harness-bounded-contexts.png differ diff --git a/wiki/assets/harness-change-lifecycle-examples.png b/wiki/assets/harness-change-lifecycle-examples.png index 56a9a95..175ddd3 100644 Binary files a/wiki/assets/harness-change-lifecycle-examples.png and b/wiki/assets/harness-change-lifecycle-examples.png differ diff --git a/wiki/assets/harness-continuous-feedback-examples.png b/wiki/assets/harness-continuous-feedback-examples.png deleted file mode 100644 index 56a9a95..0000000 --- a/wiki/assets/harness-continuous-feedback-examples.png +++ /dev/null @@ -1,280 +0,0 @@ - - - - - - -not found - - - - - - - -

404

- -

I’m afraid this is not the document you’re looking for. Try using the -search box above, and good luck.

-
- - - - - - - - - diff --git a/wiki/assets/harness-overview.png b/wiki/assets/harness-overview.png index 56a9a95..d5ea18f 100644 Binary files a/wiki/assets/harness-overview.png and b/wiki/assets/harness-overview.png differ diff --git a/wiki/assets/harness-templates.png b/wiki/assets/harness-templates.png index 56a9a95..c862605 100644 Binary files a/wiki/assets/harness-templates.png and b/wiki/assets/harness-templates.png differ diff --git a/wiki/assets/harness-types.png b/wiki/assets/harness-types.png index 56a9a95..0110fec 100644 Binary files a/wiki/assets/harness-types.png and b/wiki/assets/harness-types.png differ diff --git a/wiki/assets/img-2.png b/wiki/assets/img-2.png deleted file mode 100644 index 4550042..0000000 Binary files a/wiki/assets/img-2.png and /dev/null differ diff --git a/wiki/assets/layered-domain-architecture.webp b/wiki/assets/layered-domain-architecture.webp deleted file mode 100644 index d9bccba..0000000 Binary files a/wiki/assets/layered-domain-architecture.webp and /dev/null differ diff --git a/wiki/assets/memory_f.png b/wiki/assets/memory_f.png new file mode 100644 index 0000000..90699d7 Binary files /dev/null and b/wiki/assets/memory_f.png differ diff --git a/wiki/assets/observability-stack.svg b/wiki/assets/observability-stack.svg deleted file mode 100644 index 5584893..0000000 --- a/wiki/assets/observability-stack.svg +++ /dev/null @@ -1,129 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/wiki/assets/protocol_f.png b/wiki/assets/protocol_f.png new file mode 100644 index 0000000..2d5d0a0 Binary files /dev/null and b/wiki/assets/protocol_f.png differ diff --git a/wiki/assets/skill_f.png b/wiki/assets/skill_f.png new file mode 100644 index 0000000..cfde184 Binary files /dev/null and b/wiki/assets/skill_f.png differ diff --git a/wiki/assets/timeline.png b/wiki/assets/timeline.png new file mode 100644 index 0000000..a748c75 Binary files /dev/null and b/wiki/assets/timeline.png differ diff --git a/wiki/assets/val_vs_test_by_dataset.png b/wiki/assets/val_vs_test_by_dataset.png new file mode 100644 index 0000000..f92cd4a Binary files /dev/null and b/wiki/assets/val_vs_test_by_dataset.png differ diff --git a/wiki/assets/x1 1.png b/wiki/assets/x1 1.png new file mode 100644 index 0000000..58f50d3 Binary files /dev/null and b/wiki/assets/x1 1.png differ diff --git a/wiki/assets/x1.png b/wiki/assets/x1.png new file mode 100644 index 0000000..0c2c2d9 Binary files /dev/null and b/wiki/assets/x1.png differ diff --git a/wiki/assets/x2.png b/wiki/assets/x2.png new file mode 100644 index 0000000..db7671d Binary files /dev/null and b/wiki/assets/x2.png differ diff --git a/wiki/assets/x3 1.png b/wiki/assets/x3 1.png new file mode 100644 index 0000000..3975283 Binary files /dev/null and b/wiki/assets/x3 1.png differ diff --git a/wiki/assets/x3.png b/wiki/assets/x3.png new file mode 100644 index 0000000..06cdc72 Binary files /dev/null and b/wiki/assets/x3.png differ diff --git a/wiki/assets/x4.png b/wiki/assets/x4.png new file mode 100644 index 0000000..7633560 Binary files /dev/null and b/wiki/assets/x4.png differ diff --git a/wiki/assets/x5.png b/wiki/assets/x5.png new file mode 100644 index 0000000..38449fe Binary files /dev/null and b/wiki/assets/x5.png differ diff --git a/wiki/compile-results.tsv b/wiki/compile-results.tsv new file mode 100644 index 0000000..9b31f92 --- /dev/null +++ b/wiki/compile-results.tsv @@ -0,0 +1,12 @@ +raw_path hash last_modified wiki_paths compile_time status +raw/Externalization in LLM Agents A Unified Review of Memory, Skills, Protocols and Harness Engineering.md sha256:initial 2026-04-11 wiki/concepts/Externalization-in-LLM-Agents.md,wiki/concepts/Harness-Engineering.md,wiki/concepts/Memory-Systems.md,wiki/concepts/Skill-Systems.md,wiki/concepts/Agent-Protocols.md,wiki/concepts/From-Weights-to-Context-to-Harness.md,wiki/INDEX.md,wiki/Glossary.md 2026-04-11 success +raw/Harness design for long-running application development.md sha256:initial 2026-04-11 wiki/practices/Long-Running-Harness-Design.md,wiki/INDEX.md,wiki/Glossary.md 2026-04-11 success +raw/工程技术：在智能体优先的世界中利用 Codex.md sha256:initial 2026-04-11 wiki/practices/OpenAI-Codex-Harness-Engineering.md,wiki/INDEX.md,wiki/Glossary.md 2026-04-11 success +raw/My AI Adoption Journey.md sha256:initial 2026-04-11 wiki/practices/Mitchellh-AI-Adoption-Journey.md,wiki/INDEX.md,wiki/Glossary.md 2026-04-11 success +raw/Harness engineering for coding agent users.md sha256:initial 2026-04-11 wiki/concepts/Harness-Engineering-for-Coding-Agent-Users.md,wiki/INDEX.md,wiki/Glossary.md 2026-04-11 success +raw/Improving Deep Agents with harness engineering.md sha256:initial 2026-04-11 wiki/practices/LangChain-Harness-Engineering.md,wiki/INDEX.md,wiki/Glossary.md 2026-04-11 success +raw/Scaling Managed Agents Decoupling the brain from the hands.md sha256:initial 2026-04-11 wiki/practices/Managed-Agents-Decoupling-Brain-from-Hands.md,wiki/INDEX.md,wiki/Glossary.md 2026-04-11 success +raw/MiniMax M2.7 开启模型的自我进化.md sha256:initial 2026-04-11 wiki/practices/MiniMax-M27-Self-Evolution.md,wiki/INDEX.md,wiki/Glossary.md 2026-04-11 success +raw/Harness Engineering The Complete Guide to Building Systems That Make AI Agents Actually Work (2026).md sha256:initial 2026-04-11 wiki/concepts/Harness-Engineering-Complete-Guide.md,wiki/INDEX.md,wiki/Glossary.md 2026-04-11 success +raw/Harness Engineering - first thoughts.md sha256:initial 2026-04-11 wiki/concepts/Harness-Engineering-First-Thoughts.md,wiki/INDEX.md,wiki/Glossary.md 2026-04-11 success +raw/Meta-Harness End-to-End Optimization of Model Harnesses.md sha256:initial 2026-04-11 wiki/concepts/Meta-Harness.md,wiki/INDEX.md,wiki/Glossary.md 2026-04-11 success diff --git a/wiki/concepts/Agent-Protocols.md b/wiki/concepts/Agent-Protocols.md new file mode 100644 index 0000000..57acb0b --- /dev/null +++ b/wiki/concepts/Agent-Protocols.md @@ -0,0 +1,142 @@ +--- +title: "智能体协议" +source: "Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering" +raw_sources: + - path: raw/Externalization in LLM Agents A Unified Review of Memory, Skills, Protocols and Harness Engineering.md + hash: "sha256:to-be-computed" +tags: + - "核心概念" + - "协议" + - "外部化" +last_updated: 2026-04-11 +--- + +# 智能体协议 + +**协议外部化**解决了智能体的交互负担问题。它定义了用于发现、调用、委托和权限管理的显式机器可读契约，而不是依赖临时提示级别的与工具、服务和其他智能体的协调。 + +## 协议外部化的四大维度 + +![协议作为外部化交互](../assets/protocol_f.png) + +### 1. 调用语法 + +每个工具调用、API 请求或委托消息都需要一种格式：参数名称、类型、排序和返回结构。 + +- **没有协议**，模型必须在每次调用时推断或重新发明这种语法 +- **协议外部化**将其化为模式和类型化接口，因此模型填充字段而不是猜测语法 + +### 2. 生命周期语义 + +多步交互需要协调：谁接下来行动，允许什么状态转换，任务何时完成或失败。 + +- **协议外部化**将这些排序规则化为显式状态机或事件流 +- **从模型的推理负担中移除它们** + +### 3. 权限和信任边界 + +现实世界的智能体行为受到谁被授权、数据可以流向哪里以及必须产生什么证据的限制。 + +- **协议外部化**将这些约束化为可检查的规则，运行时可以强制执行 +- **而不是依赖模型自我监督** + +### 4. 发现元数据 + +在智能体可以与工具或另一个智能体交互之前，它必须知道有哪些能力可用以及如何到达它们。 + +- **协议外部化**将此发现问题化为注册表、能力卡和模式端点 +- **用可查询的元数据替换隐含的提示嵌入知识** + +## 为什么协议重要 + +Agent 协议的重要性直接源于它们外部化的负担：没有它们，每次交互部分都是关于格式、合法性和协调的推理问题。 + +### 统一的交互标准 + +协议为工具、智能体和前端提供了用于发现、调用、移交和状态交换的共享语法。 + +- **没有该层**，生态系统会分裂为局部提示加解析器集成 +- **标准化交互使互操作性成为设计属性**，而不是幸运的意外 +- **稳定多智能体协作的先决条件** + +### 改进的安全性、治理和可审计性 + +一旦智能体在现实环境中运行，问题不仅是它们是否可以行动，还有这些行动是否保持有界、可检查和可恢复。 + +- **协议通过使权限、身份、执行轨迹、失败状态和责任边界显式化来提供帮助** +- **将以前隐含的粘合逻辑转化为运行时可以验证和操作员可以审计的东西** + +### 减少的供应商依赖 + +开放的交互契约还保留了架构灵活性。 + +- **如果系统在协议层累积能力**，而不是在供应商特定的接口内，模型、供应商和运行时组件可以用更少的重新布线交换 +- **协议不仅是工程便利；它们是智能体生态系统随时间保持可移植和可演化的机制的一部分** + +## Agent 协议系列 + +### 1. Agent-Tool 协议 + +Agent-Tool 协议是最早成熟的协议系列之一，因为工具访问是接口碎片化首先出现的地方。 + +**MCP (Model Context Protocol)** 是最清晰的代表： +- 提供标准化方式让智能体发现工具、检查其模式并跨异构服务调用它们 +- 通过 JSON-RPC 2.0 公开工具和上下文资源 +- 将工具生态系统与模型供应商特定的函数调用格式解耦 + +### 2. Agent-Agent 协议 + +一旦多个智能体协作，交互本身就成为系统问题。Agent-Agent 协议定义能力如何被发现、任务如何被委托、进度和部分状态如何被交换以及结果如何返回给调用者。 + +**代表性协议**： +- **A2A** - 标准化通过 Agent Card 等人工制品进行的能力发现，并支持异构智能体之间的面向任务的通信、状态更新、协商和流式进度 +- **ACP** - 通过熟悉的 REST/HTTP 模式强调轻量级采用 +- **ANP** - 推动相反方向，目标是具有去中心化身份、跨域发现和安全端到端通信的开放、互联网级互操作性 + +### 3. Agent-User 协议 + +Agent-User 协议形式化了智能体运行时和面向用户的系统之间的边界。 + +**两个主要方向**： +1. **A2UI** - 允许智能体以约束的声明格式描述 UI 结构，主机应用程序可以跨平台安全地呈现 +2. **AG-UI** - 标准化类型化执行事件，如运行开始、文本发射、工具调用参数、工具调用结果、完成和错误 + +### 4. 其他协议 + +除了一般交互系列外，一些协议还针对通用接口不够的高风险垂直工作流。 + +- **UCP (Universal Commerce Protocol)** - 为智能体商务做这件事，标准化目录、请求和结账流程 +- **AP2 (Agent Payments Protocol)** - 为支付做同样的事，强调授权、签名、可审计性和承载证明的事务对象 + +## Harness 工程中的 Agent 协议 + +如果上面的调查显示哪些交互负担在生态系统中被外部化，Harness 工程显示这些协议表面一旦智能体嵌入运行时如何成为运行智能体的一部分。 + +### 意图捕获和规范化 + +意图捕获和规范化是这些表面中的第一个。该层的工作是将模型产生的语言转换为运行时可以验证和采取行动的显式命令或事件。 + +### 能力发现和工具描述 + +能力发现和工具描述形成第二个表面。在较旧的系统中，可用工具的知识通常部分存在于提示中，部分存在于开发者假设中。协议化发现用显式元数据替换了这一点。 + +### 会话和生命周期管理 + +Harness 协议还需要显式的会话和生命周期管理，因为长视野智能体不会作为孤立的单调用运行。 + +## 协议作为认知人工制品 + +协议为交互执行这一点。没有它们，每个外部行动部分都是自然语言推理问题：模型必须推断预期的操作，猜测正确的格式，重建可接受的约束，并希望接收系统正确解释结果。 + +**协议用有界的结构化任务替换了那种开放式推理**： +- 填充类型化字段 +- 遵循声明的状态转换 +- 接收结构化反馈 + +> **模型仍然需要关于是否以及何时行动的判断，但它不再需要在每个步骤上重新发明交互的语法和语义。** + +## 相关研究 + +- [[Externalization-in-LLM-Agents|LLM Agent 中的外部化]] +- [[Harness-Engineering|Harness 工程]] +- [[Skill-Systems|技能系统]] diff --git a/wiki/concepts/Anthropic-Harness-Design.md b/wiki/concepts/Anthropic-Harness-Design.md deleted file mode 100644 index adc4e85..0000000 --- a/wiki/concepts/Anthropic-Harness-Design.md +++ /dev/null @@ -1,142 +0,0 @@ ---- -title: Anthropic Harness 设计 -tags: [核心概念, 架构, 智能体] -source: [Anthropic] -confidence_score: 高 -last_updated: 2026-04-07 ---- - -# Anthropic Harness 设计 - -**Anthropic Harness 设计**是 Anthropic 团队 Prithvi Rajasekaran 提出的多智能体架构，用于长时间运行的应用开发。该设计受生成对抗网络（GAN）启发，使用**生成器**和**评估器**智能体的组合。 - -## 问题背景 - -### 朴素实现的不足 - -之前的工作已经表明，Harness 设计对长时间运行的智能体编码的有效性有重大影响。但对于更复杂的任务，智能体仍然倾向于随着时间推移偏离轨道。观察到两种常见的失败模式： - -1. **上下文窗口填充和"上下文焦虑"**：模型在冗长任务上随着上下文窗口填充而失去连贯性。一些模型还表现出"上下文焦虑"，当它们接近认为的上下文限制时会提前结束工作。 - -2. **自我评估问题**：当被要求评估它们自己生成的作品时，智能体倾向于自信地赞扬作品——即使，对于人类观察者来说，质量明显平庸。这个问题对于主观任务（如设计）特别明显，没有二进制检查相当于可验证的软件测试。 - -## 核心架构 - -### 三智能体系统 - -Anthropic 最终构建了一个三智能体系统，每个智能体解决先前运行中观察到的特定差距： - -| 角色 | 职责 | -|------|------| -| **Planner（规划者）** | 将简单的 1-4 句话提示扩展为完整的产品规格 | -| **Generator（生成者）** | 一次一个功能地实现规格，带有 git 版本控制 | -| **Evaluator（评估器）** | 使用 Playwright MCP 测试运行中的应用，分级并提供详细反馈 | - -### 上下文重置 vs 压缩 - -**上下文重置**：清空上下文窗口并启动一个新的智能体，结合携带先前智能体状态和下一步的结构化交接。 - -**上下文压缩**：将对话的早期部分就地摘要，以便同一个智能体可以在缩短的历史上继续。 - -虽然压缩保留了连续性，但它没有给智能体一个干净的状态，这意味着上下文焦虑仍然可能存在。重置提供了一个干净的状态，但代价是交接工件必须有足够的状态让下一个智能体干净地接手工作。 - -在早期测试中，发现 Claude Sonnet 4.5 表现出足够强的上下文焦虑，以至于仅靠压缩不足以实现良好的长任务性能，因此上下文重置成为 Harness 设计的必要条件。 - -## 前端设计：使主观质量可分级 - -### 两个洞见 - -1. 虽然美学不能完全简化为分数——而且个人品味总是会变化——但它们可以通过编码设计原则和偏好的分级标准来改善。 -2. 通过将前端生成与前端分级分离，可以创建一个推动生成器朝着更强输出发展的反馈循环。 - -### 四个分级标准 - -| 标准 | 描述 | -|------|------| -| **设计质量** | 设计感觉像是一个连贯的整体而不是部分的集合吗？ | -| **原创性** | 是否有自定义决策的证据，还是只是模板布局、库默认值和 AI 生成的模式？ | -| **工艺** | 技术执行：排版层次结构、间距一致性、色彩和谐、对比度。 | -| **功能性** | 独立于美学的可用性。用户能理解界面做什么吗？ | - -强调设计质量和原创性超过工艺和功能性。Claude 已经默认在工艺和功能性上得分很高，因为所需的技术能力往往自然而然地出现在模型中。但在设计和原创性上，Claude 经常产生充其量是乏味的输出。 - -### 生成-评估循环 - -构建在 Claude Agent SDK 上的循环： - -1. 生成器智能体首先根据用户提示创建 HTML/CSS/JS 前端 -2. 给予评估器 Playwright MCP，让它在评分每个标准并编写详细批评之前直接与实时页面交互 -3. 该反馈流回生成器作为下一次迭代的输入 -4. 每次生成运行 5 到 15 次迭代 - -在每次评估后，指示生成器做出战略决策：如果分数趋势良好则改进当前方向，或者如果方法不起作用则转向完全不同的美学。 - -## 扩展到全栈编码 - -### 架构演变 - -Opus 4.5 基本上自行消除了上下文焦虑行为，因此可以完全从 Harness 中删除上下文重置。智能体作为一个连续会话在整个构建中运行，Claude Agent SDK 的自动压缩处理沿途的上下文增长。 - -### 冲刺契约 - -在每个冲刺之前，生成器和评估器协商一个**冲刺契约**：在编写任何代码之前就该工作块的"完成"看起来像什么达成一致。 - -这存在是因为产品规格故意是高级的，需要一个步骤来弥合用户故事和可测试实现之间的差距。生成器提议它将构建什么以及如何验证成功，评估器审查该提议以确保生成器正在构建正确的东西。两者迭代直到达成一致。 - -### 结果对比 - -使用"创建一个带有关卡编辑器、精灵编辑器、实体行为和可玩测试模式的 2D 复古游戏制作器"提示进行的测试： - -| Harness | 持续时间 | 成本 | -|---------|---------|------| -| Solo | 20 分钟 | $9 | -| 完整 Harness | 6 小时 | $200 | - -Harness 贵了 20 多倍，但输出质量的差异立即显而易见。 - -## Harness 简化迭代 - -### 移除冲刺构造 - -从 Harness 中完全移除了冲刺构造。保留了规划器和评估器，因为每个都继续增加明显的价值。将评估器移动到运行结束时的单次通过，而不是每个冲刺分级。 - -### 模型改进的影响 - -Opus 4.6 的发布提供了进一步减少 Harness 复杂性的动力。Opus 4.6 更仔细地规划、更长时间地维持智能体任务、可以在更大的代码库中更可靠地操作，并且具有更好的代码审查和调试技能来捕捉自己的错误。它还在长上下文检索方面有了实质性改进。 - -### 更新后的 Harness 结果 - -使用"使用 Web Audio API 在浏览器中构建功能完整的 DAW"提示进行的测试： - -- 总持续时间：约 4 小时 -- 总成本：$124.70 - -大部分时间花在构建器上，它在没有 Opus 4.5 所需的冲刺分解的情况下连贯运行了两个多小时。 - -## 关键经验教训 - -随着模型继续改进，可以大致期望它们能够工作更长时间，处理更复杂的任务。在某些情况下，这将意味着随着时间推移，模型周围的支架变得不那么重要，开发人员可以等待下一个模型并看到某些问题自行解决。另一方面，模型越好，开发能够实现超出模型在基线水平所能做到的复杂任务的 Harness 的空间就越大。 - -有几个经验教训值得向前推进： - -1. 用你正在构建的模型进行实验总是好的做法 -2. 在处理更复杂的任务时，有时可以通过分解任务并将专门的智能体应用于问题的每个方面来获得提升 -3. 当新模型落地时，重新检查 Harness 通常是好的做法，剥离那些不再对性能有负载作用的部分，并添加新部分以实现以前可能无法实现的更大能力 - -## 相关概念 - -- [[Harness-Engineering|Harness 工程]] -- [[Generator-Evaluator Loop|生成-评估循环]] -- [[Context Reset|上下文重置]] -- [[Planner|规划者]] -- [[Generator|生成者]] -- [[Evaluator|评估者]] - -## 参考来源 - -1. Anthropic - Harness design for long-running application development - ---- - -*最后更新：2026-04-07* -*本文档由 [[WikiLLM]] 编译* diff --git a/wiki/concepts/Architectural-Constraints.md b/wiki/concepts/Architectural-Constraints.md deleted file mode 100644 index 45bf92f..0000000 --- a/wiki/concepts/Architectural-Constraints.md +++ /dev/null @@ -1,106 +0,0 @@ ---- -title: 架构约束 -tags: [核心概念, 架构, Harness] -source: [OpenAI, Martin Fowler, NxCode] -confidence_score: 高 -last_updated: 2026-04-07 ---- - -# 架构约束 - -**架构约束**（Architectural Constraints）是[[Harness-Engineering|Harness 工程]]的三大支柱之一。这是 Harness 工程与传统 AI 提示最显著不同的地方。与其告诉智能体"编写好代码"，不如**机械地强制执行好代码的样子**。 - -## 核心思想 - -### 约束反而提升生产力 - -矛盾的是，约束解决方案空间使智能体**更有生产力**，而不是更少。当智能体可以生成任何东西时，它会浪费 token 探索死胡同。当 Harness 定义清晰的边界时，智能体会更快地收敛到正确的解决方案。 - -### 为智能体优化代码库可读性 - -由于代码仓库完全由智能体生成，因此首先针对 Codex 的可读性进行了优化。就像团队会努力提升代码对新入职工程师的可导航性一样，人类工程师的目标也是让智能体能够直接从代码仓库推理出完整的业务领域。 - -从智能体的角度来看，它在运行时无法在情境中访问的任何内容都是不存在的。存储在 Google Docs、聊天记录或人们头脑中的知识都无法被系统访问。代码仓库本地的、已版本化的工件（例如，代码、Markdown、模式、可执行计划）就是它所能看到的全部。 - -## 分层领域架构 - -OpenAI 围绕一个严格的架构模型构建了应用。每个业务领域都划分为一组固定的层，依赖方向经过严格验证，并且仅允许有限的一组边。这些约束是通过自定义的 linter（当然是由 Codex 生成的！）和结构测试机械地强制执行的。 - -![分层领域架构](../assets/layered-domain-architecture.webp) - -### 依赖分层规则 - -在每个业务领域内（例如应用设置），代码只能"向前"依赖于一组固定的层： - -``` -Types → Config → Repo → Service → Runtime → UI -``` - -横切关注点（认证、连接器、遥测、功能标志）通过一个单一的显式接口进入：Providers。其他任何内容都不被允许，并将通过自动化方式强制执行。 - -这种架构通常要等到你拥有数百名工程师时才会推迟。对于编码智能体来说，这是一个早期的先决条件：有了约束，速度才不会下降，架构才不会漂移。 - -## 约束强制执行工具 - -### 1. 确定性 Linters - -自定义规则，自动标记违规。由于这些 lint 是自定义的，编写错误信息时会在智能体情境中注入修复指令。 - -在以人为本的工作流程中，这些规则可能会让人感到迂腐或束缚。有了智能体，它们就成了倍增器：一旦编码，它们就能立即应用于所有地方。 - -### 2. 基于 LLM 的审计员 - -审查其他智能体代码的架构合规性的智能体。 - -### 3. 结构测试 - -像 ArchUnit，但专门用于 AI 生成的代码。 - -### 4. 预提交钩子 - -在提交任何代码前的自动检查。 - -## 品味不变式 - -OpenAI 团队辅以一小组"品味不变式"。例如： - -- 通过自定义 lint 静态地强制执行结构化日志记录 -- 模式和类型的命名约定 -- 文件大小限制 -- 特定平台的可靠性要求 - -## 明确界限 - -同时，还明确指出了哪些地方需要限制，哪些地方不需要限制。这类似于领导一个大型工程平台组织：在中央层面强制执行边界，在本地层面允许自主权。你非常重视界限、正确性和可重复性。在这些边界内，你允许团队或智能体在解决方案的表达方式上拥有很大的自由。 - -生成的代码不总是符合人类的风格偏好，这也没关系。只要输出是正确的、可维护的，并且对未来的智能体运行而言清晰易读，就可以算作达标。 - -## 依赖选择 - -这一框架明确了许多取舍。倾向于选择那些可以完全内化于在仓库中进行推理的依赖项和抽象。对智能体来说，通常被称为"枯燥"的技术，由于其可组合性、API 稳定性和在训练集里的表现，往往更容易建立模型。 - -在某些情况下，让智能体重新实现部分功能子集比绕过公共库中不透明的上游行为更便宜。例如，OpenAI 团队没有引入通用的 p-limit 风格包，而是投入使用了他们自己的带并发的 map 辅助函数：它与他们的 OpenTelemetry 仪表紧密集成，具备 100% 的测试覆盖率，并且其行为完全符合他们的运行时预期。 - -将系统的更多部分转化为智能体可以检查、验证并直接修改的形式，可以直接提高杠杆效应——这不仅适用于 Codex，也适用于其他智能体也在参与代码库的开发。 - -## 强制执行不变量 - -仅靠文档本身，是没法保持完全由智能体生成的代码库的连贯性的。通过强制执行不变量，而非对实施过程进行微观管理，令智能体能够快速交付，而且不会削弱基础。例如，要求 Codex 在边界处解析数据形状，但不规定具体实现方式（模型似乎偏好 Zod，但没有指定特定库）。 - -## 相关概念 - -- [[Harness-Engineering|Harness 工程]] -- [[Context-Engineering|上下文工程]] -- [[Entropy Management|熵管理]] -- [[Layered Domain Architecture|分层领域架构]] - -## 参考来源 - -1. OpenAI - Harness Engineering：在智能体优先的世界中利用 Codex -2. Martin Fowler - Harness engineering for coding agent users -3. NxCode - Harness Engineering: The Complete Guide - ---- - -*最后更新：2026-04-07* -*本文档由 [[WikiLLM]] 编译自多个来源* diff --git a/wiki/concepts/Context-Engineering.md b/wiki/concepts/Context-Engineering.md deleted file mode 100644 index 5991da2..0000000 --- a/wiki/concepts/Context-Engineering.md +++ /dev/null @@ -1,124 +0,0 @@ ---- -title: 上下文工程 -tags: [核心概念, 上下文, Harness] -source: [OpenAI, Anthropic, LangChain] -confidence_score: 高 -last_updated: 2026-04-07 ---- - -# 上下文工程 - -**上下文工程**（Context Engineering）是[[Harness-Engineering|Harness 工程]]的三大支柱之一，专注于确保智能体在正确的时间获得正确的信息。 - -## 核心原则 - -### 给地图，而不是一千页说明书 - -OpenAI 团队学到的最早经验之一很简单：要给 Codex 的是一张地图，而不是一本 1,000 页的说明书。 - -他们尝试了"一个大型的 AGENTS.md 方法。可想而知，这是一次失败的尝试： - -- **上下文是一种稀缺资源**：一个巨大的指令文件会挤掉任务、代码和相关文档——因此智能体要么会错过关键约束条件，要么开始针对错误的约束条件进行优化。 -- **过多的指导反而变得无效**：当一切都"重要"时，一切都不重要了。智能体最终会在本地进行模式匹配，而不是有意识地进行导航。 -- **它会立即腐烂**：一本庞杂的手册会变成陈旧规则的坟场。智能体无法判断哪些信息仍然有效，一旦人类停止维护它，此文件就会悄然成为一个颇具吸引力的麻烦源头。 -- **这很难核实**：单个 blob 不适合进行机械检查（覆盖率、新鲜度、所有权、交叉链接），因此漂移是不可避免的。 - -因此，他们不再将 AGENTS.md 视为百科全书，而是将其视为内容目录。 - -## 静态上下文与动态上下文 - -### 静态上下文 - -静态上下文是仓库中不会频繁变化的文档和规范： - -- **仓库本地文档**：架构规范、API 契约、风格指南 -- **AGENTS.md 或 CLAUDE.md 文件**：编码项目特定规则 -- **交叉链接设计文档**：由 linter 验证的设计文档 - -### 动态上下文 - -动态上下文是随时间变化的运行时信息： - -- **可观测性数据**：日志、指标、追踪——智能体可访问 -- **目录结构映射**：智能体启动时的目录映射 -- **CI/CD 流水线状态**：测试结果和流水线状态 - -## 关键规则 - -**从智能体的角度来看，它无法在上下文中访问的任何内容都不存在。** - -存储在 Google Docs、Slack 线程或人们头脑中的知识对系统是不可见的。**仓库必须是单一事实来源。** - -### 渐进式披露 - -这个框架实现了渐进式披露：智能体从一个小而稳定的切入点开始，并被指导下一步该去哪里查看，而不是一开始就被淹没。 - -OpenAI 团队严格执行这一点。专职的 linter 和 CI 作业会验证知识库的更新状况、是否已交叉链接且结构正确。一个定期运行的"doc-gardening"智能体会扫描那些不再反映真实代码行为的过时或废弃文档，并发起修复用的 Pull Request。 - -## 实践中的上下文工程 - -### OpenAI 的知识库布局 - -``` -AGENTS.md -ARCHITECTURE.md -docs/ -├── design-docs/ -│ ├── index.md -│ ├── core-beliefs.md -│ └── ... -├── exec-plans/ -│ ├── active/ -│ ├── completed/ -│ └── tech-debt-tracker.md -├── generated/ -│ └── db-schema.md -├── product-specs/ -│ ├── index.md -│ ├── new-user-onboarding.md -│ └── ... -├── references/ -│ ├── design-system-reference-llms.txt -│ ├── nixpacks-llms.txt -│ ├── uv-llms.txt -│ └── ... -├── DESIGN.md -├── FRONTEND.md -├── PLANS.md -├── PRODUCT_SENSE.md -├── QUALITY_SCORE.md -├── RELIABILITY.md -└── SECURITY.md -``` - -设计文档已被编目和索引，其中包括验证状态和一套核心理念，定义了智能体优先的操作原则。架构文档提供域和包分层的顶层地图。一份高质量的文档会对每个产品领域和架构层进行评分，并随着时间的推移追踪差距。 - -计划被视为一流的工件。临时轻量计划用于小幅变更，而复杂工作则记录在执行计划中，并附带进度和决策日志，这些日志会被提交到代码仓库。活跃计划、已完成计划和已知的技术债务都已进行版本控制并集中存放，使智能体能够在不依赖外部情境的情况下运行。 - -### LangChain 的 LocalContextMiddleware - -LangChain 使用 `LocalContextMiddleware` 在智能体启动时运行，映射 `cwd` 和其他父+子目录。他们运行 `bash` 命令来查找工具如 `Python` 安装。上下文发现和搜索容易出错，因此注入上下文减少了这个错误表面并帮助**将智能体引导到其环境中。** - -## 将更多情境推送到仓库中 - -随着时间的推移，需要将越来越多的情境推送到仓库中。那次让团队在架构模式上达成一致的 Slack 讨论？如果智能体无法发现它，那么它就会像迟了三个月入职的新员工一样，对其一无所知。 - -为 Codex 提供更多情境意味着要组织和展示正确的信息，好令智能体能够基于这些信息进行推理，而不是用临时指令使其不堪重负。就像你会在产品原则、工程规范和团队文化（包括表情符号偏好）方面为新队友提供引导一样，将这些信息提供给智能体会带来更一致的输出。 - -## 相关概念 - -- [[Harness-Engineering|Harness 工程]] -- [[Context Reset|上下文重置]] -- [[Context Anxiety|上下文焦虑]] -- [[Architectural-Constraints|架构约束]] - -## 参考来源 - -1. OpenAI - Harness Engineering：在智能体优先的世界中利用 Codex -2. LangChain - Improving Deep Agents with harness engineering -3. Anthropic - Effective context engineering for AI agents - ---- - -*最后更新：2026-04-07* -*本文档由 [[WikiLLM]] 编译自多个来源* diff --git a/wiki/concepts/Externalization-in-LLM-Agents.md b/wiki/concepts/Externalization-in-LLM-Agents.md new file mode 100644 index 0000000..ba8b2af --- /dev/null +++ b/wiki/concepts/Externalization-in-LLM-Agents.md @@ -0,0 +1,106 @@ +--- +title: "LLM Agent 中的外部化" +source: "Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering" +raw_sources: + - path: raw/Externalization in LLM Agents A Unified Review of Memory, Skills, Protocols and Harness Engineering.md + hash: "sha256:to-be-computed" +tags: + - "核心概念" + - "外部化" + - "Harness工程" +last_updated: 2026-04-11 +--- + +# LLM Agent 中的外部化 + +**外部化 (Externalization)** 是 LLM Agent 设计的核心组织原则。它指的是将认知负担从模型的内部计算逐步迁移到持久、可检查和可重用的外部结构中的过程。 + +## 核心论点 + +本文的中心论点是：**外部化是统一近期 LLM Agent 在记忆、技能、协议和 Harness 工程方面进展的过渡逻辑**。 + +这不仅仅是一个关于工程便利性的主张，更是一个关于可靠智能体来源的主张：可靠的智能体不仅来自更强大的模型，还来自对任务需求的系统性重组，使得内部能力和外部基础设施共同覆盖所需的全部能力范围。 + +## 人类认知外部化的平行 + +人类文明史可以被解读为认知外部化的历史： + +1. **语言** - 将私人思想转化为可共享的符号形式 +2. **文字** - 将知识从脆弱的生物记忆转移到持久的物质记录 +3. **印刷术** - 在社会规模上机械化知识的再生产 +4. **数字计算** - 将算术和符号操作从神经劳动转移到可编程机器 + +在这些转变中，关键的变化不是人类在没有人工制品的情况下能力下降了，而是人工制品通过将选定的负担向外转移，释放了有限的内部资源用于规划、抽象和创造力。 + +## 认知人工制品理论 + +这一视角在**认知人工制品 (cognitive artifacts)** 思想中有着自然的理论锚点。中心洞见是：外部辅助工具不仅仅是放大不变的内部能力，它们经常改变任务本身。 + +- **购物清单** 不扩展生物记忆容量，它将困难的回忆问题转化为识别问题 +- **地图** 不简单地使导航"更强"，它将隐藏的空间关系转化为可见的结构 + +人工制品的力量在于**表征转换**：它重组了问题，使智能体能够用它已经拥有的能力更可靠地解决问题。 + +## LLM Agent 的三大外部化维度 + +![外部化架构](../assets/x1.png) + +LLM Agent 通过三个不同但相互耦合的外部化维度实现可靠的智能： + +### 1. 记忆系统：跨时间外部化状态 + +记忆系统允许积累的知识——用户偏好、先前轨迹、已解决的歧义、领域事实——在单个会话之外持续存在，并在相关时被选择性检索。 + +**核心转换**：从回忆到识别——智能体不再需要从潜在权重中再生过去的知识；它从持久、可搜索的存储中检索。 + +记忆的四个维度： +- **工作上下文** - 当前任务的实时中间状态 +- **情景经验** - 记录先前运行中发生的事情 +- **语义知识** - 存储在任何单个情节之外都存在的抽象 +- **个性化记忆** - 跟踪关于特定用户、团队或环境的稳定信息 + +### 2. 技能系统：外部化程序专长 + +技能系统将程序、最佳实践和操作指导打包成可重用的人工制品，而不是依赖模型的权重在每次调用时重新生成特定任务的知识。 + +**核心转换**：从生成到组合——智能体从预先验证的组件组装行为，而不是从头即兴创作每个步骤。 + +技能的三个组成部分： +- **操作程序** - 任务骨架：将复杂工作分解为步骤、阶段、依赖关系和停止条件 +- **决策启发式** - 管理分支处发生的情况 +- **规范性约束** - 程序被视为可接受的条件 + +### 3. 协议：外部化交互结构 + +协议定义了用于发现、调用、委托和权限管理的显式机器可读契约，而不是依赖临时提示级别的与工具、服务和其他智能体的协调。 + +**核心转换**：从临时到结构化——模糊、脆弱的通信变得可互操作、可治理的交换。 + +协议外部化的四个维度： +- **调用语法** - 每个工具调用、API 请求或委托消息都需要一种格式 +- **生命周期语义** - 多步交互需要协调 +- **权限和信任边界** - 现实世界的智能体行为受到谁被授权的限制 +- **发现元数据** - 在智能体可以与工具或另一个智能体交互之前，它必须知道有哪些能力可用 + +## Harness 作为统一层 + +Harness 是承载所有三个维度并提供编排逻辑、约束、可观测性和反馈循环的工程层，使外部化认知在实践中连贯一致。 + +Harness 的六大分析维度： +1. **智能体循环和控制流** - 智能体循环是 Harness 的时间骨干 +2. **沙箱和执行隔离** - 创建受控的执行边界 +3. **人工监督和审批门** - 在智能体循环中插入干预点 +4. **可观测性和结构化反馈** - 使智能体的内部轨迹对开发者、操作员和智能体本身可见 +5. **配置、权限和策略编码** - 编码不仅是智能体可以做什么，还有它在什么条件下被允许做什么 +6. **上下文预算管理** - 上下文窗口仍然是任何智能体系统中最稀缺的共享资源 + +## 相关研究 + +- [[Harness-Engineering|Harness 工程]] +- [[Memory-Systems|记忆系统]] +- [[Skill-Systems|技能系统]] +- [[Agent-Protocols|智能体协议]] + +## 参考文档 + +- 原始论文：Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering diff --git a/wiki/concepts/From-Weights-to-Context-to-Harness.md b/wiki/concepts/From-Weights-to-Context-to-Harness.md new file mode 100644 index 0000000..22f92aa --- /dev/null +++ b/wiki/concepts/From-Weights-to-Context-to-Harness.md @@ -0,0 +1,106 @@ +--- +title: "从权重到上下文到 Harness" +source: "Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering" +raw_sources: + - path: raw/Externalization in LLM Agents A Unified Review of Memory, Skills, Protocols and Harness Engineering.md + hash: "sha256:to-be-computed" +tags: + - "核心概念" + - "历史演进" + - "外部化" +last_updated: 2026-04-11 +--- + +# 从权重到上下文到 Harness + +LLM Agent 的近期历史可以被理解为从模型本身逐步向外移动的过程。能力首先被视为权重的属性，然后被视为提示和上下文窗口的属性，现在越来越被视为模型运行所在的更广泛基础设施的属性。 + +![社区主题演进](../assets/timeline.png) + +## 能力在权重中 + +Weights 层对应于现代 LLM 部署的最早浪潮，其中能力几乎完全与模型参数等同。 + +### 核心优势 + +- **快速推理**，无需外部查找 +- **紧凑部署** +- **在许多任务中强大的泛化**，无需特定任务的管道 + +### 限制 + +- **知识、程序和策略与静态人工制品耦合太紧** +- **更新单个事实需要重新训练、知识编辑或通过额外对齐层进行修补** +- **审计为什么模型以某种方式表现很困难**，因为相关知识分布在数十亿个参数上 +- **个性化很尴尬** - 单一权重集被要求为数百万具有不同历史、偏好和约束的用户服务 + +> **参数知识的中心限制是难以选择性地更新、组合和治理。** + +## 能力在上下文中 + +Context 层代表了注意力从模型修改转向输入设计的阶段。 + +### 关键发展 + +- **提示工程** 证明了模型行为可以在不接触权重的情况下大幅改变 +- **思维链 (Chain-of-Thought)** 使中间推理显式化 +- **ReAct** 在单个生成循环中将推理轨迹与工具行动交错 +- **思想树 (Tree of Thoughts)** 将思维链推广为对中间推理状态的刻意搜索 +- **Self-Refine** 引入了迭代自我批评 +- **检索增强生成 (RAG)** 通过在查询时动态将外部文档注入上下文，引入了更系统的外部化形式 + +### 优势 + +- **更灵活的 Agent 设计** +- **开发者可以在运行时附加本地指令、领域知识、输出模式和检索证据**，无需任何梯度更新 +- **迭代提示和检索管道通常比微调更便宜、更快** +- **模型可以保持冻结，而周围的提示模板、检索逻辑和工具规范快速演变** + +### 表示转换 + +这可以通过 Norman 的认知人工制品概念来解释： + +> 困难的回忆问题——"模型知道事实 X 吗？"——被转换为识别问题："鉴于事实 X 已放置在上下文中，模型可以使用它吗？" + +### 约束 + +- **上下文窗口是有限的**，在规模上成本高昂，并且在过度加载边缘相关材料时经常有噪声 +- **长提示会降低性能**而不是提高它："中间丢失"现象表明模型在长输入中不均匀地参与 +- **上下文也是短暂的**：除非状态在其他地方显式外部化，否则每个新会话都以部分健忘症开始 + +## 能力通过基础设施 + +Harness 层代表了当前阶段，其中能力延伸超出提示管理进入持久基础设施。 + +### 早期表现 + +项目如 Auto-GPT 和 BabyAGI 用任务队列、持久记忆和网络访问将 LLM 包裹在循环中，表明即使是最小的 harness 也可以维持没有单个提示可以的行为。 + +### 更原则性的框架 + +- **AutoGen** 形式化了多智能体消息交换 +- **MetaGPT** 添加了基于角色的协作和显式程序 +- **CAMEL** 探索了用于任务分解的结构化对话 +- **Reflexion** 在剧集之间持续反馈 + +### 部署域中的模式 + +- **编码 Agent** 将模型嵌入开发 harness 中，带有文件、shell、版本控制、测试和可重用技能人工制品 +- **研究和企业 Agent** 添加检索、批准、浏览和长视野编排管道 +- **具身和工作流系统** 同样使控制流、环境访问和重用显式化 + +> **反复出现的模式是，可靠性问题越来越多地通过改变环境而不是仅通过提示来解决。** + +## 外部化作为过渡逻辑 + +总之，从权重到上下文到 harness 的路径是 Norman 意义上的外部化故事： + +- **可变知识**从权重移动到检索系统和运行时上下文，将回忆转换为识别 +- **可重用程序**从隐含习惯移动到显式技能，将即兴生成转换为结构化组合 +- **交互规则**从临时提示移动到协议，将模糊协调转换为受治理的契约 +- **运行时可靠性**移动到 harness 逻辑，其中约束、可观测性和反馈循环可以显式化 + +## 相关研究 + +- [[Externalization-in-LLM-Agents|LLM Agent 中的外部化]] +- [[Harness-Engineering|Harness 工程]] diff --git a/wiki/concepts/Harness-Engineering-Complete-Guide.md b/wiki/concepts/Harness-Engineering-Complete-Guide.md new file mode 100644 index 0000000..793d207 --- /dev/null +++ b/wiki/concepts/Harness-Engineering-Complete-Guide.md @@ -0,0 +1,319 @@ +--- +title: "Harness 工程完整指南" +source: "https://www.nxcode.io/resources/news/harness-engineering-complete-guide-ai-agent-codex-2026" +author: "NxCode Team" +published: 2026-03-01 +last_updated: 2026-04-11 +tags: + - concepts + - harness-engineering +raw_sources: + - path: raw/Harness Engineering The Complete Guide to Building Systems That Make AI Agents Actually Work (2026).md + hash: "sha256:initial" +--- + +# Harness 工程完整指南 + +如果说 2025 年是 AI 智能体证明它们可以编写代码的一年，那么 2026 年就是我们认识到**智能体不是难点——Harness 才是**的一年。 + +OpenAI 的 Codex 团队刚刚构建了一个生产应用程序，**超过 100 万行代码**，其中**零行是由人工手写的**。工程师没有编写代码。他们设计了让 AI 可靠编写代码的系统。那个系统——约束、反馈循环、文档、linter 和生命周期管理——就是行业现在所称的 **Harness**。 + +**Harness 工程**是设计这些系统的新学科。它正在改变软件工程师的含义。 + +--- + +## 什么是 Harness 工程？ + +### 马的隐喻 + +"Harness" 一词来自马具——缰绳、马鞍、衔铁——用于将强大但不可预测的动物引导到正确方向的完整设备集。这个隐喻是有意的： + +- **马**是 AI 模型——强大、快速，但它自己不知道去哪里 +- **Harness**是基础设施——约束、护栏、反馈循环，以富有成效地引导模型的力量 +- **骑手**是人类工程师——提供方向，而不是亲自奔跑 + +没有 Harness，AI 智能体就像开放领域中的纯种马。快速、令人印象深刻，但完全无法完成任何有用的事情。 + +### 正式定义 + +**Harness 工程**是设计和实现以下系统的学科： + +1. **约束** AI 智能体可以做什么（架构边界、依赖规则） +2. **告知** 智能体应该做什么（上下文工程、文档） +3. **验证** 智能体正确完成了工作（测试、linting、CI 验证） +4. **纠正** 智能体出错时的问题（反馈循环、自我修复机制） + +Martin Fowler 将其描述为*"我们可以用来控制 AI 智能体的工具和实践"*——但这不仅仅是安全。一个好的 Harness 使智能体**更有能力**，而不仅仅是更受控制。 + +--- + +## 为什么 Harness 工程现在很重要 + +### 模型是商品，Harness 是护城河 + +AI 行业正在面对的令人不安的事实是：**底层模型的重要性不如其周围的系统。** + +LangChain 明确证明了这一点。他们的编码智能体在 Terminal Bench 2.0 上从 **52.8% 提升到 66.5%**——从 **前 30 名跃升至前 5 名**——通过只改变 Harness，模型本身没有任何变化： + +| 更改 | 他们做了什么 | 影响 | +|------|-------------|------| +| 自我验证循环 | 添加完成前检查清单中间件 | 在提交前捕获错误 | +| 上下文工程 | 启动时映射目录结构 | 智能体从一开始就理解代码库 | +| 循环检测 | 跟踪重复的文件编辑 | 防止"末日循环" | +| 推理三明治 | 规划/验证使用高推理，实现使用中推理 | 在时间预算内提高质量 | + +**相同的模型。不同的 Harness。显著更好的结果。** + +### OpenAI 的 100 万行代码证明点 + +OpenAI 的实验是迄今为止最引人注目的证据： + +- **5 个月**的开发 +- 最终产品中**超过 100 万行代码** +- **零手动编写的行**——每一行都是由 Codex 智能体生成的 +- **以人类所需时间的约 1/10 构建** +- 产品有**内部日常用户和外部 alpha 测试者** +- 它**交付、部署、崩溃并得到修复**——所有这些都由 Harness 内的智能体完成 + +工程师的工作？设计 Harness。指定意图。提供反馈。不编写代码。 + +--- + +## Harness 工程的三大支柱 + +OpenAI 的框架将 Harness 工程组织为三个核心类别： + +### 1. 上下文工程 + +上下文工程是关于确保智能体在正确的时间拥有正确的信息。 + +**静态上下文**： +- 存储库本地文档（架构规范、API 契约、风格指南） +- 编码项目特定规则的 `AGENTS.md` 或 `CLAUDE.md` 文件 +- 由 linter 验证的交叉链接设计文档 + +**动态上下文**： +- 智能体可访问的可观测性数据（日志、指标、追踪） +- 智能体启动时的目录结构映射 +- CI/CD 管道状态和测试结果 + +**关键规则**：从智能体的角度来看，它无法在上下文中访问的任何内容都不存在。Google Docs、Slack 线程或人们头脑中的知识对系统是不可见的。**存储库必须是唯一的真实来源。** + +### 2. 架构约束 + +这是 Harness 工程与传统 AI 提示最显著不同的地方。与其告诉智能体"编写好的代码"，不如**机械地强制执行好代码的样子。** + +**依赖分层**： +``` +Types → Config → Repo → Service → Runtime → UI +``` + +每一层只能从其左侧的层导入。这不是建议——它由结构测试和 CI 验证强制执行。 + +**约束强制执行工具**： +- **确定性 linter**——自动标记违规的自定义规则 +- **基于 LLM 的审计器**——审查其他智能体代码的架构合规性的智能体 +- **结构测试**——像 ArchUnit，但用于 AI 生成的代码 +- **预提交钩子**——任何代码提交前的自动检查 + +**为什么约束可以改善输出**：矛盾的是，约束解决方案空间使智能体**更有生产力**，而不是更少。当智能体可以生成任何东西时，它会浪费 token 探索死胡同。当 Harness 定义清晰的边界时，智能体会更快地收敛到正确的解决方案。 + +### 3. 熵管理（"垃圾收集"） + +这是最被低估的组件。随着时间的推移，AI 生成的代码库会积累熵——文档与现实脱节、命名约定发散、死代码积累。 + +Harness 工程通过**定期清理智能体**来解决这个问题： +- **文档一致性智能体**——验证文档与当前代码匹配 +- **约束违规扫描器**——找到通过早期检查的代码 +- **模式强制执行智能体**——识别并修复与既定模式的偏差 +- **依赖审计器**——跟踪并解决循环或不必要的依赖 + +这些智能体按计划运行——每天、每周或由特定事件触发——保持代码库对人类审查者和未来 AI 智能体都健康。 + +--- + +## Harness 工程实践：团队实际如何做 + +### OpenAI 方法：零人工代码 + +OpenAI 的 Harness 工程团队结构： + +| 角色 | 传统 | Harness 工程 | +|------|------|-------------| +| 编写代码 | 主要工作 | 从不 | +| 设计架构 | 工作的一部分 | 主要工作 | +| 编写文档 | 事后考虑 | 关键基础设施 | +| 审查 PR | 代码审查 | 审查智能体输出 + Harness 有效性 | +| 调试 | 阅读代码 | 分析智能体行为模式 | +| 测试 | 编写测试 | 设计智能体执行的测试策略 | + +### Stripe 方法：规模化的 Minions + +Stripe 的内部编码智能体，称为 **Minions**，现在每周产生**超过 1,000 个合并的拉取请求**： + +1. 开发者在 Slack 中发布任务 +2. Minion 编写代码 +3. Minion 通过 CI +4. Minion 打开 PR +5. 人类审查并合并 + +第 1 步和第 5 步之间没有开发者交互。Harness 处理一切——测试执行、CI 验证、风格合规和文档更新。 + +### LangChain 方法：中间件优先 + +LangChain 将他们的 Harness 构建为可组合的中间件层： + +``` +Agent Request + → LocalContextMiddleware (映射代码库) + → LoopDetectionMiddleware (防止重复) + → ReasoningSandwichMiddleware (优化计算) + → PreCompletionChecklistMiddleware (强制执行验证) + → Agent Response +``` + +每个中间件层都添加特定功能，而不修改核心智能体逻辑。这种模块化方法使 Harness 可测试且可演进。 + +--- + +## 构建你的第一个 Harness：实用框架 + +### 级别 1：基础 Harness（单个开发者） + +如果你正在使用 Claude Code、Cursor 或 Codex 进行个人项目： + +**需要设置什么**： +- 带有项目约定的 `CLAUDE.md` 或 `.cursorrules` 文件 +- 用于 linting 和格式化的预提交钩子 +- 智能体可以运行以自我验证的测试套件 +- 具有一致命名的清晰目录结构 + +**设置时间**：1-2 小时 **影响**：防止最常见的智能体错误 + +### 级别 2：团队 Harness（小团队） + +对于 3-10 个共享代码库的开发者团队： + +**添加到级别 1**： +- 带有团队范围约定的 `AGENTS.md` +- 由 CI 强制执行的架构约束 +- 常见任务的共享提示模板 +- 由 linter 验证的文档即代码 +- 专门针对智能体生成 PR 的代码审查检查清单 + +**设置时间**：1-2 天 **影响**：跨团队一致的智能体行为 + +### 级别 3：生产 Harness（工程组织） + +对于运行数十个并发智能体的组织： + +**添加到级别 2**： +- 自定义中间件层（循环检测、推理优化） +- 可观测性集成（智能体读取日志和指标） +- 计划运行的熵管理智能体 +- Harness 版本控制和 A/B 测试 +- 智能体性能监控仪表板 +- 智能体卡住时的升级策略 + +**设置时间**：1-2 周 **影响**：智能体作为自主贡献者运作 + +--- + +## 常见的 Harness 工程错误 + +### 1. 过度工程化控制流 + +> *"如果你过度工程化控制流，下一次模型更新会破坏你的系统。"* + +模型改进迅速。2024 年需要复杂管道的能力现在由单个上下文窗口提示处理。构建你的 Harness 以**可剥离**——当模型变得足够智能不需要时，你应该能够删除"智能"逻辑。 + +### 2. 将 Harness 视为静态的 + +Harness 需要与模型一起演进。当新模型版本改进推理时，你的推理优化中间件可能会适得其反。每次重大模型更新时审查和更新 Harness 组件。 + +### 3. 忽略文档层 + +最有影响力的 Harness 改进通常是最简单的：**更好的文档**。如果你的 `AGENTS.md` 含糊不清，你的智能体输出也会含糊不清。投资于精确、机器可读的文档，作为智能体的基础事实。 + +### 4. 没有反馈循环 + +没有反馈的 Harness 是一个笼子，而不是指南。智能体需要知道它何时成功何时失败。内置： +- 任务完成前的自我验证步骤 +- 作为智能体工作流一部分的测试执行 +- 按任务类型划分的智能体成功率指标 + +### 5. 只有人类的文档 + +如果你的架构决策存在于人们的头脑中或智能体无法访问的 Confluence 页面中，Harness 就有差距。**智能体需要的一切都必须在存储库中。** + +--- + +## Harness 工程与相关概念 + +| 概念 | 范围 | 焦点 | +|------|------|------| +| **提示工程** | 单次交互 | 制作有效的提示 | +| **上下文工程** | 模型上下文窗口 | 模型看到什么信息 | +| **Harness 工程** | 整个智能体系统 | 环境、约束、反馈、生命周期 | +| **智能体工程** | 智能体架构 | 内部智能体设计和路由 | +| **平台工程** | 基础设施 | 部署、扩展、运营 | + +Harness 工程**包括**上下文工程并借鉴提示工程，但它在更高层次上运作——它是关于使智能体可靠的完整系统，而不仅仅是单次交互的输入。 + +--- + +## 这对软件工程师意味着什么 + +### 工作正在改变 + +Harness 工程代表了软件工程师工作的真正演变： + +| 之前 | 之后 | +|------|------| +| 编写代码 | 设计 AI 编写代码的环境 | +| 调试代码 | 调试智能体行为 | +| 审查代码 | 审查智能体输出 + Harness 有效性 | +| 编写测试 | 设计测试策略 | +| 维护文档 | 构建文档作为机器可读基础设施 | + +这并不意味着工程师变得不那么技术。如果说有什么不同的话，Harness 工程需要**更深**的架构思考——你正在设计必须在没有你持续干预的情况下工作的系统。 + +### 重要的技能 + +基于在 NxCode 构建 AI 驱动产品的所见： + +1. **系统思维**——理解约束、反馈循环和文档如何交互 +2. **架构设计**——定义可强制执行且富有成效的边界 +3. **规范编写**——足够精确地表达意图，让智能体可以执行 +4. **可观测性**——构建揭示智能体行为模式的监控 +5. **迭代速度**——快速测试和完善 Harness 配置 + +### 实践经验：什么有效 + +使用多个智能体系统（Claude Code、Codex、Cursor）构建 AI 驱动的 Web 应用程序。产生最大差异的模式： + +- **存储库优先文档**：每个架构决策、命名约定和部署过程都在 repo 中。没有任何东西存在于 Slack 或 Google Docs 中。 +- **增量约束构建**：从基本 linting 开始，随着模式出现添加架构约束，不要尝试预先设计完美的 Harness。 +- **智能体特定审查检查清单**：AI 生成的代码与人类代码有不同的失败模式。审查过程考虑常见的智能体模式（过度抽象、不必要的错误处理、文档漂移）。 +- **多提供商 Harness 设计**：Harness 适用于 Claude、GPT 和 Gemini 模型。提供商无关设计意味着可以切换模型而无需重建整个系统。 + +--- + +## 关键要点 + +1. **Harness 工程是新学科**——设计使 AI 智能体可靠的系统——约束、反馈循环、文档和生命周期管理 +2. **模型是商品；Harness 是护城河**——LangChain 通过只改变 Harness 从基准前 30 名跃升至前 5 名 +3. **OpenAI 用零人工代码构建了 100 万+行**——证明 Harness 工程在生产规模上有效 +4. **三大支柱**：上下文工程、架构约束和熵管理 +5. **简单开始**：一个好的 `AGENTS.md` 和预提交钩子比复杂中间件更有影响力 +6. **工程师的工作正在演变**——从编写代码到设计 AI 编写代码的环境 +7. **构建可剥离的 Harness**——当模型改进时，过度工程化会破裂；保持适应性 + +--- + +## 相关研究 + +- [[Harness-Engineering|Harness 工程]] +- [[OpenAI-Codex-Harness-Engineering|OpenAI Codex Harness 工程]] +- [[LangChain-Harness-Engineering|LangChain Harness 工程实践]] +- [[Harness-Engineering-for-Coding-Agent-Users|面向编码智能体用户的 Harness 工程]] diff --git a/wiki/concepts/Harness-Engineering-First-Thoughts.md b/wiki/concepts/Harness-Engineering-First-Thoughts.md new file mode 100644 index 0000000..77cdb10 --- /dev/null +++ b/wiki/concepts/Harness-Engineering-First-Thoughts.md @@ -0,0 +1,99 @@ +--- +title: "Harness 工程：最初的思考" +source: "https://martinfowler.com/articles/exploring-gen-ai/harness-engineering-memo.html" +author: "Birgitta Böckeler" +last_updated: 2026-04-11 +tags: + - concepts + - harness-engineering +raw_sources: + - path: raw/Harness Engineering - first thoughts.md + hash: "sha256:initial" +--- + +# Harness 工程：最初的思考 + +本文是 Martin Fowler 团队关于 Harness 工程的早期思考备忘录。在撰写本文后，Birgitta Böckeler 进一步分析了这个主题，并撰写了一篇[更深入的文章描述 Harness 工程](https://martinfowler.com/articles/harness-engineering.html)。 + +--- + +## 背景 + +阅读 OpenAI 最近关于["Harness 工程"的文章](https://openai.com/index/harness-engineering/)非常有趣。这篇文章描述了一个团队如何使用"完全没有手动输入的代码"作为强制函数，来构建一个用于用 AI 智能体维护大型应用程序的 Harness。经过 5 个月，他们构建了一个现在超过 100 万行代码的真实产品。 + +这篇文章的标题是"Harness 工程：在智能体优先的世界中利用 Codex"，但在正文中只提到了一次"harness"。也许这个术语是受到 [Mitchell Hashimoto](https://mitchellh.com/writing/my-ai-adoption-journey#step-5-engineer-the-harness) 最近博客文章的启发。无论如何，我喜欢"harness"这个词来描述我们可以用来控制 AI 智能体的工具和实践。 + +--- + +## OpenAI 团队的 Harness 三大类别 + +OpenAI 团队的 Harness 组件混合了确定性和基于 LLM 的方法，分为 3 个类别（基于我的解释）： + +### 1. 上下文工程 +- 代码库中持续增强的知识库 +- 智能体访问动态上下文，如可观测性数据和浏览器导航 + +### 2. 架构约束 +- 不仅由基于 LLM 的智能体监控，还包括确定性的自定义 linter 和结构测试 + +### 3. "垃圾收集" +- 定期运行的智能体，用于查找文档中的不一致性或架构约束的违规行为 +- 对抗熵和衰减 + +他们还强调了这是多么迭代："当智能体遇到困难时，我们将其视为一个信号：识别缺少什么——工具、护栏、文档——并将其反馈回存储库，始终通过让 Codex 自己编写修复。" + +所有描述的措施都专注于提高长期内部质量和可维护性。我在这篇文章中缺少的是对功能和行为的验证。 + +--- + +## 思考与假设 + +撇开这个差距不谈，假设我们可以相信 OpenAI 对这一成功的描述（关于作者和团队，OpenAI 确实有让我们相信 AI 可维护代码的既得利益）——以下是我对文章中*实际*内容的思考。 + +### Harness——未来的服务模板？ + +大多数组织只有两三个主要技术栈——不是每个应用程序都是自己的雪花。这篇文章让我想象一个未来，团队从一组用于常见应用拓扑的 Harness 中挑选来开始。这唤起了今天的服务模板，它们帮助团队在"黄金路径"上实例化新服务。Harness——带有自定义 linter、结构测试、基本上下文和知识文档以及额外的上下文提供者——会成为新的服务模板吗？团队会将它们用作起点，然后随着时间的推移根据其应用程序的具体情况塑造它们吗？ + +使用服务模板，团队随着获得经验而贡献回来，然后其他团队经常在合并更新时遇到困难。我们会看到 Harness 类似的分叉和同步挑战吗？ + +### 运行时必须受到约束以获得更多 AI 自主权？ + +很多早期和当前的 AI 编码炒作假设 LLM 会给我们目标运行时的无限灵活性。生成任何语言、任何模式，没有约束——LLM 会弄清楚的。但对于我们可以信任的大规模、可维护的 AI 生成代码，必须有所放弃。 + +描述的 Harness 表明，增加信任和可靠性需要约束解决方案空间：特定的架构模式、强制执行的边界、标准化的结构。这意味着放弃一些"生成任何东西"的灵活性，以换取充满技术细节的提示、规则和 Harness。 + +### 收敛到有限数量的技术栈和拓扑？ + +随着编码变得越来越少关于输入代码，而越来越多关于指导其生成，AI 可能会推动我们走向更少的技术栈。框架和 SDK 的可用性仍然很重要——我们反复看到，对人类有益的对 AI 也有益。但开发者的品味在那个细节级别上就不那么重要了。接口中的小低效和特质不会那么烦人，因为我们不会直接处理它们。我们可能会选择有好的 Harness 可用的栈，并优先考虑"AI 友好性"。 + +这可能不仅适用于技术栈，还适用于代码库结构和拓扑。我们可能会默认使用更容易用 AI 维护的结构，因为它们更容易被 Harness。OpenAI 团队讨论了架构刚性和强制执行规则。我可以看到的主要焦点领域是保持数据结构稳定以及定义和强制执行模块边界。听起来合理——但没有具体示例，我仍然很难想象"我们要求 Codex 在边界解析数据形状"在他们的 Harness 中实际上是什么样子。 + +但如果我们能广泛弄清楚如何利用代码库设计模式的 Harness，这些拓扑会成为新的抽象层，而不是像那么多 AI 爱好者希望的自然语言本身吗？ + +### 两个未来世界：AI 前 vs AI 后应用维护？ + +假设我们开发了好的 Harness 技术，将 AI 自主权调到 9 并增加我们对结果的信心。哪些技术可以应用于现有应用程序，哪些只适用于从头开始构建并考虑 Harness 的应用程序？ + +对于较旧的代码库，我们需要考虑改造 Harness 是否值得付出努力。AI 可以帮助我们更快地做到这一点，但这些应用程序通常如此非标准化且充满熵，可能不值得。这让我想到在一个从未有过静态代码分析工具的代码库上运行它，然后淹没在警报中。 + +### 你今天的 Harness 是什么？ + +这个团队在他们的 Harness 上工作了 5 个月，这表明这不是你可以为快速结果而跳入的事情。但值得反思你今天的 Harness 是什么。你有预提交钩子吗？里面有什么？你对自定义 linter 有想法吗？你想对你的代码库施加什么架构约束？你尝试过像 ArchUnit 这样的结构测试框架吗？ + +--- + +## 最后的思考 + +不出所料，他们描述的听起来比仅仅生成和维护一堆 Markdown 规则文件要多得多的工作。他们为 Harness 的确定性部分构建了广泛的工具。他们的上下文工程不仅涉及策划知识库，还涉及重要的设计工作——代码设计本身就是上下文的巨大组成部分。 + +OpenAI 团队说："我们最困难的挑战现在集中在设计环境、反馈循环和控制系统上。"这让我想起了 [Chad Fowler 最近关于"重新定位严谨性"的文章](https://aicoding.leaflet.pub/3mbrvhyye4k2e)。听到关于那个严谨性可能去向的具体想法和经验，而不仅仅是希望"更好的模型"会神奇地解决可维护性问题，这令人耳目一新。 + +最后，就这一次，我喜欢这个领域的一个术语。虽然它只有 2 周大——我可能可以屏住呼吸，直到有人把他们的单提示、基于 LLM 的代码审查智能体称为 Harness…… + +--- + +## 相关研究 + +- [[Harness-Engineering-for-Coding-Agent-Users|面向编码智能体用户的 Harness 工程]] +- [[Harness-Engineering|Harness 工程]] +- [[OpenAI-Codex-Harness-Engineering|OpenAI Codex Harness 工程]] diff --git a/wiki/concepts/Harness-Engineering-for-Coding-Agent-Users.md b/wiki/concepts/Harness-Engineering-for-Coding-Agent-Users.md new file mode 100644 index 0000000..b392f78 --- /dev/null +++ b/wiki/concepts/Harness-Engineering-for-Coding-Agent-Users.md @@ -0,0 +1,163 @@ +--- +title: "面向编码智能体用户的 Harness 工程" +source: "https://martinfowler.com/articles/harness-engineering.html" +author: "Birgitta Böckeler" +published: 2026-04-02 +last_updated: 2026-04-11 +tags: + - concepts + - harness-engineering +raw_sources: + - path: raw/Harness engineering for coding agent users.md + hash: "sha256:initial" +--- + +# 面向编码智能体用户的 Harness 工程 + +本文由 Martin Fowler 团队的 Birgitta Böckeler 撰写，提供了一个在编码智能体语境下构建信任的思维模型——通过前馈指南、反馈传感器和迭代式 Harness 工程。 + +## 核心定义 + +**Harness** 这个术语已经成为 AI 智能体中除模型本身外一切的简写——[智能体 = 模型 + Harness](https://blog.langchain.com/the-anatomy-of-an-agent-harness/)。这是一个非常宽泛的定义，因此值得为常见的智能体类别缩小范围。 + +在**编码智能体**的语境下，Harness 的一部分已经内置（例如通过系统提示词、选择的代码检索机制，甚至是[复杂的编排系统](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents)）。但编码智能体也为用户提供了许多功能来构建专门针对用例和系统的**外层 Harness**。 + +![Harness 边界上下文](../assets/harness-bounded-contexts.png) + +**图 1**："Harness" 一词的含义取决于边界上下文。三个同心圆：模型在核心（最终被 harness 的事物），然后是编码智能体的构建者 harness，再外面是编码智能体的用户 harness。 + +--- + +## Harness 的双重目标 + +一个构建良好的外层 Harness 服务于两个目标： + +1. **提高智能体第一次就做对的概率** +2. **提供一个反馈循环，在问题到达人眼之前自我纠正尽可能多的问题** + +最终，它应该减少审查工作并提高系统质量，同时还能减少浪费的 token。 + +![Harness 概述](../assets/harness-overview.png) + +--- + +## 计算型 vs 推理型 + +指南和传感器有两种执行类型： + +| 类型 | 描述 | 示例 | 速度 | 可靠性 | +|------|------|------|------|--------| +| **计算型（Computational）** | 确定性且快速，由 CPU 运行 | 测试、lint、类型检查器、结构分析 | 毫秒到秒 | 可靠 | +| **推理型（Inferential）** | 语义分析、AI 代码审查、"LLM 作为法官" | 语义分析、AI 代码审查 | 更慢更昂贵 | 更不确定 | + +**计算型指南**通过确定性工具提高良好结果的概率。**计算型传感器**足够便宜和快速，可以在每次更改时与智能体一起运行。**推理型控制**当然更昂贵且更不确定，但允许我们既提供丰富的指导，又添加额外的语义判断。尽管具有不确定性，当使用强大的模型或适合手头任务的模型时，推理型传感器尤其可以增加我们的信任。 + +--- + +## 引导循环 + +人类在这其中的工作是通过迭代 Harness 来**引导**智能体。每当问题多次发生时，应该改进前馈和反馈控制，以使问题将来发生的可能性降低，甚至完全防止它。 + +在引导循环中，当然也可以使用 AI 来改进 Harness。编码智能体现在使构建更多自定义控制和更多自定义静态分析变得便宜得多。智能体可以帮助编写结构测试、从观察到的模式生成规则草案、搭建自定义 linter，或从代码库考古中创建操作指南。 + +--- + +## 时机：保持质量向左 + +持续集成的团队一直面临着根据成本、速度和关键程度在开发时间线上分布测试、检查和人工审查的挑战。当渴望持续交付时，理想情况下甚至希望每个提交状态都可部署。你希望检查尽可能靠近生产路径的左侧，因为越早发现问题，修复成本就越低。反馈传感器，包括新的推理型传感器，需要相应地分布在整个生命周期中。 + +**变更生命周期中的前馈和反馈**： +- 什么是足够快的，甚至应该在集成之前运行，甚至在创建提交之前？（例如 linter、快速测试套件、基本代码审查智能体） +- 什么更昂贵，因此只应在集成后在管道中运行，除了重复快速控制之外？（例如突变测试、可以考虑更大图景的更广泛代码审查） + +**持续漂移和健康传感器**： +- 什么类型的漂移会逐渐累积，应该通过在变更生命周期之外持续运行的传感器来监控？（例如死代码检测、测试覆盖质量分析、依赖扫描器） +- 智能体可以监控什么运行时反馈？（例如让它们查找恶化的 SLO 以提出改进建议，或 AI 法官持续采样响应质量并标记日志异常） + +--- + +## 监管类别 + +智能体 Harness 就像一个[控制论](https://en.wikipedia.org/wiki/Cybernetics)调速器，结合前馈和反馈来调节代码库朝其期望状态发展。区分期望状态的多个维度是有用的，按 Harness 应该监管的内容分类。区分这些类别有帮助，因为 Harness 能力和复杂性在它们之间变化，限定这个词为我们提供了更精确的语言，否则这个术语会非常通用。 + +### 可维护性 Harness（Maintainability harness） + +本文中给出的几乎所有示例都是关于监管内部代码质量和可维护性的。这目前是最容易的 Harness 类型，因为我们有很多可以用于此的预先存在的工具。 + +**计算型传感器**可靠地捕获结构性内容：重复代码、循环复杂度、缺失的测试覆盖、架构漂移、风格违规。这些便宜、经过验证且确定。 + +**LLM 可以部分解决需要语义判断的问题**——语义重复代码、冗余测试、暴力修复、过度工程化的解决方案——但昂贵且概率性。不是在每个提交上。 + +**两者都不能可靠地捕获一些更高影响的问题**：问题的误诊、过度工程化和不必要的功能、误解的指令。它们有时会捕获，但不够可靠以减少监督。如果人类首先没有清楚地指定他们想要什么，正确性就不在任何传感器的范围内。 + +### 架构适用性 Harness（Architecture fitness harness） + +这组指南和传感器定义和检查应用程序的架构特征。基本上：[适用性函数](https://www.thoughtworks.com/en-de/radar/techniques/architectural-fitness-function)。 + +**示例**： +- 前馈我们性能要求的技能，以及反馈给智能体它是改进还是降低了性能的性能测试 +- 描述更好可观测性编码约定的技能（如日志标准），以及要求智能体反思它可用的日志质量的调试指令 + +### 行为 Harness（Behaviour harness） + +这是房间里的大象——我们如何引导和感知应用程序是否按照我们需要的方式功能运行？目前，看到大多数给编码智能体高度自主权的人都这样做： + +- **前馈**：功能规范（详细程度不一，从简短提示到多文件描述） +- **反馈**：检查 AI 生成的测试套件是否通过，具有合理的高覆盖，有些人甚至可能用突变测试监控其质量。然后将其与手动测试结合。 + +这种方法对 AI 生成的测试寄予了很大信任，这还不够好。一些同事在[批准夹具](https://lexler.github.io/augmented-coding-patterns/patterns/approved-fixtures/)模式上看到了良好结果，但它在某些领域比其他领域更容易应用。他们有选择地在适合的地方使用它，这不是测试质量问题的全面答案。 + +因此总的来说，我们仍有很多工作要做，以找出功能行为的良好 Harness，增加我们的信心足以减少监督和手动测试。 + +![Harness 类型](../assets/harness-types.png) + +--- + +## Harness 能力 + +并非每个代码库都同样适合 Harnessing。用强类型语言编写的代码库自然有类型检查作为传感器；可明确定义的模块边界提供架构约束规则；像 Spring 这样的框架抽象掉了智能体甚至不必担心的细节，因此隐式地提高了智能体成功的机会。没有这些属性，这些控制就无法构建。 + +这在绿地与遗留项目中表现不同。绿地团队可以从第一天就内置 Harness 能力——技术决策和架构选择决定了代码库的可治理程度。遗留团队，尤其是那些积累了大量技术债务的应用程序，面临更难的问题：Harness 最需要的地方也是最难构建的地方。 + +--- + +## Harness 模板 + +大多数企业都有一些常见的服务拓扑，覆盖了他们需要的 80%——通过 API 暴露数据的业务服务；事件处理服务；数据仪表板。在许多成熟的工程组织中，这些拓扑已经在服务模板中编码。这些将来可能会演变成 Harness 模板：一组指南和传感器的捆绑包，将编码智能体拴在拓扑的结构、约定和技术栈上。团队可能会部分基于已经有哪些可用的 Harness 来选择技术栈和结构。 + +![Harness 模板](../assets/harness-templates.png) + +我们当然会面临与服务模板类似的挑战。一旦团队实例化它们，它们就开始与上游改进不同步。Harness 模板将面临相同的版本控制和贡献问题，对于非确定性且更难测试的指南和传感器可能甚至更糟。 + +--- + +## 人类的角色 + +作为人类开发者，我们将我们的技能和经验作为隐式 Harness 带到每个代码库。我们吸收了约定和良好实践，我们感受到了复杂性的认知痛苦，我们知道我们的名字在提交上。我们还携带着组织一致性——意识到团队正在努力实现什么，哪些技术债务因业务原因被容忍，以及在这个特定上下文中"好"是什么样子。我们以小步骤和我们人类的速度前进，这为该经验被触发和应用创造了思考空间。 + +编码智能体没有这一切：没有社会责任感，对 300 行函数没有美学厌恶，没有"我们这里不那样做"的直觉，也没有组织记忆。它不知道哪个约定是承载负载的，哪个只是习惯，或者技术上正确的解决方案是否适合团队正在努力做的事情。 + +Harness 是尝试外化并明确人类开发者经验带来的东西，但它只能走这么远。构建一个连贯的指南和传感器系统以及自我纠正循环是昂贵的，因此我们必须以明确的目标优先考虑：一个好的 Harness 不一定旨在完全消除人类输入，而是将其引导到我们的输入最重要的地方。 + +--- + +## 起点——和开放问题 + +这里阐述的思维模型描述了已经在实践中发生的技术，并帮助框架讨论我们仍需要弄清楚的内容。其目标是将对话提升到功能级别之上——从技能和 MCP 服务器到我们如何战略性地设计一个控制系统，让我们对智能体产生的内容真正有信心。 + +以下是当前话语中与 Harness 相关的一些示例： +- [OpenAI 团队记录了他们的 Harness 是什么样子](https://openai.com/index/harness-engineering/)：由自定义 linter 和结构测试强制执行的分层架构，以及定期的"垃圾收集"来扫描漂移并让智能体建议修复。他们的结论："我们最困难的挑战现在集中在设计环境、反馈循环和控制系统上。" +- [Stripe 关于他们的 minions 的文章](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents)描述了诸如基于启发式运行相关 linter 的预推送钩子之类的事情，他们强调了"向左反馈"对他们有多重要，他们的"蓝图"展示了他们如何将反馈传感器集成到智能体工作流中。 +- 突变和结构测试是计算型反馈传感器的示例，它们在过去未被充分利用，但现在正在复兴。 +- 开发者之间关于在编码智能体中集成 LSP 和代码智能的讨论越来越多，这是计算型前馈指南的示例。 +- 听到 Thoughtworks 团队关于使用计算型和推理型传感器解决架构漂移的故事，例如通过智能体和自定义 linter 的混合提高 API 质量，或通过"清洁大军"提高代码质量。 + +还有很多需要弄清楚的，不仅仅是已经提到的行为 Harness。我们如何保持 Harness 连贯随着它的增长，指南和传感器同步，不相互矛盾？当指令和反馈信号指向不同方向时，我们能在多大程度上信任智能体做出明智的权衡？如果传感器从不触发，这是高质量的标志还是检测机制不足？我们需要一种类似于测试的代码覆盖和突变测试的方法来评估 Harness 覆盖和质量。前馈和反馈控制目前分散在交付步骤中，工具有真正的潜力来帮助配置、同步和推理它们作为一个系统。构建这个外层 Harness 正在成为一个持续的工程实践，而不是一次性配置。 + +--- + +## 相关研究 + +- [[Harness-Engineering|Harness 工程]] +- [[Mitchellh-AI-Adoption-Journey|Mitchell Hashimoto 的 AI 采用之旅]] +- [[OpenAI-Codex-Harness-Engineering|OpenAI Codex Harness 工程]] diff --git a/wiki/concepts/Harness-Engineering.md b/wiki/concepts/Harness-Engineering.md index fe3cfe4..cfb51e2 100644 --- a/wiki/concepts/Harness-Engineering.md +++ b/wiki/concepts/Harness-Engineering.md @@ -1,137 +1,146 @@ --- -title: Harness 工程 -tags: [核心概念, Harness, 架构] -source: [OpenAI, Anthropic, Martin Fowler, LangChain, NxCode] -confidence_score: 高 -last_updated: 2026-04-07 +title: "Harness 工程" +source: "Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering" +raw_sources: + - path: raw/Externalization in LLM Agents A Unified Review of Memory, Skills, Protocols and Harness Engineering.md + hash: "sha256:to-be-computed" +tags: + - "核心概念" + - "Harness工程" + - "智能体架构" +last_updated: 2026-04-11 --- # Harness 工程 -**Harness 工程**（Harness Engineering）是设计和实现使 AI 智能体可靠工作的系统的新学科。如果说 2025 年是 AI 智能体验证它们能够编写代码的一年，那么 2026 年就是我们认识到**智能体不是难点——Harness 才是**的一年。 +**Harness 工程**是构建外部化 Agent 运行时环境的综合学科。它将记忆、技能和协议这三个外部化模块统一成一个连贯的认知环境。 -## 核心定义 +## 什么是 Harness？ -### 什么是 Harness？ +Harness 是将原始模型能力转化为可靠 Agent 行为的脚手架。实用的 Agent 最好被理解为在 Harness 内部运行的模型，而不是带有外围能力的模型。 -术语 "Harness" 来源于马具——缰绳、马鞍、马嚼子——用于引导强大但不可预测的动物朝正确方向前进的全套设备。这个比喻是有意为之的： +> **核心洞见**：Harness 不仅仅是实现便利性，它是设计的认知环境，外部化模块在其中共同生效。 -- **马**是 AI 模型——强大、快速，但自己不知道要去哪里 -- **Harness**是基础设施——约束、护栏、反馈循环，用于高效地引导模型的能力 -- **骑手**是人类工程师——提供方向，而不是亲自奔跑 +### Harness 的功能组成 -没有 Harness，AI 智能体就像开放田野里的纯种马。快速、令人印象深刻，但对于完成任何事情来说完全无用。 +Harness 包含使这种耦合成为可能的外部系统： +- 持久记忆和项目级上下文 +- 可重用技能和可执行例程 +- 用于与工具和服务进行确定性交互的协议化接口 +- 更广泛的运行时基础设施 -### Harness 工程的正式定义 +## Harness 的六大分析维度 -**Harness 工程**是设计和实现以下系统的学科： +![Harness 作为认知环境](../assets/x2.png) -1. **约束** AI 智能体可以做什么（架构边界、依赖规则） -2. **告知** 智能体应该做什么（上下文工程、文档） -3. **验证** 智能体正确执行了（测试、linting、CI 验证） -4. **纠正** 智能体出错时（反馈循环、自我修复机制） +### 1. 智能体循环和控制流 -Martin Fowler 将其描述为"我们可以用来控制 AI 智能体的工具和实践"——但它不仅仅是安全。一个好的 Harness 使智能体**更有能力**，而不仅仅是更受控制。 +智能体循环是 Harness 的时间骨干。最简单的形式是实现感知-检索-计划-行动-观察周期。 -## 为什么 Harness 工程现在如此重要 +**关键控制机制**： +- 最大步数限制 +- 递归深度边界 +- 每步成本上限 +- 超时约束 -### 模型是商品，Harness 是护城河 +这些控制定义了模型推理展开的操作 envelope。 -AI 行业正面临一个令人不安的事实：**底层模型的重要性不如围绕它的系统。** +### 2. 沙箱和执行隔离 -LangChain 明确证明了这一点。他们的编码智能体在 Terminal Bench 2.0 上从 **52.8% 提升到 66.5%**——从 **Top 30 跃升至 Top 5**——对模型没有做任何改变。他们只改变了 Harness： +每当 Agent 对世界采取行动时，Harness 必须决定暴露多少环境以及如何包含意外的副作用。 -| 改变 | 他们做了什么 | 影响 | -|------|-------------|------| -| 自我验证循环 | 添加完成前检查清单中间件 | 在提交前捕获错误 | -| 上下文工程 | 启动时映射目录结构 | 智能体从一开始就理解代码库 | -| 循环检测 | 跟踪重复的文件编辑 | 防止"末日循环" | -| 推理三明治 | 规划/验证使用高推理，实现使用中等推理 | 在时间预算内获得更好的质量 | +**隔离粒度**： +- **云沙箱** - 每个任务在专用云沙箱中运行，带有自己的文件系统快照、网络限制和资源配额 +- **分级权限模式** - 暴露从完全自主执行到每个工具调用都需要强制用户批准的分级权限模式 -**相同的模型。不同的 Harness。显著更好的结果。** +**沙箱的双重作用**： +1. 安全围栏 - 限制危险操作 +2. 认知边界 - 通过移除不相关状态简化 Agent 的操作环境 -### OpenAI 的 100 万行代码证明点 +### 3. 人工监督和审批门 -OpenAI 的实验是迄今为止最令人信服的证据： +完全自主很少适合部署的 Agent。大多数生产系统在 Agent 循环中插入干预点。 -- **5 个月**的开发 -- 最终产品中有 **100 万+ 行代码** -- **零行手动编写的代码**——每一行都是由 Codex 智能体生成的 -- **用人类所需时间的约 1/10 构建** -- 产品有**内部日常用户和外部 Alpha 测试者** -- 它**交付、部署、出故障并得到修复**——全部由 Harness 内的智能体完成 +**常见模式**： +- **执行前批准** - 在每个可能有后果的行动之前暂停 Agent 并请求明确确认 +- **执行后审查** - 让 Agent 行动但在提交或继续之前将结果浮出水面进行检查 +- **升级触发器** - 允许 Agent 在正常条件下自主运行，但在检测到特定风险信号时暂停并请求人工输入 +- **Hook 系统** - 允许操作员将任意逻辑附加到 Agent 循环中的特定生命周期事件 -工程师的工作？设计 Harness。指定意图。提供反馈。不是编写代码。 +### 4. 可观测性和结构化反馈 -## 三大支柱 +不留下可检查轨迹的 Agent 是无法调试、审计或改进的 Agent。 -OpenAI 的框架将 Harness 工程组织为三个核心类别： +**可观测性的实现**： +- 每个模型调用、工具调用、内存读/写和决策分支的结构化日志 +- 将每个行动与其因果前因联系起来的执行轨迹 +- 聚合指标，如步数、令牌消耗、错误率和延迟分布 -### 1. 上下文工程 +**双重目的**： +1. **外部** - 支持调试、合规审计和事件后分析 +2. **内部** - 关闭将执行结果连接回产生它们的模块的反馈循环 -上下文工程是关于确保智能体在正确的时间获得正确的信息。 +### 5. 配置、权限和策略编码 -**静态上下文：** -- 仓库本地文档（架构规范、API 契约、风格指南） -- 编码项目特定规则的 `AGENTS.md` 或 `CLAUDE.md` 文件 -- 由 linter 验证的交叉链接设计文档 +Harness 必须不仅编码 Agent 可以做什么，还要编码它在什么条件下被允许做什么。 -**动态上下文：** -- 智能体可访问的可观测性数据（日志、指标、追踪） -- 智能体启动时的目录结构映射 -- CI/CD 流水线状态和测试结果 +**分层配置**： +- **用户级设置** - 编码个人偏好和信任边界 +- **项目级设置** - 指定哪些工具可用、哪些文件路径可访问以及哪些命令需要批准 +- **组织级设置** - 施加合规约束、成本上限和单个项目无法覆盖的数据处理规则 -**关键规则：** 从智能体的角度来看，它无法在上下文中访问的任何内容都不存在。Google Docs、Slack 线程或人们头脑中的知识对系统是不可见的。**仓库必须是单一事实来源。** +### 6. 上下文预算管理 -### 2. 架构约束 +上下文窗口仍然是任何 Agent 系统中最稀缺的共享资源。 -这是 Harness 工程与传统 AI 提示最显著不同的地方。与其告诉智能体"编写好代码"，不如**机械地强制执行好代码的样子**。 +**有效管理策略**： +- **摘要** - 将较旧的对话轮次和执行历史压缩为更短的表示 +- **基于优先级的驱逐** - 移除或降级与活动子任务相关性已衰减的上下文条目 +- **分阶段加载** - 确保详细的程序指导仅在检测到匹配的任务模式时才进入上下文 -**依赖分层：** -``` -Types → Config → Repo → Service → Runtime → UI -``` +## 生产系统中的 Harness -每层只能从其左侧的层导入。这不是建议——它由结构测试和 CI 验证强制执行。 +成熟的 Agent 系统在以下方面收敛于惊人相似的 Harness 结构集： -**约束强制执行工具：** -- **确定性 linters**——自动标记违规的自定义规则 -- **基于 LLM 的审计员**——审查其他智能体代码的架构合规性 -- **结构测试**——像 ArchUnit，但用于 AI 生成的代码 -- **预提交钩子**——在提交任何代码前的自动检查 +| 维度 | 共同模式 | +|------|----------| +| **循环和控制流** | 围绕显式循环组织执行，带有终止控制 | +| **沙箱** | 在不同粒度实现执行隔离 | +| **人工监督** | 实现可配置的审批门和 hook 系统 | +| **可观测性** | 产生结构化执行轨迹和日志 | +| **配置和治理** | 跨多个范围分层配置 | +| **上下文预算** | 通过摘要、分阶段加载和驱逐主动管理 | -**为什么约束能改善输出：** 矛盾的是，约束解决方案空间使智能体**更有生产力**，而不是更少。当智能体可以生成任何东西时，它会浪费 token 探索死胡同。当 Harness 定义清晰的边界时，智能体会更快地收敛到正确的解决方案。 +## Harness 作为认知环境 -### 3. 熵管理（"垃圾回收"） +Harness 的重要性超出了普通软件工程意义上的基础设施。它通过确定推理展开的环境来塑造 Agent 的有效认知。 -这是最被低估的组件。随着时间的推移，AI 生成的代码库会积累熵——文档与现实脱节、命名约定分歧、死代码堆积。 +> **关键主张**：Agent 可以知道、记住和做什么，不仅由模型权重固定，还由周围系统提供的访问、持久性和行动条件固定。 -Harness 工程通过**定期清理智能体**来解决这个问题： -- **文档一致性智能体**——验证文档与当前代码匹配 -- **约束违规扫描器**——找到逃过早期检查的代码 -- **模式强制执行智能体**——识别并修复与既定模式的偏差 -- **依赖审计员**——跟踪并解决循环或不必要的依赖 +### 理论视角 -这些智能体按计划运行——每日、每周，或由特定事件触发——保持代码库对人类审查者和未来 AI 智能体都健康。 +1. **Norman 的认知人工制品** - Harness 在系统级别符合此描述。它不仅仅是用更多上下文或更多工具增强模型；它重组了模型面临的表征问题。 -## 相关概念 +2. **Kirsh 的空间智能使用** - Harness 为 Agent 发挥类似作用。它是一个认知生态位，其中信息、工具、权限和程序被安排成使得期望行为更容易执行而不期望行为更难产生。 -- [[Context-Engineering|上下文工程]] -- [[Architectural-Constraints|架构约束]] -- [[Entropy Management|熵管理]] -- [[OpenAI Harness Engineering|OpenAI Harness 工程]] -- [[Anthropic-Harness-Design|Anthropic Harness 设计]] -- [[LangChain Harness Engineering|LangChain Harness 工程]] +3. **分布式认知** - 操作智能分布在模型参数、外部记忆存储、可执行技能、协议定义、工具表面、监控系统和管理它们交互的运行时约束之间。 -## 参考来源 +## 模块交互图 -1. OpenAI - Harness Engineering：在智能体优先的世界中利用 Codex -2. Anthropic - Harness design for long-running application development -3. Martin Fowler - Harness engineering for coding agent users -4. LangChain - Improving Deep Agents with harness engineering -5. NxCode - Harness Engineering: The Complete Guide +![模块间的耦合](../assets/x3.png) ---- +记忆、技能和协议在 Harness 内部通过六个主要流动相互强化： -*最后更新：2026-04-07* -*本文档由 [[WikiLLM]] 编译自多个来源* +1. **记忆到技能** - 经验蒸馏 +2. **技能到记忆** - 执行记录 +3. **技能到协议** - 能力调用 +4. **协议到技能** - 能力生成 +5. **记忆到协议** - 策略选择 +6. **协议到记忆** - 结果同化 + +## 相关研究 + +- [[Externalization-in-LLM-Agents|LLM Agent 中的外部化]] +- [[Memory-Systems|记忆系统]] +- [[Skill-Systems|技能系统]] +- [[Agent-Protocols|智能体协议]] diff --git a/wiki/concepts/Memory-Systems.md b/wiki/concepts/Memory-Systems.md new file mode 100644 index 0000000..948816c --- /dev/null +++ b/wiki/concepts/Memory-Systems.md @@ -0,0 +1,118 @@ +--- +title: "记忆系统" +source: "Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering" +raw_sources: + - path: raw/Externalization in LLM Agents A Unified Review of Memory, Skills, Protocols and Harness Engineering.md + hash: "sha256:to-be-computed" +tags: + - "核心概念" + - "记忆系统" + - "外部化" +last_updated: 2026-04-11 +--- + +# 记忆系统 + +**记忆外部化**解决了智能体的时间负担问题。它将 Agent 的状态跨时间与短暂上下文解耦，使得连续性不再依赖于短暂的提示。 + +## 记忆的四大维度 + +![记忆作为外部化状态](../assets/memory_f.png) + +### 1. 工作上下文 + +工作上下文是当前任务的实时中间状态：打开的文件、临时变量、活跃假设、部分计划和执行检查点。 + +- **快速变化**，如果过时就失去价值 +- **没有外部化**，一旦上下文窗口重置或进程被中断就会消失 +- **示例**：编码 Agent 中的草稿、终端状态和工作区工件 + +### 2. 情景经验 + +情景经验记录先前运行中发生的事情：决策点、工具调用、失败、结果和反思。 + +- **价值不仅是档案性的** - 检索的情节可以作为具体先例 +- **帮助 Agent 避免重复已知错误** +- **为以后的抽象提供原材料** +- **示例**：Reflexion 将失败尝试的反思摘要存储为可重用经验 + +### 3. 语义知识 + +语义知识存储在任何单个情节之外都存在的抽象：领域事实、一般启发式、项目约定和稳定的世界知识。 + +- **不是围绕特定时间和地点组织的** +- **情景记忆说案例中发生了什么；语义记忆说什么倾向于在案例中成立** +- **示例**：知识库和检索增强生成 (RAG) 语料库 + +### 4. 个性化记忆 + +个性化记忆跟踪关于特定用户、团队或环境的稳定信息：偏好、习惯、重复出现的约束和先前的交互。 + +- **不应折叠到 Agent 的一般自我改进存储中** +- **用户特定的轨迹遵循不同的保留、检索和隐私规则** +- **示例**：IFRAgent 从移动环境中的演示构建用户习惯存储库 + +## 记忆架构的演进 + +记忆架构可以被理解为四种广泛的范式： + +### 1. 单片上下文 + +早期系统依赖单片上下文：所有相关历史或其摘要都直接保留在提示中。 + +- **优点**：透明且易于原型设计 +- **限制**：容量扩展性差，摘要漂移，状态随会话消失 + +### 2. 带有检索存储的上下文 + +占主导地位的下一步是仅将近期工作状态保留在上下文中，同时将长期轨迹存储在外部并按需检索。 + +- **解决原始容量问题** +- **但将记忆质量转化为检索问题** +- **改进方向**：GraphRAG 添加图结构、ENGRAM 压缩为潜在状态表示、SYNAPSE 在统一情景-语义图上使用扩散激活 + +### 3. 分层记忆和编排 + +一旦平面检索证明不足，系统就会转向分层记忆和编排。关键思想是，并非每个轨迹都应获得相同的保留策略或检索路径。 + +**两种主要设计趋势**： +1. **时空维度中的资源解耦** - 借用操作系统逻辑，将热工作状态与较冷的长尾存储分离，并在任务需求变化时跨层交换信息 +2. **认知功能维度中的语义解耦** - 按功能或内容类型组织记忆，以便异构记录不会都通过同一通道路由 + +### 4. 自适应记忆系统 + +上述架构仍然严重依赖人工设计的启发式。自适应记忆系统更进一步，使模块、路由决策或检索策略对经验做出响应。 + +**两个方向**： +1. **动态模块** - 一些系统在运行时调整架构本身 +2. **基于反馈的策略优化** - 其他系统保持架构相对固定但学习更好的控制策略 + +## Harness 时代的记忆需求 + +随着 Agent 演变为 Harness 时代，记忆系统不再仅仅是孤立的存储模块；相反，它们成为运行时协调连续性、程序重用和治理交互的基板。 + +**关键要求**： +1. **状态与上下文的显式分离** - 不受限制的会话历史累积会导致模型失去跟踪 +2. **与技能系统集成** - 记忆存储先前执行的证据；技能仅在某些证据被提升为显式可重用程序时才开始 +3. **协议耦合** - 工具结果、批准、委托事件和外部状态转换通过协议化接口到达，但只有在被规范化并写入持久状态后才会成为记忆 +4. **共享和治理机制** - 一旦多个 Agent 依赖公共外部化状态，建立读写权限、解决冲突和控制访问配额就变得必要 + +## 记忆作为认知人工制品 + +现代 LLM 是无状态生成器：每次调用都以新的上下文开始，因此必须重建连续性而不是向前携带。 + +**记忆外部化改变了该任务的结构**： +- 从**内部回忆问题**转变为**外部识别和检索问题** +- 模型不再必须从其参数中恢复相关历史；它必须识别并使用记忆系统已经浮出水面的历史切片 + +**这解释了为什么检索质量比原始存储容量更重要**： +- 拥有庞大存储但检索薄弱的系统仍然向模型呈现错误的问题表示 +- 具有强大索引、摘要和上下文选择的适度存储可以使下游推理显著更容易 + +**成功标准**：不是"我们保存了多少？"，而是"我们是否使当前决策清晰可辨？" + +## 相关研究 + +- [[Externalization-in-LLM-Agents|LLM Agent 中的外部化]] +- [[Harness-Engineering|Harness 工程]] +- [[Skill-Systems|技能系统]] diff --git a/wiki/concepts/Meta-Harness.md b/wiki/concepts/Meta-Harness.md new file mode 100644 index 0000000..491c242 --- /dev/null +++ b/wiki/concepts/Meta-Harness.md @@ -0,0 +1,324 @@ +--- +title: "Meta-Harness：模型 Harness 的端到端优化" +source: "https://arxiv.org/html/2603.28052v1" +author: "Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn" +published: 2026 +last_updated: 2026-04-11 +tags: + - concepts + - meta-harness + - research +raw_sources: + - path: raw/Meta-Harness End-to-End Optimization of Model Harnesses.md + hash: "sha256:initial" +--- + +# Meta-Harness：模型 Harness 的端到端优化 + +Meta-Harness 是斯坦福大学和 MIT 联合提出的一个外环系统，用于搜索 LLM 应用的 Harness 代码。它使用一个智能体提议者，通过文件系统访问所有先前候选的源代码、分数和执行轨迹。 + +> **核心洞察**：改变固定 LLM 周围的 Harness 可以在同一基准上产生 **6 倍的性能差距**。Harness——决定存储、检索和向模型显示什么的代码——通常与模型本身一样重要。 + +![Meta-Harness 实验结果](../assets/x1%201.png) + +**图 1**：（左）在文本分类上，Meta-Harness 优于最佳的先前人工设计的 Harness（ACE）和现有的文本优化器（TTT-Discover、OpenEvolve），仅在 4 次评估后就匹配了次优方法的最终准确度。（右）在 TerminalBench-2 上，Meta-Harness 优于所有报告的 Claude Haiku 4.5 harness。 + +--- + +## 问题与动机 + +### Harness 工程的现状 + +Harness 工程是改进 LLM 周围代码以提高整体系统性能的实践。然而，尽管其重要性，Harness 工程仍然主要是人工的：从业者检查失败、调整启发式，并在少量设计上迭代。 + +### 现有文本优化方法的局限性 + +文本优化是一个自然的起点，但这些方法与 Harness 工程不匹配，因为它们通常在短期或严重压缩的反馈下操作： + +- 一些方法仅以当前候选为条件 +- 其他方法主要依赖标量分数 +- 还有一些方法将反馈限制为短模板或 LLM 生成的摘要 + +这是一个实用的可扩展性选择，而不是长期依赖没有信息的证据。Harness 在长期范围内运作：关于存储什么、何时检索它或如何呈现它的单个选择可能在许多推理步骤后影响行为。 + +### 反馈规模对比 + +| 方法 | 历史 | 日志内容 | MTok/iter | +|------|------|----------|-----------| +| OPRO | 窗口 | 过去的（解决方案，分数）对 | 0.002 | +| TextGrad | 最后 | 当前工件的文本反馈 | 0.015 | +| AlphaEvolve | 窗口 | 程序数据库 + 评估分数 | 0.022 | +| GEPA | 摘要 | 来自回滚轨迹的反思反馈 | 0.008 | +| Feedback Descent | 摘要 | 比较 + 文本反馈 | 0.012 | +| TTT-Discover | 窗口 | 先前的解决方案片段 | 0.026 | +| **Meta-Harness** | **完整** | **所有日志和分数** | **10.0** | + +在本文研究的设置中，单个评估可以产生多达 **10,000,000 个 token 的诊断信息**，大约比先前文本优化设置中使用的最大反馈预算高出三个数量级。 + +![Meta-Harness 搜索循环](../assets/x3%201.png) + +**图 2**：Meta-Harness 搜索循环。（1）智能体读取包含所有先前候选的源代码、执行轨迹和分数的文件系统，并提出新的 Harness。（2）我们在评估任务上评估提议的 Harness。（3）所有日志（提议的代码、推理轨迹、评估分数）都存储在文件系统的新目录中，循环重复。 + +--- + +## Meta-Harness 设计 + +Meta-Harness 是一个用于通过端到端搜索优化 Harness 的智能体 Harness。 + +### 目标形式化 + +Harness 是一个有状态的程序，它包装语言模型并确定模型在每一步看到什么上下文。目标很简单：找到使底层模型在目标任务分布上表现最佳的 Harness。 + +形式化地说，令 $M$ 表示固定语言模型，$\mathcal{X}$ 表示任务分布。对于 Harness $H$ 和任务实例 $x\sim\mathcal{X}$，我们执行回滚轨迹 $\tau\sim p_{M}(H,x)$。Harness 为 $M$ 构建提示，模型响应，Harness 在每次交互后更新其状态。特定任务的奖励函数 $r(\tau,x)$ 对轨迹进行评分。Harness 优化的目标是找到最大化期望最终奖励的 Harness： + +$$ +H^{*}=\operatorname*{arg\,max}_{H}\mathbb{E}_{x\sim\mathcal{X},\tau\sim p_{M}(H,x)}\;r(\tau,x), +$$ + +### 关键设计选择 + +1. **通过文件系统暴露完整历史** - 使能选择性诊断原始先前代码和执行轨迹，而不是从压缩的每候选摘要优化 +2. **智能体提议者** - 编码智能体决定检查什么并通过与代码库的直接交互验证编辑 +3. **无父代选择规则** - 提议者可以自由检查任何先前的 Harness 及其执行轨迹 + +### 文件系统存储 + +对于每个先前的候选 Harness，文件系统存储： +- **源代码** +- **评估分数** +- **执行轨迹**（提示、工具调用、模型输出、状态更新） + +文件系统通常远大于提议者的上下文窗口，因此提议者通过终端工具（如 grep 和 cat）查询它，而不是将其作为单个提示摄入。 + +### Meta-Harness 搜索循环 + +1. **智能体读取文件系统** - 包含所有先前候选的源代码、执行轨迹和分数 +2. **提议新 Harness** - 基于检查的历史提出改进 +3. **评估候选** - 在评估任务上测试提议的 Harness +4. **存储日志** - 将所有日志（提议的代码、推理轨迹、评估分数）存储在文件系统中 +5. **重复循环** + +### 算法伪代码 + +``` +算法 1 Meta-Harness 外环 Harness 优化 + +输入: 任务 X, LLM M, 提议者 P, 迭代次数 N + +初始化: 种群 H ▶ 初始有效 Harness 集合 +初始化: 文件系统 D ← ∅ ▶ 存储代码、分数、轨迹 + +for H ∈ H do + E_H ← Evaluate(H, M, X) + D ← D ∪ {(H, E_H)} + +for t = 1 ... N do + 提议者 P 查询文件系统 D ▶ 检查先前的 Harness 和分数 + 提议者 P 提出 k 个新 Harness {H_1,…,H_k} + for H ∈ {H_1,…,H_k} do + if H 通过接口验证 then + D ← D ∪ {(H, Evaluate(H, M, X))} + +返回存储在 D 中的 Harness 的 Pareto 前沿 +``` + +--- + +## 代码空间搜索的优势 + +Harness 优化发生在代码空间中，对检索、记忆或提示构建逻辑的小更改可能在许多步骤后影响行为，使得局部搜索启发式与此问题不匹配。 + +### 因果推理能力 + +通过检查执行轨迹，提议者通常可以推断**为什么**一个 Harness 失败以及哪些早期设计选择可能导致了失败，而不仅仅是**它**失败了。 + +提议者可以在算法结构级别修改 Harness，范围从对检索、记忆或提示构建逻辑的更改到完整程序重写，而不是填写模板或应用预定义的变异算子。 + +### 实际实现细节 + +在实验中，每个 Harness 是一个单文件 Python 程序，修改特定任务的提示、检索、记忆和编排逻辑。 + +- **提议者 P**：带有 Opus-4.6 的 Claude Code +- **指导**：最小的特定领域技能，描述在哪里编写新 Harness、如何检查先前的 Harness 及其执行轨迹，以及它可以和不能修改哪些文件 +- **基础模型 M**：因领域而异，始终冻结 +- **典型运行**：在 20 次迭代中评估约 60 个 Harness + +--- + +## 实验结果 + +### 4.1 在线文本分类 + +遵循在线文本分类设置：LLM 一次接收一个标记示例，更新其记忆，并在预留测试集上评估。使用 GPT-OSS-120B 作为 LLM 文本分类器。 + +**三个数据集**： +1. **LawBench (Law)** - 从案例描述预测刑事指控（215 个类别） +2. **Symptom2Disease (S2D)** - 从症状描述预测疾病（22 个类别） +3. **USPTO-50k** - 从产物分子预测前体反应物（180 个类别） + +**搜索设置**： +- 从主要基线 Harness 初始化搜索种群：zero-shot、few-shot、ACE 和 MCE +- 运行 20 次进化迭代，每次迭代两个候选，产生 40 个候选 Harness + +![在线文本分类测试集指标](../assets/x4.png) + +**表 2**：所有 Harness 在三个数据集上的测试集指标。Ctx 表示上下文中的额外输入 token（千）。†：来自 51 的实现。↓：越低越好。Meta-Harness 在使用更小输入上下文的同时提高了在线文本分类准确度。 + +**结果**： +- Meta-Harness 在 ACE（Agentic Context Engineering）的基础上提高了 **7.7 分**，同时使用的上下文 token 减少了 **4 倍** +- 仅用 **4 次评估**就匹配了次优文本优化器的最终性能 +- 其最终准确度比所有基线高出 **10 分以上** + +**消融研究**： + +| 方法 | 分数 | 代码 | 摘要 | 轨迹 | 中位数 ↑ | 最佳准确度 ↑ | > ZS | +|------|------|------|------|------|---------|-------------|------| +| 仅分数 | ✓ | ✓ | × | × | 34.6 | 41.3 | 26 | +| 分数 + 摘要 | ✓ | ✓ | ✓ | × | 34.9 | 38.7 | 23 | +| Meta-Harness (完整) | ✓ | ✓ | - | ✓ | 50.0 | 56.7 | 39 | + +**关键发现**：完整访问执行轨迹是使能 Harness 搜索的关键成分。摘要不能恢复丢失的信号，甚至可能通过压缩掉诊断有用的细节而造成伤害。 + +--- + +### 4.2 检索增强推理的 Harness + +研究一个有点非标准的奥林匹克数学解决设置：用从大型语料库检索示例的能力增强模型。 + +**检索语料库**： +- ≥ 500,000 个已解决问题，来自八个开源数据集 +- 仔细去重和去污染 +- 手动检查预留示例的顶级 BM25 检索 + +**搜索设置**： +- 使用 Meta-Harness 在 250 道奥林匹克难度数学问题的搜索集上优化 40 次迭代 +- 产生 109 个候选检索 Harness +- 从零样本、少样本和 ACE 初始化搜索种群 +- 基于搜索集性能使用 GPT-OSS-20B 选择单个 Harness + +**评估设置**： +- 在 200 道以前未见的 IMO 级问题上评估（IMO-AnswerBench、IMO-ProofBench、ArXivMath） +- 除了 GPT-OSS-20B 外，还在搜索期间未见的四个模型上评估相同的检索 Harness：GPT-5.4-nano、GPT-5.4-mini、Gemini-3.1-Flash-Lite、Gemini-3-Flash + +**结果**： + +| 方法 | GPT-5.4n | GPT-5.4m | Gem-3.1FL | Gem-3F | GPT-20B | 平均 | +|------|-----------|-----------|------------|---------|----------|------| +| 无检索器 | 23.0 | 28.8 | 28.6 | 42.6 | 47.6 | 34.1 | +| 密集检索 (k=1) | 27.1 (+4.1) | 24.5 (-4.3) | 31.3 (+2.7) | 42.3 (-0.3) | 46.9 (-0.7) | 34.4 (+0.3) | +| 密集检索 (k=5) | 31.1 (+8.1) | 28.3 (-0.5) | 37.1 (+8.5) | 47.2 (+4.6) | 46.7 (-0.9) | 38.1 (+4.0) | +| 随机少样本 | 23.1 (+0.1) | 24.5 (-4.3) | 31.0 (+2.4) | 40.4 (-2.2) | 41.8 (-5.8) | 32.2 (-1.9) | +| BM25 检索 | 30.2 (+7.2) | 29.2 (+0.4) | 32.8 (+4.2) | 46.6 (+4.0) | 48.9 (+1.3) | 37.5 (+3.4) | +| Meta-Harness | 31.7 (+8.7) | 30.4 (+1.6) | 34.9 (+6.3) | 46.3 (+3.7) | 50.6 (+3.0) | 38.8 (+4.7) | + +**关键发现**：发现的检索 Harness 在所有五个预留模型上都优于无检索基线，平均增益为 **4.7 分**。它还匹配或超过了最强的固定基线，总体上比 BM25 检索高出 1.3 分，同时避免了在几个模型上观察到的密集检索和随机少样本提示的回归。 + +--- + +### 4.3 在 TerminalBench-2 上评估智能体编码 Harness + +TerminalBench-2 在 89 个具有挑战性的任务上评估 LLM 智能体，这些任务需要在复杂依赖下的长期、完全自主执行，以及大量领域知识。 + +**搜索设置**： +- 从两个强大的开放基线初始化搜索：Terminus 2 和 Terminus-KIRA +- 在相同的 89 任务基准上执行搜索和最终评估 +- 将此基准用作发现问题 +- 通过手动检查和基于正则表达式的审计检查过拟合，以查找任务特定字符串泄漏到进化的 Harness 中 + +**结果**： + +| Harness | 自动 | 通过率 (%) | +|---------|------|-----------| +| **Claude Opus 4.6** | | | +| Claude Code | × | 58.0 | +| Terminus 2 | × | 62.9 | +| Mux | × | 66.5 | +| Droid | × | 69.9 | +| TongAgents | × | 71.9 | +| MAYA-V2 | × | 72.1 | +| Terminus-KIRA | × | 74.7 | +| Capy | × | 75.3 | +| ForgeCode | × | 81.8 | +| Meta-Harness | ✓ | **76.4** | +| **Claude Haiku 4.5** | | | +| OpenHands | × | 13.9 | +| Claude Code | × | 27.5 | +| Terminus 2 | × | 28.3 | +| Mini-SWE-Agent | × | 29.8 | +| Terminus-KIRA | × | 33.7 | +| Goose | × | 35.5 | +| Meta-Harness | ✓ | **37.6** | + +**关键发现**： +- 在 Opus 4.6 上，Meta-Harness 发现的 Harness 达到 **76.4%** 通过率，超过了人工设计的 Terminus-KIRA（74.7%），在 TerminalBench-2 排行榜上所有 Opus 4.6 智能体中排名 **#2** +- 在较弱的 Haiku 4.5 模型上，改进更大：Meta-Harness 达到 **37.6%**，比次优报告的智能体（Goose，35.5%）高出 **2.1 分** + +--- + +## 提议者的定性行为 + +### A.1 文件访问统计 + +为验证提议者实质性地使用文件系统而不是默认局部编辑，记录了每次迭代的所有文件读取。 + +**结果摘要**： +- 提议者每次迭代读取**中位数 82 个文件**（范围 69–99） +- 大致均匀地分配在先前 Harness 源代码（41%）和执行轨迹（40%）之间 +- 其余部分用于分数摘要（6%）和其他文件（13%） + +这证实了提议者的访问模式是非马尔可夫的：它例行检查大多数可用历史，而不是仅以最近的父代为条件。 + +### A.2 定性行为：对先前失败的因果推理 + +TerminalBench-2 搜索日志揭示了一个清晰的叙事弧线，其中提议者从自己的回归中学习。它不是通过局部编辑随机漫步，而是形成对早期候选为什么失败的明确诊断，然后转向更安全的设计模式。 + +#### 迭代 1–2：有希望的 bug 修复与提示编辑混淆 + +前两次迭代都将看似合理的结构修复与提示模板修改捆绑在一起，并且都从 64.4% 的 Terminus-KIRA 基线急剧回归。 + +#### 迭代 3：提议者识别混淆 + +到迭代 3，提议者明确推断回归主要不是由于结构 bug 修复本身： + +> **假设**：__CMDEND__ 标记片段在长期任务上泄漏到 LLM 观察中，导致模型混淆并进入无限无工具调用循环。剥离这些标记 + 添加循环断路器将恢复浪费的步骤。 + +提议者注意到前两次失败的共同因素不是特定的 bug 修复，而是繁重的清理提示重写。因此，它恢复到原始提示并仅测试标记剥离和循环断路器。 + +#### 迭代 4–6：对诊断失败模式的直接修复仍然回归 + +接下来的三次迭代继续探测设计空间的相同部分，但现在有了关于为什么完成逻辑脆弱的更明确理论。 + +#### 迭代 7：获胜候选 + +在六次连续回归后，提议者从修改控制循环转向在循环开始前添加信息： + +> **策略转变**：所有 6 次先前迭代都从 64.4% 基线回归，因为它们修改了完成流程、提示模板或观察处理。evo_env_bootstrap 采取不同的方法——纯粹是加法的。它在第一次 LLM 调用之前通过单个 shell 命令收集环境快照，并将其附加到初始提示。不更改其他方法。这应该在依赖繁重的任务上消除 3–5 次浪费的探索回合，而不会冒着在已经通过的任务上回归的风险。 + +这个候选是迄今为止最好的结果。重要的一点不仅是迭代 7 获胜，而且提议者阐明了**为什么**它应该更安全：它避免触摸先前脆弱的完成机制，而是添加主要在困难任务上有用的信息。 + +--- + +## 讨论 + +除了优于现有 Harness 外，Meta-Harness 还有几个实际优势： + +1. **泛化能力**：发现的 Harness 泛化到分布外分类数据集和数学设置中未见的基础模型 +2. **时间效率**：一次搜索运行在几小时的挂钟时间内完成，但产生可读、可转移的策略，可以跨模型重用，包括未来更强的模型 +3. **可检查性**：代码空间中的过拟合也更可检查：脆弱的 if 链或硬编码的类映射在检查中可见，而权重空间过拟合则不是 + +更广泛地说，我们的结果表明 Meta-Harness 的主要优势不仅是代码搜索，而且是**对先前诊断经验的选择性访问**搜索。提议者不限于标量奖励或固定摘要；它可以检查原始代码、执行轨迹和先前失败，然后使用该信息形成和测试关于更改什么的假设。 + +--- + +## 相关研究 + +- [[Harness-Engineering|Harness 工程]] +- [[Managed-Agents-Decoupling-Brain-from-Hands|Managed Agents：将大脑与手分离]] +- [[Externalization-in-LLM-Agents|LLM Agent 中的外部化]] + +--- + +## 项目资源 + +- 项目页面与交互式演示：[https://yoonholee.com/meta-harness/](https://yoonholee.com/meta-harness/) +- 优化的 Harness：[https://github.com/stanford-iris-lab/meta-harness-tbench2-artifact](https://github.com/stanford-iris-lab/meta-harness-tbench2-artifact) diff --git a/wiki/concepts/Self-Verification.md b/wiki/concepts/Self-Verification.md deleted file mode 100644 index e013254..0000000 --- a/wiki/concepts/Self-Verification.md +++ /dev/null @@ -1,110 +0,0 @@ ---- -title: 自我验证 -tags: [核心概念, 验证, Harness] -source: [LangChain, Anthropic] -confidence_score: 高 -last_updated: 2026-04-07 ---- - -# 自我验证 - -**自我验证**（Self-Verification）是[[Harness-Engineering|Harness 工程]]中的关键技术，使智能体能够通过运行中的反馈自我改进。当今的模型是卓越的自我改进机器，但它们没有自然地倾向于进入这种**构建-验证循环**。 - -## 核心概念 - -### 最常见的失败模式 - -LangChain 团队观察到的最常见失败模式是：智能体编写解决方案、重读自己的代码、确认看起来没问题，然后停止。 - -测试是自主智能体编码的关键部分。它有助于测试整体正确性，同时为智能体提供信号来进行爬山优化。 - -## 构建-验证循环 - -LangChain 向系统提示中添加了关于如何解决问题的指导： - -### 阶段 1：规划与发现 -阅读任务、扫描代码库，并基于任务规格和如何验证解决方案构建初始计划。 - -### 阶段 2：构建 -在考虑验证的情况下实现计划。如果测试不存在则构建测试，并测试愉快路径和边缘情况。 - -### 阶段 3：验证 -运行测试、阅读完整输出、与要求的内容进行比较（而不是与你自己的代码）。 - -### 阶段 4：修复 -分析任何错误、重新访问原始规格，并修复问题。 - -## LangChain 的实现 - -### PreCompletionChecklistMiddleware - -除了提示外，确定性上下文注入有助于智能体验证它们的工作。LangChain 使用 `PreCompletionChecklistMiddleware` 在智能体退出前拦截它，并提醒它对任务规格运行验证通过。 - -这类似于 [Ralph Wiggum Loop|Ralph Wiggum 循环]]，其中一个钩子在智能体退出时强制其继续执行，用于验证。 - -![自我验证循环](../assets/self-verification-loop.png) - -### 强调测试 - -LangChain 真正专注于测试，因为它在每次迭代中推动变化。他们发现，测试必须是智能体工作流程的核心部分，而不是事后想法。 - -## 环境上下文工程 - -Harness 工程的一部分是**为上下文工程构建良好的交付机制。** Terminal Bench 任务带有目录结构、内置工具和严格的超时。 - -### 1. 目录上下文与工具 -`LocalContextMiddleware` 在智能体启动时运行，映射 `cwd` 和其他父+子目录。运行 `bash` 命令来查找工具如 `Python` 安装。上下文发现和搜索容易出错，因此注入上下文减少了这个错误表面并帮助**将智能体引导到其环境中。** - -### 2. 教导智能体编写可测试的代码 -智能体不知道它们的代码需要如何可测试。添加提示说它们的工作将根据程序化测试来衡量，类似于提交代码时。例如，提及文件路径的任务规格应该被精确遵循，以便解决方案在自动化评分步骤中工作。强调边缘情况的提示有助于智能体避免仅检查"愉快路径"情况。强制模型符合测试标准是避免随着时间推移"烂泥堆积"的强大策略。 - -### 3. 时间预算 -注入时间预算警告以推动智能体完成工作并转向验证。智能体在时间估计方面是出了名的糟糕，因此这种启发式在这种环境中有所帮助。现实世界编码通常没有严格的时间限制，但如果不添加任何约束知识，智能体就不会在时间范围内工作。 - -## 循环检测 - -智能体一旦决定了一个计划就可能变得短视，导致"末日循环"，对同一破碎方法进行小幅变异（在某些轨迹中超过 10 次以上）。 - -LangChain 使用 `LoopDetectionMiddleware`，通过工具调用钩子跟踪每个文件的编辑次数。它在对同一文件进行 `N` 次编辑后添加上下文如"……考虑重新考虑你的方法"。这可以帮助智能体从末日循环中恢复，尽管如果模型认为正确的话它可以继续沿着相同的路径前进。 - -重要提示：这是一种设计启发式，围绕当今感知的模型问题进行工程设计。随着模型改进，这些护栏可能会变得不必要，但今天帮助智能体正确和自主地执行。 - -## 推理三明治 - -推理模型可以自主运行数小时，因此必须决定在每个子任务上花费多少计算。你可以在每个任务上使用最大推理预算，但大多数工作可以从优化推理计算支出中受益。 - -Terminal Bench 超时限制创造了一个权衡。更多推理有助于智能体评估每个步骤，但可以燃烧超过 2 倍以上的 token/时间。`gpt-5.2-codex` 有 4 种推理模式，`low`、`medium`、`high` 和 `xhigh`。 - -LangChain 发现推理有助于规划以充分理解问题，一些 Terminal Bench 任务非常困难。一个好的计划有助于更快地得到可行的解决方案。 - -后期验证也从更多推理中受益，以捕获错误并提交解决方案。作为一种启发式，选择 xhigh-high-xhigh"**推理三明治**"作为基线。 - -![推理三明治](../assets/the-reasoning-sandwich.png) - -**在规划和验证上花费更多的推理计算** - -仅在 `xhigh` 运行由于智能体超时而得分较差，为 `53.9%`，相比之下在 `high` 为 `63.6%`。在推理预算分割的试验运行中没有大的差异，因此坚持使用他们的方法，将分数推到 `66.5%`。 - -## Harness 工程师的目的 - -**Harness 工程师的目的：准备和交付上下文，使智能体能够自主完成工作。** - -智能体对其环境、约束和评估标准了解得越多，它们就能越好地自主自我指导工作。 - -## 相关概念 - -- [[Build-Verify Loop|构建-验证循环]] -- [[Reasoning Sandwich|推理三明治]] -- [[Doom Loop|末日循环]] -- [[Loop Detection|循环检测]] -- [[LangChain Harness Engineering|LangChain Harness 工程]] - -## 参考来源 - -1. LangChain - Improving Deep Agents with harness engineering -2. Anthropic - Harness design for long-running application development - ---- - -*最后更新：2026-04-07* -*本文档由 [[WikiLLM]] 编译自多个来源* diff --git a/wiki/concepts/Skill-Systems.md b/wiki/concepts/Skill-Systems.md new file mode 100644 index 0000000..f71f333 --- /dev/null +++ b/wiki/concepts/Skill-Systems.md @@ -0,0 +1,148 @@ +--- +title: "技能系统" +source: "Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering" +raw_sources: + - path: raw/Externalization in LLM Agents A Unified Review of Memory, Skills, Protocols and Harness Engineering.md + hash: "sha256:to-be-computed" +tags: + - "核心概念" + - "技能系统" + - "外部化" +last_updated: 2026-04-11 +--- + +# 技能系统 + +**技能外部化**解决了智能体的程序负担问题。它将任务特定的知识打包成显式的可发现、可加载、可修订和可组合的人工制品，而不是要求模型在每次尝试任务时重新生成工作流、默认值和约束。 + +## 技能的三大组成部分 + +![技能作为外部化专长](../assets/skill_f.png) + +### 1. 操作程序 + +操作程序是任务骨架：将复杂工作分解为步骤、阶段、依赖关系和停止条件。 + +- **解决 LLM Agent 中的常见失败模式**：许多错误不是来自行动级别的无能，而是来自流程级别的不稳定 +- **使执行更少即兴创作**：Agent 可以在中断后恢复，跨上下文或协作者移交工作，并在不从记忆重建整个工作流的情况下恢复状态 +- **在长视野、多 Agent 和生产设置中最重要**，其中流程稳定性通常比瞬间流畅性更重要 + +### 2. 决策启发式 + +如果程序定义了执行的骨架，决策启发式管理分支处发生的情况。 + +- **实际任务很少作为固定管道展开**：工具失败、观察有噪声、几个局部合理的行动可能竞争 +- **在这些条件下，良好的性能依赖于从经验中得出的实用经验法则**，而不是仅靠穷举搜索 +- **捕获专家风格**：首先尝试什么，何时后退，什么证据足够，以及当多个路径仍然可行时偏好哪些权衡 + +### 3. 规范性约束 + +第三个组成部分是规范性约束：程序被视为可接受的条件。 + +- **工作流在技术上可能有效但仍然不合规、不安全或操作上错误** +- **在实际部署中，执行受到测试要求、范围限制、访问限制、可追溯性期望和特定领域操作规则的约束** +- **一旦外部化，这些约束就不再仅仅是事后评估标准**，而是成为技能本身的一部分 +- **编码不仅是如何执行任务，还有如何在组织和安全边界内执行任务** + +## 从执行原语到能力包 + +技能系统不是孤立出现的，但也不应与工具使用混为一谈。历史上，技能在两个早期发展的下游：可靠的行动调用和大规模行动选择。 + +### 阶段 1：原子执行原语 + +第一阶段为语言模型配备可靠的行动执行，例如通过结构化工具调用和函数调用接口。 + +- **关键成就**：稳定访问原子行动单元 +- **不提供**：完成更广泛任务类别的显式可重用程序 +- **单元是行动原语**，而不是技能 + +### 阶段 2：大规模原语选择 + +随着可调用工具数量的增长，问题从调用转移到选择。 + +- **主要进步**：模型可以检索、排名和动态在大型工具集合中选择 +- **单元仍然是工具**，而不是程序 +- **即使多步行为开始出现，完成任务类别的知识在很大程度上仍然隐含在提示或参数中**，而不是外部化为有界的可重用人工制品 + +### 阶段 3：作为打包专长的技能 + +第三阶段标志着抽象的进一步转变。中心问题不再是模型是否可以调用函数或检索适当的 API，而是完成一类任务所需的知识是否可以打包成可重用的能力单元。 + +**关键转换**： +- 能力不再主要被视为对工具或 API 的访问 +- 能力越来越被视为打包的程序知识，可以跨任务加载、重用和组合 +- 基础能力单元不再是孤立的工具调用，而是以可重用程序指导和执行结构为中心的更高级别人工制品 + +## 技能如何外部化 + +技能外部化不仅限于写下指令。在成熟的 Agent 系统中，关键问题是程序专长是否可以以在运行时可发现、可加载、可解释、可绑定和可执行的形式表示。 + +### 1. 规范 + +技能的外部化始于规范层。典型形式包括 SKILL.md、指令文件、清单或其他声明性规范人工制品。 + +**格式良好的技能规范应涵盖五类信息**： +1. 能力边界 +2. 适用范围 +3. 前置条件 +4. 执行约束 +5. 示例和反例 + +### 2. 发现 + +一旦技能成为显式人工制品，它们自然会引入注册和发现问题。 + +**发现机制**： +- 本地存储库、组织注册表或平台级市场 +- 基于任务目标、上下文状态和环境条件的语义检索、结构化元数据、任务分解或这些策略的组合 + +### 3. 渐进式披露 + +技能的发现并不意味着其全部内容应立即注入到活动上下文中。 + +**分层形式**： +1. **最小级别** - 模型仅看到技能的名称和简要描述 +2. **更深级别** - 公开清单类信息，如适用条件、所需的先决条件和主要约束 +3. **最深级别** - 系统加载完整指南，包括详细程序、异常处理、示例和支持文件 + +### 4. 执行绑定 + +技能仍然是认知级别的描述，除非它连接到可执行行动。实际任务完成取决于绑定过程，该过程将技能的自然语言或结构化规范转换为当前环境中的具体操作。 + +### 5. 组合 + +技能系统的价值在技能可以组合时得到最充分的体现。与原子工具不同，技能可以参与更高级别的结构化协调。 + +**常见组合模式**： +- 串行执行 +- 并行分工 +- 条件路由 +- 在更高级别技能中递归调用子技能 + +## 技能获取和演化 + +技能系统之所以重要，不仅因为它存储了创作的指令，还因为它提供了将成功行为转化为可重用专长的途径。 + +**四种获取途径**： + +1. **创作** - 仍然是技能进入当前系统的最常见和最稳定的途径 +2. **蒸馏** - 也可以从历史轨迹、实践痕迹或其他存储经验中诱导 +3. **发现** - 除了手动创作和事后蒸馏，Agent 还可以通过环境交互自主发现新技能 +4. **组合** - 最后，技能可以通过组合演化 + +## 边界条件 + +技能外部化改进了重用和治理，但它不能保证可靠性。 + +**主要边界条件**： +1. **语义对齐** - 模型可能遵循技能的字面措辞，同时仍然缺少任务的真正目标 +2. **可移植性和陈旧性** - 技能在环境中的有效性不能被假设 +3. **不安全组合** - 组合使技能更强大，但也创造了新的风险 +4. **上下文相关的降级** - 技能执行可能在长时间交互中降级 + +## 相关研究 + +- [[Externalization-in-LLM-Agents|LLM Agent 中的外部化]] +- [[Harness-Engineering|Harness 工程]] +- [[Memory-Systems|记忆系统]] +- [[Agent-Protocols|智能体协议]] diff --git a/wiki/practices/Building-Your-First-Harness.md b/wiki/practices/Building-Your-First-Harness.md deleted file mode 100644 index 2fb2a24..0000000 --- a/wiki/practices/Building-Your-First-Harness.md +++ /dev/null @@ -1,201 +0,0 @@ ---- -title: 构建你的第一个 Harness -tags: [实践指南, 入门, Harness] -source: [NxCode, OpenAI, Martin Fowler] -confidence_score: 高 -last_updated: 2026-04-07 ---- - -# 构建你的第一个 Harness - -本文提供了构建 Harness 的实用框架，从个人开发者到工程组织的三个级别。 - ---- - -## Level 1：基础 Harness（单个开发者） - -如果你正在为个人项目使用 Claude Code、Cursor 或 Codex： - -### 需要设置的内容 - -- `CLAUDE.md` 或 `.cursorrules` 文件，包含项目约定 -- 用于 linting 和格式化的预提交钩子 -- 智能体可以运行以自我验证的测试套件 -- 具有一致命名的清晰目录结构 - -**设置时间：** 1-2 小时 -**影响：** 防止最常见的智能体错误 - -### 快速入门清单 - -1. **创建 CLAUDE.md** - ```markdown - # 项目约定 - - ## 代码风格 - - 使用 TypeScript，严格模式 - - 使用 2 空格缩进 - - 函数名使用驼峰命名 - - ## 测试 - - 所有新代码必须有测试 - - 提交前运行 `npm test` - - ## 目录结构 - - src/ - 源代码 - - tests/ - 测试文件 - - docs/ - 文档 - ``` - -2. **设置预提交钩子** - ```bash - # 使用 husky 或 pre-commit - npm install husky --save-dev - npx husky install - ``` - -3. **确保测试存在** - - 即使是简单的集成测试也比没有好 - - 智能体需要可以运行的东西来验证其工作 - ---- - -## Level 2：团队 Harness（小团队） - -对于共享代码库的 3-10 名开发者的团队： - -### 在 Level 1 基础上添加 - -- 包含团队范围约定的 `AGENTS.md` -- CI 强制执行的架构约束 -- 常见任务的共享提示模板 -- 由 linter 验证的文档即代码 -- 专门针对智能体生成的 PR 的代码审查清单 - -**设置时间：** 1-2 天 -**影响：** 整个团队的智能体行为一致 - -### AGENTS.md 结构建议 - -```markdown -# AGENTS.md - -本文档包含智能体在此代码库中工作的约定。 - -## 1. 架构原则 -- 我们使用分层架构：Types → Config → Repo → Service → API -- 不要跨层导入 -- 所有外部依赖通过 Providers 访问 - -## 2. 编码标准 -- [具体规则...] - -## 3. 测试策略 -- [具体指南...] - -## 4. 审查清单 -- [智能体在提交前应检查的内容...] -``` - -### 共享提示模板 - -创建一个 `prompts/` 目录，包含常见任务： -- `prompts/add-new-feature.md` -- `prompts/fix-bug.md` -- `prompts/write-tests.md` -- `prompts/refactor-code.md` - ---- - -## Level 3：生产 Harness（工程组织） - -对于运行数十个并发智能体的组织： - -### 在 Level 2 基础上添加 - -- 自定义中间件层（循环检测、推理优化） -- 可观测性集成（智能体读取日志和指标） -- 计划运行的熵管理智能体 -- Harness 版本控制和 A/B 测试 -- 智能体性能监控仪表板 -- 智能体陷入困境时的升级策略 - -**设置时间：** 1-2 周 -**影响：** 智能体作为自主贡献者运作 - ---- - -## 常见 Harness 工程错误 - -### 1. 过度工程化控制流 - -> "如果你过度工程化控制流，下一个模型更新会破坏你的系统。" - -模型快速改进。2024 年需要复杂管道的能力现在由单个上下文窗口提示处理。构建你的 Harness 为**可剥离的**——当模型变得足够智能不需要时，你应该能够移除"智能"逻辑。 - -### 2. 将 Harness 视为静态的 - -Harness 需要随模型一起演进。当新模型版本改进推理时，你的推理优化中间件可能会适得其反。每次重大模型更新时审查和更新 Harness 组件。 - -### 3. 忽略文档层 - -最有影响力的 Harness 改进通常是最简单的：**更好的文档**。如果你的 `AGENTS.md` 含糊，你的智能体输出也会含糊。投资于精确、机器可读的文档，作为智能体的地面真理。 - -### 4. 没有反馈循环 - -没有反馈的 Harness 是一个笼子，而不是指南。智能体需要知道它何时成功何时失败。内置： -- 任务完成前的自我验证步骤 -- 作为智能体工作流一部分的测试执行 -- 按任务类型的智能体成功率指标 - -### 5. 仅人类可读的文档 - -如果你的架构决策存在于人们的头脑中或智能体无法访问的 Confluence 页面中，Harness 就有差距。**智能体需要的一切都必须在仓库中。** - ---- - -## 从哪里开始 - -### 如果你是单个开发者 - -1. 从 Level 1 开始 -2. 创建一个简单的 `CLAUDE.md` -3. 设置基础预提交钩子 -4. 添加一个简单的测试套件 -5. 迭代——观察智能体失败的地方并加以修复 - -### 如果你是一个团队 - -1. 首先就 Level 1 基础达成一致 -2. 一起编写 `AGENTS.md` 的初稿 -3. 识别最常见的 3 个智能体失败模式 -4. 为这些模式构建前 3 个约束/检查 -5. 每 2 周回顾一次什么有效/什么无效 - -### 如果你是一个组织 - -1. 从一些试点团队开始 -2. 收集什么有效的模式 -3. 构建可重用的 Harness 组件库 -4. 投资于可观测性和指标 -5. 建立 Harness 迭代的反馈循环 - ---- - -## 相关概念 - -- [[Harness-Engineering|Harness 工程]] -- [[Context-Engineering|上下文工程]] -- [[Architectural-Constraints|架构约束]] -- [[Mitchellh-Adoption-Journey|Mitchellh AI 采用之旅]] - -## 参考来源 - -1. NxCode - Harness Engineering: The Complete Guide -2. OpenAI - Harness Engineering：在智能体优先的世界中利用 Codex -3. Martin Fowler - Harness engineering for coding agent users - ---- - -*最后更新：2026-04-07* -*本文档由 [[WikiLLM]] 编译自多个来源* diff --git a/wiki/practices/LangChain-Harness-Engineering.md b/wiki/practices/LangChain-Harness-Engineering.md new file mode 100644 index 0000000..66f4052 --- /dev/null +++ b/wiki/practices/LangChain-Harness-Engineering.md @@ -0,0 +1,158 @@ +--- +title: "LangChain Harness 工程实践" +source: "https://blog.langchain.com/improving-deep-agents-with-harness-engineering/" +author: "LangChain" +published: 2026-02-18 +last_updated: 2026-04-11 +tags: + - practices + - harness-engineering + - langchain +raw_sources: + - path: raw/Improving Deep Agents with harness engineering.md + hash: "sha256:initial" +--- + +# LangChain Harness 工程实践 + +本文记录了 LangChain 团队如何通过仅改进 Harness，将他们的编码智能体在 Terminal Bench 2.0 上从第 30 名提升到第 5 名。核心要点：**自我验证和追踪帮助巨大**。 + +## 核心成果 + +- **基准测试**：Terminal Bench 2.0 +- **提升幅度**：从 52.8 分提升到 66.5 分（+13.7 分） +- **模型保持不变**：gpt-5.2-codex +- **仅修改 Harness**：系统提示词、工具、中间件 + +![基准测试结果](../assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png) + +--- + +## Harness 工程的目标 + +Harness 的目标是塑造模型固有的尖峰智力以适应我们关心的任务。**Harness 工程**是关于系统的，你在模型周围构建工具以优化目标，如任务性能、token 效率、延迟等。设计决策包括系统提示词、工具选择和执行流程。 + +但你应该如何改变 Harness 以改进智能体？ + +在 LangChain，他们使用 [Traces](https://docs.langchain.com/langsmith/observability-quickstart) 来大规模理解智能体失败模式。今天的模型在很大程度上是黑盒，它们的内部机制难以解释。但我们可以在文本空间中看到它们的输入和输出，然后将其用于我们的改进循环。 + +--- + +## 实验设置与 Harness 上的旋钮 + +使用 Terminal Bench 2.0（一个现在标准的基准测试来评估智能体编码）。它有 89 个任务，涵盖机器学习、调试和生物学等领域。使用 Harbor 来编排运行。它启动沙箱（Daytona），与智能体循环交互，并运行验证 + 评分。 + +每个智能体操作都存储在 LangSmith 中。它还包括延迟、token 计数和成本等指标。 + +### 我们可以转动的旋钮 + +智能体 Harness 有很多旋钮：系统提示词、工具、钩子/中间件、技能、子智能体委托、记忆系统等等。LangChain 有意压缩优化空间，专注于三个：**系统提示词、工具**和[**中间件**](https://docs.langchain.com/oss/python/langchain/middleware/overview)（他们对模型和工具调用周围钩子的术语）。 + +从默认提示词和标准工具+中间件开始。这在 GPT-5.2-Codex 上得分 52.8%。一个可靠的分数，刚好在今天排行榜前 30 名之外，但有增长空间。 + +![实验设置与追踪](../assets/Screenshot-2026-02-16-at-12.50.00---PM.png) + +### 追踪分析器技能 + +LangChain 希望追踪分析可重复，因此将其做成了智能体技能。这作为他们的**跨运行分析错误并对 Harness 进行改进**的方案。流程是： + +1. 从 LangSmith 获取实验追踪 +2. 生成并行错误分析智能体 → 主智能体综合发现 + 建议 +3. 聚合反馈并对 Harness 进行有针对性的更改 + +这类似于 [boosting](https://en.wikipedia.org/wiki/Boosting_\(machine_learning\))，它专注于先前运行的错误。人类在第 3 步中可能非常有帮助（尽管不是必需的）来验证和讨论提议的更改。过度拟合到任务的更改对泛化不利，并可能导致其他任务的回归。 + +![LangSmith 追踪分析器技能](../assets/langsmith_trace_analyzer_skill.png) + +--- + +## 实际改进智能体性能的因素 + +自动化追踪分析允许 LangChain [调试智能体出错的地方](https://www.langchain.com/conceptual-guides/agent-observability-powers-agent-evaluation)。问题包括推理错误、不遵循任务指令、缺少测试和验证、超时等。 + +### 构建与自我验证 + +今天的模型是卓越的自我改进机器。 + +**自我验证允许智能体通过运行内的反馈自我改进。** 然而，它们没有自然倾向进入这个**构建-验证循环。** + +最常见的失败模式是，智能体编写了一个解决方案，重新阅读自己的代码，确认看起来没问题，然后停止。测试是自主智能体编码的关键部分。它有助于测试整体正确性，同时为智能体提供信号以进行爬山优化。 + +LangChain 在系统提示词中添加了关于如何解决问题的指导： + +1. **规划与发现**：阅读任务，扫描代码库，基于任务规范和如何验证解决方案构建初始计划。 +2. **构建**：考虑验证来实现计划。如果测试不存在则构建测试，测试快乐路径和边缘情况。 +3. **验证**：运行测试，阅读完整输出，与要求的内容进行比较（不是与你自己的代码比较）。 +4. **修复**：分析任何错误，重新审视原始规范，并修复问题。 + +他们真正专注于测试，因为它在每次迭代中推动更改。发现除了提示词之外，确定性上下文注入有助于智能体验证它们的工作。使用 `PreCompletionChecklistMiddleware` 在智能体退出前拦截它，并提醒它对任务规范运行验证传递。这类似于 [Ralph Wiggum Loop](https://ghuntley.com/loop/)，其中一个钩子强制智能体在退出时继续执行，他们用它来进行验证。 + +![自我验证循环](../assets/self-verification-loop.png) + +--- + +### 为智能体提供关于其环境的上下文 + +Harness 工程的一部分是**构建一个良好的上下文工程交付机制。** Terminal Bench 任务带有目录结构、内置工具和严格的超时。 + +1. **目录上下文与工具**：`LocalContextMiddleware` 在智能体启动时运行以映射 `cwd` 和其他父/子目录。运行 `bash` 命令来查找工具，如 Python 安装。上下文发现和搜索容易出错，因此注入上下文减少了这个错误表面并帮助**将智能体引导到其环境中。** +2. **教智能体编写可测试的代码**：智能体不知道它们的代码需要如何可测试。添加提示词说它们的工作将根据编程测试来衡量，类似于提交代码时。例如，提到文件路径的任务规范应该完全遵循，以便解决方案在自动化评分步骤中工作。强调边缘情况的提示词有助于智能体避免只检查"快乐路径"情况。强制模型符合测试标准是避免随着时间推移"残渣 buildup"的强大策略。 +3. **时间预算**：注入时间预算警告以推动智能体完成工作并转向验证。智能体在时间估计方面出了名的差，因此这种启发式在这种环境中有帮助。现实世界的编码通常没有严格的时间限制，但如果不添加任何约束知识，智能体就不会在时间范围内工作。 + +智能体对其环境、约束和评估标准了解得越多，它们就能越好地自主自我指导工作。 + +> **Harness 工程师的目的**：准备并交付上下文，以便智能体可以自主完成工作。 + +--- + +### 鼓励智能体退后一步并重新考虑计划 + +智能体一旦决定了一个计划可能会短视，导致"末日循环"，对相同的破碎方法进行小的变化（在某些追踪中 10+ 次）。 + +使用 `LoopDetectionMiddleware` 通过工具调用钩子跟踪每个文件的编辑计数。它在对同一文件进行 `N` 次编辑后添加诸如"……考虑重新考虑你的方法"之类的上下文。这可以帮助智能体从末日循环中恢复，尽管如果模型认为正确，它可以继续沿着相同的路径前进。 + +重要提示。这是一种设计启发式，围绕今天感知到的模型问题进行工程设计。随着模型改进，这些护栏可能会变得不必要，但今天有助于智能体正确且自主地执行。 + +--- + +### 选择在推理上花费多少计算 + +推理模型可以自主运行数小时，因此必须决定在每个子任务上花费多少计算。可以在每个任务上使用最大推理预算，但大多数工作可以从优化推理计算花费中受益。 + +Terminal Bench 超时限制创造了一个权衡。更多推理帮助智能体评估每个步骤，但可能燃烧超过 2 倍的 token/时间。`gpt-5.2-codex` 有 4 种推理模式：`low`、`medium`、`high` 和 `xhigh`。 + +发现推理有助于规划以完全理解问题，一些 Terminal Bench 任务非常困难。一个好的计划有助于更快地达到可行的解决方案。 + +后期验证也受益于更多推理以捕获错误并提交解决方案。作为一种启发式，选择 xhigh-high-xhigh"**推理三明治**"作为基线。 + +![推理三明治](../assets/the-reasoning-sandwich.png) + +仅在 `xhigh` 运行得分很差，为 `53.9%`，因为智能体超时，相比之下在 `high` 运行得分为 `63.6%`。在推理预算拆分之间的试运行中没有大的差异，因此坚持使用将分数推到 `66.5%` 的方法。 + +模型的自然方法是**自适应推理**，在 [Claude](https://platform.claude.com/docs/en/build-with-claude/adaptive-thinking) 和 [Gemini](https://ai.google.dev/gemini-api/docs/thinking) 模型中看到，模型决定在推理上花费多少计算。 + +在多模型 Harness 中，平衡推理预算可以表现为使用大模型进行规划并[交接](https://docs.langchain.com/oss/python/langchain/multi-agent/handoffs)给小模型进行实现。 + +--- + +## 构建智能体 Harness 的实用要点 + +智能体的设计空间很大。以下是 LangChain 从实验和整体构建 deepagents 中得出的一些一般原则： + +1. **代表智能体进行上下文工程**。上下文组装对于今天的智能体仍然很困难，尤其是在看不见的环境中。用目录结构、可用工具、编码最佳实践和问题解决策略等上下文引导模型，有助于减少搜索不佳和规划中可避免错误的错误表面。 +2. **帮助智能体自我验证其工作**。模型偏向于它们第一个看似合理的解决方案。强烈提示它们通过运行测试和改进解决方案来验证其工作。这在没有人类在循环中的自主编码系统中尤其重要。 +3. **追踪作为反馈信号**。追踪允许智能体自我评估和调试自己。一起调试工具和推理很重要（例如：模型走错路径是因为它们缺少工具或关于如何做某事的指令）。 +4. **在短期内检测并修复坏模式**。今天的模型并不完美。Harness 设计者的工作是围绕今天的缺点进行设计，同时为未来更智能的模型做计划。盲目重试和不验证工作是很好的例子。这些护栏几乎肯定会随着时间推移而消失，但要在今天构建健壮的智能体应用程序，它们是值得实验的有用工具。 +5. **为模型定制 Harness**。[Codex](https://developers.openai.com/cookbook/examples/gpt-5/codex_prompting_guide/) 和 [Claude](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices) 提示指南显示模型需要不同的提示。使用 Claude Opus 4.6 的测试运行在早期 Harness 版本中得分 `59.6%`，有竞争力但比 Codex 差，因为他们没有与 Claude 运行相同的改进循环。许多原则可以推广，如良好的上下文准备和对验证的关注，但为你的任务运行几轮 Harness 迭代有助于跨任务最大化智能体性能。 + +在 Harness 设计中还有更多开放研究要做。有趣的途径包括多模型系统（Codex、Gemini 和 Claude 一起）、用于持续学习的记忆原语，以便智能体可以在任务上自主改进，以及跨模型测量 Harness 更改。 + +对于改进智能体的外层循环，LangChain 正在研究像 [RLMs](https://alexzhang13.github.io/blog/2025/rlm/) 这样的方法来更有效地挖掘追踪。他们将继续改进 Harness 并公开分享他们的研究。 + +--- + +## 相关研究 + +- [[Harness-Engineering|Harness 工程]] +- [[OpenAI-Codex-Harness-Engineering|OpenAI Codex Harness 工程]] +- [[Long-Running-Harness-Design|长运行应用的 Harness 设计]] diff --git a/wiki/practices/Long-Running-Harness-Design.md b/wiki/practices/Long-Running-Harness-Design.md new file mode 100644 index 0000000..449e8f0 --- /dev/null +++ b/wiki/practices/Long-Running-Harness-Design.md @@ -0,0 +1,145 @@ +--- +title: "长运行应用的 Harness 设计" +source: "Harness design for long-running application development" +raw_sources: + - path: raw/Harness design for long-running application development.md + hash: "sha256:initial" +tags: + - "实践指南" + - "Harness工程" + - "多智能体" +last_updated: 2026-04-11 +--- + +# 长运行应用的 Harness 设计 + +本文基于 Anthropic 团队的实践经验，探讨如何设计有效的 Harness 来支持长运行、自主的应用开发任务。 + +## 核心问题 + +在长运行 Agent 任务中，简单的实现通常会遇到两个持续存在的问题： + +### 1. 上下文窗口填充导致的连贯性丧失 + +随着任务进行，模型往往会在冗长任务上失去连贯性。一些模型还表现出"**上下文焦虑**"——当它们接近认为的上下文限制时，会过早地结束工作。 + +**解决方案对比**： +- **压缩 (Compaction)**：总结对话的早期部分，使同一个 Agent 可以在缩短的历史上继续。保留了连续性，但没有给 Agent 一个干净的状态。 +- **上下文重置 (Context Reset)**：完全清除上下文窗口并启动一个新的 Agent，结合结构化移交来携带前一个 Agent 的状态和下一步。提供了干净的状态，但增加了编排复杂性、令牌开销和延迟。 + +### 2. 自我评估问题 + +当被要求评估自己产生的工作时，Agent 往往会自信地赞扬工作——即使在人类观察者看来质量明显一般。对于像设计这样的主观任务，这个问题尤其明显。 + +**解决方案**：将执行工作的 Agent 与评判工作的 Agent 分离。调整独立的评估器使其持怀疑态度，比让生成器批评自己的工作要容易得多。 + +## 前端设计：使主观质量可评分 + +受生成对抗网络 (GANs) 的启发，设计了一个带有**生成器**和**评估器** Agent 的多 Agent 结构。 + +### 四个评分标准 + +1. **设计质量**：设计感觉像是一个连贯的整体而不是部分的集合吗？ +2. **原创性**：有自定义决策的证据吗，还是模板布局、库默认值和 AI 生成的模式？ +3. **工艺**：技术执行：排版层次结构、间距一致性、色彩和谐、对比度。 +4. **功能性**：独立于美学的可用性。 + +### 评估器校准 + +使用带有详细分数分解的少样本示例来校准评估器，确保评估器的判断与人类偏好一致，并减少迭代之间的分数漂移。 + +### 迭代循环 + +1. 生成器 Agent 首先根据用户提示创建 HTML/CSS/JS 前端 +2. 评估器使用 Playwright MCP 与实时页面交互 +3. 评估器对每个标准进行评分并编写详细的批评 +4. 反馈流回生成器作为下一次迭代的输入 +5. 每次生成运行 5 到 15 次迭代 + +## 扩展到全栈编码 + +将这种 GAN 启发的模式应用于全栈开发，生成器-评估器循环自然地映射到软件开发生命周期。 + +### 三 Agent 架构 + +**Planner (规划器)**： +- 将简单的 1-4 句提示扩展为完整的产品规范 +- 被提示要对范围保持雄心勃勃 +- 专注于产品上下文和高级技术设计，而不是详细的技术实现 +- 寻找机会将 AI 功能编织到产品规范中 + +**Generator (生成器)**： +- 一次从规范中提取一个功能 +- 每个冲刺使用 React、Vite、FastAPI 和 SQLite 堆栈实现应用 +- 被指示在每个冲刺结束时自我评估其工作，然后移交给 QA +- 有 git 用于版本控制 + +**Evaluator (评估器)**： +- 使用 Playwright MCP 像用户一样点击运行中的应用 +- 测试 UI 功能、API 端点和数据库状态 +- 根据发现的 bug 和一组标准对每个冲刺进行评分 +- 每个标准都有硬阈值，如果任何一个低于它，冲刺就失败 + +### 冲刺契约 + +在每个冲刺之前，生成器和评估器协商一个**冲刺契约**：在编写任何代码之前，就该工作块的"完成"是什么样子达成一致。 + +## 结果对比 + +### 复古游戏制作器示例 + +| Harness | 时长 | 成本 | 结果 | +|---------|------|------|------| +| 单独 Agent | 20 分钟 | $9 | 界面看起来符合预期，但实际游戏坏了，实体不响应输入 | +| 完整 Harness | 6 小时 | $200 | 16 个功能规范，分布在十个冲刺中，包含精灵动画系统、行为模板、音效和音乐、AI 辅助的精灵生成器和关卡设计器，以及可分享链接的游戏导出 | + +### 关键发现 + +- 单独运行的输出最初看起来令人印象深刻，但深入研究后问题开始出现 +- Harness 运行的应用立即显示出比单独运行更多的打磨和流畅性 +- 评估器使实现与规范保持一致，契约是细粒度的，评估器的发现足够具体，可以采取行动 + +## 简化 Harness + +随着模型的改进，值得重新检查 Harness，剥离不再对性能有负载作用的部分，并添加新部分以实现以前不可能的更大能力。 + +### Opus 4.6 的改进 + +- 更仔细地规划 +- 更长时间地维持 Agent 任务 +- 可以在更大的代码库中更可靠地操作 +- 具有更好的代码审查和调试技能来发现自己的错误 +- 长上下文检索方面有了实质性改进 + +### 移除冲刺结构 + +保持规划器和评估器，因为每个都继续增加明显的价值。将评估器移动到运行结束时的单次通过，而不是每个冲刺评分。 + +### DAW 示例结果 + +使用更新的 Harness 生成数字音频工作站 (DAW)： + +| Agent & Phase | 时长 | 成本 | +|---------------|------|------| +| Planner | 4.7 分钟 | $0.46 | +| Build (Round 1) | 2 小时 7 分钟 | $71.08 | +| QA (Round 1) | 8.8 分钟 | $3.24 | +| Build (Round 2) | 1 小时 2 分钟 | $36.89 | +| QA (Round 2) | 6.8 分钟 | $3.09 | +| Build (Round 3) | 10.9 分钟 | $5.88 | +| QA (Round 3) | 9.6 分钟 | $4.06 | +| **总计 V2 Harness** | **3 小时 50 分钟** | **$124.70** | + +## 经验教训 + +1. **实验**：与你正在构建的模型一起实验，阅读其在现实问题上的轨迹，并调整其性能以实现你想要的结果 +2. **分解**：在处理更复杂的任务时，有时可以通过分解任务并将专门的 Agent 应用于问题的每个方面来获得提升 +3. **重新检查**：当新模型落地时，通常很好的做法是重新检查 Harness，剥离不再对性能有负载作用的部分，并添加新部分以实现以前不可能的更大能力 + +> 随着模型的改进，有趣的 Harness 组合空间不会缩小。相反，它会移动，而 AI 工程师的有趣工作是继续找到下一个新颖的组合。 + +## 相关研究 + +- [[Harness-Engineering|Harness 工程]] +- [[Externalization-in-LLM-Agents|LLM Agent 中的外部化]] +- [[Skill-Systems|技能系统]] diff --git a/wiki/practices/Managed-Agents-Decoupling-Brain-from-Hands.md b/wiki/practices/Managed-Agents-Decoupling-Brain-from-Hands.md new file mode 100644 index 0000000..fad8b4e --- /dev/null +++ b/wiki/practices/Managed-Agents-Decoupling-Brain-from-Hands.md @@ -0,0 +1,111 @@ +--- +title: "Managed Agents：将大脑与手分离" +source: "https://www.anthropic.com/engineering/managed-agents" +author: "Anthropic" +published: 2026 +last_updated: 2026-04-11 +tags: + - practices + - managed-agents + - anthropic +raw_sources: + - path: raw/Scaling Managed Agents Decoupling the brain from the hands.md + hash: "sha256:initial" +--- + +# Managed Agents：将大脑与手分离 + +本文描述了 Anthropic 如何构建 Managed Agents——一个在 Claude 平台中代表你运行长 horizon 智能体的托管服务。核心思想是通过一组旨在超越任何特定实现的接口来虚拟化智能体的组件。 + +## 核心论点 + +工程博客上的一个持续主题是如何[构建有效的智能体](https://www.anthropic.com/engineering/building-effective-agents)和[为长运行工作设计 Harness](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents)。这项工作的一个共同线索是，Harness 编码了关于 Claude 自己不能做什么的假设。然而，这些假设需要经常被质疑，因为随着模型改进，它们可能会[过时](http://www.incompleteideas.net/IncIdeas/BitterLesson.html)。 + +仅举一个例子，在先前的工作中[发现](https://www.anthropic.com/engineering/harness-design-long-running-apps)，Claude Sonnet 4.5 会在感知到其上下文限制临近时过早结束任务——一种有时称为"上下文焦虑"的行为。通过向 Harness 添加上下文重置来解决这个问题。但是当在 Claude Opus 4.5 上使用相同的 Harness 时，发现这种行为已经消失了。重置变成了死重。 + +期望 Harness 继续演进。因此构建了 Managed Agents：Claude 平台中的一个托管服务，通过一组旨在超越任何特定实现的接口——包括今天运行的那些——来代表你运行长 horizon 智能体。 + +![Managed Agents 概述](../assets/903b624ada206b10753a24c6a1367e74a869165d-1080x1080.webp) + +--- + +## 操作系统的启发 + +构建 Managed Agents 意味着解决计算中的一个老问题：如何为"尚未想到的程序"设计一个系统。几十年前，操作系统通过将硬件虚拟化为抽象——*进程、文件*——来解决这个问题，这些抽象足够通用，适用于尚不存在的程序。这些抽象比硬件更持久。`read()` 命令与它是访问 70 年代的磁盘组还是现代 SSD 无关。上面的抽象保持稳定，而下面的实现可以自由更改。 + +Managed Agents 遵循相同的模式。虚拟化了智能体的组件：会话（发生的一切的仅追加日志）、Harness（调用 Claude 并将 Claude 的工具调用路由到相关基础设施的循环）和沙箱（Claude 可以运行代码和编辑文件的执行环境）。这允许每个的实现被交换而不干扰其他的。对这些接口的形状有意见，但对它们后面运行的东西没有意见。 + +--- + +## 不要收养宠物 + +首先将所有智能体组件放入单个容器中，这意味着会话、智能体 Harness 和沙箱都共享一个环境。这种方法有好处，包括文件编辑是直接的系统调用，并且没有服务边界需要设计。 + +但是通过将所有东西耦合到一个容器中，遇到了一个老的基础设施问题：收养了一个[*宠物*](https://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/)。在宠物与牛的类比中，宠物是一个命名的、手工照料的个体，你不能失去它，而牛是可互换的。在他们的情况下，服务器变成了那个宠物；如果容器失败，会话就丢失了。如果容器没有响应，必须护理它恢复健康。 + +护理容器意味着调试无响应的卡住会话。唯一的窗口是 WebSocket 事件流，但这不能告诉我们*哪里*出现了故障，这意味着 Harness 中的错误、事件流中的数据包丢弃或容器脱机都呈现相同的情况。为了弄清楚出了什么问题，工程师必须在容器内打开一个 shell，但因为该容器通常也持有用户数据，这种方法基本上意味着缺乏调试能力。 + +第二个问题是，Harness 假设 Claude 处理的任何东西都与它一起生活在容器中。当客户要求将 Claude 连接到他们的虚拟私有云时，他们必须要么将他们的网络与我们的对等，要么在他们自己的环境中运行我们的 Harness。Harness 中烘焙的假设在想要将其连接到不同的基础设施时变成了一个问题。 + +--- + +## 将大脑与手分离 + +到达的解决方案是将"大脑"（Claude 及其 Harness）与"手"（执行操作的沙箱和工具）和"会话"（会话事件的日志）两者分离。每个都成为一个对其他组件做出很少假设的接口，并且每个都可以独立失败或被替换。 + +**Harness 离开容器。** 将大脑与手分离意味着 Harness 不再生活在容器内。它像调用任何其他工具一样调用容器：`execute(name, input) → string`。容器变成了牛。如果容器死亡，Harness 将失败捕获为工具调用错误并将其传递回 Claude。如果 Claude 决定重试，可以使用标准配方重新初始化一个新容器：`provision({resources})`。不再需要护理失败的容器恢复健康。 + +**从 Harness 失败中恢复。** Harness 也变成了牛。因为会话日志位于 Harness 之外，Harness 中没有任何东西需要在崩溃后幸存。当一个失败时，可以用 `wake(sessionId)` 重新启动一个新的，使用 `getSession(id)` 取回事件日志，并从最后一个事件恢复。在智能体循环期间，Harness 使用 `emitEvent(id, event)` 写入会话，以保持事件的持久记录。 + +![将大脑与手分离](../assets/73e900af5b9d6ed8c64db0a8e74d4465963556b7-1640x1596.webp) + +**安全边界。** 在耦合设计中，Claude 生成的任何不受信任的代码都在与凭证相同的容器中运行——因此提示注入只需要说服 Claude 读取自己的环境。一旦攻击者拥有这些令牌，他们就可以生成新的、不受限制的会话并将工作委托给它们。狭窄的范围是一个明显的缓解措施，但这编码了关于 Claude 不能用有限令牌做什么的假设——而 Claude 正变得越来越智能。结构性修复是确保令牌永远无法从 Claude 生成代码运行的沙箱访问到。 + +使用两种模式来确保这一点。Auth 可以与资源捆绑在一起，也可以保存在沙箱外的保险库中。对于 Git，使用每个存储库的访问令牌在沙箱初始化期间克隆存储库并将其连接到本地 git remote。Git `push` 和 `pull` 可以从沙箱内部工作，而智能体永远不会自己处理令牌。对于自定义工具，支持 MCP 并将 OAuth 令牌存储在安全保险库中。Claude 通过专用代理调用 MCP 工具；这个代理接受与会话关联的令牌。然后代理可以从保险库获取相应的凭证并调用外部服务。Harness 永远不会知道任何凭证。 + +--- + +## 会话不是 Claude 的上下文窗口 + +长 horizon 任务通常超过 Claude 上下文窗口的长度，解决这个问题的标准方法都涉及关于保留什么的不可逆决策。在关于上下文工程的[先前工作](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)中探索了这些技术。例如，压缩让 Claude 保存其上下文窗口的摘要，记忆工具让 Claude 将上下文写入文件，从而实现跨会话学习。这可以与上下文修剪配对，上下文修剪选择性地删除令牌，如旧工具结果或思考块。 + +但是选择性保留或丢弃上下文的不可逆决策可能导致失败。很难知道未来回合将需要哪些令牌。如果消息被压缩步骤转换，Harness 会从 Claude 的上下文窗口中删除压缩的消息，并且只有在存储时才能恢复这些消息。先前的工作[已经探索](https://arxiv.org/pdf/2512.24601)通过将上下文存储为生活在*上下文窗口之外*的对象来解决这个问题的方法。例如，上下文可以是 REPL 中的一个对象，LLM 通过编写代码来过滤或切片它来以编程方式访问它。 + +![会话不是 Claude 的上下文窗口](../assets/cf0719d7832b1f577b7393c84a7c53eecc725ca4-760x200.webp) + +在 Managed Agents 中，会话提供了相同的好处，充当生活在 Claude 上下文窗口之外的上下文对象。但不是存储在沙箱或 REPL 中，上下文被持久存储在会话日志中。接口 `getEvents()` 允许大脑通过选择事件流的位置切片来询问上下文。该接口可以灵活使用，允许大脑从上次停止阅读的地方继续，倒回到特定时刻之前的几个事件以查看前导，或在特定操作之前重新阅读上下文。 + +任何获取的事件也可以在传递给 Claude 的上下文窗口之前在 Harness 中进行转换。这些转换可以是 Harness 编码的任何东西，包括上下文组织以实现高提示缓存命中率和上下文工程。分离了会话中可恢复的上下文存储和 Harness 中任意上下文管理的关注点，因为无法预测未来模型将需要什么特定的上下文工程。接口将该上下文管理推送到 Harness 中，并且只保证会话是持久的并且可用于询问。 + +--- + +## 多大脑，多手 + +**多大脑。** 将大脑与手分离解决了最早的客户投诉之一。当团队希望 Claude 针对他们自己 VPC 中的资源工作时，唯一的路径是将他们的网络与我们的对等，因为持有 Harness 的容器假设每个资源都在它旁边。一旦 Harness 不再在容器中，那个假设就消失了。相同的更改带来了性能回报。当最初将大脑放在容器中时，意味着许多大脑需要同样多的容器。对于每个大脑，在该容器被配置之前不能进行推理；每个会话都预先支付完整的容器设置成本。每个会话，即使是那些永远不会接触沙箱的会话，都必须克隆存储库、启动进程、从我们的服务器获取待处理事件。 + +那段死时间表示为首次 token 时间（TTFT），它衡量会话在接受工作和产生其第一个响应令牌之间等待的时间。TTFT 是用户最敏锐地*感受到*的延迟。 + +将大脑与手分离意味着容器仅在需要时才由大脑通过工具调用 `(execute(name, input) → string)` 来配置。因此，不需要立即容器的会话不必等待一个。一旦编排层从会话日志中提取待处理事件，推理就可以开始。使用这种架构，p50 TTFT 下降了大约 60%，p95 下降了超过 90%。扩展到许多大脑只意味着启动许多无状态 Harness，并且仅在需要时将它们连接到手。 + +**多手。** 还希望能够将每个大脑连接到多只手。在实践中，这意味着 Claude 必须推理许多执行环境并决定在哪里发送工作——这比在单个 shell 中操作更难的认知任务。从单个容器中的大脑开始，因为早期的模型没有能力做到这一点。随着智能的扩展，单个容器变成了限制而不是：当该容器失败时，失去了大脑正在伸向的每只手的状态。 + +将大脑与手分离使每只手成为一个工具，`execute(name, input) → string`：一个名称和输入进去，一个字符串返回。该接口支持任何自定义工具、任何 MCP 服务器和我们自己的工具。Harness 不知道沙箱是容器、电话还是 Pokémon 模拟器。并且因为没有手耦合到任何大脑，大脑可以将手彼此传递。 + +![多大脑，多手](../assets/4f67b1c10566552aec514a716ea43544ab330e0b-668x243.webp) + +--- + +## 结论 + +面临的挑战是一个老问题：如何为"尚未想到的程序"设计一个系统。操作系统通过将硬件虚拟化为足够通用的抽象，适用于尚不存在的程序，已经持续了几十年。使用 Managed Agents，旨在设计一个适应 Claude 周围未来 Harness、沙箱或其他组件的系统。 + +Managed Agents 是一个具有相同精神的元 Harness，对 Claude 未来将需要的*特定*Harness 没有意见。相反，它是一个具有允许许多不同 Harness 的通用接口的系统。例如，Claude Code 是一个在任务中广泛使用的优秀 Harness。还表明特定任务的智能体 Harness 在狭窄领域表现出色。Managed Agents 可以容纳其中任何一个，随着时间推移匹配 Claude 的智能。 + +元 Harness 设计意味着对 Claude 周围的接口有意见：期望 Claude 将需要操纵状态（会话）和执行计算（沙箱）的能力。还期望 Claude 将需要扩展到多大脑和多手的能力。设计了接口，以便这些可以在长时间范围内可靠且安全地运行。但对 Claude 将需要的大脑或手的数量或位置没有做出任何假设。 + +--- + +## 相关研究 + +- [[Harness-Engineering|Harness 工程]] +- [[Long-Running-Harness-Design|长运行应用的 Harness 设计]] diff --git a/wiki/practices/MiniMax-M27-Self-Evolution.md b/wiki/practices/MiniMax-M27-Self-Evolution.md new file mode 100644 index 0000000..9e3afab --- /dev/null +++ b/wiki/practices/MiniMax-M27-Self-Evolution.md @@ -0,0 +1,221 @@ +--- +title: "MiniMax M2.7：开启模型的自我进化" +source: "https://www.minimaxi.com/news/minimax-m27-zh" +author: "MiniMax" +published: 2026-04-11 +last_updated: 2026-04-11 +tags: + - practices + - model-evolution + - minimax +raw_sources: + - path: raw/MiniMax M2.7 开启模型的自我进化.md + hash: "sha256:initial" +--- + +# MiniMax M2.7：开启模型的自我进化 + +本文介绍了 MiniMax M2.7——第一个模型深度参与迭代自己的模型。M2.7 能够自行构建复杂 Agent Harness，并基于 Agent Teams、复杂 Skills、Tool Search Tool 等能力，完成高度复杂的生产力任务。 + +## 核心亮点 + +在 M2 系列模型发布后的几个月，MiniMax 收到了大量热心用户的反馈和建议，这促使他们进一步加速模型的迭代效率。除了更加认真工作之外，能找到的唯一途径就是**开启模型和组织的自我进化**。 + +**关键声明**：MiniMax M2.7 是第一个模型深度参与迭代自己的模型。 + +![MiniMax M2.7](../assets/Image.png) + +--- + +## 三大核心能力 + +### 1. 真实软件工程能力 + +M2.7 在真实的软件工程中有优异的表现，包括端到端的完整项目交付，分析日志排查 Bug、代码安全，机器学习等。 + +**基准测试成绩**： +- **SWE-Pro**：56.22%，几乎接近 Opus 最好的水平 +- **VIBE-Pro**：55.6%（端到端完整项目交付场景） +- **Terminal Bench 2**：57.0%（对复杂工程系统的深层理解） +- **SWE Multilingual**：76.5 +- **Multi SWE Bench**：52.7 + +这一能力延伸到了端到端的完整项目交付场景——无论是 Web、Android、iOS 还是 Simulation 类需求，都可以直接交给 M2.7 完成。 + +**线上生产环境故障调试示例**： +面对实际的生产环境告警，M2.7 能： +- 关联监控指标与部署时间线做因果推理 +- 对轨迹采样做统计分析并提出精准假设 +- 主动连接数据库执行验证根因 +- 定位到代码仓库中缺失的索引迁移文件 +- 知道用非阻塞建索引先止血，再提 MR + +> **结果**：从可观测性分析、数据库专业知识到 SRE 级别的决策判断——这不只是一个会写代码的模型，而是一个真正理解生产系统的模型。相比传统的人工排障流程，基于 M2.7，已多次将线上生产系统故障的恢复时间缩短到三分钟以内。 + +**原生 Agent Teams 能力**： +Agent Teams 对模型提出了范式级要求：角色边界、对抗性推理、协议遵循、行为分化——这些无法通过提示词，必须内化为模型的原生能力。Agent Teams 场景下，模型需要稳定锚定角色身份、主动挑战队友的逻辑与伦理盲区、在复杂状态机中自主决策。 + +![原生 Agent Teams 能力](../assets/dd83d19a-156b-481f-9835-9576293c323a.gif) + +--- + +### 2. 专业办公能力 + +在专业办公领域，提升了模型在各领域的专业知识和任务交付能力。 + +**基准测试成绩**： +- **GDPval-AA**：ELO 得分 1495-1500，为开源最高，仅次于 Opus 4.6、Sonnet 4.6 和 GPT5.4，超过了 GPT5.3 +- **Toolathon**：46.3% 正确率，达到全球第一梯队水平 +- **MM Claw**：62.7% 正确率，接近最新的 Sonnet 4.6 + +**核心办公能力**： +1. **专业知识与任务交付能力** - 模型需要具备各领域的专业知识，理解用户的需求 +2. **与复杂环境的交互能力** - 泛化的日常场景意味着模型需要灵活适应各类上下文、调用各种 skills 和工具、并在长程交互中保持稳定的指令遵循 + +**Office 套件能力**： +- 对 Office 三件套 Excel/PPT/Word 的复杂编辑能力显著提升 +- 能更好地完成多轮修改和高保真的编辑 +- 既能够基于模版和 skills 直接生成文件，也能够遵从用户的交互指令，对已有的文件做多轮的高保真编辑 +- 在 40 个复杂 skills (> 2000 Token) 的 case 上，仍能保持 97% 的 skills 遵循率 + +**Finance 领域示例**： +在阅读研报并建模公司未来营收的场景，M2.7 可以： +- 自主阅读公司的年报与业绩沟通会纪要 +- 交叉比对多篇研报 +- 独立设计假设并构建营收预测模型 +- 再基于模版产出 PPT 和研究报告 + +从业者的评价是：产出物已经可以作为初稿直接进入后续工作流程。 + +--- + +### 3. 互动娱乐能力 + +M2.7 具备优秀的身份保持能力和情商，除了生产力使用外，给互动娱乐场景的创新也准备了空间。 + +在 OpenClaw 等 Agent 脚手架的使用过程中，不少用户在使用 Agent 完成工作的同时，还希望模型具备比较高的情商和复杂人设保持能力。在有人设的情况下，用户不再只是让模型机械完成任务，而是开始自然于与 Agent"相处"。 + +这促使 MiniMax 思考，产品与交互设计、内容创作、甚至娱乐体验的构建，都可以被 AI 原生驱动的可能性。这会让 Agentic 模型的使用从单纯的生产力能进一步拓展到互动娱乐。 + +**OpenRoom：Agent 交互系统** +基于此，MiniMax 构建了一个 Agent 交互系统 OpenRoom，它将 AI 互动置入一个万物皆可互动的 Web GUI 空间。在这里： +- 对话即驱动，实时产生视觉反馈与场景交互 +- 角色可以主动地与环境交互 +- 框架扩展性较高，能够随着模型 Agentic 能力的提升和社区的共建持续进化 +- 探索出更多人与 Agent 之间全新的交互方式 + +**开源信息**： +- 项目地址：[github.com/MiniMax-AI/OpenRoom](https://github.com/MiniMax-AI/OpenRoom) +- 立即体验：[openroom.ai](https://openroom.ai/) +- 代码大部分也是 AI 写的 + +--- + +## 模型自我进化智能体 + +MiniMax 分享了一个内部让 M2 系列模型自我进化的实践，这也是对模型 Agent 能力边界的探索。 + +### 研究型 Agent Harness + +Agent Harness 通常依赖复杂的 Skills、记忆系统和其他组件来提升模型对不同工作环境的适应能力。在此基础上，MiniMax 在 M2 的早期版本中，将其引导为一个**研究型 Agent Harness**——它能够与不同的研究项目组进行交互和协作。 + +**系统覆盖**： +- 数据流水线 +- 训练环境 +- 评测基础设施 +- 跨团队协作 +- 持久化记忆 + +让研究员可以驱动它来交付更好的模型。研究 Agent 驱动着产出下一代模型的迭代循环。研究员在每一层引导方向，模型在每一层负责构建。 + +### RL 场景示例 + +以一个 RL 场景为例： +1. 研究员从一个实验想法出发，与 Agent 展开讨论 +2. Agent 协助进行文献调研，持续跟踪预设的实验规格，完成数据流水线及其他对接工作，并启动实验 +3. 实验运行期间，它会自动监控和分析实验状态，并自动触发日志读取、问题排查、指标分析、代码修复、合并请求以及冒烟测试 +4. 识别并配置那些细微但关键的变更 + +这些工作过去可能需要来自不同团队的多位同事协作完成，而现在研究员只需在关键决策和讨论时介入。这大幅加速了问题发现和实验迭代，从而更快地交付模型。 + +**结果**：在这个场景下，M2.7 能够胜任 30-50% 的工作流。 + +![研究型 Agent Harness](../assets/d080353d-31aa-4642-8586-3b7f2201bf94.png) + +--- + +## Harness 自我进化 + +MiniMax 在迭代过程中也意识到，**模型自主迭代 harness 的能力也至关重要**。 + +内部的 harness 会： +- 自主收集反馈 +- 建立内部任务的评测集 +- 基于此不断迭代自己的 Agent 架构、Skills/MCP 实现和记忆机制 +- 更好和更高效的完成任务 + +### 内部脚手架优化示例 + +让 M2.7 优化一个内部脚手架上模型的软件工程开发表现。M2.7 全程自主运行，执行以下迭代循环超过 100 轮： + +``` +分析失败轨迹 → 规划改动 → 修改脚手架代码 → 运行评测 → 对比结果 → 决定保留或回退 +``` + +**M2.7 发现的有效优化**： +- 系统性搜索温度、频率惩罚、存在惩罚等采样参数的最优组合 +- 为模型设计更具体的工作流指引（如修复后自动搜索其他文件中的相同 bug 模式） +- 在脚手架的 Agent Loop 中添加循环检测等优化 + +**最终结果**：在内部评测集上效果提升 30%。 + +--- + +## MLE Bench Lite 测试 + +MiniMax 相信，未来的 AI 自我进化会逐步向完全自动化过渡，包括完全自主的协调数据构建、模型训练、推理架构、评测等等。 + +**测试设置**： +- 用 M2.7 参与了 MLE Bench Lite 的 22 个机器学习任务测试 +- 几乎囊括了研发的所有环节 +- 设计和实现了一个简易的脚手架来引导 Agent 进行自主优化 +- 核心模块包括：短时记忆、自反馈以及自优化三个模块 +- 总共测试三次，每次有 24 小时来迭代进化 + +**具体运作方式**： +- Agent 完成每轮迭代后会形成一个短时记忆文件 +- 同时对当前轮次的结果进行自反馈，从而给下一轮次提供潜在的优化方向 +- 下一轮次基于所有历史轮次的记忆及自反馈链进行下一步的自优化 + +**成绩**： +- 最好的一次取得 9 枚金牌，5 枚银牌，1 枚铜牌 +- 三次平均是 66.6% 的得牌率 +- 此成绩仅次于： + - Opus-4.6 (75.7%) + - GPT-5.4 (71.2%) +- 和 Gemini-3.1 (66.6%) 持平 + +![MLE Bench Lite 测试成绩](../assets/img-3.png) + +--- + +## 组织进化 + +基于上述能力，M2.7 也在显著加速 MiniMax 自身向一个 AI Native 组织的进化。 + +![组织进化](../assets/img-1.png) + +**产品可用性**： +- MiniMax M2.7 已在 MiniMax Agent 与开放平台上全量上线 +- MiniMax Agent：[agent.minimaxi.com](https://agent.minimaxi.com/) +- API 服务：[platform.minimaxi.com](https://platform.minimaxi.com/) +- Coding Plan 订阅：[platform.minimaxi.com/subscribe/coding-plan](https://platform.minimaxi.com/subscribe/coding-plan) + +> **Intelligence with Everyone.** + +--- + +## 相关研究 + +- [[Harness-Engineering|Harness 工程]] +- [[Meta-Harness|元 Harness]] diff --git a/wiki/practices/Mitchellh-AI-Adoption-Journey.md b/wiki/practices/Mitchellh-AI-Adoption-Journey.md new file mode 100644 index 0000000..595ae7e --- /dev/null +++ b/wiki/practices/Mitchellh-AI-Adoption-Journey.md @@ -0,0 +1,153 @@ +--- +title: "Mitchell Hashimoto 的 AI 采用之旅" +source: "https://mitchellh.com/writing/my-ai-adoption-journey" +author: "Mitchell Hashimoto" +published: 2026-02-05 +last_updated: 2026-04-11 +tags: + - practices + - adoption + - harness-engineering +raw_sources: + - path: raw/My AI Adoption Journey.md + hash: "sha256:initial" +--- + +# Mitchell Hashimoto 的 AI 采用之旅 + +本文记录了 HashiCorp 创始人 Mitchell Hashimoto 从 AI 怀疑论者到深度用户的六个阶段采用之旅。他的经历为如何有效采用 AI 工具提供了实用的指导框架。 + +## 核心论点 + +Mitchell 认为，任何有意义的工具采用都必然经历三个阶段： + +1. **低效期** - 初期感觉不如现有工作流高效 +2. **胜任期** - 逐渐掌握，达到与手动工作相当的效率 +3. **变革期** - 发现新的工作方式，彻底改变工作流和生活 + +> **关键洞察**：必须强迫自己度过前两个阶段，才能发现 AI 的真正价值。 + +--- + +## 六个采用阶段 + +### 阶段 1：放弃聊天机器人界面 + +**立即停止尝试通过聊天机器人执行有意义的工作。** + +聊天机器人（如 ChatGPT、Gemini 网页版）有其实际价值，是日常 AI 工作流的一部分，但在编码方面的效用非常有限。你大多只是希望它们基于先前训练得出正确结果，而纠正它们需要你（人类）反复告诉它们错了。这种方式效率低下。 + +**关键发现**： +- 聊天界面在棕色地带（brownfield）项目中效果很差 +- 复制粘贴代码和命令输出的过程令人沮丧 +- 必须使用**智能体（Agent）**才能找到价值 + +**智能体的最低要求**： +- 读取文件 +- 执行程序 +- 发出 HTTP 请求 + +> **个人转折点**：将 Zed 命令面板的截图粘贴到 Gemini，要求它用 SwiftUI 重现，结果非常好。Ghostty 的 macOS 命令面板仅在此基础上做了轻微修改。 + +--- + +### 阶段 2：重现自己的工作 + +**强迫自己用智能体重现所有手动提交。** + +Mitchell 最初使用 Claude Code 时并不满意，觉得必须润色它生成的所有内容，这比自己动手花的时间还多。但他没有放弃，而是**强迫自己用智能体重现所有手动提交**—— literally 把工作做两遍。先手动完成工作，然后与智能体斗争以产生相同质量和功能的结果（当然不让它看到手动解决方案）。 + +**这一过程的收获**： +1. **分解任务** - 将会话拆分为独立、清晰、可操作的任务。不要试图在一个大型会话中"画出整只猫头鹰"。 +2. **规划与执行分离** - 对于模糊的请求，将工作分为规划会话和执行会话。 +3. **自我验证** - 如果给智能体一种验证其工作的方法，它通常会修正自己的错误并防止回归。 + +**更重要的收获**： +- 了解了智能体擅长什么、不擅长什么 +- 对于擅长的任务，掌握了如何获得想要的结果 +- 知道了**什么时候不应该使用智能体**——这本身就是巨大的时间节省 + +--- + +### 阶段 3：日终智能体 + +**每天最后 30 分钟启动一个或多个智能体。** + +Mitchell 的下一个模式是：**留出每天最后 30 分钟来启动一个或多个智能体。** 他的假设是，也许可以让智能体在他无法工作的时间取得一些积极进展。基本上：不要试图在已有时间内做更多事情，而是尝试在没有的时间内做更多事情。 + +**发现的高价值任务类别**： +- **深度研究会话** - 要求智能体调查某个领域，例如查找特定语言中具有特定许可证类型的所有库，并为每个库生成多页摘要，包括优缺点、开发活动、社会情绪等。 +- **并行智能体尝试不同的模糊想法** - 不期望它们产生可交付的成果，但也许可以在第二天处理任务时揭示一些未知的未知。 +- **Issue 和 PR 分类/审查** - 智能体擅长使用 `gh`（GitHub CLI），因此手动编写了一个快速方法来并行启动一堆智能体来分类问题。不允许智能体回应，只需要第二天的报告来指导高价值或低工作量的任务。 + +> **关键洞察**：这不是让智能体整夜循环运行。大多数情况下，智能体在半小时内完成任务。但在工作日的后半段，通常会感到疲惫，脱离心流状态，个人效率低下，因此将精力转移到启动这些智能体上，第二天早上可以"热启动"，比平时更快地开始工作。 + +--- + +### 阶段 4：外包确定的任务 + +**让智能体做所有它几乎肯定能正确完成的工作，同时你处理其他任务。** + +到了这个阶段，Mitchell 对 AI 擅长和不擅长的任务非常有信心。对某些任务，AI 会取得基本正确的解决方案有很高的信心。因此，旅程的下一步是：**让智能体做所有这些工作，同时你处理其他任务。** + +**具体做法**： +- 每天从昨晚分类智能体的结果开始 +- 手动过滤出智能体几乎肯定能很好解决的问题 +- 让它们在后台运行（一次一个，不是并行） +- **同时处理其他事情**——不是刷社交媒体，而是处于正常的、AI 前的深度思考模式 + +**关键规则**： +- **关闭智能体桌面通知** - 上下文切换非常昂贵。为了保持效率，人类应该控制何时中断智能体，而不是相反。 +- 在工作的自然休息时间，切换过去检查一下，然后继续。 + +> **技能形成的权衡**：这很好地抵消了 [Anthropic 技能形成论文](https://www.anthropic.com/research/AI-assistance-coding-skills) 中提出的问题。你在权衡：不为委托给智能体的任务形成技能，同时继续在手动处理的任务中自然形成技能。 + +--- + +### 阶段 5：工程化 Harness + +**每次发现智能体犯错时，花时间设计一个解决方案，让智能体永远不会再犯那个错误。** + +> **显而易见的事实**：当智能体第一次就产生正确结果，或最坏情况下只需要最少润色时，效率会高得多。实现这一点的最可靠方法是给智能体快速、高质量的工具来自动告诉它什么时候错了。 + +**Harness 工程的两种形式**： + +1. **更好的隐式提示（AGENTS.md）** - 对于简单的事情，比如智能体反复运行错误的命令或找到错误的 API，更新 `AGENTS.md`（或等效文件）。每一行都基于一个糟糕的智能体行为，而且几乎完全解决了所有问题。 + +2. **实际的编程工具** - 例如，截图脚本、运行筛选测试等。这通常与 AGENTS.md 更改配对，让它知道这个工具的存在。 + +> **这就是我今天的位置**：每当看到智能体做了坏事，就真诚地努力防止它再做那件坏事。或者，相反地，真诚地努力让智能体验证它们在做正确的事。 + +--- + +### 阶段 6：始终有一个智能体在运行 + +**如果智能体没有在运行，问问自己"现在有什么是智能体可以为我做的吗？"** + +Mitchell 同时也在**始终有一个智能体在运行**的目标下运作。他特别喜欢将这与较慢、更周到的模型（如 Amp 的深度模式，本质上就是 GPT-5.2-Codex）结合使用，这些模型可能需要 30 多分钟来进行小的更改。但另一面是，它确实倾向于产生非常好的结果。 + +**当前状态**： +- 还没有运行多个智能体，目前也不想这么做 +- 有一个智能体在运行是一个很好的平衡 +- 目前正常工作日中可能只有 10-20% 的时间有效地让后台智能体运行 +- 正在积极努力改善这一点 + +> **关键原则**：不想为了运行智能体而运行智能体。只有当有一个认为对自己真正有帮助的任务时才运行它们。这个目标的一部分挑战是改善自己的工作流和工具，以便可以有源源不断的高质量工作来委托。即使没有 AI，这也很重要！ + +--- + +## 今天的状态 + +这就是 Mitchell 今天的位置。 + +通过这段旅程，他个人达到了一个点：成功使用现代 AI 工具，并相信自己以基于现实的适当、审慎的观点来处理它。他完全不在乎 AI 是否会留下来——他是一个软件工匠，只是出于对这个行业的热爱而构建东西。 + +整个领域发展得如此之快，他确信很快就会回头看这篇文章，嘲笑自己的天真。但正如他们所说，如果你不能为过去的自己感到尴尬，你可能就没有成长。只希望能朝着正确的方向成长！ + +--- + +## 相关研究 + +- [[Harness-Engineering|Harness 工程]] +- [[Long-Running-Harness-Design|长运行应用的 Harness 设计]] +- [[OpenAI-Codex-Harness-Engineering|OpenAI Codex Harness 工程]] diff --git a/wiki/practices/Mitchellh-Adoption-Journey.md b/wiki/practices/Mitchellh-Adoption-Journey.md deleted file mode 100644 index 561046e..0000000 --- a/wiki/practices/Mitchellh-Adoption-Journey.md +++ /dev/null @@ -1,150 +0,0 @@ ---- -title: Mitchellh AI 采用之旅 -tags: [实践指南, 采用路径, 个人工作流] -source: [Mitchell Hashimoto] -confidence_score: 高 -last_updated: 2026-04-07 ---- - -# Mitchellh AI 采用之旅 - -本文总结了 HashiCorp 创始人 Mitchell Hashimoto 的个人 AI 工具采用历程。他分享了从怀疑论者到深度用户的六个阶段，为其他开发者提供了实用的参考路径。 - -## 核心观点 - -> "我采用任何有意义的工具的经验是，我必然经历三个阶段：(1) 低效率时期 (2) 胜任时期，最后 (3) 工作流和生活改变发现时期。" - ---- - -## 六个阶段 - -### 阶段 1：放弃聊天机器人 - -**立即停止尝试通过聊天机器人执行有意义的工作**（例如 ChatGPT、网页版 Gemini 等）。 - -聊天机器人确实有价值，而且是 Mitchell AI 工作流的日常组成部分，但它们在编码中的效用非常有限，因为你大多希望它们根据先前训练得出正确结果，而纠正它们涉及人类（你）反复告诉它们错了。这是低效的。 - -**关键点：** -- 每个人的第一次 AI 体验都是聊天界面 -- 每个人第一次尝试用 AI 编码都是要求聊天界面写代码 -- 要找到价值，你**必须**使用**智能体**（Agent） - -**什么是智能体？** -智能体是行业采用的术语，指能够聊天并循环调用外部行为的 LLM。至少，智能体必须具备以下能力： -- 读取文件 -- 执行程序 -- 发起 HTTP 请求 - ---- - -### 阶段 2：重现你自己的工作 - -Mitchell 最初尝试 Claude Code 时并没有留下深刻印象。他只是没有从会话中得到好的结果。他觉得必须润色它生成的所有内容，这个过程比他自己做要花更多时间。 - -**解决方案：** -> "我没有放弃，而是**强迫自己用智能体重现所有手动提交**。我真的做了两次工作。我会手动完成工作，然后与智能体斗争以产生相同的质量和功能结果（当然，不让它看到我的手动解决方案）。" - -**这是痛苦的**，因为它妨碍了简单地完成事情。但 Mitchell 从第一性原理中自己发现了别人已经在说的东西，但自己发现它产生了更强的基本理解： - -1. 将会话分解为单独清晰、可操作的任务。不要试图在一个大型会话中"画猫头鹰"。 -2. 对于模糊的请求，将工作分成单独的规划与会话。 -3. 如果你给智能体验证其工作的方法，它往往会修复自己的错误并防止回归。 - -更一般地说，他还发现了智能体当时擅长什么、不擅长什么的边界，以及对于它们擅长的任务如何实现想要的结果。 - -所有这些都导致了显著的效率提升，以至于他开始自然地使用智能体，感觉不比自己做慢（但他仍然不觉得更快，因为他主要在照看智能体）。 - ---- - -### 阶段 3：日终智能体 - -为了尝试找到一些效率，Mitchell 开始了一个新模式：**每天划出最后 30 分钟来启动一个或多个智能体。** - -他的假设是，如果智能体能在他无论如何都不能工作的时间里取得一些**积极进展**，也许他可以获得一些效率。基本上：与其试图在他拥有的时间里做更多，不如尝试在他没有的时间里做更多。 - -**发现的有价值的工作类别：** - -1. **深度研究会话**：要求智能体调查某个领域，例如查找特定许可证类型的特定语言的所有库，并为每个库生成关于其优缺点、开发活动、社会情绪等的多页摘要。 - -2. **并行智能体尝试他没有时间开始的不同模糊想法**：不期望它们产生他会交付的东西，但也许可以在第二天他处理任务时阐明一些未知的未知。 - -3. **问题和 PR 分类/审查**：智能体擅长使用 `gh`（GitHub CLI），因此手动编写了一个快速方法来并行启动一堆来分类问题。不会允许智能体回复，只想要第二天的报告来尝试引导他走向高价值或低工作量的任务。 - -**结果：** -他开始感觉自己比 AI 之前做的更多，哪怕只是一点点。 - ---- - -### 阶段 4：外包稳操胜券的任务 - -到了这个阶段，Mitchell 对他的 AI 擅长和不擅长什么任务非常有信心。他对某些任务 AI 会实现基本正确的解决方案有非常高的信心。 - -**所以旅程的下一步是：让智能体做所有这些工作，而他做其他任务。** - -更具体地说： -- 每天从查看前一晚分类智能体的结果开始 -- 手动过滤以找到智能体几乎肯定会很好解决的问题 -- 让它们在后台继续（一次一个，不是并行） - -**同时，他做其他事情。** 不是去社交媒体（比平时不用 AI 更多），不是看视频等。他处于自己的、正常的、AI 之前的深度思考模式，处理他想做或必须做的事情。 - -**非常重要：关闭智能体桌面通知。** 上下文切换非常昂贵。为了保持效率，他发现作为人类控制何时中断智能体是他的工作，而不是相反。不要让智能体通知你。在工作的自然休息时间，切换标签检查它，然后继续。 - -**结果：** -他坚定地处于"我无法回去"的境地。他觉得更有效率，但即使不是，他最喜欢的是他现在可以将编码和思考集中在他真正喜欢的任务上，同时仍然充分完成他不喜欢的任务。 - ---- - -### 阶段 5：设计 Harness - -冒着陈述显而易见的风险：智能体在第一次产生正确结果时效率要高得多，或者最坏情况下产生需要最少润色的结果。实现这一点的最可靠方法是给智能体快速、高质量的工具来自动告诉它什么时候错了。 - -Mitchell 不知道是否有一个广泛的行业接受的术语，但他已经逐渐称之为"**Harness 工程**"。这是这样一种想法：任何时候你发现智能体犯了错误，你都花时间设计一个解决方案，使智能体永远不会再犯那个错误。 - -**这有两种形式：** - -1. **更好的隐式提示（AGENTS.md）**：对于简单的事情，比如智能体反复运行错误的命令或找到错误的 API，更新 `AGENTS.md`（或等效文件）。每一行都是基于坏的智能体行为，并且几乎完全解决了所有问题。 - -2. **实际的编程工具**：例如，截图脚本、运行过滤测试等。这通常与 AGENTS.md 更改配对，让它知道这存在。 - -**这就是 Mitchell 今天所处的位置。** 他正在真诚地努力，每当看到智能体做坏事时，防止它再做那件坏事。或者，相反，他正在真诚地努力让智能体验证它们在做好事。 - ---- - -### 阶段 6：始终运行一个智能体 - -与阶段 5 同时，Mitchell 也在**始终运行一个智能体**的目标下运作。如果智能体没有运行，他会问自己："现在有什么智能体可以为我做的吗？" - -他特别喜欢将其与较慢、更深思熟虑的模型结合使用，比如 Amp 的 [deep mode](https://ampcode.com/news/deep-mode)（基本上只是 GPT-5.2-Codex），这可能需要 30 多分钟来做小的更改。另一面是它确实倾向于产生非常好的结果。 - -**重要说明：** -- 他还没有运行多个智能体，目前也真的不想。 -- 他觉得现在有一个智能体运行是一个很好的平衡，既能做他觉得愉快的深度手动工作，又能照看他有点愚蠢但又神秘高效的机器人朋友。 - -"始终运行一个智能体"的目标仍然只是一个目标。他会说现在他在正常工作日的 10 到 20% 时间里有效地运行后台智能体。但他正在积极努力改善这一点。 - -> "我不想为了运行智能体而运行智能体。我只希望在有任务我认为对我真正有帮助时运行它们。这个目标的一部分挑战是改善我自己的工作流和工具，以便我能有源源不断的高质量工作可以委托。即使没有 AI，这也很重要！" - ---- - -## 今天 - -这就是 Mitchell 今天所处的位置。 - -通过这段旅程，他个人已经达到了一个点，他在现代 AI 工具上取得了成功，并且相信他正在以基于现实的适当衡量观点来处理它。他真的不在乎 AI 是否会留下来，他是一个软件工匠，只是为了热爱而构建东西。 - -整个格局变化如此之快，他相信他会很快回头看这篇帖子，嘲笑自己的天真。但正如他们所说，如果你不能为过去的自己感到尴尬，你可能就没有成长。他只希望他会朝着正确的方向成长！ - -## 相关概念 - -- [[Harness-Engineering|Harness 工程]] -- [[Agent|智能体]] - -## 参考来源 - -1. Mitchell Hashimoto - My AI Adoption Journey - ---- - -*最后更新：2026-04-07* -*本文档由 [[WikiLLM]] 编译* diff --git a/wiki/practices/OpenAI-Codex-Harness-Engineering.md b/wiki/practices/OpenAI-Codex-Harness-Engineering.md new file mode 100644 index 0000000..c56cfcd --- /dev/null +++ b/wiki/practices/OpenAI-Codex-Harness-Engineering.md @@ -0,0 +1,257 @@ +--- +title: "OpenAI Codex Harness 工程" +source: "工程技术：在智能体优先的世界中利用 Codex" +raw_sources: + - path: raw/工程技术：在智能体优先的世界中利用 Codex.md + hash: "sha256:initial" +tags: + - "实践指南" + - "OpenAI" + - "Codex" + - "智能体优先" +last_updated: 2026-04-11 +--- + +# OpenAI Codex Harness 工程 + +本文基于 OpenAI 团队的实践经验，探讨如何构建一个**完全由智能体生成代码**的产品开发环境。 + +## 核心实验 + +在过去五个月里，OpenAI 团队进行了一项实验：构建并交付一款软件产品的内部 beta 版，**其中没有一行代码是人工编写的**。 + +### 关键数据 + +- **代码量**：约一百万行代码 +- **时间**：五个月（2025年8月下旬开始） +- **团队规模**：从 3 名工程师扩展到 7 名 +- **PR 吞吐量**：平均每位工程师每天处理 3.5 个 PRs +- **效率提升**：只用了手工编写代码所需的大约 1/10 的时间 + +> **核心理念**：人类掌舵。智能体执行。 + +## 工程师角色的重新定义 + +由于缺乏人工编码的实践，**工程师工作的重点转向了系统、架构和杠杆作用**。 + +### 深度优先工作方式 + +将更大的目标拆解为更小的构建模块（设计、代码、评审、测试等），提示智能体去构建这些模块，并使用它们去解锁更复杂的任务。 + +> **关键洞见**：当事情进行不顺利时，解决方案基本上再也不会是"再努力一点"。因为取得进展的唯一方式是让 Codex 来完成工作，而人类工程师则总是介入这项任务并追问："究竟还需要什么样的能力，我们又该如何让这个能力对智能体来说既清晰可读又可强制执行？" + +### 交互方式 + +人类几乎完全通过提示与系统交互： +- 工程师描述任务 +- 运行智能体，允许其打开一个 Pull Request +- 指示 Codex 在本地审核其自身的更改 +- 请求额外的特定智能体审查 +- 对任何人工或智能体给出的反馈做出响应 +- 循环往复，直到所有智能体审核人员都满意为止（这实际上是一个 [Ralph Wiggum 循环](https://ghuntley.com/loop/)） + +## 提高应用程序的可读性 + +随着代码吞吐量的增加，瓶颈变成了人工 QA 能力。解决方案是：**令应用程序的 UI、日志和应用指标等内容对 Codex 直接可读**。 + +### Chrome DevTools 集成 + +- 令应用程序可以根据 git worktree 启动 +- 将 Chrome DevTools 协议接入智能体运行时 +- 创建用于处理 DOM 快照、屏幕截图和导航的技能 +- 使 Codex 能够复现错误、验证修复，并直接推理 UI 的行为 + +![Codex 使用 Chrome DevTools MCP 驱动应用程序以验证其工作](../assets/OAI_Harness_engineering_Codex_drives_the_app_with_Chrome_DevTools_MCP_to_validate_its_work_desktop-dark.png) + +### 完整的可观测性堆栈 + +日志、指标和追踪记录会通过一个本地可观测性堆栈展示给 Codex： +- 使用 LogQL 查询日志 +- 使用 PromQL 查询指标 +- 示例提示："确保服务启动在 800ms 内完成"或"这四个关键用户旅程中的任何跨度都不得超过两秒" + +![赋予 Codex 完整的可观察性堆栈](../assets/OAI_Harness_engineering_Giving_Codex_a_full_observability_stack_desktop-dark.png) + +## 将代码仓库设为记录系统 + +情境管理是使智能体在大型和复杂任务中有效发挥作用的最大挑战之一。 + +> **早期经验教训**：要给 Codex 的是一张地图，而不是一本 1,000 页的说明书。 + +### "大型 AGENTS.md"方法的失败 + +尝试了"一个大型的 AGENTS.md"方法，结果失败了： + +1. **情境是一种稀缺资源** - 一个巨大的指令文件会挤掉任务、代码和相关文档 +2. **过多的指导反而变得无效** - 当一切都"重要"时，一切都不重要了 +3. **它会立即腐烂** - 一本庞杂的手册会变成陈旧规则的坟场 +4. **这很难核实** - 单个 blob 不适合进行机械检查 + +### 结构化知识库布局 + +因此，不再将 `AGENTS.md` 视为百科全书，而是将其视为**内容目录**： + +``` +AGENTS.md +ARCHITECTURE.md +docs/ +├── design-docs/ +│ ├── index.md +│ ├── core-beliefs.md +│ └── ... +├── exec-plans/ +│ ├── active/ +│ ├── completed/ +│ └── tech-debt-tracker.md +├── generated/ +│ └── db-schema.md +├── product-specs/ +│ ├── index.md +│ ├── new-user-onboarding.md +│ └── ... +├── references/ +│ ├── design-system-reference-llms.txt +│ ├── nixpacks-llms.txt +│ ├── uv-llms.txt +│ └── ... +├── DESIGN.md +├── FRONTEND.md +├── PLANS.md +├── PRODUCT_SENSE.md +├── QUALITY_SCORE.md +├── RELIABILITY.md +└── SECURITY.md +``` + +### 渐进式披露 + +智能体从一个小而稳定的切入点开始，并被指导下一步该去哪里查看，而不是一开始就被淹没。 + +- 专职的 linter 和 CI 作业会验证知识库的更新状况、是否已交叉链接且结构正确 +- 定期运行的"doc-gardening"智能体会扫描过时或废弃文档，并发起修复用的 Pull Request + +## 目标是智能体的可读性 + +由于该代码仓库完全由智能体生成，因此首先针对 Codex 的可读性进行了优化。 + +### 智能体知识的局限性 + +从智能体的角度来看，它在运行时无法在情境中访问的任何内容都是不存在的。 + +- 存储在 Google Docs、聊天记录或人们头脑中的知识都无法被系统访问 +- 代码仓库本地的、已版本化的工件（代码、Markdown、模式、可执行计划）就是它所能看到的全部 + +![智能体知识的局限性：Codex看不到的东西就不存在](../assets/OAI_Harness_engineering_The_limits_of_agent_knowledge_desktop-dark.png) + +### 将更多情境推送到仓库中 + +那次让团队在架构模式上达成一致的 Slack 讨论？如果智能体无法发现它，那么它就会像迟了三个月入职的新员工一样，对其一无所知。 + +> **关键原则**：倾向于选择那些可以完全内化于在仓库中进行推理的依赖项和抽象。对智能体来说，通常被称为"枯燥"的技术，由于其可组合性、API 稳定性和在训练集里的表现，往往更容易建立模型。 + +## 规范架构与品味 + +仅靠文档本身，是没法保持完全由智能体生成的代码库的连贯性的。 + +> **核心策略**：通过强制执行不变量，而非对实施过程进行微观管理，我们令智能体能够快速交付，而且不会削弱基础。 + +### 示例：在边界处解析数据形状 + +要求 Codex [在边界处解析数据形状](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/)，但不规定具体实现方式（模型似乎偏好 Zod，但没有指定特定库）。 + +### 分层领域架构 + +智能体在具有严格边界和可预测结构的环境中最为高效，因此围绕一个严格的架构模型构建了该应用： + +- 每个业务域都划分为一组固定的层 +- 依赖方向经过严格验证 +- 仅允许有限的一组边 +- 通过自定义的 linter 和结构测试机械地强制执行这些约束 + +规则：在每个业务领域内（例如应用设置），代码只能"向前"依赖于一组固定的层（Types → Config → Repo → Service → Runtime → UI）。横切关注点（认证、连接器、遥测、功能标志）通过一个单一的显式接口进入：Providers。 + +![具有明确交叉界限的分层领域架构](../assets/OAI_Harness_engineering_Layered_domain_architecture_with_explicit_cross-cutting_boundries_desktop-dark.png) + +### 品味不变式 + +通过自定义的代码检查器和结构测试来强制执行规则，并辅以一小组"品味不变式"： + +- 结构化日志记录 +- 模式和类型的命名约定 +- 文件大小限制 +- 特定平台的可靠性要求 + +> **领导类比**：这类似于领导一个大型工程平台组织：在中央层面强制执行边界，在本地层面允许自主权。你非常重视界限、正确性和可重复性。在这些边界内，你允许团队或智能体在解决方案的表达方式上拥有很大的自由。 + +## 吞吐量改变了合并的理念 + +随着 Codex 的吞吐量增加，许多传统的工程规范变得不再有效。 + +- 代码仓库在运行过程中尽量减少阻塞合并门 +- Pull Request 的生命周期很短 +- 测试偶发失败通常通过后续重跑来解决，而不是无限期地阻碍进展 + +> **成本权衡**：在一个智能体吞吐量远超人类注意力的系统中，纠错成本低，而等待成本高。在低吞吐量环境中，这样做是不负责任的。而在这里，这通常是正确的选择。 + +## "智能体生成"实际上意味着什么 + +当说代码库是由 Codex 智能体生成的，指的是整个代码库： + +- 产品代码与测试 +- CI 配置和发布工具 +- 内部开发者工具 +- 文档和设计历史 +- 评估框架 +- 审阅评论和回复 +- 管理代码仓库本身的脚本 +- 生产仪表板定义文件 + +> **人类的角色**：人类始终参与其中，但工作的抽象层次与过去不同。我们优先处理工作，将用户反馈转化为验收标准，并对结果进行验证。当智能体遇到困难时，我们将其视为一个信号：识别缺失的内容 — 工具、指导与约束、文档 — 并将其反馈到代码仓库中，始终由 Codex 自己编写修复。 + +## 不断提高的自主水平 + +随着越来越多的开发环节被直接编码到系统中，该代码仓库最近跨过了一个重要门槛，使 Codex 能够端到端地驱动一个新功能。 + +给定一个提示，智能体现在可以： + +1. 验证代码库的当前状态 +2. 重现已报告的漏洞 +3. 录制一个演示故障的视频 +4. 实施修复措施 +5. 通过运行应用程序来验证修复 +6. 录制第二个视频，演示解决方案 +7. 打开 Pull Request +8. 回应智能体和人类反馈 +9. 检测并修复构建故障 +10. 仅在需要判断时才交由人工处理 +11. 合并更改 + +## 熵与垃圾收集 + +**完全自主的智能体也引入了新的问题**。Codex 会复现代码仓库中已存在的模式 — 甚至包括那些不均衡或不够理想的模式。随着时间的推移，这不可避免地导致漂移。 + +### 从人工清理到自动化循环 + +最初，人类是手动处理这个问题的：团队过去每周五（占一周的 20%）都要花时间清理"AI 残渣"。不出所料，那并不具备可扩展性。 + +### 黄金原则与循环清理流程 + +相反，开始将"黄金原则"直接编码到代码仓库中，并建立了一个循环清理流程： + +1. **更倾向于使用共享的实用程序包**，而不是手工编写的辅助工具，以便将不变式集中管理 +2. **不会使用"YOLO 式"探测数据** — 验证边界，或依赖类型化的 SDK +3. **定期运行一组后台 Codex 任务**，扫描偏差、更新质量等级，并发起有针对性的重构 Pull Request + +> **垃圾收集类比**：技术债务就像一笔高息贷款：不断地以小额贷款的方式偿还债务，总比让债务不断累积，再痛苦地一次解决要好得多。 + +## 仍在学习的内容 + +显而易见的是：构建软件仍然需要纪律，但纪律更多地体现在支撑结构上，而不是代码上。保持代码库一致性的工具、抽象和反馈回路变得越发重要。 + +> **当前最棘手的挑战**：集中在设计环境、反馈回路和控制系统方面，帮助智能体实现我们的目标：大规模构建和维护复杂、可靠的软件。 + +## 相关研究 + +- [[Harness-Engineering|Harness 工程]] +- [[Long-Running-Harness-Design|长运行应用的 Harness 设计]] +- [[Externalization-in-LLM-Agents|LLM Agent 中的外部化]] diff --git a/wiki/sources.md b/wiki/sources.md new file mode 100644 index 0000000..a2efce2 --- /dev/null +++ b/wiki/sources.md @@ -0,0 +1,28 @@ +--- +title: "来源索引" +last_updated: 2026-04-11 +--- + +# 来源索引 + +本页面列出 WikiLLM 知识库中所有内容的原始来源。 + +## 学术论文 + +- [Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering](https://arxiv.org/html/2604.08224v1) +- [Meta-Harness: End-to-End Optimization of Model Harnesses](https://arxiv.org/html/2603.28052v1) + +## 概念文章 + +- [Harness engineering for coding agent users](https://martinfowler.com/articles/harness-engineering.html) +- [Harness Engineering - first thoughts](https://martinfowler.com/articles/exploring-gen-ai/harness-engineering-memo.html) +- [Harness Engineering: The Complete Guide to Building Systems That Make AI Agents Actually Work (2026)](https://www.nxcode.io/resources/news/harness-engineering-complete-guide-ai-agent-codex-2026) + +## 实践指南 + +- [Harness design for long-running application development](https://www.anthropic.com/engineering/harness-design-long-running-apps) +- [Scaling Managed Agents: Decoupling the brain from the hands](https://www.anthropic.com/engineering/managed-agents) +- [工程技术：在智能体优先的世界中利用 Codex](https://openai.com/zh-Hans-CN/index/harness-engineering/) +- [My AI Adoption Journey](https://mitchellh.com/writing/my-ai-adoption-journey) +- [Improving Deep Agents with harness engineering](https://blog.langchain.com/improving-deep-agents-with-harness-engineering/) +- [MiniMax M2.7 开启模型的自我进化](https://www.minimaxi.com/news/minimax-m27-zh)