diff --git a/.gitignore b/.gitignore index 9a5aced..c085c06 100644 --- a/.gitignore +++ b/.gitignore @@ -137,3 +137,6 @@ dist # Vite logs files vite.config.js.timestamp-* vite.config.ts.timestamp-* + +.DS_Store +.obsidian diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..753a2cd --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,30 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +WikiLLM is a system for building LLM-powered personal knowledge bases. The workflow consists of: + +1. **Data Ingest**: Source documents (articles, papers, repos, datasets, images) are indexed into a `raw/` directory +2. **Wiki Compilation**: An LLM incrementally "compiles" the raw data into a wiki of markdown files with summaries, backlinks, categorized concepts, and interlinked articles +3. **IDE**: Obsidian is used as the frontend to view raw data, compiled wiki, and visualizations +4. **Q&A**: The LLM can answer complex questions against the wiki by researching the related data +5. **Output**: Results are rendered as markdown files, Marp slides, or matplotlib images, viewable in Obsidian +6. **Linting**: LLM "health checks" find inconsistencies, impute missing data, suggest new article candidates +7. **Extra Tools**: Additional tools like a naive search engine over the wiki + +## Directory Structure + +The project will eventually include these key directories: + +- `raw/` - Source documents and unprocessed data +- `wiki/` - LLM-compiled markdown wiki with articles, summaries, and links +- `tools/` - CLI tools for searching, processing, and enhancing the wiki + +## Core Principles + +- The LLM writes and maintains all wiki data; manual edits are rare +- User explorations and queries are filed back into the wiki to enhance it +- The system focuses on markdown files and Obsidian-compatible formats +- Images are downloaded locally for easy LLM reference diff --git a/raw/anthropic-harness-design.md b/raw/anthropic-harness-design.md new file mode 100644 index 0000000..163421d --- /dev/null +++ b/raw/anthropic-harness-design.md @@ -0,0 +1,294 @@ +# Harness design for long-running application development + +*Written by Prithvi Rajasekaran, a member of our [Labs](https://www.anthropic.com/news/introducing-anthropic-labs) team.* + +Over the past several months I’ve been working on two interconnected problems: getting Claude to produce high-quality frontend designs, and getting it to build complete applications without human intervention. This work originated with earlier efforts on our [frontend design skill](https://github.com/anthropics/claude-code/blob/main/plugins/frontend-design/skills/frontend-design/SKILL.md) and [long-running coding agent harness](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents), where my colleagues and I were able to improve Claude’s performance well above baseline through prompt engineering and harness design—but both eventually hit ceilings. + +To break through, I sought out novel AI engineering approaches that held across two quite different domains, one defined by subjective taste, the other by verifiable correctness and usability. Taking inspiration from [Generative Adversarial Networks](https://en.wikipedia.org/wiki/Generative_adversarial_network) (GANs), I designed a multi-agent structure with a **generator** and **evaluator** agent. Building an evaluator that graded outputs reliably—and with taste—meant first developing a set of criteria that could turn subjective judgments like “is this design good?” into concrete, gradable terms. + +I then applied these techniques to long-running autonomous coding, carrying over two lessons from our earlier harness work: decomposing the build into tractable chunks, and using structured artifacts to hand off context between sessions. The final result was a three-agent architecture—planner, generator, and evaluator—that produced rich full-stack applications over multi-hour autonomous coding sessions. + +## Why naive implementations fall short + +We've previously shown that harness design has a substantial impact on the effectiveness of long running agentic coding. In an earlier [experiment](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents), we used an initializer agent to decompose a product spec into a task list, and a coding agent that implemented the tasks one feature at a time before handing off artifacts to carry context across sessions. The broader developer community has converged on similar insights, with approaches like the "[Ralph Wiggum](https://ghuntley.com/ralph/)" method using hooks or scripts to keep agents in continuous iteration cycles. + +But some problems remained persistent. For more complex tasks, the agent still tends to go off the rails over time. While decomposing this issue, we observed two common failure modes with agents executing these sorts of tasks. + +First is that models tend to lose coherence on lengthy tasks as the context window fills (see our post on [context engineering](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)). Some models also exhibit "context anxiety," in which they begin wrapping up work prematurely as they approach what they believe is their context limit. Context resets—clearing the context window entirely and starting a fresh agent, combined with a structured handoff that carries the previous agent's state and the next steps—addresses both these issues. + +This differs from compaction, where earlier parts of the conversation are summarized in place so the same agent can keep going on a shortened history. While compaction preserves continuity, it doesn't give the agent a clean slate, which means context anxiety can still persist. A reset provides a clean slate, at the cost of the handoff artifact having enough state for the next agent to pick up the work cleanly. In our earlier testing, we found Claude Sonnet 4.5 exhibited context anxiety strongly enough that compaction alone wasn't sufficient to enable strong long task performance, so context resets became essential to the harness design. This solves the core issue, but adds orchestration complexity, token overhead, and latency to each harness run. + +A second issue, which we haven’t previously addressed, is self-evaluation. When asked to evaluate work they've produced, agents tend to respond by confidently praising the work—even when, to a human observer, the quality is obviously mediocre. This problem is particularly pronounced for subjective tasks like design, where there is no binary check equivalent to a verifiable software test. Whether a layout feels polished or generic is a judgment call, and agents reliably skew positive when grading their own work. + +However, even on tasks that do have verifiable outcomes, agents still sometimes exhibit poor judgment that impedes their performance while completing the task. Separating the agent doing the work from the agent judging it proves to be a strong lever to address this issue. The separation doesn't immediately eliminate that leniency on its own; the evaluator is still an LLM that is inclined to be generous towards LLM-generated outputs. But tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work, and once that external feedback exists, the generator has something concrete to iterate against. + +## Frontend design: making subjective quality gradable + +I started by experimenting on frontend design, where the self-evaluation issue was most visible. Absent any intervention, Claude normally gravitates toward safe, predictable layouts that are technically functional but visually unremarkable. + +Two insights shaped the harness I built for frontend design. First, while aesthetics can’t be fully reduced to a score—and individual tastes will always vary—they can be improved with grading criteria that encode design principles and preferences. "Is this design beautiful?" is hard to answer consistently, but "does this follow our principles for good design?" gives Claude something concrete to grade against. Second, by separating frontend generation from frontend grading, we can create a feedback loop that drives the generator toward stronger outputs. + +With this in mind, I wrote four grading criteria that I gave to both the generator and evaluator agents in their prompts: + +- **Design quality:** Does the design feel like a coherent whole rather than a collection of parts? Strong work here means the colors, typography, layout, imagery, and other details combine to create a distinct mood and identity. +- **Originality:** Is there evidence of custom decisions, or is this template layouts, library defaults, and AI-generated patterns? A human designer should recognize deliberate creative choices. Unmodified stock components—or telltale signs of AI generation like purple gradients over white cards—fail here. +- **Craft:** Technical execution: typography hierarchy, spacing consistency, color harmony, contrast ratios. This is a competence check rather than a creativity check. Most reasonable implementations do fine here by default; failing means broken fundamentals. +- **Functionality:** Usability independent of aesthetics. Can users understand what the interface does, find primary actions, and complete tasks without guessing? + +I emphasized design quality and originality over craft and functionality. Claude already scored well on craft and functionality by default, as the required technical competence tended to come naturally to the model. But on design and originality, Claude often produced outputs that were bland at best. The criteria explicitly penalized highly generic “AI slop” patterns, and by weighting design and originality more heavily it pushed the model toward more aesthetic risk-taking. + +I calibrated the evaluator using few-shot examples with detailed score breakdowns. This ensured the evaluator’s judgment aligned with my preferences, and reduced score drift across iterations. + +I built the loop on the [Claude Agent SDK](https://platform.claude.com/docs/en/agent-sdk/overview), which kept the orchestration straightforward. A generator agent first created an HTML/CSS/JS frontend based on a user prompt. I gave the evaluator the Playwright MCP, which let it interact with the live page directly before scoring each criterion and writing a detailed critique. In practice, the evaluator would navigate the page on its own, screenshotting and carefully studying the implementation before producing its assessment. That feedback flowed back to the generator as input for the next iteration. I ran 5 to 15 iterations per generation, with each iteration typically pushing the generator in a more distinctive direction as it responded to the evaluator's critique. Because the evaluator was actively navigating the page rather than scoring a static screenshot, each cycle took real wall-clock time. Full runs stretched up to four hours. I also instructed the generator to make a strategic decision after each evaluation: refine the current direction if scores were trending well, or pivot to an entirely different aesthetic if the approach wasn't working. + +Across runs, the evaluator's assessments improved over iterations before plateauing, with headroom still remaining. Some generations refined incrementally. Others took sharp aesthetic turns between iterations. + +The wording of the criteria steered the generator in ways I didn't fully anticipate. Including phrases like "the best designs are museum quality" pushed designs toward a particular visual convergence, suggesting that the prompting associated with the criteria directly shaped the character of the output. + +While scores generally improved over iterations, the pattern was not always cleanly linear. Later implementations tended to be better as a whole, but I regularly saw cases where I preferred a middle iteration over the last one. Implementation complexity also tended to increase across rounds, with the generator reaching for more ambitious solutions in response to the evaluator’s feedback. Even on the first iteration, outputs were noticeably better than a baseline with no prompting at all, suggesting the criteria and associated language themselves steered the model away from generic defaults before any evaluator feedback led to further refinement. + +In one notable example, I prompted the model to create a website for a Dutch art museum. By the ninth iteration, it had produced a clean, dark-themed landing page for a fictional museum. The page was visually polished but largely in line with my expectations. Then, on the tenth cycle, it scrapped the approach entirely and reimagined the site as a spatial experience: a 3D room with a checkered floor rendered in CSS perspective, artwork hung on the walls in free-form positions, and doorway-based navigation between gallery rooms instead of scroll or click. It was the kind of creative leap that I hadn't seen before from a single-pass generation. + +## Scaling to full-stack coding + +With these findings in hand, I applied this GAN-inspired pattern to full-stack development. The generator-evaluator loop maps naturally onto the software development lifecycle, where code review and QA serve the same structural role as the design evaluator. + +### The architecture + +In our earlier [long-running harness](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents), we had solved for coherent multi-session coding with an initializer agent, a coding agent that worked one feature at a time, and context resets between sessions. Context resets were a key unlock: the harness used Sonnet 4.5, which exhibited the “context anxiety” tendency mentioned earlier. Creating a harness that worked well across context resets was key to keeping the model on task. Opus 4.5 largely removed that behavior on its own, so I was able to drop context resets from this harness entirely. The agents were run as one continuous session across the whole build, with the [Claude Agent SDK](https://platform.claude.com/docs/en/agent-sdk/overview)'s automatic compaction handling context growth along the way. + +For this work I built on the foundation from the original harness with a three-agent system, with each agent addressing a specific gap I'd observed in prior runs. The system contained the following agent personas: + +**Planner:** Our previous long-running harness required the user to provide a detailed spec upfront. I wanted to automate that step, so I created a planner agent that took a simple 1-4 sentence prompt and expanded it into a full product spec. I prompted it to be ambitious about scope and to stay focused on product context and high level technical design rather than detailed technical implementation. This emphasis was due to the concern that if the planner tried to specify granular technical details upfront and got something wrong, the errors in the spec would cascade into the downstream implementation. It seemed smarter to constrain the agents on the deliverables to be produced and let them figure out the path as they worked. I also asked the planner to find opportunities to weave AI features into the product specs. (See example in the Appendix at the bottom.) + +**Generator:** The one-feature-at-a-time approach from the earlier harness worked well for scope management. I applied a similar model here, instructing the generator to work in sprints, picking up one feature at a time from the spec. Each sprint implemented the app with a React, Vite, FastAPI, and SQLite (later PostgreSQL) stack, and the generator was instructed to self-evaluate its work at the end of each sprint before handing off to QA. It also had git for version control. + +**Evaluator:** Applications from earlier harnesses often looked impressive but still had real bugs when you actually tried to use them. To catch these, the evaluator used the Playwright MCP to click through the running application the way a user would, testing UI features, API endpoints, and database states. It then graded each sprint against both the bugs it had found and a set of criteria modeled on the frontend experiment, adapted here to cover product depth, functionality, visual design, and code quality. Each criterion had a hard threshold, and if any one fell below it, the sprint failed and the generator got detailed feedback on what went wrong. + +Before each sprint, the generator and evaluator negotiated a sprint contract: agreeing on what "done" looked like for that chunk of work before any code was written. This existed because the product spec was intentionally high-level, and I wanted a step to bridge the gap between user stories and testable implementation. The generator proposed what it would build and how success would be verified, and the evaluator reviewed that proposal to make sure the generator was building the right thing. The two iterated until they agreed. + +Communication was handled via files: one agent would write a file, another agent would read it and respond either within that file or with a new file that the previous agent would read in turn. The generator then built against the agreed-upon contract before handing the work off to QA. This kept the work faithful to the spec without over-specifying implementation too early. + +### Running the harness + +For the first version of this harness, I used Claude Opus 4.5, running user prompts against both the full harness and a single-agent system for comparison. I used Opus 4.5 since this was our best coding model when I began these experiments. + +I wrote the following prompt to generate a retro video game maker: + +> *Create a 2D retro game maker with features including a level editor, sprite editor, entity behaviors, and a playable test mode.* + +The table below shows the harness type, length it ran for, and the total cost. + +**Harness** + +**Duration** + +**Cost** + +Solo + +20 min + +$9 + +Full harness + +6 hr + +$200 + +The harness was over 20x more expensive, but the difference in output quality was immediately apparent. + +I was expecting an interface where I could construct a level and its component parts (sprites, entities, tile layout) then hit play to actually play the level. I started by opening the solo run’s output, and the initial application seemed in line with those expectations. + +As I clicked through, however, issues started to emerge. The layout wasted space, with fixed-height panels leaving most of the viewport empty. The workflow was rigid. Trying to populate a level prompted me to create sprites and entities first, but nothing in the UI guided me toward that sequence. More to the point, the actual game was broken. My entities appeared on screen but nothing responded to input. Digging into the code revealed that the wiring between entity definitions and the game runtime was broken, with no surface indication of where. + +After evaluating the solo run, I turned my attention to the harness run. This run started from the same one-sentence prompt, but the planner step expanded that prompt into a 16-feature spec spread across ten sprints. It went well beyond what the solo run attempted. In addition to the core editors and play mode, the spec called for a sprite animation system, behavior templates, sound effects and music, an AI-assisted sprite generator and level designer, and game export with shareable links. I gave the planner access to our [frontend design skill](https://github.com/anthropics/claude-code/blob/main/plugins/frontend-design/skills/frontend-design/SKILL.md), which it read and used to create a visual design language for the app as part of the spec. For each sprint, the generator and evaluator negotiated a contract defining the specific implementation details for the sprint, and the testable behaviors that would be tested to verify completion. + +The app immediately showed more polish and smoothness than the solo run. The canvas used the full viewport, the panels were sized sensibly, and the interface had a consistent visual identity that tracked the design direction from the spec. Some of the clunkiness I'd seen in the solo run did remain—the workflow still didn't make it clear that you should build sprites and entities before trying to populate a level, and I had to figure that out by poking around. This read as a gap in the base model’s product intuition rather than something the harness was designed to address, though it did suggest a place where targeted iteration inside the harness could help to further improve output quality. + +Working through the editors, the new run's advantages over solo became more apparent. The sprite editor was richer and more fully featured, with cleaner tool palettes, a better color picker, and more usable zoom controls. + +Because I'd asked the planner to weave AI features into its specs, the app also came with a built-in Claude integration that let me generate different parts of the game through prompting. This significantly sped up the workflow. + +The biggest difference was in play mode. I was actually able to move my entity and play the game. The physics had some rough edges—my character jumped onto a platform but ended up overlapping with it, which felt intuitively wrong—but the core thing worked, which the solo run did not manage. After moving around a bit, I did hit some limitations with the AI’s game level construction. There was a large wall that I wasn’t able to jump past, so I was stuck. This suggested there were some common sense improvements and edge cases that the harness could handle to further refine the app. + +Reading through the logs, it was clear that the evaluator kept the implementation in line with the spec. Each sprint, it walked through the sprint contract's test criteria and exercised the running application through Playwright, filing bugs against anything that diverged from expected behavior. The contracts were granular—Sprint 3 alone had 27 criteria covering the level editor—and the evaluator's findings were specific enough to act on without extra investigation. The table below shows several examples of issues our evaluator identified: + +**Contract criterion** + +**Evaluator finding** + +Rectangle fill tool allows click-drag to fill a rectangular area with selected tile + +**FAIL** — Tool only places tiles at drag start/end points instead of filling the region. `fillRectangle` function exists but isn't triggered properly on mouseUp. + +User can select and delete placed entity spawn points + +**FAIL** — Delete key handler at `LevelEditor.tsx:892` requires both `selection` and `selectedEntityId` to be set, but clicking an entity only sets `selectedEntityId`. Condition should be `selection || (selectedEntityId && activeLayer === 'entity')`. + +User can reorder animation frames via API + +**FAIL** — `PUT /frames/reorder` route defined after `/{frame_id}` routes. FastAPI matches 'r`eorder`' as a frame\_id integer and returns 422: "unable to parse string as an integer." + +Getting the evaluator to perform at this level took work. Out of the box, Claude is a poor QA agent. In early runs, I watched it identify legitimate issues, then talk itself into deciding they weren't a big deal and approve the work anyway. It also tended to test superficially, rather than probing edge cases, so more subtle bugs often slipped through. The tuning loop was to read the evaluator's logs, find examples where its judgment diverged from mine, and update the QAs prompt to solve for those issues. It took several rounds of this development loop before the evaluator was grading in a way that I found reasonable. Even then, the harness output showed the limits of the model’s QAing capabilities: small layout issues, interactions that felt unintuitive in places, and undiscovered bugs in more deeply nested features that the evaluator hadn't exercised thoroughly. There was clearly more verification headroom to capture with further tuning. But compared to the solo run, where the central feature of the application simply didn't work, the lift was obvious. + +### +Iterating on the harness + +The first set of harness results was encouraging, but it was also bulky, slow, and expensive. The logical next step was to find ways to simplify the harness without degrading its performance. This was partly common sense and partly a function of a more general principle: every component in a harness encodes an assumption about what the model can't do on its own, and those assumptions are worth stress testing, both because they may be incorrect, and because they can quickly go stale as models improve. Our blog post [Building Effective Agents](https://www.anthropic.com/research/building-effective-agents) frames the underlying idea as "find the simplest solution possible, and only increase complexity when needed," and it's a pattern that shows up consistently for anyone maintaining an agent harness. + +In my first attempt to simplify, I cut the harness back radically and tried a few creative new ideas, but I wasn't able to replicate the performance of the original. It also became difficult to tell which pieces of the harness design were actually load-bearing, and in what ways. Based on that experience, I moved to a more methodical approach, removing one component at a time and reviewing what impact it had on the final result. + +As I was going through these iteration cycles, we also released Opus 4.6, which provided further motivation to reduce harness complexity. There was good reason to expect 4.6 would need less scaffolding than 4.5 did. From our [launch blog:](https://www.anthropic.com/news/claude-opus-4-6) "\[Opus 4.6\] plans more carefully, sustains agentic tasks for longer, can operate more reliably in larger codebases, and has better code review and debugging skills to catch its own mistakes." It also improved substantially on long-context retrieval. These were all capabilities the harness had been built to supplement. + +### Removing the sprint construct + +I started by removing the sprint construct entirely. The sprint structure had helped to decompose work into chunks for the model to work coherently. Given the improvements in Opus 4.6, there was good reason to believe that the model could natively handle the job without this sort of decomposition. + +I kept both the planner and evaluator, as each continued to add obvious value. Without the planner, the generator under-scoped: given the raw prompt, it would start building without first speccing its work, and end up creating a less feature-rich application than the planner did. + +With the sprint construct removed, I moved the evaluator to a single pass at the end of the run rather than grading per sprint. Since the model was much more capable, it changed how load-bearing the evaluator was for certain runs, with its usefulness depending on where the task sat relative to what the model could do reliably on its own. On 4.5, that boundary was close: our builds were at the edge of what the generator could do well solo, and the evaluator caught meaningful issues across the build. On 4.6, the model's raw capability increased, so the boundary moved outward. Tasks that used to need the evaluator's check to be implemented coherently were now often within what the generator handled well on its own, and for tasks within that boundary, the evaluator became unnecessary overhead. But for the parts of the build that were still at the edge of the generator’s capabilities, the evaluator continued to give real lift. + +The practical implication is that the evaluator is not a fixed yes-or-no decision. It is worth the cost when the task sits beyond what the current model does reliably solo. + +Alongside the structural simplification, I also added prompting to improve how the harness built AI features into each app, specifically getting the generator to build a proper agent that could drive the app's own functionality through tools. That took real iteration, since the relevant knowledge is recent enough that Claude's training data covers it thinly. But with enough tuning, the generator was building agents correctly. + +### Results from the updated harness + +To put the updated harness to the test, I used the following prompt to generate a Digital Audio Workstation (DAW), a music production program for composing, recording, and mixing songs: + +> *Build a fully featured DAW in the browser using the Web Audio API.* + +The run was still lengthy and expensive, at about 4 hours and $124 in token costs. + +Most of the time went to the builder, which ran coherently for over two hours without the sprint decomposition that Opus 4.5 had needed. + +**Agent & Phase** + +**Duration** + +**Cost** + +Planner + +4.7 min + +$0.46 + +Build (Round 1) + +2 hr 7 min + +$71.08 + +QA (Round 1) + +8.8 min + +$3.24 + +Build (Round 2) + +1 hr 2 min + +$36.89 + +QA (Round 2) + +6.8 min + +$3.09 + +Build (Round 3) + +10.9 min + +$5.88 + +QA (Round 3) + +9.6 min + +$4.06 + +**Total V2 Harness** + +**3 hr 50 min** + +**$124.70** + +As with the previous harness, the planner expanded the one-line prompt into a full spec. From the logs, I could see the generator model did a good job planning the app and the agent design, wiring the agent up, and testing it before handing off to QA. + +That being said, the QA agent still caught real gaps. In its first-round feedback, it noted: + +> This is a strong app with excellent design fidelity, solid AI agent, and good backend. The main failure point is Feature Completeness — while the app looks impressive and the AI integration works well, several core DAW features are display-only without interactive depth: clips can't be dragged/moved on the timeline, there are no instrument UI panels (synth knobs, drum pads), and no visual effect editors (EQ curves, compressor meters). These aren't edge cases — they're the core interactions that make a DAW usable, and the spec explicitly calls for them. + +In its second round feedback, it again caught several functionality gaps: + +> Remaining gaps: +> \- Audio recording is still stub-only (button toggles but no mic capture) +> \- Clip resize by edge drag and clip split not implemented +> \- Effect visualizations are numeric sliders, not graphical (no EQ curve) + +The generator was still liable to miss details or stub features when left to its own devices, and the QA still added value in catching those last mile issues for the generator to fix. + +Based on the prompt, I was expecting a program where I could create melodies, harmonies, and drum patterns, arrange them into a song, and get help from an integrated agent along the way. The video below shows the result. + +The app is far from a professional music production program, and the agent's song composition skills could clearly use a lot of work. Additionally, Claude can’t actually hear, which made the QA feedback loop less effective with respect to musical taste. + +But the final app had all the core pieces of a functional music production program: a working arrangement view, mixer, and transport running in the browser. Beyond that, I was able to put together a short song snippet entirely through prompting: the agent set the tempo and key, laid down a melody, built a drum track, adjusted mixer levels, and added reverb. The core primitives for song composition were present, and the agent could drive them autonomously, using tools to create a simple production from end to end. You might say it’s not pitch-perfect yet—but it’s getting there. + +## What comes next + +As models continue to improve, we can roughly expect them to be capable of working for longer, and on more complex tasks. In some cases, that will mean the scaffold surrounding the model matters less over time, and developers can wait for the next model and see certain problems solve themselves. On the other hand, the better the models get, the more space there is to develop harnesses that can achieve complex tasks beyond what the model can do at baseline. + +With this in mind, there are a few lessons from this work worth carrying forward. It is always good practice to experiment with the model you're building against, read its traces on realistic problems, and tune its performance to achieve your desired outcomes. When working on more complex tasks, there is sometimes headroom from decomposing the task and applying specialized agents to each aspect of the problem. And when a new model lands, it is generally good practice to re-examine a harness, stripping away pieces that are no longer load-bearing to performance and adding new pieces to achieve greater capability that may not have been possible before. + +From this work, my conviction is that the space of interesting harness combinations doesn't shrink as models improve. Instead, it moves, and the interesting work for AI engineers is to keep finding the next novel combination. + +## +Acknowledgements + +Special thanks to Mike Krieger, Michael Agaby, Justin Young, Jeremy Hadfield, David Hershey, Julius Tarng, Xiaoyi Zhang, Barry Zhang, Orowa Sidker, Michael Tingley, Ibrahim Madha, Martina Long, and Canyon Robbins for their contributions to this work. + +Thanks also to Jake Eaton, Alyssa Leonard, and Stef Sequeira for their help shaping the post. + +## +Appendix + +Example plan generated by planner agent. + +``` +RetroForge - 2D Retro Game Maker + +Overview +RetroForge is a web-based creative studio for designing and building 2D retro-style video games. It combines the nostalgic charm of classic 8-bit and 16-bit game aesthetics with modern, intuitive editing tools—enabling anyone from hobbyist creators to indie developers to bring their game ideas to life without writing traditional code. + +The platform provides four integrated creative modules: a tile-based Level Editor for designing game worlds, a pixel-art Sprite Editor for crafting visual assets, a visual Entity Behavior system for defining game logic, and an instant Playable Test Mode for real-time gameplay testing. By weaving AI assistance throughout (powered by Claude), RetroForge accelerates the creative process—helping users generate sprites, design levels, and configure behaviors through natural language interaction. + +RetroForge targets creators who love retro gaming aesthetics but want modern conveniences. Whether recreating the platformers, RPGs, or action games of their childhood, or inventing entirely new experiences within retro constraints, users can prototype rapidly, iterate visually, and share their creations with others. + +Features +1. Project Dashboard & Management +The Project Dashboard is the home base for all creative work in RetroForge. Users need a clear, organized way to manage their game projects—creating new ones, returning to works-in-progress, and understanding what each project contains at a glance. + +User Stories: As a user, I want to: + +- Create a new game project with a name and description, so that I can begin designing my game +- See all my existing projects displayed as visual cards showing the project name, last modified date, and a thumbnail preview, so that I can quickly find and continue my work +- Open any project to enter the full game editor workspace, so that I can work on my game +- Delete projects I no longer need, with a confirmation dialog to prevent accidents, so that I can keep my workspace organized +- Duplicate an existing project as a starting point for a new game, so that I can reuse my previous work + +Project Data Model: Each project contains: + +Project metadata (name, description, created/modified timestamps) +Canvas settings (resolution: e.g., 256x224, 320x240, or 160x144) +Tile size configuration (8x8, 16x16, or 32x32 pixels) +Color palette selection +All associated sprites, tilesets, levels, and entity definitions + +... + +``` \ No newline at end of file diff --git a/raw/harness-assets/agent-knowledge-limits.webp b/raw/harness-assets/agent-knowledge-limits.webp new file mode 100644 index 0000000..afa0592 Binary files /dev/null and b/raw/harness-assets/agent-knowledge-limits.webp differ diff --git a/raw/harness-assets/fig1-codex-drives-app.webp b/raw/harness-assets/fig1-codex-drives-app.webp new file mode 100644 index 0000000..52a2249 Binary files /dev/null and b/raw/harness-assets/fig1-codex-drives-app.webp differ diff --git a/raw/harness-assets/layered-domain-architecture.webp b/raw/harness-assets/layered-domain-architecture.webp new file mode 100644 index 0000000..d9bccba Binary files /dev/null and b/raw/harness-assets/layered-domain-architecture.webp differ diff --git a/raw/harness-assets/observability-stack.svg b/raw/harness-assets/observability-stack.svg new file mode 100644 index 0000000..5584893 --- /dev/null +++ b/raw/harness-assets/observability-stack.svg @@ -0,0 +1,129 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/raw/harness-engineering-openai-zh.md b/raw/harness-engineering-openai-zh.md new file mode 100644 index 0000000..039750a --- /dev/null +++ b/raw/harness-engineering-openai-zh.md @@ -0,0 +1,139 @@ +# Harness Engineering:在智能体优先的世界中利用 Codex | OpenAI + +(原文来源:https://openai.com/zh-Hans-CN/index/harness-engineering/) + +在过去五个月里,我们的团队一直在进行一项实验:构建并交付一款软件产品的内部 beta 版,其中没有一行代码是人工编写的。 +该产品有内部日常活跃用户和外部 Alpha 测试者。它经历了交付、部署、故障和修复的整个过程。与众不同的是,每一行代码 — 从应用逻辑、测试、CI 配置、文档、可观察性到内部工具 — 全都是由 Codex 编写的。据估计,我们只用了手工编写代码所需的大约 1/10 的时间就完成了这项工作。 +人类掌舵。智能体执行。 +我们有意选择这一限制,以便构建必要的内容,从而将工程速度提升数个数量级。我们用了几周的时间来交付最终达到一百万行代码的项目。为此,我们需要了解,当软件工程团队的主要工作不再是编写代码,而是设计环境、明确意图和构建反馈回路,从而使 Codex 智能体能够可靠地工作时,会发生哪些变化。 +这个帖子要说的是,在我们与智能体团队一起从零开始打造一款全新产品的过程中,所能学到的经验教训 — 哪些地方出了问题,哪些问题相互叠加,以及如何最大化利用我们唯一真正稀缺的资源:人类的时间和注意力。 +首次提交到一个空的代码仓库是在 2025 年 8 月下旬。 +初始架构 — 包括代码仓库结构、CI 配置、格式化规则、包管理器设置和应用框架 — 是在一小套现有模板的指导下,由 Codex CLI 使用 GPT‑5 生成的。就连指导智能体如何在代码仓库中工作的初始 AGENTS.md 文件本身也是由 Codex 编写的。 +该系统没有预存任何人工编写的代码。从一开始,代码仓库就由智能体塑造。 +五个月后,该代码仓库已经拥有约一百万行代码,从应用逻辑、基础设施、工具、文档到内部开发者工具应有尽有。在那段时间内,大约有 1,500 个 Pull Request 被打开与合并,而推动 Codex 的仅仅是一个由三名工程师组成的小团队。这相当于平均每位工程师每天处理 3.5 个 PRs 的吞吐量,而且令人惊讶的是,随着团队规模扩大到现在的七名工程师,吞吐量甚至还增加了。重要的是,这并非为了输出而输出:该产品已在数百名内测用户那里投入使用,其中包括每天都在使用的内测高级用户。 +在整个开发过程中,人类从未直接直接贡献过任何代码。这成为团队的核心理念:不手动编写代码。 +由于缺乏人工编码的实践,工程师工作的重点转向了系统、架构和杠杆作用。 +早期进展比我们所预期的要慢,而这并不是因为 Codex 不具备相应的能力,而是因为环境的规范不够明确。该智能体缺乏实现高级目标所需的工具、抽象层和内部结构,因而无法取得进展。我们工程团队的主要任务成了协助智能体完成有用的工作。 +在实践中,这意味着采用深度优先的工作方式:将更大的目标拆解为更小的构建模块(设计、代码、评审、测试等),提示智能体去构建这些模块,并使用它们去解锁更复杂的任务。当事情进行不顺利时,解决方案基本上再也不会是“再努力一点”。因为取得进展的唯一方式是让 Codex 来完成工作,而人类工程师则总是介入这项任务并追问:“究竟还需要什么样的能力,我们又该如何让这个能力对智能体来说既清晰可读又可强制执行?” +人类几乎完全通过提示与系统交互:工程师描述任务,运行智能体,并允许其打开一个 Pull Request。为了推动 PR 的完成,我们会指示 Codex 在本地审核其自身的更改,在本地和云端请求额外的特定智能体审查,对任何人工或智能体给出的反馈做出响应,并循环往复,直到所有智能体审核人员都满意为止(这实际上是一个 [Ralph Wiggum 循环](https://ghuntley.com/loop/))。Codex 直接使用我们的标准开发工具(gh、本地脚本和嵌入代码仓库的技能)来收集情境,而无需人工将内容复制粘贴到 CLI 中。 +人类可以审核 Pull Request(合并请求),但并非必须这样做。随着时间的推移,我们已将几乎所有的审核工作调整为用智能体对智能体的方式来处理。 +随着代码吞吐量的增加,我们的瓶颈变成了人工 QA 能力。由于人类的时间和注意力是固定的限制因素,我们一直在努力通过令应用程序的 UI、日志和应用指标等内容对 Codex 直接可读,从而为智能体增加更多功能。 +例如,我们令应用程序可以根据 git worktree 启动,因此 Codex 可以为每次更改启动并驱动一个实例。我们还将 Chrome DevTools 协议接入智能体运行时,并创建了用于处理 DOM 快照、屏幕截图和导航的技能。这使 Codex 能够复现错误、验证修复,并直接推理 UI 的行为。 +我们对可观测性工具也做了同样的处理。日志、指标和追踪记录会通过一个本地可观测性堆栈展示给 Codex,对任何给定的工作树来说,该堆栈都是临时的。Codex 在该应用程序的一个完全独立的版本上运行,一旦任务完成,该版本的所有内容,包括日志和指标,都会被删除。智能体可以使用 LogQL 查询日志,使用 PromQL 查询指标。有了这些情境,像“确保服务启动在 800ms 内完成”或“这四个关键用户旅程中的任何跨度都不得超过两秒”这样的提示就变得可行了。 +我们经常看到单次 Codex 运行在单个任务上持续工作超过六个小时(通常是在人类睡眠时间)。 +情境管理是使智能体在大型和复杂任务中有效发挥作用的最大挑战之一。我们学到的最早经验教训之一很简单:要给 Codex 的是一张地图,而不是一本 1,000 页的说明书。 + +![智能体知识局限边界](./harness-assets/agent-knowledge-limits.webp) + +我们尝试了“一个大型的 [AGENTS.md](https://agents.md/)”方法。可想而知,这是一次失败的尝试: + +- 情境是一种稀缺资源。一个巨大的指令文件会挤掉任务、代码和相关文档 — 因此智能体要么会错过关键约束条件,要么开始针对错误的约束条件进行优化。 +- 过多的指导反而变得无效。当一切都 "重要"时,一切都不重要了。智能体最终会在本地进行模式匹配,而不是有意识地进行导航。 +- 它会立即腐烂。一本庞杂的手册会变成陈旧规则的坟场。智能体无法判断哪些信息仍然有效,一旦人类停止维护它,此文件就会悄然成为一个颇具吸引力的麻烦源头。 +- 这很难核实。单个 blob 不适合进行机械检查(覆盖率、新鲜度、所有权、交叉链接),因此漂移是不可避免的。 + +因此,我们不再将 AGENTS.md 视为百科全书,而是将其视为内容目录。 +代码仓库的知识库位于一个结构化了的 docs/ 目录中,此目录被当作记录系统来使用。一份简短的 AGENTS.md(大约 100 行)被注入到情境中,主要用作地图,并指向其他地方更深层次的真实信息来源。 +代码仓库内知识存储布局。 + +![Codex 驱动应用架构图](./harness-assets/fig1-codex-drives-app.webp) + +``` +AGENTS.md +ARCHITECTURE.md +docs/ +├── design-docs/ +│ ├── index.md +│ ├── core-beliefs.md +│ └── ... +├── exec-plans/ +│ ├── active/ +│ ├── completed/ +│ └── tech-debt-tracker.md +├── generated/ +│ └── db-schema.md +├── product-specs/ +│ ├── index.md +│ ├── new-user-onboarding.md +│ └── ... +├── references/ +│ ├── design-system-reference-llms.txt +│ ├── nixpacks-llms.txt +│ ├── uv-llms.txt +│ └── ... +├── DESIGN.md +├── FRONTEND.md +├── PLANS.md +├── PRODUCT_SENSE.md +├── QUALITY_SCORE.md +├── RELIABILITY.md +└── SECURITY.md +``` + +![Codex 全栈可观测性架构](./harness-assets/observability-stack.svg) + +设计文档已被编目和索引,其中包括验证状态和一套核心理念,定义了智能体优先的操作原则。[架构文档](https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html)提供域和包分层的顶层地图。一份高质量的文档会对每个产品领域和架构层进行评分,并随着时间的推移追踪差距。 +计划被视为一流的工件。临时轻量计划用于小幅变更,而复杂工作则记录在[执行计划](https://cookbook.openai.com/articles/codex_exec_plans)中,并附带进度和决策日志,这些日志会被提交到代码仓库。活跃计划、已完成计划和已知的技术债务都已进行版本控制并集中存放,使智能体能够在不依赖外部情境的情况下运行。 +这实现了渐进式披露:智能体从一个小而稳定的切入点开始,并被指导下一步该去哪里查看,而不是一开始就被淹没。 +我们严格执行这一点。专职的 linter 和 CI 作业会验证知识库的更新状况、是否已交叉链接且结构正确。一个定期运行的“doc-gardening”智能体会扫描那些不再反映真实代码行为的过时或废弃文档,并发起修复用的 Pull Request。 +随着代码库的发展,Codex 的设计决策框架也需要随之演变。 +由于该代码仓库完全由智能体生成,因此我们首先针对 Codex 的可读性进行了优化。就像团队会努力提升代码对新入职工程师的可导航性一样,我们的人类工程师的目标也是让智能体能够直接从代码仓库推理出完整的业务领域。 +从智能体的角度来看,它在运行时无法在情境中访问的任何内容都是不存在的。存储在 Google Docs、聊天记录或人们头脑中的知识都无法被系统访问。代码仓库本地的、已版本化的工件(例如,代码、Markdown、模式、可执行计划)就是它所能看到的全部。 +我们了解到,随着时间的推移,我们需要将越来越多的情境推送到仓库中。那次让团队在架构模式上达成一致的 Slack 讨论?如果智能体无法发现它,那么它就会像迟了三个月入职的新员工一样,对其一无所知。 +为 Codex 提供更多情境意味着要组织和展示正确的信息,好令智能体能够基于这些信息进行推理,而不是用临时指令使其不堪重负。就像你会在产品原则、工程规范和团队文化(包括表情符号偏好)方面为新队友提供引导一样,将这些信息提供给智能体会带来更一致的输出。 +这一框架明确了许多取舍。我们倾向于选择那些可以完全内化于在仓库中进行推理的依赖项和抽象。对智能体来说,通常被称为“枯燥”的技术,由于其可组合性、API 稳定性和在训练集里的表现,往往更容易建立模型。在某些情况下,让智能体重新实现部分功能子集比绕过公共库中不透明的上游行为更便宜。例如,我们没有引入通用的 p-limit 风格包,而是投入使用了我们自己的带并发的 map 辅助函数:它与我们的 OpenTelemetry 仪表紧密集成,具备 100% 的测试覆盖率,并且其行为完全符合我们的运行时预期。 +将系统的更多部分转化为智能体可以检查、验证并直接修改的形式,可以直接提高杠杆效应 — 这不仅适用于 Codex,也适用于其他智能体(例如[Aardvark](/index/introducing-aardvark/)) 也在参与代码库的开发。 +仅靠文档本身,是没法保持完全由智能体生成的代码库的连贯性的。通过强制执行不变量,而非对实施过程进行微观管理,我们令智能体能够快速交付,而且不会削弱基础。例如,我们要求 Codex [在边界处解析数据形状](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/),但不规定具体实现方式(模型似乎偏好 Zod,但我们没有指定特定库)。 +智能体在具有[严格边界和可预测结构](https://bits.logic.inc/p/ai-is-forcing-us-to-write-good-code)的环境中最为高效,因此我们围绕一个严格的架构模型构建了该应用。每个业务域都划分为一组固定的层,依赖方向经过严格验证,并且仅允许有限的一组边。这些约束是通过自定义的 linter(当然是由 Codex 生成的!)和结构测试机械地强制执行的。 + +![分层领域架构与跨切面边界](./harness-assets/layered-domain-architecture.webp) + +下图展示了规则:在每个业务领域内(例如应用设置),代码只能“向前”依赖于一组固定的层(Types → Config → Repo → Service → Runtime → UI)。横切关注点(认证、连接器、遥测、功能标志)通过一个单一的显式接口进入:Providers。其他任何内容都不被允许,并将通过自动化方式强制执行。 +这种架构通常要等到你拥有数百名工程师时才会推迟。对于编码智能体来说,这是一个早期的先决条件:有了约束,速度才不会下降,架构才不会漂移。 +在实践中,我们通过自定义的代码检查器和结构测试来强制执行这些规则,并辅以一小组“品味不变式”。例如,我们通过自定义 lint 静态地强制执行结构化日志记录、模式和类型的命名约定、文件大小限制,以及特定平台的可靠性要求。由于这些 lint 是自定义的,我们编写错误信息时会在智能体情境中注入修复指令。 +在以人为本的工作流程中,这些规则可能会让人感到迂腐或束缚。有了智能体,它们就成了倍增器:一旦编码,它们就能立即应用于所有地方。 +同时,我们还明确指出了哪些地方需要限制,哪些地方不需要限制。这类似于领导一个大型工程平台组织:在中央层面强制执行边界,在本地层面允许自主权。你非常重视界限、正确性和可重复性。在这些边界内,你允许团队或智能体在解决方案的表达方式上拥有很大的自由。 +生成的代码不总是符合人类的风格偏好,这也没关系。只要输出是正确的、可维护的,并且对未来的智能体运行而言清晰易读,就可以算作达标。 +人类的品味会不断反馈到系统中。审查评论、重构的 Pull Request 和面向用户的 Bug 会被记录为文档更新,或直接编码到工具中。当文档不够完善时,我们会将规则转化为代码 +随着 Codex 的吞吐量增加,许多传统的工程规范变得不再有效。 +该代码仓库在运行过程中尽量减少阻塞合并门。Pull Request 的生命周期很短。测试偶发失败通常通过后续重跑来解决,而不是无限期地阻碍进展。在一个智能体吞吐量远超人类注意力的系统中,纠错成本低,而等待成本高。 +在低吞吐量环境中,这样做是不负责任的。而在这里,这通常是正确的选择。 +当我们说代码库是由 Codex 智能体生成的,我们指的是整个代码库。 +智能体的产出包括: + +- 产品代码与测试 +- CI 配置和发布工具 +- 内部开发者工具 +- 文档和设计历史 +- 评估框架 +- 审阅评论和回复 +- 管理代码仓库本身的脚本 +- 生产仪表板定义文件 + +人类始终参与其中,但工作的抽象层次与过去不同。我们优先处理工作,将用户反馈转化为验收标准,并对结果进行验证。当智能体遇到困难时,我们将其视为一个信号:识别缺失的内容 — 工具、指导与约束、文档 — 并将其反馈到代码仓库中,始终由 Codex 自己编写修复。 +智能体可以直接使用我们的标准开发工具。他们会拉取审查反馈、在行内回复、推送更新,并且经常压缩并合并他们自己的 Pull Request(合并请求)。 +随着越来越多的开发环节被直接编码到系统中 — 包括测试、验证、审查、反馈处理和恢复 — 该代码仓库最近跨过了一个重要门槛,使 Codex 能够端到端地驱动一个新功能。 +给定一个提示,智能体现在可以: + +- 验证代码库的当前状态 +- 重现已报告的漏洞 +- 录制一个演示故障的视频 +- 实施修复措施 +- 通过运行应用程序来验证修复 +- 录制第二个视频,演示解决方案 +- 打开 Pull Request +- 回应智能体和人类反馈 +- 检测并修复构建故障 +- 仅在需要判断时才交由人工处理 +- 合并更改 + +此行为在很大程度上取决于此代码仓库的具体结构和工具,不应在没有类似投入的情况下假定它可以泛化 — 至少目前还不行。 +完全自主的智能体也引入了新的问题。Codex 会复现代码仓库中已存在的模式 — 甚至包括那些不均衡或不够理想的模式。随着时间的推移,这不可避免地导致漂移。 +最初,人类是手动处理这个问题的。我们的团队过去每周五(占一周的20%)都要花时间清理“AI 残渣”。不出所料,那并不具备可扩展性。 +相反,我们开始将我们称为“黄金原则”的内容直接编码到代码仓库中,并建立了一个循环清理流程。这些原则是带有主观意见的机械规则,旨在保持代码库的可读性和一致性,以便将来运行智能体。例如:(1) 我们更倾向于使用共享的实用程序包,而不是手工编写的辅助工具,以便将不变式集中管理;(2) 我们不会使用“YOLO 式”探测数据 — 我们会验证边界,或依赖类型化的 SDK,这样智能体就不会意外地基于猜测的结构进行构建。我们会定期运行一组后台 Codex 任务,扫描偏差、更新质量等级,并发起有针对性的重构 Pull Request。其中大多数都可以在一分钟内完成审查并自动合并。 +其功能类似于垃圾回收。技术债务就像一笔高息贷款:不断地以小额贷款的方式偿还债务,总比让债务不断累积,再痛苦地一次解决要好得多。人类的品味一旦被捕捉,就会持续应用于每一行代码。这也使我们能够每天发现并解决不良模式,而不是让它们在代码库中传播数天或数周。 +到目前为止,这一策略在 OpenAI 的内部发布和采纳过程中表现良好。为真实用户打造真实产品,帮助我们将投资锚定在现实中,并引导我们实现长期的可维护性。 +我们尚不清楚的是,在一个完全由智能体生成的系统中,架构连贯性会如何随着时间的推移而演变。我们仍在学习人类的判断力在哪些方面能发挥最大作用,以及如何对这种判断力进行编码,使其发挥更大作用。我们也不知道,随着时间的推移,模型的功能不断增强,这一系统将如何演变。 +显而易见的是:构建软件仍然需要纪律,但纪律更多地体现在支撑结构上,而不是代码上。保持代码库一致性的工具、抽象和反馈回路变得越发重要。 +我们当前最棘手的挑战集中在设计环境、反馈回路和控制系统方面,帮助智能体实现我们的目标:大规模构建和维护复杂、可靠的软件。 +随着像 Codex 这样的智能体在软件生命周期中占据越来越大的比重,这些问题将变得更加重要。我们希望通过分享一些早期的经验教训,帮助你理清投入精力的方向,以便[你可以直接开始构建](/codex/)。 diff --git a/raw/langchain-assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png b/raw/langchain-assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png new file mode 100644 index 0000000..8e28ca3 Binary files /dev/null and b/raw/langchain-assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png differ diff --git a/raw/langchain-assets/Screenshot-2026-02-16-at-12.50.00---PM.png b/raw/langchain-assets/Screenshot-2026-02-16-at-12.50.00---PM.png new file mode 100644 index 0000000..37ef0d7 Binary files /dev/null and b/raw/langchain-assets/Screenshot-2026-02-16-at-12.50.00---PM.png differ diff --git a/raw/langchain-assets/langsmith_trace_analyzer_skill.png b/raw/langchain-assets/langsmith_trace_analyzer_skill.png new file mode 100644 index 0000000..f990b8e Binary files /dev/null and b/raw/langchain-assets/langsmith_trace_analyzer_skill.png differ diff --git a/raw/langchain-assets/self-verification-loop.png b/raw/langchain-assets/self-verification-loop.png new file mode 100644 index 0000000..56a4f6d Binary files /dev/null and b/raw/langchain-assets/self-verification-loop.png differ diff --git a/raw/langchain-assets/the-reasoning-sandwich.png b/raw/langchain-assets/the-reasoning-sandwich.png new file mode 100644 index 0000000..cddc86f Binary files /dev/null and b/raw/langchain-assets/the-reasoning-sandwich.png differ diff --git a/raw/langchain-harness-engineering.md b/raw/langchain-harness-engineering.md new file mode 100644 index 0000000..442eb8c --- /dev/null +++ b/raw/langchain-harness-engineering.md @@ -0,0 +1,126 @@ +# Improving Deep Agents with harness engineering + +TLDR: Our coding agent went from Top 30 to Top 5 on [Terminal Bench 2.0](https://www.tbench.ai/leaderboard/terminal-bench/2.0?ref=blog.langchain.com). We only changed the harness. Here’s our approach to harness engineering (teaser: self-verification & tracing help a lot). + +## The Goal of Harness Engineering + +The goal of a harness is to mold the inherently spiky intelligence of a model for tasks we care about. **Harness Engineering** is about systems, you’re building tooling around the model to optimize goals like task performance, token efficiency, latency, etc. Design decisions include the system prompt, tool choice, and execution flow. + +But how should you change the harness to improve your agent? + +At LangChain, we use [Traces](https://docs.langchain.com/langsmith/observability-quickstart?ref=blog.langchain.com) to understand agent failure modes at scale. Models today are largely black-boxes, their inner mechanisms are hard to interpret. But we can see their inputs and outputs in text space which we then use in our improvement loops. + +We used a simple recipe to iteratively improve [deepagents-cli](https://github.com/langchain-ai/deepagents/tree/main/libs/cli?ref=blog.langchain.com) (our coding agent) `13.7 points` from `52.8` to `66.5` on Terminal Bench 2.0. We only tweaked the harness and kept the model fixed, `gpt-5.2-codex`. + +![](../../langchain-assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png) + +## Experiment Setup & The Knobs on a Harness + +We used [Terminal Bench 2.0](https://www.tbench.ai/?ref=blog.langchain.com), a now standard benchmark to evaluate agentic coding. It has 89 tasks across domains like machine learning, debugging, and biology. We use [Harbor](https://harborframework.com/?ref=blog.langchain.com) to orchestrate the runs. It spins up sandboxes ([Daytona](https://www.daytona.io/?ref=blog.langchain.com)), interacts with our agent loop, and runs verification + scoring. + +Every agent action is stored in [LangSmith](https://smith.langchain.com/?ref=blog.langchain.com). It also includes metrics like latency, token counts, and costs. + +### **The Knobs we can Turn** + +An agent harness has a lot of knobs: system prompts, tools, hooks/middleware, skills, sub-agent delegation, memory systems, and more. We deliberately compress the optimization space and focus on three: **System Prompt, Tools,** and [**Middleware**](https://docs.langchain.com/oss/python/langchain/middleware/overview?ref=blog.langchain.com#the-agent-loop) (our term for hooks around model and tool calls). + +We start with a default prompt and standard tools+middleware. This scores 52.8% with GPT-5.2-Codex. A solid score, just outside the Top 30 of the leaderboard today, but room to grow. + +![](../../langchain-assets/Screenshot-2026-02-16-at-12.50.00---PM.png) + +### **The Trace Analyzer Skill** + +We wanted trace analysis to be repeatable so we made it into an Agent Skill. This serves as our recipe to **analyze errors across runs and make improvements to the harness**. The flow is: + +1. Fetch experiment traces from LangSmith +2. Spawn parallel error analysis agents → main agent synthesizes findings + suggestions +3. Aggregate feedback and make targeted changes to the harness. + +This works similarly to [boosting](https://en.wikipedia.org/wiki/Boosting_\(machine_learning\)?ref=blog.langchain.com) which focuses on mistakes from previous runs. A human can be pretty helpful in Step 3 (though not required) to verify and discuss proposed changes. Changes that overfit to a task are bad for generalization and can lead to regressions in other Tasks. + +Automated trace analysis saves hours of time and made it easy to quickly try experiments. We’ll be publishing this skill soon, we’re currently testing it for prompt optimization generally. + +![](../../langchain-assets/langsmith_trace_analyzer_skill.png) + +## What Actually Improved Agent Performance + +Automated Trace analysis allowed us to [debug where agents were going wrong](https://www.langchain.com/conceptual-guides/agent-observability-powers-agent-evaluation?ref=blog.langchain.com). Issues included reasoning errors, not following task instructions, missing testing and verification, running out of time, etc. We go into these improvements in more details in the sections below. + +### Build & Self-Verify + +Today’s models are exceptional self-improvement machines. + +**Self-verification allows agents to self-improve via feedback within a run**. However, they don’t have a natural tendency to enter this **build-verify loop.** + +The most common failure pattern was that the agent wrote a solution, re-read its own code, confirmed it looks ok, and stopped. Testing is a key part of autonomous agentic coding. It helps test for overall correctness and simultaneously gives agents signal to hill-climb against. + +We added guidance to the system prompt on how to approach problem solving. + +1. **Planning & Discovery:** Read the task, scan the codebase, and build an initial plan based on the task specification and how to verify the solution. +2. **Build:** Implement the plan with verification in mind. Build tests, if they don’t exist and test both happy paths and edge cases. +3. **Verify:** Run tests, read the full output, compare against what was asked (not against your own code). +4. **Fix:** Analyze any errors, revisit the original spec, and fix issues. + +We really focus on testing because it powers the changes in every iteration. We found that alongside prompting, deterministic context injection helps agents verify their work. We use a `PreCompletionChecklistMiddleware` that intercepts the agent before it exits and reminds it to run a verification pass against the Task spec. This is similar to a [Ralph Wiggum Loop](https://ghuntley.com/loop/?ref=blog.langchain.com) where a hook forces the agent to continue executing on exit, we use this for verification. + +![](../../langchain-assets/self-verification-loop.png) + +### Giving Agents Context about their Environment + +Part of harness engineering is **building a good delivery mechanism for context engineering.** Terminal Bench tasks come with directory structures, built-in tooling, and strict timeouts. + +1. **Directory Context & Tooling:** A `LocalContextMiddleware` runs on agent start to map the `cwd` and other parent+children directories. We run `bash` commands to find tools like `Python` installations. Context discovery and search are error prone, so injecting context reduces this error surface and helps **onboard the agent into its environment.** +2. **Teaching Agents to Write Testable Code:** Agents don’t know how their code needs to be testable. We add prompting say their work will be measured against programatic tests, similar to when committing code. For example, Task specs that mention file paths should be followed exactly so the solutions works in an automated scoring step. Prompting that stresses edge-cases helps the agent avoid only checking “happy path” cases. Forcing models to conform to testing standards is a powerful strategy to avoid “slop buildup” over time. +3. **Time Budgeting:** We inject time budget warnings to nudge the agent to finish work and shift to verification. Agents are famously bad at time estimation so this heuristic helps in this environment. Real world coding usually doesn’t have strict time limits, but without adding any knowledge of constraints, agents won’t work within time bounds. + +The more that agents know about their environment, constraints, and evaluation criteria, the better they can autonomously self-direct their work. + +**The purpose of the harness engineer: prepare and deliver context so agents can autonomously complete work.** + +### Encouraging Agents to Step Back & Reconsider Plans + +Agents can be myopic once they’ve decided on a plan which results in “doom loops” that make small variations to the same broken approach (10+ times in some traces). + +We use a `LoopDetectionMiddleware` that tracks per-file edit counts via tool call hooks. It adds context like “…consider reconsidering your approach” after `N` edits to the same file. This can help agents recover from doom loops, though the model can continue down the same path if it thinks it’s correct. + +Important note. This is a design heuristic that engineers around today’s perceived model issues. As models improve, these guardrails will likely be unnecessary, but today helps agents execute correctly and autonomously. + +### Choosing How Much Compute to Spend on Reasoning + +Reasoning models can run autonomously for hours so we have to decide how much compute to spend on every subtask. You can use the max reasoning budget on every task, but most work can benefit from optimizing reasoning compute spend. + +Terminal Bench timeout limits create a tradeoff. More reasoning helps agents evaluate each step, but can burn over `2x` more tokens/time. `gpt-5.2-codex` has 4 reasoning modes, `low`, `medium`, `high`, and `xhigh`. + +We found that reasoning helps with planning to fully understand the problem, some Terminal Bench tasks are very difficult. A good plan helps get to a working solution more quickly. + +Later stage verification also benefits from more reasoning to catch mistakes and get a solution submitted. As a heuristic, we choose a xhigh-high-xhigh "**reasoning sandwich**" as a baseline. + +![](../../langchain-assets/the-reasoning-sandwich.png) + +**Spending more reasoning compute on planning and verification** + +Running only at `xhigh` scored poorly at `53.9%` due to agent timeouts compared to `63.6%` at `high`. There weren’t large differences in trial runs across reasoning budget splits so we stuck with our approach which pushed the score to `66.5%`. + +The natural approach for models is **Adaptive Reasoning,** seen with [Claude](https://platform.claude.com/docs/en/build-with-claude/adaptive-thinking?ref=blog.langchain.com) and [Gemini](https://ai.google.dev/gemini-api/docs/thinking?ref=blog.langchain.com) models where the model decides how much compute to spend on reasoning. + +In a multi-model harness, balancing reasoning budgets could play out as using a large model for planning and [handing off](https://docs.langchain.com/oss/python/langchain/multi-agent/handoffs?ref=blog.langchain.com) to a smaller model for implementation. + +## Practical Takeaways for Building Agent Harnesses + +The design space of agents is big. Here are some general principles from our experiments and building deepagents overall. + +1. **Context Engineering on Behalf of Agents.** Context assembly is still difficult for agents today, especially in unseen environments. Onboarding models with context like directory structures, available tools, coding best practices, and problem solving strategies helps reduce the error surface for poor search and avoidable errors in planning. +2. **Help agents self-verify their work.** Models are biased towards their first plausible solution. Prompt them aggressively to verify their work by running tests and refining solutions. This is especially important in autonomous coding systems that don’t have humans in the loop. +3. **Tracing as a feedback signal.** Traces allow agents to self-evaluate and debug themselves. It’s important to debug tooling and reasoning together (ex: models go down wrong paths because they lack a tool or instructions how to do something). +4. **Detect and fix bad patterns in the short term.** Models today aren’t perfect. The job of the harness designer is to design around today’s shortcomings while planning for smarter models in the future. Blind retries and not verifying work are good examples. These guardrails will almost surely dissolve over time, but to build robust agent applications today, they’re useful tools to experiment with. +5. **Tailor Harnesses to Models. T**he [Codex](https://developers.openai.com/cookbook/examples/gpt-5/codex_prompting_guide/?ref=blog.langchain.com) and [Claude](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices?ref=blog.langchain.com) prompting guides show that models require different prompting. A test run with Claude Opus 4.6 scored `59.6%` with an earlier harness version, competitive but worse than Codex because we didn’t run the same Improvement Loop with Claude. Many principles generalize like good context preparation and a focus on verification, but running a few rounds of harness iterations for your task helps maximize agent performance across tasks. + +There’s more open research to do in harness design. Interesting avenues include multi-model systems (Codex, Gemini, and Claude together), memory primitives for continual learning so agents can autonomously improve on tasks, and measuring harness changes across models. + +For the outer loop of improving agents, we’re looking at methods like [RLMs](https://alexzhang13.github.io/blog/2025/rlm/?ref=blog.langchain.com) to more efficiently mine traces. We’ll be continuing work to improve the harness and openly share our research. + +We created [a dataset of our Traces](https://smith.langchain.com/public/29393299-8f31-48bb-a949-5a1f5968a744/d?tab=2&ref=blog.langchain.com) to share with the community. + +Deep Agents is open source. [Python](https://github.com/langchain-ai/deepagents?ref=blog.langchain.com) and [Javascript](https://github.com/langchain-ai/deepagentsjs?ref=blog.langchain.com). + +**To more hill climbing and open research.** \ No newline at end of file diff --git a/raw/martinfowler-assets/harness-bounded-contexts.png b/raw/martinfowler-assets/harness-bounded-contexts.png new file mode 100644 index 0000000..56a9a95 --- /dev/null +++ b/raw/martinfowler-assets/harness-bounded-contexts.png @@ -0,0 +1,280 @@ + + + + + + +not found + + + + + + + +

404

+ +

I’m afraid this is not the document you’re looking for. Try using the +search box above, and good luck.

+
+ + + + + + + + + diff --git a/raw/martinfowler-assets/harness-change-lifecycle-examples.png b/raw/martinfowler-assets/harness-change-lifecycle-examples.png new file mode 100644 index 0000000..56a9a95 --- /dev/null +++ b/raw/martinfowler-assets/harness-change-lifecycle-examples.png @@ -0,0 +1,280 @@ + + + + + + +not found + + + + + + + +

404

+ +

I’m afraid this is not the document you’re looking for. Try using the +search box above, and good luck.

+
+ + + + + + + + + diff --git a/raw/martinfowler-assets/harness-continuous-feedback-examples.png b/raw/martinfowler-assets/harness-continuous-feedback-examples.png new file mode 100644 index 0000000..56a9a95 --- /dev/null +++ b/raw/martinfowler-assets/harness-continuous-feedback-examples.png @@ -0,0 +1,280 @@ + + + + + + +not found + + + + + + + +

404

+ +

I’m afraid this is not the document you’re looking for. Try using the +search box above, and good luck.

+
+ + + + + + + + + diff --git a/raw/martinfowler-assets/harness-overview.png b/raw/martinfowler-assets/harness-overview.png new file mode 100644 index 0000000..56a9a95 --- /dev/null +++ b/raw/martinfowler-assets/harness-overview.png @@ -0,0 +1,280 @@ + + + + + + +not found + + + + + + + +

404

+ +

I’m afraid this is not the document you’re looking for. Try using the +search box above, and good luck.

+
+ + + + + + + + + diff --git a/raw/martinfowler-assets/harness-templates.png b/raw/martinfowler-assets/harness-templates.png new file mode 100644 index 0000000..56a9a95 --- /dev/null +++ b/raw/martinfowler-assets/harness-templates.png @@ -0,0 +1,280 @@ + + + + + + +not found + + + + + + + +

404

+ +

I’m afraid this is not the document you’re looking for. Try using the +search box above, and good luck.

+
+ + + + + + + + + diff --git a/raw/martinfowler-assets/harness-types.png b/raw/martinfowler-assets/harness-types.png new file mode 100644 index 0000000..56a9a95 --- /dev/null +++ b/raw/martinfowler-assets/harness-types.png @@ -0,0 +1,280 @@ + + + + + + +not found + + + + + + + +

404

+ +

I’m afraid this is not the document you’re looking for. Try using the +search box above, and good luck.

+
+ + + + + + + + + diff --git a/raw/martinfowler-harness-engineering.md b/raw/martinfowler-harness-engineering.md new file mode 100644 index 0000000..ca4c9fc --- /dev/null +++ b/raw/martinfowler-harness-engineering.md @@ -0,0 +1,179 @@ +# Harness engineering for coding agent users + +The term harness has emerged as a shorthand to mean everything in an AI agent except the model itself - [Agent = Model + Harness](https://blog.langchain.com/the-anatomy-of-an-agent-harness/). That is a very wide definition, and therefore worth narrowing down for common categories of agents. I want to take the liberty here of defining its meaning in the bounded context of using a coding agent. In coding agents, part of the harness is already built in (e.g. via the system prompt, or the chosen code retrieval mechanism, or even a [sophisticated orchestration system](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents)). But coding agents also provide us, their users, with many features to build an outer harness specifically for our use case and system. + +![Three concentric circles, with the model in the core (the ultimate thing being harnessed), then the coding agent's builder harness next circle out, and the coding agent's user harness as the outermost ring](../../martinfowler-assets/harness-bounded-contexts.png) + +Figure 1: The term “harness” means different things depending on the bounded context. + +A well-built outer harness serves two goals: it increases the probability that the agent gets it right in the first place, and it provides a feedback loop that self-corrects as many issues as possible before they even reach human eyes. Ultimately it should reduce the review toil and increase the system quality, all with the added benefit of fewer wasted tokens along the way. + +![Title "Harness engineering for coding agent users". Overview of guides (examples shown are \[inferential\] principles, CfRs, Rules, Ref Docs, How-tos; \[computational\] Language Servers, CLIs, scripts, codemods) that feedforward into a coding agent; and feedback sensors (examples shown are \[inferential\] review agents; \[computational\] static analysis, logs, browser). The feedback sensors point at the coding agent as well as input into its self-correcting loop. On the left side of it all we see a box with a human who steers both the guides and sensors.](../../martinfowler-assets/harness-overview.png) + +## Feedforward and Feedback + +To harness a coding agent we both anticipate unwanted outputs and try to prevent them, and we put sensors in place to allow the agent to self-correct: + +- **Guides (feedforward controls)** - anticipate the agent's behaviour and aim to steer it *before* it acts. Guides increase the probability that the agent creates good results in the first attempt +- **Sensors (feedback controls)** - observe *after* the agent acts and help it self-correct. Particularly powerful when they produce signals that are optimised for LLM consumption, e.g. custom linter messages that include instructions for the self-correction - a positive kind of prompt injection. + +Separately, you get either an agent that keeps repeating the same mistakes (feedback-only) or an agent that encodes rules but never finds out whether they worked (feed-forward-only). + +## Computational vs Inferential + +There are two execution types of guides and sensors: + +- **Computational** - deterministic and fast, run by the CPU. Tests, linters, type checkers, structural analysis. Run in milliseconds to seconds; results are reliable. +- **Inferential** - Semantic analysis, AI code review, “LLM as judge”. Typically run by a GPU or NPU. Slower and more expensive; results are more non-deterministic. + +Computational guides increase the probability of good results with deterministic tooling. Computational sensors are cheap and fast enough to run on every change, alongside the agent. Inferential controls are of course more expensive and non-deterministic, but allow us to both provide rich guidance, and add additional semantic judgment. In spite of their non-determinism, inferential sensors can particularly increase our trust when used with a strong model, or rather a model that is suitable to the task at hand. + +**Examples** + +Direction + +Computational / Inferential + +Example implementations + +Coding conventions + +feedforward + +Inferential + +AGENTS.md, Skills + +Instructions how to bootstrap a new project + +feedforward + +Both + +Skill with instructions and a bootstrap script + +Code mods + +feedforward + +Computational + +A tool with access to OpenRewrite recipes + +Structural tests + +feedback + +Computational + +A pre-commit (or coding agent) hook running ArchUnit tests that check for violations of module boundaries + +Instructions how to review + +feedback + +Inferential + +Skills + +## The steering loop + +The human's job in this is to **steer** the agent by iterating on the harness. Whenever an issue happens multiple times, the feedforward and feedback controls should be improved to make the issue less probable to occur in the future, or even prevent it. + +In the steering loop, we can of course also use AI to improve the harness. Coding agents now make it much cheaper to build more custom controls and more custom static analysis. Agents can help write structural tests, generate draft rules from observed patterns, scaffold custom linters, or create how-to guides from codebase archaeology. + +## Timing: Keep quality left + +Teams who are [continuously integrating](https://martinfowler.com/articles/continuousIntegration.html) have always faced the challenge of spreading tests, checks and human reviews across the development timeline according to their cost, speed and criticality. When you aspire to [continuously deliver](https://martinfowler.com/bliki/ContinuousDelivery.html), you ideally even want every commit state to be deployable. You want to have checks as far left in the path to production as possible, since the earlier you find issues, the cheaper they are to fix. Feedback sensors, including the new inferential ones, need to be distributed across the lifecycle accordingly. + +**Feedforward and feedback in the change lifecycle** + +- What is reasonably fast and should be run even before integration, or even before a commit is even created? (e.g. linters, fast test suites, basic code review agent) +- What is more expensive and should therefore only be run post-integration in the pipeline, in addition to a repetition of the fast controls? (e.g. mutation testing, a more broad code review that can take into account the bigger picture) + +![Examples of feedforward and feedback in a change's lifecycle. Feedforward: LSP, architecture.md, /how-to-test skill, AGENTS.md, MCP server that can access a team's knowledge management tool, /xyz-api-docs skill; they feed into the agent's initial generation; feedback sensor examples for first self-correction loop are /code-review, npx eslint, semgrep, npm run coverage, npm run dep-cruiser; then human review is an additional feedback sensor; then integration happens; after integration, examples shown in the pipeline, which reruns all the previous sensors, and additional examples for more expensive sensors are /architecture-review skill, /detailed-review skill, mutation testing. An arrow shows that the feedback can then lead to new commits by agents or humans.](../../martinfowler-assets/harness-change-lifecycle-examples.png) + +**Continuous drift and health sensors** + +- What type of drift accumulates gradually and should be monitored by sensors running continuously against the codebase, outside the change lifecycle? (e.g. dead code detection, analysis of the quality of the test coverage, dependency scanners) +- What runtime feedback could agents be monitoring? (e.g. having them look for degrading SLOs to make suggestions how to improve them, or AI judges continuously sampling response quality and flagging log anomalies) + +![Shows examples of continuous feedback sensors after change integration. Continuous drift detection in the codebase, e.g. /find-dead-code, /code-coverage-quality, dependabot; or Continuous runtime feedback, e.g. latency, error rate or availability SLOs leading to coding agent suggestions, or /response-quality-sampling, /log-anomalies AI judges.](../../martinfowler-assets/harness-continuous-feedback-examples.png) + +## Regulation categories + +The agent harness acts like a [cybernetic](https://en.wikipedia.org/wiki/Cybernetics) governor, combining feed-forward and feedback to regulate the codebase towards its desired state. It's useful to distinguish between multiple dimensions of that desired state, categorised by what the harness is supposed to regulate. Distinguishing between these categories helps because harnessability and complexity vary across them, and qualifying the word gives us more precise language for a term that is otherwise very generic. + +The following are three categories that seem useful to me as of now: + +### Maintainability harness + +More or less all of the examples I am giving in this article are about regulating internal code quality and maintainability. This is at the moment the easiest type of harness, as we have a lot of pre-existing tooling that we can use for this. + +To reflect on how much these aforementioned maintainability harness ideas increase my trust in agents, I mapped [common coding agent failure modes that I catalogued before](https://martinfowler.com/articles/exploring-gen-ai/13-role-of-developer-skills.html) against it. + +Computational sensors catch the structural stuff reliably: duplicate code, cyclomatic complexity, missing test coverage, architectural drift, style violations. These are cheap, proven, and deterministic. + +LLMs can partially address problems that require semantic judgment - semantically duplicate code, redundant tests, brute-force fixes, over-engineered solutions - but expensively and probabilistically. Not on every commit. + +Neither catches reliably some of the higher-impact problems: Misdiagnosis of issues, overengineering and unnecessary features, misunderstood instructions. They'll sometimes catch them, but not reliably enough to reduce supervision. Correctness is outside any sensor's remit if the human didn't clearly specify what they wanted in the first place. + +### Architecture fitness harness + +This groups guides and sensors that define and check the architecture characteristics of the application. Basically: [Fitness Functions](https://www.thoughtworks.com/en-de/radar/techniques/architectural-fitness-function). + +Examples: + +- Skills that feed forward our performance requirements, and performance tests that feed back to the agent if it improved or degraded them. +- Skills that describe coding conventions for better observability (like logging standards), and debugging instructions that ask the agent to reflect on the quality of the logs it had available. + +### Behaviour harness + +This is the elephant in the room - how do we guide and sense if the application functionally behaves the way we need it to? At the moment, I see most people who give high autonomy to their coding agents do this: + +- Feed-forward: A functional specification (of varying levels of detail, from a short prompt to multi-file descriptions) +- Feed-back: Check if the AI-generated test suite is green, has reasonably high coverage, some might even monitor its quality with mutation testing. Then combine that with manual testing. + +This approach puts a lot of faith into the AI-generated tests, that's not good enough yet. Some of my colleagues are seeing good results with the [approved fixtures](https://lexler.github.io/augmented-coding-patterns/patterns/approved-fixtures/) pattern, but it's easier to apply in some areas than others. They use it selectively where it fits, it's not a wholesale answer to the test quality problem. + +So overall, we still have a lot to do to figure out good harnesses for functional behaviour that increase our confidence enough to reduce supervision and manual testing. + +![Simplified overview of a harness showing guides and sensors in horizontal, and then the regulation dimensions maintainability, architecture fitness, and behaviour, in vertical. Examples shown for the behaviour harness, spec as feedforward guide, test suite as feedback sensor that is a mix of inferential and computational, plus a human icon indicating human review and manual tests as main additional feedback sensor.](../../martinfowler-assets/harness-types.png) + +## Harnessability + +Not every codebase is equally amenable to harnessing. A codebase written in a strongly typed language naturally has type-checking as a sensor; clearly definable module boundaries afford architectural constraint rules; frameworks like Spring abstract away details the agent doesn't even have to worry about and therefore implicitly increase the agent's chances of success. Without those properties, those controls aren't available to build. + +This plays out differently for greenfield versus legacy. Greenfield teams can bake harnessability in from day one - technology decisions and architecture choices determine how governable the codebase will be. Legacy teams, especially with applications that have accrued a lot of technical debt, face the harder problem: the harness is most needed where it is hardest to build. + +## Harness templates + +Most enterprises have a few common topologies of services that cover 80% of what they need - business services that exposes data via APIs; event processing services; data dashboards. In many mature engineering organizations these topologies are already codified in service templates. These might evolve into harness templates in the future: a bundle of guides and sensors that leash a coding agent to the structure, conventions and tech stack of a topology. Teams may start picking tech stacks and structures partly based on what harnesses are already available for them. + +![A stack of examples of topologies (Data dashboard in Node, CRUD business service on JVM, event processor in Golang). The top one, data dashboard, is shown in detail, as a combination of structure definition and tech stack. The graphic indicates a "harness template" with guides and sensors for each topology, which can be instantiated.](../../martinfowler-assets/harness-templates.png) + +We would of course face similar challenges as with service templates. As soon as teams instantiate them, they start fall out of sync with upstream improvements. Harness templates would face the same versioning and contribution problems, maybe even worse with non-deterministic guides and sensors that are harder to test. + +## The role of the human + +As human developers we bring our skills and experience as an implicit harness to every codebase. We absorbed conventions and good practices, we have felt the cognitive pain of complexity, and we know that our name is on the commit. We also carry organisational alignment - awareness of what the team is trying to achieve, which technical debt is tolerated for business reasons, and what “good” looks like in this specific context. We go in small steps and at our human pace, which creates the thinking space for that experience to get triggered and applied. + +A coding agent has none of this: no social accountability, no aesthetic disgust at a 300-line function, no intuition that “we don't do it that way here,” and no organisational memory. It doesn't know which convention is load-bearing and which is just habit, or whether the technically correct solution fits what the team is trying to do. + +Harnesses are an attempt to externalise and make explicit what human developer experience brings to the table, but it can only go so far. Building a coherent system of guides and sensors and self-correction loops is expensive, so we have to prioritise with a clear goal in mind: A good harness should not necessarily aim to fully eliminate human input, but to direct it to where our input is most important. + +## A starting point - and open questions + +The mental model I've laid out here describes techniques that are already happening in practice and helps frame discussions about what we still need to figure out. Its goal is to raise the conversation above the feature level - from skills and MCP servers to how we strategically design a system of controls that gives us genuine confidence in what agents produce. + +Here are some harness-related examples from the current discourse: + +- [An OpenAI team documented what their harness looks like](https://openai.com/index/harness-engineering/): layered architecture enforced by custom linters and structural tests, and recurring “garbage collection” that scans for drift and has agents suggest fixes. Their conclusion: “Our most difficult challenges now center on designing environments, feedback loops, and control systems.” +- [Stripe's write-up about their minions](https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents) describes things like pre-push hooks that run relevant linters based on a heuristic, they highlight how important “shift feedback left” is to them, and their “blueprints” show how they're integrating feedback sensors into the agent workflows. +- Mutation and structural testing are examples of computational feedback sensors that have been underused in the past, but are now having a resurgence. +- There is increased chatter among developers about the integration of LSPs and code intelligence in coding agents, examples of computational feedforward guides. +- I hear stories from teams at Thoughtworks about tackling architecture drift with both computational and inferential sensors, e.g. increasing API quality with a mix of agents and custom linters, or increasing code quality with a “janitor army”. + +There's plenty still to figure out, not just the already mentioned behavioural harness. How do we keep a harness coherent as it grows, with guides and sensors in sync, not contradicting each other? How far can we trust agents to make sensible trade-offs when instructions and feedback signals point in different directions? If sensors never fire, is that a sign of high quality or inadequate detection mechanisms? We need a way to evaluate harness coverage and quality similar to what code coverage and mutation testing do for tests. Feedforward and feedback controls are currently scattered across delivery steps, there's real potential for tooling that helps configure, sync, and reason about them as a system. Building this outer harness is emerging as an ongoing engineering practice, not a one-time configuration. + +* * * \ No newline at end of file diff --git a/raw/minimax-assets/d92a6eb4-a4b8-4906-b76a-d627c814a2c0.gif b/raw/minimax-assets/d92a6eb4-a4b8-4906-b76a-d627c814a2c0.gif new file mode 100644 index 0000000..e8f5576 Binary files /dev/null and b/raw/minimax-assets/d92a6eb4-a4b8-4906-b76a-d627c814a2c0.gif differ diff --git a/raw/minimax-assets/img-1.png b/raw/minimax-assets/img-1.png new file mode 100644 index 0000000..77bbb22 Binary files /dev/null and b/raw/minimax-assets/img-1.png differ diff --git a/raw/minimax-assets/img-2.png b/raw/minimax-assets/img-2.png new file mode 100644 index 0000000..4550042 Binary files /dev/null and b/raw/minimax-assets/img-2.png differ diff --git a/raw/minimax-assets/img-3.png b/raw/minimax-assets/img-3.png new file mode 100644 index 0000000..80ff7f3 Binary files /dev/null and b/raw/minimax-assets/img-3.png differ diff --git a/raw/minimax-m27.md b/raw/minimax-m27.md new file mode 100644 index 0000000..d5ef2aa --- /dev/null +++ b/raw/minimax-m27.md @@ -0,0 +1,99 @@ +# MiniMax M2.7: Early Echoes of Self-Evolution + +In the months following the first release of our M2-series models, we received a large volume of feedback and suggestions from enthusiastic users and developers, which drove us to further accelerate the efficiency of our model iterations. With human productivity already fully unleashed, the natural next step was to initiate self-evolution of both the model and the organization. M2.7 is our first model deeply participating in its own evolution. + +M2.7 is capable of building complex agent harnesses and completing highly elaborate productivity tasks, leveraging capabilities such as Agent Teams, complex Skills, and dynamic tool search. For example, when developing M2.7, we let the model update its own memory and build dozens of complex skills in its harness to help with reinforcement learning experiments. We further let the model improve its learning process and harness based on the experiment results. This process initiates a cycle of model self-evolution. + +1\. M2.7 delivers outstanding performance in real-world software engineering, including end-to-end full project delivery, log analysis, bug troubleshooting, code security, machine learning, and more. On the SWE-Pro benchmark, M2.7 scored 56.22%, nearly approaching Opus's best level. This capability also extends to end-to-end full project delivery scenarios (VIBE-Pro 55.6%) and deep understanding of complex engineering systems on Terminal Bench 2 (57.0%). + +2\. We have also enhanced the model's expertise and task delivery capabilities across various fields in the professional office software domain. Its ELO score on GDPval-AA is 1495, the highest among open-source models. M2.7 shows significantly improved ability for complex editing in the Office suite — Excel, PPT, and Word — and can better handle multi-round revisions and high-fidelity editing. M2.7 is capable of interacting with complex environments: It maintains a 97% skill adherence rate while working with over 40 complex skills, each exceeding 2,000 tokens. + +3\. M2.7 exhibits excellent character consistency and emotional intelligence, opening up more room for product innovation. + +Based on these capabilities, M2.7 is also significantly accelerating our own evolution into an AI-native organization. + +![MiniMax M2.7 benchmark overview](minimax-assets/img-1.png) + +## Building an agent for model self-evolution + +We first share an internal workflow that enables the M2-series models to self-evolve. This workflow also serves as an exploration of the boundaries of the model's agentic capabilities. + +Modern agent harness utilizes a combination of complex skills, memory, and other external modules to help improve its adaptability to various workspace environments. In MiniMax, our agents are routinely faced with very complex and disparate working environments spanning multiple departments. As such, to improve the robustness of our agents in these heterogeneous environments, we tasked an internal version of M2.7 to build a research agent harness that interacts and collaborates with different research project groups. The harness supports data pipelines, training environments, infrastructure, cross-team collaboration, and persistent memory — enabling researchers to drive it to deliver better models. The research agent harness drives the iteration cycle that produces the next generation of models under the guidance set by researchers. + +An exemplary workflow lies in the daily routine of our RL team. A researcher starts by discussing an experimental idea with the agent, who helps with literature review, tracks a pre-set experiment spec, pipelines data and other artifacts, and launches experiments. During the experiments, the agent monitors and profiles the experiment's progress and automatically triggers log reading, debugging, metric analysis, code fixes, merge requests, and smoke tests, identifying and configuring subtle yet key changes. These could have required the collaboration of multiple human researchers from different teams before, but now human researchers only interact for critical decisions and discussions. This accelerates problem discovery and experimentation, delivering models faster. Here, M2.7 is capable of handling 30%-50% of the workflow. + +![Research agent harness architecture](minimax-assets/img-2.png) + +During the iteration process, we realized that the model's ability to recursively evolve its own harness is also critical. Our internal harness autonomously collects feedback, builds evaluation sets for internal tasks, and based on this continuously iterates its own architecture, skills/MCP implementation, and memory mechanisms to complete tasks better and more efficiently. + +For example, we had M2.7 optimize a model's programming performance on an internal scaffold. M2.7 ran entirely autonomously, executing an iterative loop of "analyze failure trajectories → plan changes → modify scaffold code → run evaluations → compare results → decide to keep or revert changes" for over 100 rounds. During this process, M2.7 discovered effective optimizations for the model: systematically searching for the optimal combination of sampling parameters such as temperature, frequency penalty, and presence penalty; designing more specific workflow guidelines for the model (e.g., automatically searching for the same bug patterns in other files after a fix); and adding loop detection and other optimizations to the scaffold's agent loop. Ultimately, this achieved a 30% performance improvement on internal evaluation sets. + +We believe that future AI self-evolution will gradually transition towards full autonomy, coordinating data construction, model training, inference architecture, evaluation, and other stages without human involvement. + +To this end, we conducted preliminary exploratory tests in low-resource scenarios. We had M2.7 participate in 22 machine learning competitions at the MLE Bench Lite level open-sourced by OpenAI. These competitions can be run on a single A30 GPU, yet they cover virtually all stages of machine learning workflow. + +We designed and implemented a simple harness to guide the agent in autonomous optimization. The core modules include three components: short-term memory, self-feedback, and self-optimization. Specifically, after each iteration round, the agent generates a short-term memory markdown file and simultaneously performs self-criticism on the current round's results, thereby providing potential optimization directions for the next round. The next round then conducts further self-optimization based on the memory and self-feedback chain from all previous rounds. We ran a total of three trials, each with 24 hours for iterative evolution. From the figure below, one can see that the ML models trained by M2.7 continuously achieved higher medal rates over time. In the end, the best run achieved 9 gold medals, 5 silver medals, and 1 bronze medal. The average medal rate across the three runs was 66.6%, a result second only to Opus-4.6 (75.7%) and GPT-5.4 (71.2%), tying with Gemini-3.1 (66.6%). + +![MLE Bench Lite performance chart](minimax-assets/img-3.png) + +## Professional Software Engineering + +In software engineering tasks, M2.7 more deeply explores real-world programming abilities, including log analysis for bug hunting, refactoring, code security, machine learning, Android development, and more. + +Take a common production scenario as an example: debugging in a live environment. This requires not just code generation, but strong comprehensive reasoning abilities. When faced with alerts in production, M2.7 can correlate monitoring metrics with deployment timelines to perform causal reasoning, conduct statistical analysis on trace sampling and propose precise hypotheses, proactively connect to databases to verify root causes, pinpoint missing index migration files in the code repository, and even have the awareness to use non-blocking index creation to stop the bleeding first before submitting a merge request. From observability analysis and database expertise to SRE-level decision-making — this is not merely a model that can write code, but one that truly understands production systems. Compared to traditional manual troubleshooting processes, using M2.7, we have on multiple occasions reduced the recovery time for live production system incidents to under three minutes. + +Your browser does not support video playback. Please switch to a different browser. + +Live production environment debugging + +In terms of raw programming capabilities, M2.7 has reached the level of SOTA models. On SWE-Pro, which covers multiple programming languages, M2.7 achieved a 56.22% accuracy rate, matching GPT-5.3-Codex. It demonstrates an even more notable advantage on benchmarks closer to real-world engineering scenarios, such as SWE Multilingual (76.5) and Multi SWE Bench (52.7). + +This capability also extends to end-to-end full project delivery scenarios. On the repo-level code generation benchmark VIBE-Pro, M2.7 scored 55.6%, nearly on par with Opus 4.6 — meaning that whether the requirement involves Web, Android, iOS, or simulation tasks, they can be handed directly to M2.7 to complete. + +What deserves even more attention is its deep understanding of complex engineering systems. On Terminal Bench 2 (57.0%) and NL2Repo (39.8%), both of which demand a high degree of system-level comprehension, M2.7 also performs solidly. This further confirms that it excels not only at code generation but can also deeply understand the operational logic and collaborative dynamics of software systems. + +Your browser does not support video playback. Please switch to a different browser. + +WildGuard demo webpage generated by M2.7 + +To improve development efficiency, one particularly important feature is native Agent Teams (multi-agent collaboration). Agent Teams impose paradigm-level demands on the model: role boundaries, adversarial reasoning, protocol adherence, and behavioral differentiation — these cannot be achieved through prompting alone and must be internalized as native capabilities of the model. In Agent Teams scenarios, the model needs to stably anchor its role identity, proactively challenge teammates' logical and ethical blind spots, and make autonomous decisions within complex state machines. Below is an Agent Teams setup we use internally for product prototype development, which contains a minimal organization for building product prototypes. + +![Research agent harness architecture](minimax-assets/d92a6eb4-a4b8-4906-b76a-d627c814a2c0.gif) + +Agent Teams multi-agent collaboration demo + +## Professional Work + +Beyond software engineering, agents are becoming increasingly useful in office scenarios. We believe this comes down to two core capabilities: + +Domain expertise and task delivery capability. The model needs to possess professional knowledge across various fields and understand user requirements. In the GDPval-AA evaluation, which measures this capability, M2.7 achieved an ELO score of 1495 among 45 models, second only to Opus 4.6, Sonnet 4.6, and GPT5.4, and surpassing GPT5.3. For the most common office document processing tasks, we systematically optimized the model's ability to handle Word, Excel, and PPT. Across various agent harnesses, M2.7 can both generate files directly based on templates and skills, and follow users' interactive instructions to perform multiple rounds of high-fidelity editing on existing files, ultimately producing editable deliverables. + +Ability to interact with complex environments. Generalized everyday scenarios mean the model must flexibly adapt to various contexts, invoke diverse skills and tools, and maintain stable instruction adherence throughout extended interactions. M2.7 has made substantial improvements in these areas. On Toolathon, M2.7 achieved an accuracy of 46.3%, reaching the global top tier. Agent harnesses in real-world work scenarios also often require understanding and invoking a large number of complex skills. In MM Claw testing, M2.7 maintained a 97% skill compliance rate across 40 complex skills (each exceeding 2,000 tokens). + +We tested the model's professional proficiency in finance, and compared to the previous generation, the improvement in capability is significant. For example, in a scenario involving reading research reports and modeling a company's future revenue, M2.7 can autonomously read a company's annual reports and earnings call minutes, cross-reference multiple research reports, independently design assumptions and build a revenue forecast model, and then produce a PPT and research report based on templates — understanding, making judgments, and producing output like a junior analyst, while self-correcting through multiple rounds of interaction. The feedback from practitioners is that the output can already serve as a first draft and go directly into subsequent workflows. Below is an example for TSMC. + +Task: Based on TSMC's annual report and earnings call information, build a revenue model for TSMC. Read multiple research reports, design corresponding assumptions, model TSMC's revenue based on the latest information, then produce a PPT based on a PPT template, and write a Word document research report. + +您的浏览器不支持播放此视频,请更换浏览器。 + +The recent surge in popularity of OpenClaw is representative of a thriving agent ecosystem, and we are pleased that our M2-series models have contributed to the community's flourishing. Based on commonly used tasks in OpenClaw, we built an evaluation set called MM Claw, covering a wide range of real-world needs in both work and life — from personal learning planning, to office document processing and delivery, scheduled professional research and investment advice, and code development and maintenance. M2.7 achieved a level close to Sonnet 4.6 on this test, with an accuracy of 62.7%. + +## Entertainment + +With OpenClaw and similar personal agents, we noticed that beyond getting work done, many users also want the model to have high emotional intelligence and character consistency. With a persona in place, users start interacting with OpenClaw like a friend. We believe this presents an opportunity to extend the use of agentic models beyond pure productivity into interactive entertainment. To this end, we strengthened character consistency and conversational capabilities in M2.7. + +Based on this, we built a preliminary demo: OpenRoom, an interaction system based on an agent harness that liberates AI interaction from plain text streams and places it within a Web GUI space where everything is interactive. Here, character settings are no longer cold chunks of prompts; conversation drives the experience, generating real-time visual feedback and scene interactions, with characters proactively engaging with their environment. We believe this framework is highly extensible and can continue to evolve alongside improvements in agentic capabilities and community development, exploring entirely new ways for humans and agents to interact. + +To encourage exploration in this area, we have open-sourced the initial demo (of which most of the code was written by AI): + +Your browser does not support video playback. Please switch to a different browser. + +MiniMax M2.7 is now fully available on MiniMax Agent and the MiniMax API Platform. We look forward to users and developers exploring even more interesting use cases with M2.7. + +MiniMax Agent: [agent.minimax.io](https://agent.minimax.io/) + +API: [platform.minimax.io](https://platform.minimax.io/) + +Coding Plan: [platform.minimax.io/subscribe/coding-plan](https://platform.minimax.io/subscribe/coding-plan) + +Intelligence with Everyone. \ No newline at end of file diff --git a/raw/mitchellh-ai-adoption-journey.md b/raw/mitchellh-ai-adoption-journey.md new file mode 100644 index 0000000..753b810 --- /dev/null +++ b/raw/mitchellh-ai-adoption-journey.md @@ -0,0 +1,142 @@ +# My AI Adoption Journey + +Table of Contents + +- [Step 1: Drop the Chatbot](#step-1-drop-the-chatbot) +- [Step 2: Reproduce Your Own Work](#step-2-reproduce-your-own-work) +- [Step 3: End-of-Day Agents](#step-3-end-of-day-agents) +- [Step 4: Outsource the Slam Dunks](#step-4-outsource-the-slam-dunks) +- [Step 5: Engineer the Harness](#step-5-engineer-the-harness) +- [Step 6: Always Have an Agent Running](#step-6-always-have-an-agent-running) +- [Today](#today) + +My experience adopting any meaningful tool is that I've necessarily gone through three phases: (1) a period of inefficiency (2) a period of adequacy, then finally (3) a period of workflow and life-altering discovery. + +In most cases, I have to force myself through phase 1 and 2 because I usually have a workflow I'm already happy and comfortable with. Adopting a tool feels like work, and I *do not* want to put in the effort, but I usually do in an effort to be a well-rounded person of my craft. + +This is my journey of how I found value in AI tooling and what I'm trying next with it. In an ocean of overly dramatic, hyped takes, I hope this represents a more nuanced, measured approach to my views on AI and how they've changed over time. + +This blog post was fully written by hand, in my own words. I hate that I have to say that but especially given the subject matter, I want to be explicit about it. + +* * * + +## Step 1: Drop the Chatbot + +Immediately cease trying to perform meaningful work via a chatbot (e.g. ChatGPT, Gemini on the web, etc.). Chatbots have real value and are a daily part of my AI workflow, but their utility in coding is highly limited because you're mostly hoping they come up with the right results based on their prior training, and correcting them involves a human (you) to tell them they're wrong repeatedly. It is inefficient. + +I think everyone's first experience with AI is a chat interface. And I think everyone's first experience trying to code with AI has been asking a chat interface to write code. + +While I was still a heavy AI skeptic, my first "oh wow" moment was pasting a screenshot of Zed's command palette into Gemini, asking it to reproduce it with SwiftUI, and being truly flabbergasted that it did it *very well*. The command palette that ships for macOS in Ghostty today is only very lightly modified from what Gemini produced for me in seconds. + +But when I tried to reproduce that behavior for other tasks, I was left disappointed. In the context of brownfield projects, I found the chat interface produced poor results very often, and I found myself very frustrated copying and pasting code and command output to and from the interface. It was very obviously far less efficient than me doing the work myself. + +To find value, you *must* use an **agent**. An agent is the industry-adopted term for an LLM that can chat and invoke external behavior in a loop[1](#user-content-fn-1) At a bare minimum, the agent must have the ability to: read files, execute programs, and make HTTP requests. + +* * * + +## Step 2: Reproduce Your Own Work + +The next phase on my journey I tried [Claude Code](https://github.com/anthropics/claude-code). I'll cut to the chase: I initially wasn't impressed. I just wasn't getting good results out of my sessions. I felt I had to touch up everything it produced and this process was taking more time than if I had just done it myself. I read blog posts, watched videos, but just wasn't that impressed. + +Instead of giving up, I **forced myself to reproduce all my manual commits with agentic ones.** I literally did the work twice. I'd do the work manually, and then I'd fight an agent to produce identical results in terms of quality and function (without it being able to see my manual solution, of course). + +This was *excruciating*, because it got in the way of simply getting things done. But I've been around the block with non-AI tools enough to know that friction is natural, and I can't come to a firm, defensible conclusion without exhausting my efforts. + +But, expertise formed. I quickly discovered for myself from first principles what others were already saying, but discovering it myself resulted in a stronger fundamental understanding. + +1. Break down sessions into separate clear, actionable tasks. Don't try to "draw the owl" in one mega session. +2. For vague requests, split the work into separate planning vs. execution sessions. +3. If you give an agent a way to verify its work, it more often than not fixes its own mistakes and prevents regressions. + +More generally, I also found the edges of what agents -- at the time -- were good at, what they weren't good at, and for the tasks they were good at how to achieve the results I wanted. + +All of this led to significant efficiency gains, to the point where I was starting to naturally use agents in a way that I felt was no slower than doing it myself (but I still didn't feel it was any faster, since I was mostly babysitting an agent). + +The negative space here is worth reiterating: part of the efficiency gains here were understanding when *not* to reach for an agent. Using an agent for something it'll likely fail at is obviously a big waste of time and having the knowledge to avoid that completely leads to time savings[2](#user-content-fn-3). + +At this stage, I was finding adequate value with agents that I was happy to use them in my workflow, but still didn't feel like I was seeing any net efficiency gains. I didn't care though, I was content at this point with AI as a tool. + +* * * + +## Step 3: End-of-Day Agents + +To try to find some efficiency, I next started up a new pattern: **block out the last 30 minutes of every day to kick off one or more agents.** My hypothesis was that *perhaps* I could gain some efficiency if the agent can make some *positive progress* in the times I can't work anyways. Basically: instead of trying to do more in the time I have, try to do more in the time I don't have. + +Similar to the previous task, I at first found this both unsuccessful and annoying. But, I once again quickly found different categories of work that were really helpful: + +- **Deep research sessions** where I'd ask agents to survey some field, such as finding all libraries in a specific language with a specific license type and producing multi-page summaries for each on their pros, cons, development activity, social sentiment, etc. +- **Parallel agents attempting different vague ideas I had but didn't have time to get started on.** I didn't expect them to produce something I'd ever ship here, but perhaps could illuminate some unknown unknowns when I got to the task the next day. +- **Issue and PR triage/review.** Agents are good at using `gh` (GitHub CLI), so I manually scripted a quick way to spin up a bunch in parallel to triage issues. I would NOT allow agents to respond, I just wanted reports the next day to try to guide me towards high value or low effort tasks. + +To be clear, I did not go as far as others went to have agents running in loops all night. In most cases, agents completed their tasks in less than half an hour. But, the latter part of the working day, I'm usually tired and coming out of flow and find myself too personally inefficient, so shifting my effort to spinning up these agents I found gave me a "warm start" the next morning that got me working more quickly than I would've otherwise. + +I was happy, and I was starting to feel like I was doing more than I was doing prior to AI, if only slightly. + +* * * + +## Step 4: Outsource the Slam Dunks + +By this point, I was getting very confident about what tasks my AI was and wasn't great at. I had really high confidence with certain tasks that the AI would achieve a mostly-correct solution. So the next step on my journey was: **let agents do all of that work while I worked on other tasks.** + +More specifically, I would start each day by taking the results of my prior night's triage agents, filter them manually to find the issues that an agent will almost certainly solve well, and then keep them going in the background (one at a time, not in parallel). + +Meanwhile, **I'd work on something else.** I wasn't going to social media (any more than usual without AI), I wasn't watching videos, etc. I was in my own, normal, pre-AI deep thinking mode working on something I wanted to work on or had to work on. + +**Very important at this stage: turn off agent desktop notifications.** Context switching is very expensive. In order to remain efficient, I found that it was my job as a human to be in control of when I interrupt the agent, not the other way around. Don't let the agent notify you. During natural breaks in your work, tab over and check on it, then carry on. + +Importantly, I think the "work on something else" helps counteract the highly publicized [Anthropic skill formation paper](https://www.anthropic.com/research/AI-assistance-coding-skills). Well, you're trading off: not forming skills for the tasks you're delegating to the agent while continuing to form skills naturally in the tasks you continue to work on manually. + +At this point I was firmly in the "no way I can go back" territory. I felt more efficient, but even if I wasn't, the thing I liked the most was that I could now focus my coding and thinking on tasks I really loved while still adequately completing the tasks I didn't. + +* * * + +## Step 5: Engineer the Harness + +At risk of stating the obvious: agents are much more efficient when they produce the right result the first time, or at worst produce a result that requires minimal touch-ups. The most sure-fire way to achieve this is to give the agent fast, high quality tools to automatically tell it when it is wrong. + +I don't know if there is a broad industry-accepted term for this yet, but I've grown to calling this "harness engineering." It is the idea that anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again. I don't need to invent any new terms here; if another one exists, I'll jump on the bandwagon. + +This comes in two forms: + +1. **Better implicit prompting (AGENTS.md).** For simple things, like the agent repeatedly running the wrong commands or finding the wrong APIs, update the `AGENTS.md` (or equivalent). Here is [an example from Ghostty](https://github.com/ghostty-org/ghostty/blob/ca07f8c3f775fe437d46722db80a755c2b6e6399/src/inspector/AGENTS.md). Each line in that file is based on a bad agent behavior, and it almost completely resolved them all. + +2. **Actual, programmed tools.** For example, scripts to take screenshots, run filtered tests, etc etc. This is usually paired with an AGENTS.md change to let it know about this existing. + + +**This is where I'm at today.** I'm making an earnest effort whenever I see an agent do a Bad Thing to prevent it from ever doing that bad thing again. Or, conversely, I'm making an earnest effort for agents to be able to verify they're doing a Good Thing. + +* * * + +## Step 6: Always Have an Agent Running + +Simultaneous to step 5, I'm also operating under the goal of **having an agent running at all times.** If an agent isn't running, I ask myself "is there something an agent could be doing for me right now?" + +I particularly like to combine this with slower, more thoughtful models like Amp's [deep mode](https://ampcode.com/news/deep-mode) (which is basically just GPT-5.2-Codex) which can take upwards of 30+ minutes to make small changes. The flip side of that is that it does tend to produce very good results. + +**I'm not \[yet?\] running multiple agents, and currently don't really want to.** I find having the one agent running is a good balance for me right now between being able to do deep, manual work I find enjoyable, and babysitting my kind of stupid and yet mysteriously productive robot friend. + +The "have an agent running at all times" goal is still just a goal. I'd say right now I'm maybe effective at having a background agent running 10 to 20% of a normal working day. But, I'm actively working to improve that. + +**I don't want to run agents for the sake of running agents.** I only want to run them when there is a task I think would be truly helpful to me. Part of the challenge of this goal is improving my own workflows and tools so that I can have a constant stream of high quality work to do that I can delegate. Which, even without AI, is important! + +* * * + +## Today + +And that's where I'm at today. + +Through this journey, I've personally reached a point where I'm having success with modern AI tooling and I believe I'm approaching it with the proper measured view that is grounded in reality. I really don't care one way or the other if AI is here to stay[3](#user-content-fn-4), I'm a software craftsman that just wants to build stuff for the love of the game. + +The whole landscape is moving so rapidly that I'm sure I'll look back at this post very quickly and laugh at my naivete. But, as they say, if you can't be embarassed about your past self, you're probably not growing. I just hope I'll grow in the right direction! + +I have no skin in the game here[4](#user-content-fn-5), and there are of course other reasons behind utility to avoid using AI. I fully respect anyone's individual decisions regarding it. I'm not here to convince you! For those interested, I just wanted to share my personal approach to navigating these new tools and give a glimpse about how I approach new tools *in general*, regardless of AI. + +## Footnotes + +1. Modern coding models like Opus and Codex are specifically trained to bias towards using tools compared to conversational models. [↩](#user-content-fnref-1) + +2. Due to the rapid pace of innovation in models, I have to constantly revisit my priors on this one. [↩](#user-content-fnref-3) + +3. The skill formation issues particularly in juniors without a strong grasp of fundamentals deeply worries me, however. [↩](#user-content-fnref-4) + +4. I don't work for, invest in, or advise any AI companies. [↩](#user-content-fnref-5) \ No newline at end of file diff --git a/raw/nxcode-harness-engineering-complete-guide.md b/raw/nxcode-harness-engineering-complete-guide.md new file mode 100644 index 0000000..083df0d --- /dev/null +++ b/raw/nxcode-harness-engineering-complete-guide.md @@ -0,0 +1,417 @@ +# Harness Engineering: The Complete Guide to Building Systems That Make AI Agents Actually Work (2026) + +## Harness Engineering: The Complete Guide to Building Systems That Make AI Agents Actually Work + +**March 2026** — If 2025 was the year AI agents proved they could write code, 2026 is the year we learned that **the agent isn't the hard part — the harness is.** + +OpenAI's Codex team just built a production application with **over 1 million lines of code** where **zero lines were written by human hands**. The engineers didn't write code. They designed the system that let AI write code reliably. That system — the constraints, feedback loops, documentation, linters, and lifecycle management — is what the industry now calls a **harness**. + +**Harness engineering** is the new discipline of designing these systems. And it's changing what it means to be a software engineer. + +* * * + +## What Is Harness Engineering? + +### The Horse Metaphor + +The term "harness" comes from horse tack — reins, saddle, bit — the complete set of equipment for channeling a powerful but unpredictable animal in the right direction. The metaphor is deliberate: + +- The **horse** is the AI model — powerful, fast, but it doesn't know where to go on its own +- The **harness** is the infrastructure — constraints, guardrails, feedback loops that channel the model's power productively +- The **rider** is the human engineer — providing direction, not doing the running + +Without a harness, an AI agent is a thoroughbred in an open field. Fast, impressive, and completely useless for getting anything done. + +### The Formal Definition + +**Harness engineering** is the design and implementation of systems that: + +1. **Constrain** what an AI agent can do (architectural boundaries, dependency rules) +2. **Inform** the agent about what it should do (context engineering, documentation) +3. **Verify** that the agent did it correctly (testing, linting, CI validation) +4. **Correct** the agent when it goes wrong (feedback loops, self-repair mechanisms) + +Martin Fowler describes it as *"the tooling and practices we can use to keep AI agents in check"* — but it's more than just safety. A good harness makes agents **more capable**, not just more controlled. + +* * * + +## Why Harness Engineering Matters Now + +### The Model Is Commodity. The Harness Is Moat. + +Here's the uncomfortable truth the AI industry is confronting: **the underlying model matters less than the system around it.** + +LangChain proved this definitively. Their coding agent went from **52.8% to 66.5%** on Terminal Bench 2.0 — jumping from **Top 30 to Top 5** — by changing nothing about the model. They only changed the harness: + +Change + +What They Did + +Impact + +Self-verification loop + +Added pre-completion checklist middleware + +Caught errors before submission + +Context engineering + +Mapped directory structures at startup + +Agent understood codebase from the start + +Loop detection + +Tracked repeated file edits + +Prevented "doom loops" + +Reasoning sandwich + +High reasoning for planning/verification, medium for implementation + +Better quality within time budgets + +**Same model. Different harness. Dramatically better results.** + +### OpenAI's 1 Million Line Proof Point + +OpenAI's experiment is the most compelling evidence yet: + +- **5 months** of development +- **1 million+ lines of code** in the final product +- **Zero manually written lines** — every line was produced by Codex agents +- **Built in ~1/10th the time** it would have taken humans +- The product has **internal daily users and external alpha testers** +- It **ships, deploys, breaks, and gets fixed** — all by agents within the harness + +The engineers' job? Designing the harness. Specifying intent. Providing feedback. Not writing code. + +* * * + +## The Three Pillars of Harness Engineering + +OpenAI's framework organizes harness engineering into three core categories: + +### 1\. Context Engineering + +Context engineering is about ensuring the agent has the right information at the right time. + +**Static context:** + +- Repository-local documentation (architecture specs, API contracts, style guides) +- `AGENTS.md` or `CLAUDE.md` files that encode project-specific rules +- Cross-linked design documents validated by linters + +**Dynamic context:** + +- Observability data (logs, metrics, traces) accessible to agents +- Directory structure mapping at agent startup +- CI/CD pipeline status and test results + +**The critical rule:** From the agent's perspective, anything it can't access in-context doesn't exist. Knowledge in Google Docs, Slack threads, or people's heads is invisible to the system. **The repository must be the single source of truth.** + +### 2\. Architectural Constraints + +This is where harness engineering diverges most sharply from traditional AI prompting. Instead of telling the agent "write good code," you **mechanically enforce what good code looks like.** + +**Dependency layering:** + +``` +Types → Config → Repo → Service → Runtime → UI + +``` + +Each layer can only import from layers to its left. This isn't a suggestion — it's enforced by structural tests and CI validation. + +**Constraint enforcement tools:** + +- **Deterministic linters** — Custom rules that flag violations automatically +- **LLM-based auditors** — Agents that review other agents' code for architectural compliance +- **Structural tests** — Like ArchUnit, but for AI-generated code +- **Pre-commit hooks** — Automated checks before any code is committed + +**Why constraints improve output:** Paradoxically, constraining the solution space makes agents **more productive**, not less. When an agent can generate anything, it wastes tokens exploring dead ends. When the harness defines clear boundaries, the agent converges faster on correct solutions. + +### 3\. Entropy Management ("Garbage Collection") + +This is the most underappreciated component. Over time, AI-generated codebases accumulate entropy — documentation drifts from reality, naming conventions diverge, dead code accumulates. + +Harness engineering addresses this with **periodic cleanup agents:** + +- **Documentation consistency agents** — Verify that docs match current code +- **Constraint violation scanners** — Find code that slipped past earlier checks +- **Pattern enforcement agents** — Identify and fix deviations from established patterns +- **Dependency auditors** — Track and resolve circular or unnecessary dependencies + +These agents run on schedules — daily, weekly, or triggered by specific events — keeping the codebase healthy for both human reviewers and future AI agents. + +* * * + +## Harness Engineering in Practice: How Teams Actually Do It + +### The OpenAI Approach: Zero Human Code + +OpenAI's team structure for harness engineering: + +Role + +Traditional + +Harness Engineering + +Writing code + +Primary job + +Never + +Designing architecture + +Part of the job + +Primary job + +Writing documentation + +Afterthought + +Critical infrastructure + +Reviewing PRs + +Code review + +Reviewing agent output + harness effectiveness + +Debugging + +Reading code + +Analyzing agent behavior patterns + +Testing + +Writing tests + +Designing test strategies agents execute + +### The Stripe Approach: Minions at Scale + +Stripe's internal coding agents, called **Minions**, now produce **over 1,000 merged pull requests per week**: + +1. Developer posts a task in Slack +2. Minion writes the code +3. Minion passes CI +4. Minion opens a PR +5. Human reviews and merges + +No developer interaction between step 1 and step 5. The harness handles everything — test execution, CI validation, style compliance, and documentation updates. + +### The LangChain Approach: Middleware-First + +LangChain structures their harness as composable middleware layers: + +``` +Agent Request + → LocalContextMiddleware (maps codebase) + → LoopDetectionMiddleware (prevents repetition) + → ReasoningSandwichMiddleware (optimizes compute) + → PreCompletionChecklistMiddleware (enforces verification) + → Agent Response + +``` + +Each middleware layer adds a specific capability without modifying the core agent logic. This modular approach makes the harness testable and evolvable. + +* * * + +## Building Your First Harness: A Practical Framework + +### Level 1: Basic Harness (Single Developer) + +If you're using Claude Code, Cursor, or Codex for individual projects: + +**What to set up:** + +- `CLAUDE.md` or `.cursorrules` file with project conventions +- Pre-commit hooks for linting and formatting +- A test suite the agent can run to self-verify +- Clear directory structure with consistent naming + +**Time to set up:** 1-2 hours **Impact:** Prevents the most common agent mistakes + +### Level 2: Team Harness (Small Team) + +For teams of 3-10 developers sharing a codebase: + +**Add to Level 1:** + +- `AGENTS.md` with team-wide conventions +- Architectural constraints enforced by CI +- Shared prompt templates for common tasks +- Documentation-as-code validated by linters +- Code review checklists specifically for agent-generated PRs + +**Time to set up:** 1-2 days **Impact:** Consistent agent behavior across the team + +### Level 3: Production Harness (Engineering Organization) + +For organizations running dozens of concurrent agents: + +**Add to Level 2:** + +- Custom middleware layers (loop detection, reasoning optimization) +- Observability integration (agents read logs and metrics) +- Entropy management agents on scheduled runs +- Harness versioning and A/B testing +- Agent performance monitoring dashboards +- Escalation policies for when agents get stuck + +**Time to set up:** 1-2 weeks **Impact:** Agents operate as autonomous contributors + +* * * + +## Common Harness Engineering Mistakes + +### 1\. Over-Engineering the Control Flow + +> *"If you over-engineer the control flow, the next model update will break your system."* + +Models improve rapidly. Capabilities that required complex pipelines in 2024 are now handled by a single context-window prompt. Build your harness to be **rippable** — you should be able to remove "smart" logic when the model gets smart enough to not need it. + +### 2\. Treating the Harness as Static + +The harness needs to evolve with the model. When a new model release improves reasoning, your reasoning-optimization middleware might become counterproductive. Review and update harness components with every major model update. + +### 3\. Ignoring the Documentation Layer + +The most impactful harness improvement is often the simplest: **better documentation**. If your `AGENTS.md` is vague, your agent output will be vague. Invest in precise, machine-readable documentation that serves as the agent's ground truth. + +### 4\. No Feedback Loop + +A harness without feedback is a cage, not a guide. The agent needs to know when it's succeeding and when it's failing. Build in: + +- Self-verification steps before task completion +- Test execution as part of the agent workflow +- Metrics on agent success rates by task type + +### 5\. Human-Only Documentation + +If your architectural decisions live in people's heads or in Confluence pages the agent can't access, the harness has a gap. **Everything the agent needs must be in the repository.** + +* * * + +## Harness Engineering vs. Related Concepts + +Concept + +Scope + +Focus + +**Prompt Engineering** + +Single interaction + +Crafting effective prompts + +**Context Engineering** + +Model context window + +What information the model sees + +**Harness Engineering** + +Entire agent system + +Environment, constraints, feedback, lifecycle + +**Agent Engineering** + +Agent architecture + +Internal agent design and routing + +**Platform Engineering** + +Infrastructure + +Deployment, scaling, operations + +Harness engineering **includes** context engineering and draws from prompt engineering, but it operates at a higher level — it's about the complete system that makes agents reliable, not just the inputs to a single interaction. + +* * * + +## What This Means for Software Engineers + +### The Job Is Changing + +Harness engineering represents a genuine evolution in what software engineers do: + +Before + +After + +Write code + +Design environments where AI writes code + +Debug code + +Debug agent behavior + +Review code + +Review agent output + harness effectiveness + +Write tests + +Design test strategies + +Maintain docs + +Build documentation as machine-readable infrastructure + +This doesn't mean engineers become less technical. If anything, harness engineering requires **deeper** architectural thinking — you're designing systems that must work without your constant intervention. + +### The Skills That Matter + +Based on what we've seen building AI-powered products at [NxCode](https://www.nxcode.io/): + +1. **Systems thinking** — Understanding how constraints, feedback loops, and documentation interact +2. **Architecture design** — Defining boundaries that are enforceable and productive +3. **Specification writing** — Articulating intent precisely enough for agents to execute +4. **Observability** — Building monitoring that reveals agent behavior patterns +5. **Iteration speed** — Rapidly testing and refining harness configurations + +### Our Experience: What Works in Practice + +We've been building AI-powered web applications using multiple agent systems (Claude Code, Codex, Cursor). The patterns that have made the biggest difference for us: + +- **Repository-first documentation**: Every architectural decision, naming convention, and deployment process is in the repo. Nothing lives in Slack or Google Docs. +- **Incremental constraint building**: Start with basic linting, add architectural constraints as patterns emerge, don't try to design the perfect harness upfront. +- **Agent-specific review checklists**: AI-generated code has different failure modes than human code. Our review process accounts for common agent patterns (over-abstraction, unnecessary error handling, documentation drift). +- **Multi-provider harness design**: Our harness works with Claude, GPT, and Gemini models. Provider-agnostic design means we can switch models without rebuilding the entire system. + +* * * + +## Key Takeaways + +1. **Harness engineering is the new discipline** of designing systems that make AI agents reliable — constraints, feedback loops, documentation, and lifecycle management +2. **The model is commodity; the harness is moat** — LangChain jumped from Top 30 to Top 5 on benchmarks by only changing the harness +3. **OpenAI built 1M+ lines with zero human code** — proving harness engineering works at production scale +4. **Three pillars**: Context engineering, architectural constraints, and entropy management +5. **Start simple**: A good `AGENTS.md` and pre-commit hooks are more impactful than complex middleware +6. **The engineer's job is evolving** — from writing code to designing environments where AI writes code +7. **Build rippable harnesses** — over-engineering breaks when models improve; keep it adaptable + +* * * + +## Related Articles + +- [Best AI for Coding in 2026: 10 Tools Ranked by Real-World Performance](https://www.nxcode.io/resources/news/best-ai-for-coding-2026-complete-ranking) +- [OpenAI Frontier Guide: Enterprise AI Agent Platform for Building AI Coworkers (2026)](https://www.nxcode.io/resources/news/openai-frontier-enterprise-ai-agent-platform-guide-2026) +- [Cursor Tutorial 2026: Learn AI Coding in 15 Minutes (Beginner Guide)](https://www.nxcode.io/resources/news/cursor-tutorial-beginners-2026) \ No newline at end of file diff --git a/skills/SKILL.md b/skills/SKILL.md new file mode 100644 index 0000000..73ddbe1 --- /dev/null +++ b/skills/SKILL.md @@ -0,0 +1,118 @@ +# 使用 LLM 生成高质量中文 Wiki 知识库 + +## 1. Skill 概述 +本 Skill 旨在利用大模型(LLM)将原始文档和图像(`raw/`)增量“编译”为 **结构化、交叉链接、高质量的中文 Wiki 知识库(`wiki/`)** 的完整思路和方法。 + +* **核心逻辑**:人工不直接编写 Wiki,仅负责投放素材和发起查询;LLM 负责理解、重写、链接与维护。 +* **适配工具**:Obsidian(IDE 前端),通过插件支持更多格式: + - Markdown(文本内容) + - Matplotlib(数据可视化) + - Marp(幻灯片渲染) + - Mermaid(架构图) + +--- + +## 2. 标准文件系统架构 +严格遵循 I/O 分离原则,确保知识库的纯净度与可迁移性: + +```text +📁 wikillm +├── 📁 raw/ # 【输入层】原始素材(只读) +└── 📁 wiki/ # 【输出层】编译器生成的知识产物 + ├── 📁 concepts/ # 核心概念、原理分析 + ├── 📁 practices/ # 部署指南、最佳实践 + ├── 📁 visual/ # Marp 幻灯片、Matplotlib 趋势图 + ├── 📁 queries/ # 高价值 Q&A 的沉淀归档 + ├── INDEX.md # 动态索引与学习路径 + └── Glossary.md # 统一术语表与双链枢纽 +``` + +--- + +## 3. 核心工作流 (The "Compilation" Loop) + +### 阶段 1:多模态解构 (Ingest & Analyze) +* **任务**:解析 `raw/` 目录下的新增内容。 +* **视觉解析**:对图片进行深度 OCR 与逻辑识别。将架构图转化为文字描述及 **Mermaid** 代码块,存入对应 Wiki 页面。 +* **元数据提取**:为每篇文档生成 YAML Frontmatter(包含:`tags`, `source`, `confidence_score`, `last_updated`)。 + +### 阶段 2:增量编译 (Incremental Writing) +* **非线性重构**:不进行 1:1 翻译,而是基于源文档的“核心贡献”进行重写。 +* **中文化增强**: + * 消除翻译腔:使用行业专业术语(如将 "Agent" 译为 "智能体")。 + * 添加上下文:为中文读者补充必要的背景知识或行业对比。 +* **可视化输出**:若涉及多步流程或对比,自动生成 **Marp** 格式的幻灯片文件(`.md`),以便在 Obsidian 中演示。 + +### 阶段 3:网络化链接 (Wikilinks & Indexing) +* **双链注入**:全文检索 `Glossary.md` 中的术语,使用 `[[术语名]]` 自动包裹。 +* **Wikilink 格式规范**: + - 文件名使用 kebab-case(连字符分隔),例如:`Harness-Engineering.md` + - Wikilink 格式为 `[[文件名|显示文本]]`,其中**文件名部分必须与实际文件名完全匹配**(不带 .md 扩展名) + - 正确示例:`[[Harness-Engineering|Harness 工程]]`(对应文件 `Harness-Engineering.md`) + - 错误示例:`[[Harness Engineering|Harness 工程]]`(文件名带空格,不匹配实际文件) + - **文章列表格式**: + - 错误写法(表格无法正确解析双链): + ``` + | 文章 | 描述 | + |------|------| + | [[Mitchellh-Adoption-Journey|Mitchellh AI 采用之旅]] | HashiCorp 创始人从怀疑论者到深度用户的六个阶段 | + | [[Building-Your-First-Harness|构建你的第一个 Harness]] | 从个人开发者到工程组织的三级实用框架 | + ``` + - 正确写法(使用无序列表): + ``` + - [[Mitchellh-Adoption-Journey|Mitchellh AI 采用之旅]] - HashiCorp 创始人从怀疑论者到深度用户的六个阶段 + - [[Building-Your-First-Harness|构建你的第一个 Harness]] - 从个人开发者到工程组织的三级实用框架 + ``` +* **反向链接**:在文末生成 `## 相关研究` 模块,强制链接到 Wiki 内部至少 2 篇关联文档。 +* **动态索引**:根据新增内容,自动更新 `INDEX.md` 中的”最新研究”与”学习路径”部分。 + +### 阶段 4:健康检查与维护 (Linting) +* **一致性检查**:扫描 `wiki/`,发现术语冲突(如 A 文档叫“智能体”,B 文档叫“代理”)时,自动统一。 +* **孤岛扫描**:识别没有任何链接指向的页面,强制将其挂载到导航树中。 +* **补丁发布**:当 `raw/` 有新版本(如论文更新)时,在对应 Wiki 页面顶部发布 `[Update Patch]` 摘要。 + +--- + +## 4. 输出质量标准 (The Gold Standard) + +### 表达标准 +> **原则**:读起来像是由该领域的资深专家直接用中文撰写的。 +* **禁止词汇**:产生、输出(作为动词)、这个、那个(指代不明)。 +* **提倡词汇**:负责构建、驱动、沉淀、权衡(Trade-off)。 + +### 技术标准 +| 维度 | 要求 | +| :--- | :--- | +| **术语表** | 必须包含 40+ 核心概念,中英对照并带有 Wikilink | +| **链接密度** | 每 500 字需包含至少 3-5 个内部链接 | +| **视觉呈现** | 复杂架构必须有 Mermaid 图,数据趋势必须有 Markdown 表格 | +| **Marp 适配** | 综述类文章必须同步生成一份 `visual/` 下的 Slide 文档 | + +--- + +## 5. Q&A 与知识沉淀 (Filing Back) + +**当用户针对 Wiki 发起复杂查询时:** +1. **Agent 模式**:LLM 检索全库文档,进行跨文档推理。 +2. **回答格式**:回答不仅要解决当前问题,还需提供“参考文档清单”。 +3. **自动归档 (Filing)**:若该次 Q&A 具有通用研究价值,LLM 需自动将其整理为一篇新的文章,存入 `wiki/queries/`,并在 `INDEX.md` 中创建入口。 + +--- + +## 6. 执行清单 (Checklist) + +* [ ] **Raw Check**: `raw/` 目录中是否包含待处理的新素材(图片/文档)? +* [ ] **Glossary Lock**: 是否已锁定全局术语表,确保翻译不漂移? +* [ ] **Multimodal Sync**: 图片是否已转化为可编辑的文字解析/Mermaid? +* [ ] **文件名规范**: 所有 wiki 页面文件是否使用 kebab-case(连字符分隔)命名? +* [ ] **Wikilink 格式检查**: 所有 `[[文件名|显示文本]]` 链接中的文件名部分是否与实际文件名完全匹配? +* [ ] **Wikilink Check**: 所有的核心概念是否都已变成 `[[可点击的链接]]`? +* [ ] **Marp Check**: 是否为需要汇报的内容生成了幻灯片格式? +* [ ] **Orphan Check**: 是否存在无法从 `INDEX.md` 触达的”孤儿页面”? + +--- + +## 7. 最佳实践提示 +* **手离开键盘**:不要手动修改 `wiki/` 目录下的内容,所有的修改应通过“向 LLM 发出 Lint 任务”或“添加 raw 素材后重新编译”来完成。 +* **搜索即创作**:把每一次对知识库的提问看作是一次“知识合成”,务必将高质量的回答存回库中。 +* **结构化思考**:在生成任何长篇文档前,先让 LLM 在内存中构建该主题的“概念地图”。 diff --git a/wiki/Glossary.md b/wiki/Glossary.md new file mode 100644 index 0000000..7875685 --- /dev/null +++ b/wiki/Glossary.md @@ -0,0 +1,294 @@ +--- +title: 术语表 +tags: [术语表, 核心概念] +last_updated: 2026-04-07 +--- + +# 术语表 + +本术语表汇总了 [[Harness-Engineering|Harness 工程]] 领域的核心概念,为知识库提供统一的术语枢纽。 + +## A + +### Agent (智能体) +**英文**:Agent +**中文**:智能体 +**定义**:能够自主感知环境、做出决策并执行行动的 AI 系统。在编码场景中,智能体通常具备读取文件、执行程序、发起 HTTP 请求等工具调用能力。 +**相关概念**:[[Agent Teams|智能体团队]], [[Coding Agent|编码智能体]] + +### Agent Teams (智能体团队) +**英文**:Agent Teams +**中文**:智能体团队 +**定义**:多个智能体通过角色分工、协作协议和差异化行为共同完成复杂任务的系统。需要模型原生支持角色锚定、对抗性推理和协议遵守。 +**相关概念**:[[MiniMax M2.7]], [[Multi-Agent Collaboration|多智能体协作]] + +### Anthropic Harness Design (Anthropic Harness 设计) +**英文**:Anthropic Harness Design +**中文**:Anthropic Harness 设计 +**定义**:Anthropic 团队提出的多智能体架构,包含 Planner(规划者)、Generator(生成者)和 Evaluator(评估者)三种角色,通过生成-评估循环提升输出质量。 +**相关概念**:[[Generator-Evaluator Loop|生成-评估循环]], [[Context Reset|上下文重置]] + +## B + +### Build-Verify Loop (构建-验证循环) +**英文**:Build-Verify Loop +**中文**:构建-验证循环 +**定义**:智能体在完成任务过程中自主进行的迭代改进流程,包括规划发现、构建实现、验证测试和修复问题四个阶段。 +**相关概念**:[[Self-Verification|自我验证]], [[Reasoning Sandwich|推理三明治]] + +## C + +### Codex (Codex 模型) +**英文**:Codex +**中文**:Codex 模型 +**定义**:OpenAI 推出的专门用于代码生成的模型系列,在 Harness 工程中被用于从零生成完整产品代码库。 +**相关概念**:[[OpenAI Harness Engineering|OpenAI Harness 工程]] + +### Coding Agent (编码智能体) +**英文**:Coding Agent +**中文**:编码智能体 +**定义**:专门用于软件工程任务的 AI 智能体,能够理解代码库、编写代码、运行测试和调试问题。 +**相关概念**:[[Harness|Harness]], [[Agent|智能体]] + +### Context Engineering (上下文工程) +**英文**:Context Engineering +**中文**:上下文工程 +**定义**:Harness 工程的三大支柱之一,专注于确保智能体在正确的时间获得正确的信息,包括静态上下文和动态上下文。 +**相关概念**:[[Harness-Engineering|Harness 工程]], [[Context Reset|上下文重置]] + +### Context Reset (上下文重置) +**英文**:Context Reset +**中文**:上下文重置 +**定义**:一种解决长任务中上下文窗口填充和"上下文焦虑"问题的技术,通过清空上下文窗口并使用结构化交接传递状态来实现。 +**相关概念**:[[Context Compaction|上下文压缩]], [[Context Anxiety|上下文焦虑]] + +### Context Anxiety (上下文焦虑) +**英文**:Context Anxiety +**中文**:上下文焦虑 +**定义**:模型在接近其认为的上下文限制时提前结束工作的倾向,Claude Sonnet 4.5 表现出较强的这种行为。 +**相关概念**:[[Context Reset|上下文重置]], [[Context Window|上下文窗口]] + +### Context Compaction (上下文压缩) +**英文**:Context Compaction +**中文**:上下文压缩 +**定义**:通过摘要方式保留对话连续性的技术,但无法为智能体提供干净的状态,上下文焦虑问题仍可能存在。 +**相关概念**:[[Context Reset|上下文重置]] + +## D + +### Doom Loop (末日循环) +**英文**:Doom Loop +**中文**:末日循环 +**定义**:智能体在陷入困境时对同一错误方法进行小幅变异的重复尝试现象,可能多达 10 次以上。 +**相关概念**:[[Loop Detection|循环检测]] + +## E + +### Entropy Management (熵管理) +**英文**:Entropy Management +**中文**:熵管理 +**定义**:Harness 工程的三大支柱之一,通过定期清理智能体来管理 AI 生成代码库中随时间积累的熵(文档漂移、命名约定分歧、死代码堆积等)。 +**相关概念**:[[Harness-Engineering|Harness 工程]], [[Garbage Collection|垃圾回收]] + +### Evaluator (评估者) +**英文**:Evaluator +**中文**:评估者 +**定义**:Anthropic Harness 设计中的三种角色之一,负责评估 Generator 的输出质量,提供具体的反馈和评分。 +**相关概念**:[[Generator|生成者]], [[Planner|规划者]] + +## F + +### Feedforward (前馈控制) +**英文**:Feedforward +**中文**:前馈控制 +**定义**:预期智能体行为并在其行动前进行引导的控制方式,提高智能体第一次尝试就产生良好结果的概率。 +**相关概念**:[[Feedback|反馈控制]], [[Guide|引导]] + +### Feedback (反馈控制) +**英文**:Feedback +**中文**:反馈控制 +**定义**:在智能体行动后进行观察并帮助其自我纠正的控制方式,特别是当产生针对 LLM 消费优化的信号时效果显著。 +**相关概念**:[[Feedforward|前馈控制]], [[Sensor|传感器]] + +## G + +### Garbage Collection (垃圾回收) +**英文**:Garbage Collection +**中文**:垃圾回收 +**定义**:OpenAI 团队采用的定期清理流程,通过"黄金原则"和后台 Codex 任务来扫描偏差、更新质量等级并发起针对性重构。 +**相关概念**:[[Entropy Management|熵管理]] + +### Generator (生成者) +**英文**:Generator +**中文**:生成者 +**定义**:Anthropic Harness 设计中的三种角色之一,负责实际创建输出(如前端代码、应用功能等)。 +**相关概念**:[[Evaluator|评估者]], [[Planner|规划者]] + +### Generator-Evaluator Loop (生成-评估循环) +**英文**:Generator-Evaluator Loop +**中文**:生成-评估循环 +**定义**:受 GAN 启发的多智能体结构,Generator 生成输出,Evaluator 评估并提供反馈,Generator 根据反馈进行迭代改进。 +**相关概念**:[[Anthropic-Harness-Design|Anthropic Harness 设计]] + +### Glossary (术语表) +**英文**:Glossary +**中文**:术语表 +**定义**:WikiLLM 知识库的核心枢纽文档,统一术语翻译、提供中英对照,并通过 wikilinks 连接所有相关概念。 +**相关概念**:[[Wikilink|Wikilink]], [[WikiLLM]] + +## H + +### Harness (Harness) +**英文**:Harness +**中文**:Harness +**定义**:AI 智能体之外的一切,包括系统提示、工具选择、执行流程、约束条件、反馈循环等。公式:Agent = Model + Harness。 +**相关概念**:[[Harness-Engineering|Harness 工程]], [[Agent|智能体]] + +### Harness Engineering (Harness 工程) +**英文**:Harness Engineering +**中文**:Harness 工程 +**定义**:设计和实现使 AI 智能体可靠工作的系统的新学科,包括约束智能体行为、告知智能体应该做什么、验证智能体正确执行、纠正智能体错误四个方面。 +**相关概念**:[[Context-Engineering|上下文工程]], [[Architectural-Constraints|架构约束]], [[Entropy Management|熵管理]] + +### Harness Template (Harness 模板) +**英文**:Harness Template +**中文**:Harness 模板 +**定义**:为常见服务拓扑(如数据仪表板、CRUD 业务服务、事件处理器)准备的引导和传感器捆绑包,可实例化用于特定项目。 +**相关概念**:[[Harness-Engineering|Harness 工程]] + +## L + +### LangChain Harness Engineering (LangChain Harness 工程) +**英文**:LangChain Harness Engineering +**中文**:LangChain Harness 工程 +**定义**:LangChain 团队通过仅改变 Harness 将编码智能体在 Terminal Bench 2.0 上的表现从 52.8% 提升到 66.5%(Top 30 到 Top 5)的实践。 +**相关概念**:[[Harness-Engineering|Harness 工程]], [[Self-Verification|自我验证]] + +### Layered Domain Architecture (分层领域架构) +**英文**:Layered Domain Architecture +**中文**:分层领域架构 +**定义**:OpenAI 采用的严格架构模型,每个业务领域划分为固定的层组(Types → Config → Repo → Service → Runtime → UI),依赖方向经过严格验证。 +**相关概念**:[[Architectural-Constraints|架构约束]] + +### Loop Detection (循环检测) +**英文**:Loop Detection +**中文**:循环检测 +**定义**:LangChain 采用的中间件,通过钩子跟踪每个文件的编辑次数,在对同一文件进行 N 次编辑后添加"考虑重新考虑你的方法"的上下文。 +**相关概念**:[[Doom Loop|末日循环]], [[Middleware|中间件]] + +## M + +### Middleware (中间件) +**英文**:Middleware +**中文**:中间件 +**定义**:LangChain 结构化 Harness 的方式,通过可组合的中间件层在不修改核心智能体逻辑的情况下添加特定功能。 +**相关概念**:[[LangChain Harness Engineering|LangChain Harness 工程]] + +### MiniMax M2.7 (MiniMax M2.7 模型) +**英文**:MiniMax M2.7 +**中文**:MiniMax M2.7 模型 +**定义**:MiniMax 推出的深度参与自我进化的模型,能够构建复杂智能体 Harness、完成高度复杂的生产力任务,包括 Agent Teams、复杂 Skills 和动态工具搜索。 +**相关概念**:[[Agent Teams|智能体团队]], [[Self-Evolution|自我进化]] + +### Mitchellh AI Adoption Journey (Mitchellh AI 采用之旅) +**英文**:Mitchellh AI Adoption Journey +**中文**:Mitchellh AI 采用之旅 +**定义**:HashiCorp 创始人 Mitchell Hashimoto 分享的个人 AI 工具采用历程,包括从聊天机器人到始终运行智能体的六个阶段。 +**相关概念**:[[Harness-Engineering|Harness 工程]] + +## N + +### NxCode Harness Engineering (NxCode Harness 工程) +**英文**:NxCode Harness Engineering +**中文**:NxCode Harness 工程 +**定义**:NxCode 团队提供的 Harness 工程完整指南,总结了三大支柱、实践框架和常见错误。 +**相关概念**:[[Harness-Engineering|Harness 工程]] + +## O + +### OpenAI Harness Engineering (OpenAI Harness 工程) +**英文**:OpenAI Harness Engineering +**中文**:OpenAI Harness 工程 +**定义**:OpenAI 团队在 5 个月内构建了超过 100 万行代码的产品,其中零行代码由人工编写,证明了 Harness 工程在生产规模上的有效性。 +**相关概念**:[[Harness-Engineering|Harness 工程]], [[Codex|Codex 模型]] + +### Observability Stack (可观测性堆栈) +**英文**:Observability Stack +**中文**:可观测性堆栈 +**定义**:OpenAI 为 Codex 提供的日志、指标和追踪记录展示系统,使智能体能够直接访问应用程序的运行状态。 +**相关概念**:[[Context-Engineering|上下文工程]] + +## P + +### Planner (规划者) +**英文**:Planner +**中文**:规划者 +**定义**:Anthropic Harness 设计中的三种角色之一,负责将简单的 1-4 句话提示扩展为完整的产品规格。 +**相关概念**:[[Generator|生成者]], [[Evaluator|评估者]] + +## R + +### Reasoning Sandwich (推理三明治) +**英文**:Reasoning Sandwich +**中文**:推理三明治 +**定义**:LangChain 采用的推理预算分配策略,在规划和验证阶段使用高推理预算,在实现阶段使用中等推理预算。 +**相关概念**:[[Build-Verify Loop|构建-验证循环]] + +### Ralph Wiggum Loop (Ralph Wiggum 循环) +**英文**:Ralph Wiggum Loop +**中文**:Ralph Wiggum 循环 +**定义**:使用钩子在智能体退出时强制其继续执行的循环模式,用于验证环节。 +**相关概念**:[[Self-Verification|自我验证]] + +## S + +### Self-Verification (自我验证) +**英文**:Self-Verification +**中文**:自我验证 +**定义**:智能体通过运行测试、阅读完整输出并与原始要求进行比较来自我改进的能力。 +**相关概念**:[[Build-Verify Loop|构建-验证循环]] + +### Self-Evolution (自我进化) +**英文**:Self-Evolution +**中文**:自我进化 +**定义**:模型深度参与自身进化的过程,包括更新自身记忆、构建复杂技能、根据实验结果改进学习过程和 Harness。 +**相关概念**:[[MiniMax M2.7|MiniMax M2.7 模型]] + +### Sensor (传感器) +**英文**:Sensor +**中文**:传感器 +**定义**:观察智能体行动后结果并帮助其自我纠正的反馈控制,包括计算型和推理型两种类型。 +**相关概念**:[[Feedback|反馈控制]] + +### Sprint Contract (冲刺契约) +**英文**:Sprint Contract +**中文**:冲刺契约 +**定义**:在每个冲刺前,Generator 和 Evaluator 协商达成的协议,定义该阶段工作的"完成"标准。 +**相关概念**:[[Anthropic-Harness-Design|Anthropic Harness 设计]] + +## T + +### Terminal Bench (终端基准测试) +**英文**:Terminal Bench +**中文**:终端基准测试 +**定义**:评估智能体编码能力的标准基准测试,包含机器学习、调试、生物学等多个领域的任务。 +**相关概念**:[[LangChain Harness Engineering|LangChain Harness 工程]] + +## W + +### Wikilink (Wikilink) +**英文**:Wikilink +**中文**:Wikilink +**定义**:使用 `[[文档标题]]` 格式的内部链接,在 Obsidian 等工具中支持双向链接和图谱视图。 +**相关概念**:[[Backlink|反向链接]], [[Glossary|术语表]] + +### WikiLLM (WikiLLM) +**英文**:WikiLLM +**中文**:WikiLLM +**定义**:本项目的名称,一个利用 LLM 构建个人知识库的系统,通过"编译"原始数据生成结构化、交叉链接的高质量中文 Wiki。 +**相关概念**:[[Glossary|术语表]], [[INDEX]] + +--- + +*最后更新:2026-04-07* +*本文档由 [[WikiLLM]] 自动生成* diff --git a/wiki/INDEX.md b/wiki/INDEX.md new file mode 100644 index 0000000..4f1be42 --- /dev/null +++ b/wiki/INDEX.md @@ -0,0 +1,84 @@ +--- +title: WikiLLM 知识库首页 +tags: [首页, 索引, 导航] +last_updated: 2026-04-07 +--- + +# WikiLLM 知识库 + +欢迎来到 **WikiLLM**——一个关于 [[Harness-Engineering|Harness 工程]] 的中文知识库。本 wiki 基于多篇权威来源编译而成,旨在为 AI 智能体时代的软件工程提供系统化的指南。 + +> **Harness 工程**是设计和实现使 AI 智能体可靠工作的系统的新学科。如果说 2025 年是 AI 智能体验证它们能够编写代码的一年,那么 2026 年就是我们认识到**智能体不是难点——Harness 才是**的一年。 + +--- + +## 📚 核心概念 + +- [[Harness-Engineering|Harness 工程]] - Harness 工程的完整概述,包括三大支柱、为什么现在重要、以及实践中的方法 +- [[Context-Engineering|上下文工程]] - 如何确保智能体在正确的时间获得正确的信息 +- [[Architectural-Constraints|架构约束]] - 如何机械地强制执行好代码的样子,而不是仅仅告诉智能体"写好代码" +- [[Anthropic-Harness-Design|Anthropic Harness 设计]] - Anthropic 的三智能体架构:Planner、Generator、Evaluator +- [[Self-Verification|自我验证]] - 让智能体通过构建-验证循环自我改进的技术 + +--- + +## 🛠️ 实践指南 + +- [[Mitchellh-Adoption-Journey|Mitchellh AI 采用之旅]] - HashiCorp 创始人从怀疑论者到深度用户的六个阶段 +- [[Building-Your-First-Harness|构建你的第一个 Harness]] - 从个人开发者到工程组织的三级实用框架 + +--- + +## 📖 学习路径 + +### 初学者路径 + +1. 首先阅读 [[Harness-Engineering|Harness 工程]] 获得概览 +2. 然后阅读 [[Mitchellh-Adoption-Journey|Mitchellh AI 采用之旅]] 了解个人采用路径 +3. 最后阅读 [[Building-Your-First-Harness|构建你的第一个 Harness]] 开始实践 + +### 深入学习路径 + +1. 从 [[Anthropic-Harness-Design|Anthropic Harness 设计]] 开始了解前沿架构 +2. 深入研究 [[Context-Engineering|上下文工程]] 和 [[Architectural-Constraints|架构约束]] +3. 学习 [[Self-Verification|自我验证]] 技术让智能体自我改进 + +--- + +## 🔗 快速导航 + +- [[Glossary|术语表]] - 40+ 核心概念的中英对照和解释 +- [概念目录](./concepts/) - 所有核心概念文章 +- [实践目录](./practices/) - 所有实践指南文章 + +--- + +## 📊 编译来源 + +本知识库基于以下权威来源编译: + +1. **OpenAI** - Harness Engineering:在智能体优先的世界中利用 Codex +2. **Anthropic** - Harness design for long-running application development +3. **Martin Fowler** - Harness engineering for coding agent users +4. **LangChain** - Improving Deep Agents with harness engineering +5. **NxCode** - Harness Engineering: The Complete Guide +6. **MiniMax** - MiniMax M2.7: Early Echoes of Self-Evolution +7. **Mitchell Hashimoto** - My AI Adoption Journey + +--- + +## 💡 关于 WikiLLM + +WikiLLM 是一个利用 LLM 构建个人知识库的系统。本项目的核心原则是: + +- **LLM 编写和维护所有 wiki 数据**;手动编辑很少见 +- **用户探索和查询被归档回 wiki** 以增强它 +- **系统专注于 markdown 文件和 Obsidian 兼容格式** +- **图像被下载到本地** 以便 LLM 轻松引用 + +查看 [[Glossary|术语表]] 了解更多核心概念,或从 [[Harness-Engineering|Harness 工程]] 开始阅读! + +--- + +*最后更新:2026-04-07* +*本文档由 [[WikiLLM]] 自动生成* diff --git a/wiki/assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png b/wiki/assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png new file mode 100644 index 0000000..8e28ca3 Binary files /dev/null and b/wiki/assets/Screenshot-2026-02-12-at-12.25.20---PM-1.png differ diff --git a/wiki/assets/Screenshot-2026-02-16-at-12.50.00---PM.png b/wiki/assets/Screenshot-2026-02-16-at-12.50.00---PM.png new file mode 100644 index 0000000..37ef0d7 Binary files /dev/null and b/wiki/assets/Screenshot-2026-02-16-at-12.50.00---PM.png differ diff --git a/wiki/assets/agent-knowledge-limits.webp b/wiki/assets/agent-knowledge-limits.webp new file mode 100644 index 0000000..afa0592 Binary files /dev/null and b/wiki/assets/agent-knowledge-limits.webp differ diff --git a/wiki/assets/d92a6eb4-a4b8-4906-b76a-d627c814a2c0.gif b/wiki/assets/d92a6eb4-a4b8-4906-b76a-d627c814a2c0.gif new file mode 100644 index 0000000..e8f5576 Binary files /dev/null and b/wiki/assets/d92a6eb4-a4b8-4906-b76a-d627c814a2c0.gif differ diff --git a/wiki/assets/fig1-codex-drives-app.webp b/wiki/assets/fig1-codex-drives-app.webp new file mode 100644 index 0000000..52a2249 Binary files /dev/null and b/wiki/assets/fig1-codex-drives-app.webp differ diff --git a/wiki/assets/harness-bounded-contexts.png b/wiki/assets/harness-bounded-contexts.png new file mode 100644 index 0000000..56a9a95 --- /dev/null +++ b/wiki/assets/harness-bounded-contexts.png @@ -0,0 +1,280 @@ + + + + + + +not found + + + + + + + +

404

+ +

I’m afraid this is not the document you’re looking for. Try using the +search box above, and good luck.

+
+ + + + + + + + + diff --git a/wiki/assets/harness-change-lifecycle-examples.png b/wiki/assets/harness-change-lifecycle-examples.png new file mode 100644 index 0000000..56a9a95 --- /dev/null +++ b/wiki/assets/harness-change-lifecycle-examples.png @@ -0,0 +1,280 @@ + + + + + + +not found + + + + + + + +

404

+ +

I’m afraid this is not the document you’re looking for. Try using the +search box above, and good luck.

+
+ + + + + + + + + diff --git a/wiki/assets/harness-continuous-feedback-examples.png b/wiki/assets/harness-continuous-feedback-examples.png new file mode 100644 index 0000000..56a9a95 --- /dev/null +++ b/wiki/assets/harness-continuous-feedback-examples.png @@ -0,0 +1,280 @@ + + + + + + +not found + + + + + + + +

404

+ +

I’m afraid this is not the document you’re looking for. Try using the +search box above, and good luck.

+
+ + + + + + + + + diff --git a/wiki/assets/harness-overview.png b/wiki/assets/harness-overview.png new file mode 100644 index 0000000..56a9a95 --- /dev/null +++ b/wiki/assets/harness-overview.png @@ -0,0 +1,280 @@ + + + + + + +not found + + + + + + + +

404

+ +

I’m afraid this is not the document you’re looking for. Try using the +search box above, and good luck.

+
+ + + + + + + + + diff --git a/wiki/assets/harness-templates.png b/wiki/assets/harness-templates.png new file mode 100644 index 0000000..56a9a95 --- /dev/null +++ b/wiki/assets/harness-templates.png @@ -0,0 +1,280 @@ + + + + + + +not found + + + + + + + +

404

+ +

I’m afraid this is not the document you’re looking for. Try using the +search box above, and good luck.

+
+ + + + + + + + + diff --git a/wiki/assets/harness-types.png b/wiki/assets/harness-types.png new file mode 100644 index 0000000..56a9a95 --- /dev/null +++ b/wiki/assets/harness-types.png @@ -0,0 +1,280 @@ + + + + + + +not found + + + + + + + +

404

+ +

I’m afraid this is not the document you’re looking for. Try using the +search box above, and good luck.

+
+ + + + + + + + + diff --git a/wiki/assets/img-1.png b/wiki/assets/img-1.png new file mode 100644 index 0000000..77bbb22 Binary files /dev/null and b/wiki/assets/img-1.png differ diff --git a/wiki/assets/img-2.png b/wiki/assets/img-2.png new file mode 100644 index 0000000..4550042 Binary files /dev/null and b/wiki/assets/img-2.png differ diff --git a/wiki/assets/img-3.png b/wiki/assets/img-3.png new file mode 100644 index 0000000..80ff7f3 Binary files /dev/null and b/wiki/assets/img-3.png differ diff --git a/wiki/assets/langsmith_trace_analyzer_skill.png b/wiki/assets/langsmith_trace_analyzer_skill.png new file mode 100644 index 0000000..f990b8e Binary files /dev/null and b/wiki/assets/langsmith_trace_analyzer_skill.png differ diff --git a/wiki/assets/layered-domain-architecture.webp b/wiki/assets/layered-domain-architecture.webp new file mode 100644 index 0000000..d9bccba Binary files /dev/null and b/wiki/assets/layered-domain-architecture.webp differ diff --git a/wiki/assets/observability-stack.svg b/wiki/assets/observability-stack.svg new file mode 100644 index 0000000..5584893 --- /dev/null +++ b/wiki/assets/observability-stack.svg @@ -0,0 +1,129 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/wiki/assets/self-verification-loop.png b/wiki/assets/self-verification-loop.png new file mode 100644 index 0000000..56a4f6d Binary files /dev/null and b/wiki/assets/self-verification-loop.png differ diff --git a/wiki/assets/the-reasoning-sandwich.png b/wiki/assets/the-reasoning-sandwich.png new file mode 100644 index 0000000..cddc86f Binary files /dev/null and b/wiki/assets/the-reasoning-sandwich.png differ diff --git a/wiki/concepts/Anthropic-Harness-Design.md b/wiki/concepts/Anthropic-Harness-Design.md new file mode 100644 index 0000000..adc4e85 --- /dev/null +++ b/wiki/concepts/Anthropic-Harness-Design.md @@ -0,0 +1,142 @@ +--- +title: Anthropic Harness 设计 +tags: [核心概念, 架构, 智能体] +source: [Anthropic] +confidence_score: 高 +last_updated: 2026-04-07 +--- + +# Anthropic Harness 设计 + +**Anthropic Harness 设计**是 Anthropic 团队 Prithvi Rajasekaran 提出的多智能体架构,用于长时间运行的应用开发。该设计受生成对抗网络(GAN)启发,使用**生成器**和**评估器**智能体的组合。 + +## 问题背景 + +### 朴素实现的不足 + +之前的工作已经表明,Harness 设计对长时间运行的智能体编码的有效性有重大影响。但对于更复杂的任务,智能体仍然倾向于随着时间推移偏离轨道。观察到两种常见的失败模式: + +1. **上下文窗口填充和"上下文焦虑"**:模型在冗长任务上随着上下文窗口填充而失去连贯性。一些模型还表现出"上下文焦虑",当它们接近认为的上下文限制时会提前结束工作。 + +2. **自我评估问题**:当被要求评估它们自己生成的作品时,智能体倾向于自信地赞扬作品——即使,对于人类观察者来说,质量明显平庸。这个问题对于主观任务(如设计)特别明显,没有二进制检查相当于可验证的软件测试。 + +## 核心架构 + +### 三智能体系统 + +Anthropic 最终构建了一个三智能体系统,每个智能体解决先前运行中观察到的特定差距: + +| 角色 | 职责 | +|------|------| +| **Planner(规划者)** | 将简单的 1-4 句话提示扩展为完整的产品规格 | +| **Generator(生成者)** | 一次一个功能地实现规格,带有 git 版本控制 | +| **Evaluator(评估器)** | 使用 Playwright MCP 测试运行中的应用,分级并提供详细反馈 | + +### 上下文重置 vs 压缩 + +**上下文重置**:清空上下文窗口并启动一个新的智能体,结合携带先前智能体状态和下一步的结构化交接。 + +**上下文压缩**:将对话的早期部分就地摘要,以便同一个智能体可以在缩短的历史上继续。 + +虽然压缩保留了连续性,但它没有给智能体一个干净的状态,这意味着上下文焦虑仍然可能存在。重置提供了一个干净的状态,但代价是交接工件必须有足够的状态让下一个智能体干净地接手工作。 + +在早期测试中,发现 Claude Sonnet 4.5 表现出足够强的上下文焦虑,以至于仅靠压缩不足以实现良好的长任务性能,因此上下文重置成为 Harness 设计的必要条件。 + +## 前端设计:使主观质量可分级 + +### 两个洞见 + +1. 虽然美学不能完全简化为分数——而且个人品味总是会变化——但它们可以通过编码设计原则和偏好的分级标准来改善。 +2. 通过将前端生成与前端分级分离,可以创建一个推动生成器朝着更强输出发展的反馈循环。 + +### 四个分级标准 + +| 标准 | 描述 | +|------|------| +| **设计质量** | 设计感觉像是一个连贯的整体而不是部分的集合吗? | +| **原创性** | 是否有自定义决策的证据,还是只是模板布局、库默认值和 AI 生成的模式? | +| **工艺** | 技术执行:排版层次结构、间距一致性、色彩和谐、对比度。 | +| **功能性** | 独立于美学的可用性。用户能理解界面做什么吗? | + +强调设计质量和原创性超过工艺和功能性。Claude 已经默认在工艺和功能性上得分很高,因为所需的技术能力往往自然而然地出现在模型中。但在设计和原创性上,Claude 经常产生充其量是乏味的输出。 + +### 生成-评估循环 + +构建在 Claude Agent SDK 上的循环: + +1. 生成器智能体首先根据用户提示创建 HTML/CSS/JS 前端 +2. 给予评估器 Playwright MCP,让它在评分每个标准并编写详细批评之前直接与实时页面交互 +3. 该反馈流回生成器作为下一次迭代的输入 +4. 每次生成运行 5 到 15 次迭代 + +在每次评估后,指示生成器做出战略决策:如果分数趋势良好则改进当前方向,或者如果方法不起作用则转向完全不同的美学。 + +## 扩展到全栈编码 + +### 架构演变 + +Opus 4.5 基本上自行消除了上下文焦虑行为,因此可以完全从 Harness 中删除上下文重置。智能体作为一个连续会话在整个构建中运行,Claude Agent SDK 的自动压缩处理沿途的上下文增长。 + +### 冲刺契约 + +在每个冲刺之前,生成器和评估器协商一个**冲刺契约**:在编写任何代码之前就该工作块的"完成"看起来像什么达成一致。 + +这存在是因为产品规格故意是高级的,需要一个步骤来弥合用户故事和可测试实现之间的差距。生成器提议它将构建什么以及如何验证成功,评估器审查该提议以确保生成器正在构建正确的东西。两者迭代直到达成一致。 + +### 结果对比 + +使用"创建一个带有关卡编辑器、精灵编辑器、实体行为和可玩测试模式的 2D 复古游戏制作器"提示进行的测试: + +| Harness | 持续时间 | 成本 | +|---------|---------|------| +| Solo | 20 分钟 | $9 | +| 完整 Harness | 6 小时 | $200 | + +Harness 贵了 20 多倍,但输出质量的差异立即显而易见。 + +## Harness 简化迭代 + +### 移除冲刺构造 + +从 Harness 中完全移除了冲刺构造。保留了规划器和评估器,因为每个都继续增加明显的价值。将评估器移动到运行结束时的单次通过,而不是每个冲刺分级。 + +### 模型改进的影响 + +Opus 4.6 的发布提供了进一步减少 Harness 复杂性的动力。Opus 4.6 更仔细地规划、更长时间地维持智能体任务、可以在更大的代码库中更可靠地操作,并且具有更好的代码审查和调试技能来捕捉自己的错误。它还在长上下文检索方面有了实质性改进。 + +### 更新后的 Harness 结果 + +使用"使用 Web Audio API 在浏览器中构建功能完整的 DAW"提示进行的测试: + +- 总持续时间:约 4 小时 +- 总成本:$124.70 + +大部分时间花在构建器上,它在没有 Opus 4.5 所需的冲刺分解的情况下连贯运行了两个多小时。 + +## 关键经验教训 + +随着模型继续改进,可以大致期望它们能够工作更长时间,处理更复杂的任务。在某些情况下,这将意味着随着时间推移,模型周围的支架变得不那么重要,开发人员可以等待下一个模型并看到某些问题自行解决。另一方面,模型越好,开发能够实现超出模型在基线水平所能做到的复杂任务的 Harness 的空间就越大。 + +有几个经验教训值得向前推进: + +1. 用你正在构建的模型进行实验总是好的做法 +2. 在处理更复杂的任务时,有时可以通过分解任务并将专门的智能体应用于问题的每个方面来获得提升 +3. 当新模型落地时,重新检查 Harness 通常是好的做法,剥离那些不再对性能有负载作用的部分,并添加新部分以实现以前可能无法实现的更大能力 + +## 相关概念 + +- [[Harness-Engineering|Harness 工程]] +- [[Generator-Evaluator Loop|生成-评估循环]] +- [[Context Reset|上下文重置]] +- [[Planner|规划者]] +- [[Generator|生成者]] +- [[Evaluator|评估者]] + +## 参考来源 + +1. Anthropic - Harness design for long-running application development + +--- + +*最后更新:2026-04-07* +*本文档由 [[WikiLLM]] 编译* diff --git a/wiki/concepts/Architectural-Constraints.md b/wiki/concepts/Architectural-Constraints.md new file mode 100644 index 0000000..45bf92f --- /dev/null +++ b/wiki/concepts/Architectural-Constraints.md @@ -0,0 +1,106 @@ +--- +title: 架构约束 +tags: [核心概念, 架构, Harness] +source: [OpenAI, Martin Fowler, NxCode] +confidence_score: 高 +last_updated: 2026-04-07 +--- + +# 架构约束 + +**架构约束**(Architectural Constraints)是[[Harness-Engineering|Harness 工程]]的三大支柱之一。这是 Harness 工程与传统 AI 提示最显著不同的地方。与其告诉智能体"编写好代码",不如**机械地强制执行好代码的样子**。 + +## 核心思想 + +### 约束反而提升生产力 + +矛盾的是,约束解决方案空间使智能体**更有生产力**,而不是更少。当智能体可以生成任何东西时,它会浪费 token 探索死胡同。当 Harness 定义清晰的边界时,智能体会更快地收敛到正确的解决方案。 + +### 为智能体优化代码库可读性 + +由于代码仓库完全由智能体生成,因此首先针对 Codex 的可读性进行了优化。就像团队会努力提升代码对新入职工程师的可导航性一样,人类工程师的目标也是让智能体能够直接从代码仓库推理出完整的业务领域。 + +从智能体的角度来看,它在运行时无法在情境中访问的任何内容都是不存在的。存储在 Google Docs、聊天记录或人们头脑中的知识都无法被系统访问。代码仓库本地的、已版本化的工件(例如,代码、Markdown、模式、可执行计划)就是它所能看到的全部。 + +## 分层领域架构 + +OpenAI 围绕一个严格的架构模型构建了应用。每个业务领域都划分为一组固定的层,依赖方向经过严格验证,并且仅允许有限的一组边。这些约束是通过自定义的 linter(当然是由 Codex 生成的!)和结构测试机械地强制执行的。 + +![分层领域架构](../assets/layered-domain-architecture.webp) + +### 依赖分层规则 + +在每个业务领域内(例如应用设置),代码只能"向前"依赖于一组固定的层: + +``` +Types → Config → Repo → Service → Runtime → UI +``` + +横切关注点(认证、连接器、遥测、功能标志)通过一个单一的显式接口进入:Providers。其他任何内容都不被允许,并将通过自动化方式强制执行。 + +这种架构通常要等到你拥有数百名工程师时才会推迟。对于编码智能体来说,这是一个早期的先决条件:有了约束,速度才不会下降,架构才不会漂移。 + +## 约束强制执行工具 + +### 1. 确定性 Linters + +自定义规则,自动标记违规。由于这些 lint 是自定义的,编写错误信息时会在智能体情境中注入修复指令。 + +在以人为本的工作流程中,这些规则可能会让人感到迂腐或束缚。有了智能体,它们就成了倍增器:一旦编码,它们就能立即应用于所有地方。 + +### 2. 基于 LLM 的审计员 + +审查其他智能体代码的架构合规性的智能体。 + +### 3. 结构测试 + +像 ArchUnit,但专门用于 AI 生成的代码。 + +### 4. 预提交钩子 + +在提交任何代码前的自动检查。 + +## 品味不变式 + +OpenAI 团队辅以一小组"品味不变式"。例如: + +- 通过自定义 lint 静态地强制执行结构化日志记录 +- 模式和类型的命名约定 +- 文件大小限制 +- 特定平台的可靠性要求 + +## 明确界限 + +同时,还明确指出了哪些地方需要限制,哪些地方不需要限制。这类似于领导一个大型工程平台组织:在中央层面强制执行边界,在本地层面允许自主权。你非常重视界限、正确性和可重复性。在这些边界内,你允许团队或智能体在解决方案的表达方式上拥有很大的自由。 + +生成的代码不总是符合人类的风格偏好,这也没关系。只要输出是正确的、可维护的,并且对未来的智能体运行而言清晰易读,就可以算作达标。 + +## 依赖选择 + +这一框架明确了许多取舍。倾向于选择那些可以完全内化于在仓库中进行推理的依赖项和抽象。对智能体来说,通常被称为"枯燥"的技术,由于其可组合性、API 稳定性和在训练集里的表现,往往更容易建立模型。 + +在某些情况下,让智能体重新实现部分功能子集比绕过公共库中不透明的上游行为更便宜。例如,OpenAI 团队没有引入通用的 p-limit 风格包,而是投入使用了他们自己的带并发的 map 辅助函数:它与他们的 OpenTelemetry 仪表紧密集成,具备 100% 的测试覆盖率,并且其行为完全符合他们的运行时预期。 + +将系统的更多部分转化为智能体可以检查、验证并直接修改的形式,可以直接提高杠杆效应——这不仅适用于 Codex,也适用于其他智能体也在参与代码库的开发。 + +## 强制执行不变量 + +仅靠文档本身,是没法保持完全由智能体生成的代码库的连贯性的。通过强制执行不变量,而非对实施过程进行微观管理,令智能体能够快速交付,而且不会削弱基础。例如,要求 Codex 在边界处解析数据形状,但不规定具体实现方式(模型似乎偏好 Zod,但没有指定特定库)。 + +## 相关概念 + +- [[Harness-Engineering|Harness 工程]] +- [[Context-Engineering|上下文工程]] +- [[Entropy Management|熵管理]] +- [[Layered Domain Architecture|分层领域架构]] + +## 参考来源 + +1. OpenAI - Harness Engineering:在智能体优先的世界中利用 Codex +2. Martin Fowler - Harness engineering for coding agent users +3. NxCode - Harness Engineering: The Complete Guide + +--- + +*最后更新:2026-04-07* +*本文档由 [[WikiLLM]] 编译自多个来源* diff --git a/wiki/concepts/Context-Engineering.md b/wiki/concepts/Context-Engineering.md new file mode 100644 index 0000000..5991da2 --- /dev/null +++ b/wiki/concepts/Context-Engineering.md @@ -0,0 +1,124 @@ +--- +title: 上下文工程 +tags: [核心概念, 上下文, Harness] +source: [OpenAI, Anthropic, LangChain] +confidence_score: 高 +last_updated: 2026-04-07 +--- + +# 上下文工程 + +**上下文工程**(Context Engineering)是[[Harness-Engineering|Harness 工程]]的三大支柱之一,专注于确保智能体在正确的时间获得正确的信息。 + +## 核心原则 + +### 给地图,而不是一千页说明书 + +OpenAI 团队学到的最早经验之一很简单:要给 Codex 的是一张地图,而不是一本 1,000 页的说明书。 + +他们尝试了"一个大型的 AGENTS.md 方法。可想而知,这是一次失败的尝试: + +- **上下文是一种稀缺资源**:一个巨大的指令文件会挤掉任务、代码和相关文档——因此智能体要么会错过关键约束条件,要么开始针对错误的约束条件进行优化。 +- **过多的指导反而变得无效**:当一切都"重要"时,一切都不重要了。智能体最终会在本地进行模式匹配,而不是有意识地进行导航。 +- **它会立即腐烂**:一本庞杂的手册会变成陈旧规则的坟场。智能体无法判断哪些信息仍然有效,一旦人类停止维护它,此文件就会悄然成为一个颇具吸引力的麻烦源头。 +- **这很难核实**:单个 blob 不适合进行机械检查(覆盖率、新鲜度、所有权、交叉链接),因此漂移是不可避免的。 + +因此,他们不再将 AGENTS.md 视为百科全书,而是将其视为内容目录。 + +## 静态上下文与动态上下文 + +### 静态上下文 + +静态上下文是仓库中不会频繁变化的文档和规范: + +- **仓库本地文档**:架构规范、API 契约、风格指南 +- **AGENTS.md 或 CLAUDE.md 文件**:编码项目特定规则 +- **交叉链接设计文档**:由 linter 验证的设计文档 + +### 动态上下文 + +动态上下文是随时间变化的运行时信息: + +- **可观测性数据**:日志、指标、追踪——智能体可访问 +- **目录结构映射**:智能体启动时的目录映射 +- **CI/CD 流水线状态**:测试结果和流水线状态 + +## 关键规则 + +**从智能体的角度来看,它无法在上下文中访问的任何内容都不存在。** + +存储在 Google Docs、Slack 线程或人们头脑中的知识对系统是不可见的。**仓库必须是单一事实来源。** + +### 渐进式披露 + +这个框架实现了渐进式披露:智能体从一个小而稳定的切入点开始,并被指导下一步该去哪里查看,而不是一开始就被淹没。 + +OpenAI 团队严格执行这一点。专职的 linter 和 CI 作业会验证知识库的更新状况、是否已交叉链接且结构正确。一个定期运行的"doc-gardening"智能体会扫描那些不再反映真实代码行为的过时或废弃文档,并发起修复用的 Pull Request。 + +## 实践中的上下文工程 + +### OpenAI 的知识库布局 + +``` +AGENTS.md +ARCHITECTURE.md +docs/ +├── design-docs/ +│ ├── index.md +│ ├── core-beliefs.md +│ └── ... +├── exec-plans/ +│ ├── active/ +│ ├── completed/ +│ └── tech-debt-tracker.md +├── generated/ +│ └── db-schema.md +├── product-specs/ +│ ├── index.md +│ ├── new-user-onboarding.md +│ └── ... +├── references/ +│ ├── design-system-reference-llms.txt +│ ├── nixpacks-llms.txt +│ ├── uv-llms.txt +│ └── ... +├── DESIGN.md +├── FRONTEND.md +├── PLANS.md +├── PRODUCT_SENSE.md +├── QUALITY_SCORE.md +├── RELIABILITY.md +└── SECURITY.md +``` + +设计文档已被编目和索引,其中包括验证状态和一套核心理念,定义了智能体优先的操作原则。架构文档提供域和包分层的顶层地图。一份高质量的文档会对每个产品领域和架构层进行评分,并随着时间的推移追踪差距。 + +计划被视为一流的工件。临时轻量计划用于小幅变更,而复杂工作则记录在执行计划中,并附带进度和决策日志,这些日志会被提交到代码仓库。活跃计划、已完成计划和已知的技术债务都已进行版本控制并集中存放,使智能体能够在不依赖外部情境的情况下运行。 + +### LangChain 的 LocalContextMiddleware + +LangChain 使用 `LocalContextMiddleware` 在智能体启动时运行,映射 `cwd` 和其他父+子目录。他们运行 `bash` 命令来查找工具如 `Python` 安装。上下文发现和搜索容易出错,因此注入上下文减少了这个错误表面并帮助**将智能体引导到其环境中。** + +## 将更多情境推送到仓库中 + +随着时间的推移,需要将越来越多的情境推送到仓库中。那次让团队在架构模式上达成一致的 Slack 讨论?如果智能体无法发现它,那么它就会像迟了三个月入职的新员工一样,对其一无所知。 + +为 Codex 提供更多情境意味着要组织和展示正确的信息,好令智能体能够基于这些信息进行推理,而不是用临时指令使其不堪重负。就像你会在产品原则、工程规范和团队文化(包括表情符号偏好)方面为新队友提供引导一样,将这些信息提供给智能体会带来更一致的输出。 + +## 相关概念 + +- [[Harness-Engineering|Harness 工程]] +- [[Context Reset|上下文重置]] +- [[Context Anxiety|上下文焦虑]] +- [[Architectural-Constraints|架构约束]] + +## 参考来源 + +1. OpenAI - Harness Engineering:在智能体优先的世界中利用 Codex +2. LangChain - Improving Deep Agents with harness engineering +3. Anthropic - Effective context engineering for AI agents + +--- + +*最后更新:2026-04-07* +*本文档由 [[WikiLLM]] 编译自多个来源* diff --git a/wiki/concepts/Harness-Engineering.md b/wiki/concepts/Harness-Engineering.md new file mode 100644 index 0000000..fe3cfe4 --- /dev/null +++ b/wiki/concepts/Harness-Engineering.md @@ -0,0 +1,137 @@ +--- +title: Harness 工程 +tags: [核心概念, Harness, 架构] +source: [OpenAI, Anthropic, Martin Fowler, LangChain, NxCode] +confidence_score: 高 +last_updated: 2026-04-07 +--- + +# Harness 工程 + +**Harness 工程**(Harness Engineering)是设计和实现使 AI 智能体可靠工作的系统的新学科。如果说 2025 年是 AI 智能体验证它们能够编写代码的一年,那么 2026 年就是我们认识到**智能体不是难点——Harness 才是**的一年。 + +## 核心定义 + +### 什么是 Harness? + +术语 "Harness" 来源于马具——缰绳、马鞍、马嚼子——用于引导强大但不可预测的动物朝正确方向前进的全套设备。这个比喻是有意为之的: + +- **马**是 AI 模型——强大、快速,但自己不知道要去哪里 +- **Harness**是基础设施——约束、护栏、反馈循环,用于高效地引导模型的能力 +- **骑手**是人类工程师——提供方向,而不是亲自奔跑 + +没有 Harness,AI 智能体就像开放田野里的纯种马。快速、令人印象深刻,但对于完成任何事情来说完全无用。 + +### Harness 工程的正式定义 + +**Harness 工程**是设计和实现以下系统的学科: + +1. **约束** AI 智能体可以做什么(架构边界、依赖规则) +2. **告知** 智能体应该做什么(上下文工程、文档) +3. **验证** 智能体正确执行了(测试、linting、CI 验证) +4. **纠正** 智能体出错时(反馈循环、自我修复机制) + +Martin Fowler 将其描述为"我们可以用来控制 AI 智能体的工具和实践"——但它不仅仅是安全。一个好的 Harness 使智能体**更有能力**,而不仅仅是更受控制。 + +## 为什么 Harness 工程现在如此重要 + +### 模型是商品,Harness 是护城河 + +AI 行业正面临一个令人不安的事实:**底层模型的重要性不如围绕它的系统。** + +LangChain 明确证明了这一点。他们的编码智能体在 Terminal Bench 2.0 上从 **52.8% 提升到 66.5%**——从 **Top 30 跃升至 Top 5**——对模型没有做任何改变。他们只改变了 Harness: + +| 改变 | 他们做了什么 | 影响 | +|------|-------------|------| +| 自我验证循环 | 添加完成前检查清单中间件 | 在提交前捕获错误 | +| 上下文工程 | 启动时映射目录结构 | 智能体从一开始就理解代码库 | +| 循环检测 | 跟踪重复的文件编辑 | 防止"末日循环" | +| 推理三明治 | 规划/验证使用高推理,实现使用中等推理 | 在时间预算内获得更好的质量 | + +**相同的模型。不同的 Harness。显著更好的结果。** + +### OpenAI 的 100 万行代码证明点 + +OpenAI 的实验是迄今为止最令人信服的证据: + +- **5 个月**的开发 +- 最终产品中有 **100 万+ 行代码** +- **零行手动编写的代码**——每一行都是由 Codex 智能体生成的 +- **用人类所需时间的约 1/10 构建** +- 产品有**内部日常用户和外部 Alpha 测试者** +- 它**交付、部署、出故障并得到修复**——全部由 Harness 内的智能体完成 + +工程师的工作?设计 Harness。指定意图。提供反馈。不是编写代码。 + +## 三大支柱 + +OpenAI 的框架将 Harness 工程组织为三个核心类别: + +### 1. 上下文工程 + +上下文工程是关于确保智能体在正确的时间获得正确的信息。 + +**静态上下文:** +- 仓库本地文档(架构规范、API 契约、风格指南) +- 编码项目特定规则的 `AGENTS.md` 或 `CLAUDE.md` 文件 +- 由 linter 验证的交叉链接设计文档 + +**动态上下文:** +- 智能体可访问的可观测性数据(日志、指标、追踪) +- 智能体启动时的目录结构映射 +- CI/CD 流水线状态和测试结果 + +**关键规则:** 从智能体的角度来看,它无法在上下文中访问的任何内容都不存在。Google Docs、Slack 线程或人们头脑中的知识对系统是不可见的。**仓库必须是单一事实来源。** + +### 2. 架构约束 + +这是 Harness 工程与传统 AI 提示最显著不同的地方。与其告诉智能体"编写好代码",不如**机械地强制执行好代码的样子**。 + +**依赖分层:** +``` +Types → Config → Repo → Service → Runtime → UI +``` + +每层只能从其左侧的层导入。这不是建议——它由结构测试和 CI 验证强制执行。 + +**约束强制执行工具:** +- **确定性 linters**——自动标记违规的自定义规则 +- **基于 LLM 的审计员**——审查其他智能体代码的架构合规性 +- **结构测试**——像 ArchUnit,但用于 AI 生成的代码 +- **预提交钩子**——在提交任何代码前的自动检查 + +**为什么约束能改善输出:** 矛盾的是,约束解决方案空间使智能体**更有生产力**,而不是更少。当智能体可以生成任何东西时,它会浪费 token 探索死胡同。当 Harness 定义清晰的边界时,智能体会更快地收敛到正确的解决方案。 + +### 3. 熵管理("垃圾回收") + +这是最被低估的组件。随着时间的推移,AI 生成的代码库会积累熵——文档与现实脱节、命名约定分歧、死代码堆积。 + +Harness 工程通过**定期清理智能体**来解决这个问题: +- **文档一致性智能体**——验证文档与当前代码匹配 +- **约束违规扫描器**——找到逃过早期检查的代码 +- **模式强制执行智能体**——识别并修复与既定模式的偏差 +- **依赖审计员**——跟踪并解决循环或不必要的依赖 + +这些智能体按计划运行——每日、每周,或由特定事件触发——保持代码库对人类审查者和未来 AI 智能体都健康。 + +## 相关概念 + +- [[Context-Engineering|上下文工程]] +- [[Architectural-Constraints|架构约束]] +- [[Entropy Management|熵管理]] +- [[OpenAI Harness Engineering|OpenAI Harness 工程]] +- [[Anthropic-Harness-Design|Anthropic Harness 设计]] +- [[LangChain Harness Engineering|LangChain Harness 工程]] + +## 参考来源 + +1. OpenAI - Harness Engineering:在智能体优先的世界中利用 Codex +2. Anthropic - Harness design for long-running application development +3. Martin Fowler - Harness engineering for coding agent users +4. LangChain - Improving Deep Agents with harness engineering +5. NxCode - Harness Engineering: The Complete Guide + +--- + +*最后更新:2026-04-07* +*本文档由 [[WikiLLM]] 编译自多个来源* diff --git a/wiki/concepts/Self-Verification.md b/wiki/concepts/Self-Verification.md new file mode 100644 index 0000000..e013254 --- /dev/null +++ b/wiki/concepts/Self-Verification.md @@ -0,0 +1,110 @@ +--- +title: 自我验证 +tags: [核心概念, 验证, Harness] +source: [LangChain, Anthropic] +confidence_score: 高 +last_updated: 2026-04-07 +--- + +# 自我验证 + +**自我验证**(Self-Verification)是[[Harness-Engineering|Harness 工程]]中的关键技术,使智能体能够通过运行中的反馈自我改进。当今的模型是卓越的自我改进机器,但它们没有自然地倾向于进入这种**构建-验证循环**。 + +## 核心概念 + +### 最常见的失败模式 + +LangChain 团队观察到的最常见失败模式是:智能体编写解决方案、重读自己的代码、确认看起来没问题,然后停止。 + +测试是自主智能体编码的关键部分。它有助于测试整体正确性,同时为智能体提供信号来进行爬山优化。 + +## 构建-验证循环 + +LangChain 向系统提示中添加了关于如何解决问题的指导: + +### 阶段 1:规划与发现 +阅读任务、扫描代码库,并基于任务规格和如何验证解决方案构建初始计划。 + +### 阶段 2:构建 +在考虑验证的情况下实现计划。如果测试不存在则构建测试,并测试愉快路径和边缘情况。 + +### 阶段 3:验证 +运行测试、阅读完整输出、与要求的内容进行比较(而不是与你自己的代码)。 + +### 阶段 4:修复 +分析任何错误、重新访问原始规格,并修复问题。 + +## LangChain 的实现 + +### PreCompletionChecklistMiddleware + +除了提示外,确定性上下文注入有助于智能体验证它们的工作。LangChain 使用 `PreCompletionChecklistMiddleware` 在智能体退出前拦截它,并提醒它对任务规格运行验证通过。 + +这类似于 [Ralph Wiggum Loop|Ralph Wiggum 循环]],其中一个钩子在智能体退出时强制其继续执行,用于验证。 + +![自我验证循环](../assets/self-verification-loop.png) + +### 强调测试 + +LangChain 真正专注于测试,因为它在每次迭代中推动变化。他们发现,测试必须是智能体工作流程的核心部分,而不是事后想法。 + +## 环境上下文工程 + +Harness 工程的一部分是**为上下文工程构建良好的交付机制。** Terminal Bench 任务带有目录结构、内置工具和严格的超时。 + +### 1. 目录上下文与工具 +`LocalContextMiddleware` 在智能体启动时运行,映射 `cwd` 和其他父+子目录。运行 `bash` 命令来查找工具如 `Python` 安装。上下文发现和搜索容易出错,因此注入上下文减少了这个错误表面并帮助**将智能体引导到其环境中。** + +### 2. 教导智能体编写可测试的代码 +智能体不知道它们的代码需要如何可测试。添加提示说它们的工作将根据程序化测试来衡量,类似于提交代码时。例如,提及文件路径的任务规格应该被精确遵循,以便解决方案在自动化评分步骤中工作。强调边缘情况的提示有助于智能体避免仅检查"愉快路径"情况。强制模型符合测试标准是避免随着时间推移"烂泥堆积"的强大策略。 + +### 3. 时间预算 +注入时间预算警告以推动智能体完成工作并转向验证。智能体在时间估计方面是出了名的糟糕,因此这种启发式在这种环境中有所帮助。现实世界编码通常没有严格的时间限制,但如果不添加任何约束知识,智能体就不会在时间范围内工作。 + +## 循环检测 + +智能体一旦决定了一个计划就可能变得短视,导致"末日循环",对同一破碎方法进行小幅变异(在某些轨迹中超过 10 次以上)。 + +LangChain 使用 `LoopDetectionMiddleware`,通过工具调用钩子跟踪每个文件的编辑次数。它在对同一文件进行 `N` 次编辑后添加上下文如"……考虑重新考虑你的方法"。这可以帮助智能体从末日循环中恢复,尽管如果模型认为正确的话它可以继续沿着相同的路径前进。 + +重要提示:这是一种设计启发式,围绕当今感知的模型问题进行工程设计。随着模型改进,这些护栏可能会变得不必要,但今天帮助智能体正确和自主地执行。 + +## 推理三明治 + +推理模型可以自主运行数小时,因此必须决定在每个子任务上花费多少计算。你可以在每个任务上使用最大推理预算,但大多数工作可以从优化推理计算支出中受益。 + +Terminal Bench 超时限制创造了一个权衡。更多推理有助于智能体评估每个步骤,但可以燃烧超过 2 倍以上的 token/时间。`gpt-5.2-codex` 有 4 种推理模式,`low`、`medium`、`high` 和 `xhigh`。 + +LangChain 发现推理有助于规划以充分理解问题,一些 Terminal Bench 任务非常困难。一个好的计划有助于更快地得到可行的解决方案。 + +后期验证也从更多推理中受益,以捕获错误并提交解决方案。作为一种启发式,选择 xhigh-high-xhigh"**推理三明治**"作为基线。 + +![推理三明治](../assets/the-reasoning-sandwich.png) + +**在规划和验证上花费更多的推理计算** + +仅在 `xhigh` 运行由于智能体超时而得分较差,为 `53.9%`,相比之下在 `high` 为 `63.6%`。在推理预算分割的试验运行中没有大的差异,因此坚持使用他们的方法,将分数推到 `66.5%`。 + +## Harness 工程师的目的 + +**Harness 工程师的目的:准备和交付上下文,使智能体能够自主完成工作。** + +智能体对其环境、约束和评估标准了解得越多,它们就能越好地自主自我指导工作。 + +## 相关概念 + +- [[Build-Verify Loop|构建-验证循环]] +- [[Reasoning Sandwich|推理三明治]] +- [[Doom Loop|末日循环]] +- [[Loop Detection|循环检测]] +- [[LangChain Harness Engineering|LangChain Harness 工程]] + +## 参考来源 + +1. LangChain - Improving Deep Agents with harness engineering +2. Anthropic - Harness design for long-running application development + +--- + +*最后更新:2026-04-07* +*本文档由 [[WikiLLM]] 编译自多个来源* diff --git a/wiki/practices/Building-Your-First-Harness.md b/wiki/practices/Building-Your-First-Harness.md new file mode 100644 index 0000000..2fb2a24 --- /dev/null +++ b/wiki/practices/Building-Your-First-Harness.md @@ -0,0 +1,201 @@ +--- +title: 构建你的第一个 Harness +tags: [实践指南, 入门, Harness] +source: [NxCode, OpenAI, Martin Fowler] +confidence_score: 高 +last_updated: 2026-04-07 +--- + +# 构建你的第一个 Harness + +本文提供了构建 Harness 的实用框架,从个人开发者到工程组织的三个级别。 + +--- + +## Level 1:基础 Harness(单个开发者) + +如果你正在为个人项目使用 Claude Code、Cursor 或 Codex: + +### 需要设置的内容 + +- `CLAUDE.md` 或 `.cursorrules` 文件,包含项目约定 +- 用于 linting 和格式化的预提交钩子 +- 智能体可以运行以自我验证的测试套件 +- 具有一致命名的清晰目录结构 + +**设置时间:** 1-2 小时 +**影响:** 防止最常见的智能体错误 + +### 快速入门清单 + +1. **创建 CLAUDE.md** + ```markdown + # 项目约定 + + ## 代码风格 + - 使用 TypeScript,严格模式 + - 使用 2 空格缩进 + - 函数名使用驼峰命名 + + ## 测试 + - 所有新代码必须有测试 + - 提交前运行 `npm test` + + ## 目录结构 + - src/ - 源代码 + - tests/ - 测试文件 + - docs/ - 文档 + ``` + +2. **设置预提交钩子** + ```bash + # 使用 husky 或 pre-commit + npm install husky --save-dev + npx husky install + ``` + +3. **确保测试存在** + - 即使是简单的集成测试也比没有好 + - 智能体需要可以运行的东西来验证其工作 + +--- + +## Level 2:团队 Harness(小团队) + +对于共享代码库的 3-10 名开发者的团队: + +### 在 Level 1 基础上添加 + +- 包含团队范围约定的 `AGENTS.md` +- CI 强制执行的架构约束 +- 常见任务的共享提示模板 +- 由 linter 验证的文档即代码 +- 专门针对智能体生成的 PR 的代码审查清单 + +**设置时间:** 1-2 天 +**影响:** 整个团队的智能体行为一致 + +### AGENTS.md 结构建议 + +```markdown +# AGENTS.md + +本文档包含智能体在此代码库中工作的约定。 + +## 1. 架构原则 +- 我们使用分层架构:Types → Config → Repo → Service → API +- 不要跨层导入 +- 所有外部依赖通过 Providers 访问 + +## 2. 编码标准 +- [具体规则...] + +## 3. 测试策略 +- [具体指南...] + +## 4. 审查清单 +- [智能体在提交前应检查的内容...] +``` + +### 共享提示模板 + +创建一个 `prompts/` 目录,包含常见任务: +- `prompts/add-new-feature.md` +- `prompts/fix-bug.md` +- `prompts/write-tests.md` +- `prompts/refactor-code.md` + +--- + +## Level 3:生产 Harness(工程组织) + +对于运行数十个并发智能体的组织: + +### 在 Level 2 基础上添加 + +- 自定义中间件层(循环检测、推理优化) +- 可观测性集成(智能体读取日志和指标) +- 计划运行的熵管理智能体 +- Harness 版本控制和 A/B 测试 +- 智能体性能监控仪表板 +- 智能体陷入困境时的升级策略 + +**设置时间:** 1-2 周 +**影响:** 智能体作为自主贡献者运作 + +--- + +## 常见 Harness 工程错误 + +### 1. 过度工程化控制流 + +> "如果你过度工程化控制流,下一个模型更新会破坏你的系统。" + +模型快速改进。2024 年需要复杂管道的能力现在由单个上下文窗口提示处理。构建你的 Harness 为**可剥离的**——当模型变得足够智能不需要时,你应该能够移除"智能"逻辑。 + +### 2. 将 Harness 视为静态的 + +Harness 需要随模型一起演进。当新模型版本改进推理时,你的推理优化中间件可能会适得其反。每次重大模型更新时审查和更新 Harness 组件。 + +### 3. 忽略文档层 + +最有影响力的 Harness 改进通常是最简单的:**更好的文档**。如果你的 `AGENTS.md` 含糊,你的智能体输出也会含糊。投资于精确、机器可读的文档,作为智能体的地面真理。 + +### 4. 没有反馈循环 + +没有反馈的 Harness 是一个笼子,而不是指南。智能体需要知道它何时成功何时失败。内置: +- 任务完成前的自我验证步骤 +- 作为智能体工作流一部分的测试执行 +- 按任务类型的智能体成功率指标 + +### 5. 仅人类可读的文档 + +如果你的架构决策存在于人们的头脑中或智能体无法访问的 Confluence 页面中,Harness 就有差距。**智能体需要的一切都必须在仓库中。** + +--- + +## 从哪里开始 + +### 如果你是单个开发者 + +1. 从 Level 1 开始 +2. 创建一个简单的 `CLAUDE.md` +3. 设置基础预提交钩子 +4. 添加一个简单的测试套件 +5. 迭代——观察智能体失败的地方并加以修复 + +### 如果你是一个团队 + +1. 首先就 Level 1 基础达成一致 +2. 一起编写 `AGENTS.md` 的初稿 +3. 识别最常见的 3 个智能体失败模式 +4. 为这些模式构建前 3 个约束/检查 +5. 每 2 周回顾一次什么有效/什么无效 + +### 如果你是一个组织 + +1. 从一些试点团队开始 +2. 收集什么有效的模式 +3. 构建可重用的 Harness 组件库 +4. 投资于可观测性和指标 +5. 建立 Harness 迭代的反馈循环 + +--- + +## 相关概念 + +- [[Harness-Engineering|Harness 工程]] +- [[Context-Engineering|上下文工程]] +- [[Architectural-Constraints|架构约束]] +- [[Mitchellh-Adoption-Journey|Mitchellh AI 采用之旅]] + +## 参考来源 + +1. NxCode - Harness Engineering: The Complete Guide +2. OpenAI - Harness Engineering:在智能体优先的世界中利用 Codex +3. Martin Fowler - Harness engineering for coding agent users + +--- + +*最后更新:2026-04-07* +*本文档由 [[WikiLLM]] 编译自多个来源* diff --git a/wiki/practices/Mitchellh-Adoption-Journey.md b/wiki/practices/Mitchellh-Adoption-Journey.md new file mode 100644 index 0000000..561046e --- /dev/null +++ b/wiki/practices/Mitchellh-Adoption-Journey.md @@ -0,0 +1,150 @@ +--- +title: Mitchellh AI 采用之旅 +tags: [实践指南, 采用路径, 个人工作流] +source: [Mitchell Hashimoto] +confidence_score: 高 +last_updated: 2026-04-07 +--- + +# Mitchellh AI 采用之旅 + +本文总结了 HashiCorp 创始人 Mitchell Hashimoto 的个人 AI 工具采用历程。他分享了从怀疑论者到深度用户的六个阶段,为其他开发者提供了实用的参考路径。 + +## 核心观点 + +> "我采用任何有意义的工具的经验是,我必然经历三个阶段:(1) 低效率时期 (2) 胜任时期,最后 (3) 工作流和生活改变发现时期。" + +--- + +## 六个阶段 + +### 阶段 1:放弃聊天机器人 + +**立即停止尝试通过聊天机器人执行有意义的工作**(例如 ChatGPT、网页版 Gemini 等)。 + +聊天机器人确实有价值,而且是 Mitchell AI 工作流的日常组成部分,但它们在编码中的效用非常有限,因为你大多希望它们根据先前训练得出正确结果,而纠正它们涉及人类(你)反复告诉它们错了。这是低效的。 + +**关键点:** +- 每个人的第一次 AI 体验都是聊天界面 +- 每个人第一次尝试用 AI 编码都是要求聊天界面写代码 +- 要找到价值,你**必须**使用**智能体**(Agent) + +**什么是智能体?** +智能体是行业采用的术语,指能够聊天并循环调用外部行为的 LLM。至少,智能体必须具备以下能力: +- 读取文件 +- 执行程序 +- 发起 HTTP 请求 + +--- + +### 阶段 2:重现你自己的工作 + +Mitchell 最初尝试 Claude Code 时并没有留下深刻印象。他只是没有从会话中得到好的结果。他觉得必须润色它生成的所有内容,这个过程比他自己做要花更多时间。 + +**解决方案:** +> "我没有放弃,而是**强迫自己用智能体重现所有手动提交**。我真的做了两次工作。我会手动完成工作,然后与智能体斗争以产生相同的质量和功能结果(当然,不让它看到我的手动解决方案)。" + +**这是痛苦的**,因为它妨碍了简单地完成事情。但 Mitchell 从第一性原理中自己发现了别人已经在说的东西,但自己发现它产生了更强的基本理解: + +1. 将会话分解为单独清晰、可操作的任务。不要试图在一个大型会话中"画猫头鹰"。 +2. 对于模糊的请求,将工作分成单独的规划与会话。 +3. 如果你给智能体验证其工作的方法,它往往会修复自己的错误并防止回归。 + +更一般地说,他还发现了智能体当时擅长什么、不擅长什么的边界,以及对于它们擅长的任务如何实现想要的结果。 + +所有这些都导致了显著的效率提升,以至于他开始自然地使用智能体,感觉不比自己做慢(但他仍然不觉得更快,因为他主要在照看智能体)。 + +--- + +### 阶段 3:日终智能体 + +为了尝试找到一些效率,Mitchell 开始了一个新模式:**每天划出最后 30 分钟来启动一个或多个智能体。** + +他的假设是,如果智能体能在他无论如何都不能工作的时间里取得一些**积极进展**,也许他可以获得一些效率。基本上:与其试图在他拥有的时间里做更多,不如尝试在他没有的时间里做更多。 + +**发现的有价值的工作类别:** + +1. **深度研究会话**:要求智能体调查某个领域,例如查找特定许可证类型的特定语言的所有库,并为每个库生成关于其优缺点、开发活动、社会情绪等的多页摘要。 + +2. **并行智能体尝试他没有时间开始的不同模糊想法**:不期望它们产生他会交付的东西,但也许可以在第二天他处理任务时阐明一些未知的未知。 + +3. **问题和 PR 分类/审查**:智能体擅长使用 `gh`(GitHub CLI),因此手动编写了一个快速方法来并行启动一堆来分类问题。不会允许智能体回复,只想要第二天的报告来尝试引导他走向高价值或低工作量的任务。 + +**结果:** +他开始感觉自己比 AI 之前做的更多,哪怕只是一点点。 + +--- + +### 阶段 4:外包稳操胜券的任务 + +到了这个阶段,Mitchell 对他的 AI 擅长和不擅长什么任务非常有信心。他对某些任务 AI 会实现基本正确的解决方案有非常高的信心。 + +**所以旅程的下一步是:让智能体做所有这些工作,而他做其他任务。** + +更具体地说: +- 每天从查看前一晚分类智能体的结果开始 +- 手动过滤以找到智能体几乎肯定会很好解决的问题 +- 让它们在后台继续(一次一个,不是并行) + +**同时,他做其他事情。** 不是去社交媒体(比平时不用 AI 更多),不是看视频等。他处于自己的、正常的、AI 之前的深度思考模式,处理他想做或必须做的事情。 + +**非常重要:关闭智能体桌面通知。** 上下文切换非常昂贵。为了保持效率,他发现作为人类控制何时中断智能体是他的工作,而不是相反。不要让智能体通知你。在工作的自然休息时间,切换标签检查它,然后继续。 + +**结果:** +他坚定地处于"我无法回去"的境地。他觉得更有效率,但即使不是,他最喜欢的是他现在可以将编码和思考集中在他真正喜欢的任务上,同时仍然充分完成他不喜欢的任务。 + +--- + +### 阶段 5:设计 Harness + +冒着陈述显而易见的风险:智能体在第一次产生正确结果时效率要高得多,或者最坏情况下产生需要最少润色的结果。实现这一点的最可靠方法是给智能体快速、高质量的工具来自动告诉它什么时候错了。 + +Mitchell 不知道是否有一个广泛的行业接受的术语,但他已经逐渐称之为"**Harness 工程**"。这是这样一种想法:任何时候你发现智能体犯了错误,你都花时间设计一个解决方案,使智能体永远不会再犯那个错误。 + +**这有两种形式:** + +1. **更好的隐式提示(AGENTS.md)**:对于简单的事情,比如智能体反复运行错误的命令或找到错误的 API,更新 `AGENTS.md`(或等效文件)。每一行都是基于坏的智能体行为,并且几乎完全解决了所有问题。 + +2. **实际的编程工具**:例如,截图脚本、运行过滤测试等。这通常与 AGENTS.md 更改配对,让它知道这存在。 + +**这就是 Mitchell 今天所处的位置。** 他正在真诚地努力,每当看到智能体做坏事时,防止它再做那件坏事。或者,相反,他正在真诚地努力让智能体验证它们在做好事。 + +--- + +### 阶段 6:始终运行一个智能体 + +与阶段 5 同时,Mitchell 也在**始终运行一个智能体**的目标下运作。如果智能体没有运行,他会问自己:"现在有什么智能体可以为我做的吗?" + +他特别喜欢将其与较慢、更深思熟虑的模型结合使用,比如 Amp 的 [deep mode](https://ampcode.com/news/deep-mode)(基本上只是 GPT-5.2-Codex),这可能需要 30 多分钟来做小的更改。另一面是它确实倾向于产生非常好的结果。 + +**重要说明:** +- 他还没有运行多个智能体,目前也真的不想。 +- 他觉得现在有一个智能体运行是一个很好的平衡,既能做他觉得愉快的深度手动工作,又能照看他有点愚蠢但又神秘高效的机器人朋友。 + +"始终运行一个智能体"的目标仍然只是一个目标。他会说现在他在正常工作日的 10 到 20% 时间里有效地运行后台智能体。但他正在积极努力改善这一点。 + +> "我不想为了运行智能体而运行智能体。我只希望在有任务我认为对我真正有帮助时运行它们。这个目标的一部分挑战是改善我自己的工作流和工具,以便我能有源源不断的高质量工作可以委托。即使没有 AI,这也很重要!" + +--- + +## 今天 + +这就是 Mitchell 今天所处的位置。 + +通过这段旅程,他个人已经达到了一个点,他在现代 AI 工具上取得了成功,并且相信他正在以基于现实的适当衡量观点来处理它。他真的不在乎 AI 是否会留下来,他是一个软件工匠,只是为了热爱而构建东西。 + +整个格局变化如此之快,他相信他会很快回头看这篇帖子,嘲笑自己的天真。但正如他们所说,如果你不能为过去的自己感到尴尬,你可能就没有成长。他只希望他会朝着正确的方向成长! + +## 相关概念 + +- [[Harness-Engineering|Harness 工程]] +- [[Agent|智能体]] + +## 参考来源 + +1. Mitchell Hashimoto - My AI Adoption Journey + +--- + +*最后更新:2026-04-07* +*本文档由 [[WikiLLM]] 编译*