主页
30 秒速览The 30-second version
Harness(直译「挽具」)是把一个语言模型接到真实世界的那层运行时——它负责循环调用模型、把模型说的「我要用这个工具」变成真的执行、管理上下文、加安全闸门、编排多个 agent、并提供扩展点。模型是发动机,harness 是传动系统与仪表盘。
A harness is the runtime that couples a language model to the world — it runs the loop that calls the model, turns "I want to use this tool" into an actual execution, manages context, enforces safety controls, orchestrates multiple agents, and exposes extension surfaces. The model is the engine; the harness is the drivetrain and the dashboard.
2026 年 9 月 1–2 日,arXiv 上同期出现两篇把 harness 当作独立工程对象研究的论文,其中一篇对 11 个生产级编码 harness 做了源码级解剖(约 400 万行代码)。它的核心论断是:「2026 年上半年,编码 harness 完成了从工具到平台的转身。」
On 1–2 September 2026, two papers treating the harness as an engineering object in its own right appeared on arXiv within a day of each other. One of them performs a source-code anatomy of eleven production coding harnesses (roughly four million lines). Its thesis: "in the first half of 2026 the coding harness completed a turn from tool to platform."
这份文档要回答五个问题:harness 到底是什么、它在补模型的哪些短板、它由哪些子系统构成、什么时候该往里加东西又什么时候该拆、以及如果你自己在搭一个(哪怕只是一套 Claude Code 的 skills + hooks + CLAUDE.md),该照着什么清单检查。
Five questions this document answers: what a harness actually is, which model weaknesses it compensates for, which subsystems it decomposes into, when to add components and when to strip them out, and — if you are building one yourself, even just a set of Claude Code skills, hooks and a CLAUDE.md — what checklist to work through.
Harness 是什么What a harness is
先讲一个大部分人已经有的直觉,再给正式定义。
Start with an intuition most people already have, then get to the formal definition.
你在网页上和一个模型对话,它只能说话。你在 Claude Code 里和同一个模型对话,它能读文件、跑测试、开 PR、连着干两小时。模型是同一个,差别全在外面那一层。那一层就是 harness。
Talk to a model in a web chat box and all it can do is talk. Talk to the same model inside Claude Code and it reads files, runs tests, opens PRs, and works for two hours straight. The model is identical; the entire difference is the layer around it. That layer is the harness.
Harness 这个词的本义是「挽具」——套在马身上、把马的力气传到车上的那套皮带和杆子。马再强,没有挽具也拉不动车;挽具设计得差,马的力气会浪费在打滑和别劲上。但反过来也成立:如果你的挽具是为一匹瘦马设计的,换上一匹强壮的马之后,那些额外的支撑杆就成了纯粹的重量。这句话是整篇文档的主线,后面会反复出现。
"Harness" originally means the straps and rigging that transfer a horse's power to a cart. No matter how strong the horse, without a harness it pulls nothing; with a badly designed one, its power is wasted on slippage and strain. But the converse holds too: if your harness was designed for a weak horse, then once you hitch up a strong one, those extra support struts are pure dead weight. That sentence is the spine of this document and will recur.
「一个 agent 是一个模型加一个 harness —— 那个通过循环、工具、上下文管理、安全控制、编排、扩展面把 LLM 耦合到世界上的运行时。Harness engineering,这个在 2026 年初被命名的学科,就是该运行时的设计与演化。」
"An agent is a model plus a harness — the runtime that couples an LLM to the world through a loop, tools, context management, safety controls, orchestration, and extension surfaces. Harness engineering, named as a discipline in early 2026, is the design and evolution of that runtime."
图 1 · 模型只输出文本;harness 的六类职责把文本变成对世界的动作,并把结果送回下一轮。子系统划分依据 arXiv:2609.00006。
Fig. 1 · The model only emits text; the harness's six responsibilities turn that text into actions on the world and feed results back. Subsystem breakdown per arXiv:2609.00006.
为什么现在才把它当成一门学科
Why it only now counts as a discipline
因为直到最近,harness 都被当成「胶水代码」——写它的人各写各的,没有共享词汇,更没有可比较的设计空间。这两篇论文做的事就是给这个空间画地图:定义子系统边界、为每个子系统标出「最小实现」和「最大实现」两个端点、然后把 11 个真实系统摆进这个坐标系里对照。有了坐标系,「我们要不要做 X」才第一次变成一个可以论证的问题,而不是各凭手感。
Because until recently the harness was treated as glue code — everyone wrote their own, with no shared vocabulary and no comparable design space. What these papers do is map that space: define subsystem boundaries, mark a minimal and a maximal implementation for each, then place eleven real systems into that coordinate system side by side. With a coordinate system, "should we build X?" becomes an arguable question for the first time rather than a matter of taste.
为什么需要它:四种失败Why you need one: four failure modes
Harness 不是为了好看而加的。它的每一个组件都是从一次具体的失败里长出来的。Anthropic 在 2025 年 11 月的《Effective harnesses for long-running agents》里,把长时程编码 agent 的失败归纳成四条,并给出了每条对应的解法——这张表是理解 harness 设计动机最快的入口。
A harness isn't decoration. Every component grew out of a specific failure. In "Effective harnesses for long-running agents" (Nov 2025), Anthropic reduced the failure modes of long-running coding agents to four, each with a corresponding fix. This table is the fastest way in to the design motivation.
| 失败模式 | 初始化 agent 该做什么 | 编码 agent 该做什么 |
|---|---|---|
| Claude 过早宣布「整个项目做完了」 | 建一个 feature list 文件:把输入 spec 展开成结构化 JSON 的端到端功能清单 | 会话开头先读 feature list,只挑一个功能开工 |
| Claude 留下带 bug 或没记录的环境 | 初始化 git 仓库 + 一个 progress notes 文件 | 开头读 progress 文件与 git log,并对开发服务器跑一次基本测试抓出未记录的 bug;结尾写 commit 和 progress 更新 |
| Claude 过早把功能标记为完成 | 建 feature list 文件 | 自验证所有功能,只有仔细测过才能标 "passing" |
| Claude 要花时间摸索怎么把应用跑起来 | 写一个能启动开发服务器的 init.sh | 会话开头先读 init.sh |
| Failure mode | Initializer agent behavior | Coding agent behavior |
|---|---|---|
| Claude declares victory on the entire project too early | Set up a feature list file: expand the input spec into a structured JSON list of end-to-end feature descriptions | Read the feature list at the start of a session. Choose a single feature to work on |
| Claude leaves the environment buggy or undocumented | Write an initial git repo and a progress notes file | Start by reading the progress file and git log, and run a basic test against the dev server to catch undocumented bugs; end by writing a commit and a progress update |
| Claude marks features as done prematurely | Set up a feature list file | Self-verify all features. Only mark them "passing" after careful testing |
| Claude spends time figuring out how to run the app | Write an init.sh that can start the dev server | Start the session by reading init.sh |
来源:Effective harnesses for long-running agents,Anthropic Engineering,2025-11-26。表格为原文表格的中译。
Source: Effective harnesses for long-running agents, Anthropic Engineering, 2025-11-26. Table reproduced from the original.
一个特别值得记住的失败:上下文焦虑
One failure worth memorising: context anxiety
模型在自认为快要用完上下文窗口时,会开始草草收尾——不是真的用完了,而是它觉得快用完了,于是提前进入「总结陈词」模式。Anthropic 明确报告:Claude Sonnet 4.5 的上下文焦虑强到单靠 compaction(压缩)不足以支撑长任务表现。
A model starts wrapping up prematurely when it believes it is approaching the context limit — not because it has actually run out, but because it thinks it is about to, and shifts into closing-remarks mode. Anthropic reports this plainly: Claude Sonnet 4.5's context anxiety was strong enough that compaction alone was insufficient for long-task performance.
这引出一个很多人分不清的区别:compaction(压缩)和 context reset(上下文重置)不是一回事。
This introduces a distinction many people blur: compaction and context reset are not the same thing.
- Compaction:就地把早期对话总结掉,同一个 agent 在缩短后的历史上继续跑。保留了连续性,但没有干净的白板——所以上下文焦虑依然存在。
- Context reset:彻底清空上下文窗口,起一个全新的 agent,靠一份结构化的 handoff(交接件)把状态和下一步传过去。给出了干净白板,代价是 handoff 必须包含足够的状态让下一个 agent 无缝接手。
- Compaction: summarise the early conversation in place; the same agent continues on a shortened history. Continuity is preserved, but there's no clean slate — so context anxiety persists.
- Context reset: clear the context window entirely and start a fresh agent, carrying state and next steps across in a structured handoff. This gives a clean slate, at the cost of the handoff artifact having to contain enough state for the next agent to pick up seamlessly.
Anthropic 后来把同一个 harness 用在 Opus 4.5 上,发现这个行为消失了。原文的措辞很不客气:「The resets had become dead weight.」(那些重置变成了死重量。)——你为一个模型缺陷做的补丁,在下一代模型上会变成纯粹的复杂度、token 开销和延迟。第 5 章整章在讲这件事怎么系统化处理。
Anthropic later ran the same harness on Opus 4.5 and found the behaviour had disappeared. The wording in the original is blunt: "The resets had become dead weight." A patch you built for one model's deficiency becomes pure complexity, token overhead and latency on the next generation. Section 5 is entirely about how to handle this systematically.
第五种失败:自评失效
A fifth failure: self-evaluation collapses
还有一条不在上表里、但同样致命:让 agent 评价自己的产出时,它们倾向于自信地夸奖自己,即使人类一眼就看出质量平庸。在主观任务(比如界面设计——没有二元的可验证测试)上尤其严重。
One more, not in the table above but equally lethal: when agents evaluate their own work, they tend to praise it confidently, even when a human immediately sees mediocre quality. This is worst on subjective tasks such as interface design, where there is no binary verifiable test.
解法是把「做事的 agent」和「评判的 agent」分开。但 Anthropic 给了一句很重要的限定,别跳过:分离本身不会立刻消除宽松——evaluator 仍然是 LLM,天然偏袒 LLM 生成的产物。真正的收益在于,把一个独立的 evaluator 调教得挑剔,远比让 generator 批判自己的作品可行;一旦外部反馈存在,generator 就有了具体的迭代靶子。
The fix is to split the agent that does the work from the agent that judges it. But Anthropic adds an important qualifier — don't skip it: separation does not by itself eliminate leniency. The evaluator is still an LLM and is naturally biased toward LLM-generated artifacts. The real gain is that tuning a separate evaluator to be skeptical is far more tractable than getting a generator to critique its own work; once external feedback exists, the generator has a concrete target to iterate against.
《The Illusion of Independent Quorums》(arXiv:2609.02925)直指多 agent 投票背后那条从未验证的假设——参与投票的 agent 之间失效是独立的。同日的《Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Models》(arXiv:2609.03422)给出了可度量的量。共享训练语料、共享偏好训练、共享 prompt 骨架的模型,错的时候大概率一起错——投票只会把错误放大成共识。所以「多加一个 evaluator」不等于「多了一层独立保险」。
"The Illusion of Independent Quorums" (arXiv:2609.02925) targets the never-validated assumption behind multi-agent voting: that failures across the voting agents are independent. The same day, "Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Models" (arXiv:2609.03422) offers a measurable quantity for it. Models that share training corpora, share preference tuning and share prompt scaffolding tend to be wrong together — voting then amplifies error into consensus. So "add another evaluator" is not the same as "add an independent layer of insurance".
解剖:七个子系统Anatomy: seven subsystems
论文对 11 个系统的解剖是按七个标准子系统切的,并且为每个子系统同时标出「最小实现」和「最大实现」两个端点。这个「最小—最大」的呈现方式比一张特性清单有用得多——它直接告诉你,不做这件事最少能省到什么程度,做到极致又是什么样,你的取舍落在中间哪个位置就一目了然。
The eleven-system anatomy is cut along seven canonical subsystems, and for each subsystem the paper marks both a minimal and a maximal implementation. This min–max framing is far more useful than a feature list: it tells you directly how little you can get away with, and what the ceiling looks like, so you can see where your own trade-off sits.
图 2 · 六类职责的「最小实现—最大实现」跨度。第七个子系统是横切的:可观测性与状态持久化。划分与采用率数字依据 arXiv:2609.00006。
Fig. 2 · The min–max span of the six responsibilities. The seventh subsystem is cross-cutting: observability and state persistence. Breakdown and adoption figures per arXiv:2609.00006.
在约 400 万行 Python、TypeScript 和 Rust 代码里,论文报告了两个经受住了语料三倍扩张的「缺席」:
Across roughly four million lines of Python, TypeScript and Rust, the paper reports two absences that survived a threefold expansion of the corpus:
- 没有任何一个 agent 运行时引入通用 agent 框架。全都跑在手写的 async 循环上。
- 没有任何一个用向量嵌入检索代码。全都用确定性检索(grep、glob、AST、语言服务器一类)。
- No agent runtime imports a general-purpose agentic framework. They all run on hand-rolled async loops.
- None retrieves code with vector embeddings. They all use deterministic retrieval — grep, glob, AST, language servers and the like.
如果你正在纠结「要不要上 LangGraph / 要不要给代码库做向量索引」,这条经验数据值得放在秤上:目前没有一个进入生产的编码 agent 是这么做的。这不构成「不该做」的证明,但它至少说明,这两条路在这个具体场景下没有被从业者验证为必需。
If you are weighing "should we adopt LangGraph / should we build a vector index over the codebase", this empirical datum belongs on the scale: no coding agent that reached production does either. That isn't proof you shouldn't — but it does mean neither path has been validated as necessary by practitioners in this specific setting.
纵向观察:一个季度里发生了什么
The longitudinal view: what changed in one quarter
这篇论文有个不常见的设计:它是四月那份八系统研究的扩充版,而原来那八个系统是重新固定版本(re-pinned)而不是替换掉的——于是它顺带包含了一个受控纵向样本:同一批 harness,隔一个季度做源码 diff。两条结论:
The study has an unusual design: it is an expanded edition of an April eight-system study, and the original eight were re-pinned rather than replaced — which incidentally gives it a controlled longitudinal sample: the same harnesses, source-diffed across one quarter. Two findings:
- 趋同正在变成模仿(convergence becoming imitation)。设计不再是各自独立收敛到相似解,而是开始直接抄。
- 行为策略正从 prompt 散文迁移到配置。以前写在 system prompt 里的一大段「你应该……」,正在变成结构化的配置文件条目。这对使用者是好消息:配置能进 git、能 diff、能 review、能在 CI 里引用同一份;prompt 散文做不到。
- Convergence becoming imitation. Designs no longer converge independently on similar solutions; they are increasingly copied outright.
- Behavioral policy migrating from prompt prose to configuration. Long "you should…" passages that used to live in the system prompt are becoming structured config entries. This is good news for users: config goes into git, diffs, gets reviewed, and can be referenced by the same file in CI. Prompt prose can do none of that.
三 agent 架构与冲刺契约Three agents and the sprint contract
上面讲的是「有哪些零件」。这一章讲一个具体的、被真实跑过并记了账的组装方式。Anthropic 2026 年 3 月的《Harness design for long-running application development》记录了一次从 generator + evaluator 到 planner + generator + evaluator 的演化,目标是让 agent 无人干预地跑几个小时,产出完整全栈应用。
The previous section covered what the parts are. This one covers one specific assembly that was actually run and accounted for. Anthropic's March 2026 post "Harness design for long-running application development" documents an evolution from generator + evaluator to planner + generator + evaluator, aimed at letting an agent run unattended for hours and produce a complete full-stack application.
灵感来源说得很直白:GAN(生成对抗网络)——一个负责生成、一个负责挑刺,互相推着走。
The stated inspiration is blunt: GANs (generative adversarial networks) — one side generates, the other picks holes, and they push each other forward.
图 3 · planner / generator / evaluator 三角,以及每个 sprint 开工前的契约谈判。依据 Anthropic《Harness design for long-running application development》(2026-03-24)。
Fig. 3 · The planner / generator / evaluator triangle, plus the contract negotiated before each sprint. Per Anthropic, "Harness design for long-running application development" (2026-03-24).
Planner:只约束「交付什么」,不指定「怎么做」
Planner: constrain WHAT, never HOW
Planner 接收 1–4 句话的 prompt,把它展开成完整产品 spec,并被明确要求对范围有野心。但关键在于一条限制:planner 被指示聚焦产品语境和高层技术设计,而不是细粒度技术实现。
The planner takes a 1–4 sentence prompt and expands it into a full product spec, and is explicitly told to be ambitious about scope. But the key is a restriction: the planner is instructed to focus on product context and high-level technical design, not fine-grained technical implementation.
如果 planner 预先指定细粒度技术细节并且弄错了,spec 里的错误会级联进下游所有实现。更聪明的做法是:约束 agent 要交付什么,让它们在过程中自己找路径。
If the planner specifies fine-grained technical details up front and gets them wrong, that error cascades into every downstream implementation. The smarter move: constrain what the agents must deliver, and let them find the path themselves.
Evaluator:开箱即用的 Claude 是个糟糕的 QA
Evaluator: out of the box, Claude is a poor QA agent
这是原文里最直白的一句自我批评:「Out of the box, Claude is a poor QA agent.」作者眼看着它识别出真实问题,然后说服自己这些问题不算大事,照样批准了工作;它还倾向于表面化测试而不去探边界情况。
The bluntest self-criticism in the piece: "Out of the box, Claude is a poor QA agent." The author watched it identify real problems, then talk itself into deciding they weren't a big deal and approve the work anyway. It also tended to test superficially rather than probe edge cases.
调教方法很朴素,但要跑好几轮:读 evaluator 的日志 → 找出它的判断与人的判断分歧的例子 → 更新 QA prompt 解决这些具体分歧。这不是一次性的 prompt 工程,是一个开发循环。
The tuning method is unglamorous but requires several rounds: read the evaluator's logs → find cases where its judgement diverged from yours → update the QA prompt to address those specific divergences. This isn't one-shot prompt engineering; it's a development loop.
调教到位之后,evaluator 报出的问题长这样(原文表格节选):
Once tuned, the issues it reports look like this (excerpted from the original table):
| 契约条款 | Evaluator 的发现 |
|---|---|
| 矩形填充工具支持拖拽填充选中区域 | FAIL —— 工具只在拖拽起点/终点放了 tile,没有填充区域。fillRectangle 函数存在,但 mouseUp 时没被正确触发。 |
| 用户可以选中并删除已放置的 entity 生成点 | FAIL —— LevelEditor.tsx:892 的 Delete 键处理要求 selection 和 selectedEntityId 同时被设置,但点击 entity 只会设置后者。条件应改为 selection || (selectedEntityId && activeLayer === 'entity')。 |
| 用户可通过 API 重排动画帧 | FAIL —— PUT /frames/reorder 路由定义在 /{frame_id} 之后,FastAPI 把 reorder 当成整型 frame_id 去匹配,返回 422:"unable to parse string as an integer"。 |
| Contract criterion | Evaluator finding |
|---|---|
| Rectangle fill tool allows click-drag to fill a rectangular area with selected tile | FAIL — Tool only places tiles at drag start/end points instead of filling the region. fillRectangle function exists but isn't triggered properly on mouseUp. |
| User can select and delete placed entity spawn points | FAIL — Delete key handler at LevelEditor.tsx:892 requires both selection and selectedEntityId to be set, but clicking an entity only sets selectedEntityId. Condition should be selection || (selectedEntityId && activeLayer === 'entity'). |
| User can reorder animation frames via API | FAIL — PUT /frames/reorder route defined after /{frame_id} routes. FastAPI matches reorder as a frame_id integer and returns 422: "unable to parse string as an integer." |
注意这三条的共同点:它们都是「代码存在、但接线断了」类型的 bug——静态读代码或跑单元测试很难发现,只有真的点一遍才暴露。这也是为什么 evaluator 必须配浏览器自动化工具(原文用的是 Playwright MCP),而不是只看 diff。
Note what these three share: they are all "the code exists but the wiring is broken" bugs — hard to catch by reading code or running unit tests, and exposed only by actually clicking through. That is why the evaluator needs browser automation (Playwright MCP in the original) rather than just a diff.
Sprint contract:动手前先谈好「done 长什么样」
The sprint contract: agree on "done" before writing code
每个 sprint 开始之前,generator 提出「我要建什么、成功如何验证」,evaluator 审阅并确认它在建正确的东西,两者迭代到达成一致——然后才写第一行代码。存在的理由是:产品 spec 被刻意保持在高层,需要一个步骤来桥接「用户故事」和「可测试的实现」之间的鸿沟。原文里 Sprint 3 一个 sprint 就有 27 条 criteria。
Before each sprint begins, the generator proposes what it will build and how success will be verified; the evaluator reviews the proposal to confirm the generator is building the right thing; the two iterate until they agree — and only then is the first line of code written. The rationale: the product spec is deliberately kept high-level, so a step is needed to bridge the gap between user stories and testable implementation. In the original, Sprint 3 alone carried 27 criteria.
对个人开发者的可移植版本很简单:让 Claude 在动手前先把「这次改完之后,我怎么向你证明它好了」写下来,你确认,再开工。这就是一个人的 sprint contract。
The portable version for a solo developer is simple: have Claude write down "here is how I will prove to you this is done" before it starts, confirm it, then let it go. That is a one-person sprint contract.
账单:harness 贵 20 倍以上,但差别是「能不能用」
The bill: 20× more expensive, but the difference is "does it work at all"
同一句 prompt(「做一个 2D 复古游戏制作器,含关卡编辑器、精灵编辑器、实体行为、可试玩模式」),模型都是 Claude Opus 4.5:
The same prompt ("Create a 2D retro game maker with features including a level editor, sprite editor, entity behaviors, and a playable test mode"), same model, Claude Opus 4.5:
| 配置 | 耗时 | 成本 | 结果 |
|---|---|---|---|
| 单 agent(solo) | 20 分钟 | $9 | 布局浪费空间;工作流僵硬;游戏是坏的——entity 出现在屏幕上但对输入无响应,实体定义与运行时之间的接线断了,而界面上没有任何提示 |
| 完整 harness | 6 小时 | $200 | planner 把一句话展开成横跨 10 个 sprint 的 16 功能 spec;含精灵动画系统、行为模板、音效音乐、AI 辅助生成器、带分享链接的导出;试玩模式真的能玩 |
| Setup | Duration | Cost | Outcome |
|---|---|---|---|
| Solo agent | 20 min | $9 | Wasted layout space; rigid workflow; the game was broken — entities appeared on screen but didn't respond to input; the wiring between entity definitions and the game runtime was severed, with nothing in the UI to indicate it |
| Full harness | 6 hr | $200 | The planner expanded one sentence into a 16-feature spec across 10 sprints; sprite animation system, behavior templates, sound and music, AI-assisted generators, export with shareable links; play mode actually plays |
原文也记下了 harness 没解决的问题:工作流仍然没说清「要先建精灵和实体再填关卡」;物理有毛边(角色跳上平台后与平台重叠);AI 生成的关卡里有一堵跳不过去的墙导致卡死。作者把第一条判为基础模型产品直觉的欠缺,而不是 harness 该解决的问题——这个归因判断本身就是 harness 设计的核心技能:分清哪些是模型的问题,哪些是你的脚手架的问题。
The post also records what the harness didn't fix: the workflow still failed to communicate that you build sprites and entities before filling in a level; physics had rough edges (a character overlapping a platform after jumping onto it); an AI-generated level contained an unjumpable wall that softlocked the game. The author attributes the first to a gap in the base model's product intuition, not something harness design should solve — and that attribution judgement is itself the core skill of harness design: telling apart what is the model's problem from what is your scaffolding's problem.
什么时候该把脚手架拆掉When to remove the scaffolding
这是整个学科最重要、也最容易被跳过的一章。
This is the most important section of the discipline, and the easiest to skip.
「Harness 里的每一个组件,都编码了一条关于模型自己做不到什么的假设。这些假设值得被压力测试——既因为它们可能本来就错,也因为它们会随模型进步迅速过期。」
"Every component in a harness encodes an assumption about what the model can't do on its own. Those assumptions are worth stress-testing — both because they may have been wrong to begin with, and because they go stale quickly as models improve."
第一版 harness 令人鼓舞,但臃肿、慢、贵。于是作者动手精简。这里有一条方法论教训值得单独记下:
The first version of the harness was encouraging but bloated, slow and expensive. So the author set about trimming it. One methodological lesson deserves its own note:
第一次尝试是激进砍掉 harness 并同时试几个有创意的新点子,结果无法复现原性能,而且很难判断哪些设计片段真正是承重的(load-bearing)、以何种方式承重。改成有条理的做法后才走通:一次移除一个组件,逐个检查对最终结果的影响。
The first attempt was to aggressively cut the harness while trying several creative new ideas at the same time. The result: the original performance couldn't be reproduced, and it was hard to tell which design pieces were genuinely load-bearing, and in what way. Only a methodical approach worked: remove one component at a time and inspect its effect on the final result.
V1 → V2 具体删了什么、留了什么
What V1 → V2 actually removed and kept
- 删掉 sprint 结构。它原本是为了把工作切成 chunk 保持连贯;Opus 4.6 之后有理由相信模型能原生处理。
- 删掉 context resets。Opus 4.5 起 context anxiety 基本自行消除,整个构建改为跑成一个连续 session,靠 Agent SDK 的自动 compaction 处理上下文增长。
- 保留 planner。理由是实测的:没有 planner 时 generator 会 under-scope——拿到原始 prompt 就不写 spec 直接开建,最终做出的应用功能少得多。
- 保留 evaluator,但降级为整个 run 结束时的单次 pass,不再按 sprint 打分。
- Removed the sprint structure. It existed to break work into chunks for coherence; after Opus 4.6 there was reason to believe the model handles that natively.
- Removed context resets. From Opus 4.5 on, context anxiety largely disappeared on its own, so the whole build runs as one continuous session with the Agent SDK's automatic compaction handling context growth.
- Kept the planner. The reason is empirical: without it the generator under-scopes — given the raw prompt it starts building without writing a spec, and ends up with a far less featured application.
- Kept the evaluator, but downgraded it to a single pass at the end of the run rather than scoring every sprint.
图 4 · 能力边界外移时,原本承重的组件会变成 overhead——但只在边界扫过的那段区间里。依据 Anthropic 2026-03 工程文的原文论断。
Fig. 4 · As the capability frontier moves out, previously load-bearing components become overhead — but only in the band the frontier sweeps across. Per the March 2026 Anthropic engineering post.
「Evaluator 值不值,不是一个固定的是非题。当任务落在当前模型单干无法可靠解决的范围之外时,它才值这个成本。」
"The evaluator is not a fixed yes-or-no decision. It is worth the cost when the task sits beyond what the current model does reliably solo."
V2 的账单:更快、更便宜、连续跑 2 小时
The V2 bill: faster, cheaper, two hours unbroken
V2 的 prompt 是「用 Web Audio API 在浏览器里做一个功能完整的 DAW(数字音频工作站)」。总计约 4 小时、$124.70。注意 Build Round 1 单独就连续跑了 2 小时 7 分钟且没有 sprint 分解——而 Opus 4.5 需要 sprint 分解才能做到。
The V2 prompt: "Build a fully featured DAW in the browser using the Web Audio API." Roughly 4 hours and $124.70 total. Note that Build Round 1 alone ran 2 hours 7 minutes continuously with no sprint decomposition — whereas Opus 4.5 required it.
| 阶段 | 耗时 | 成本 |
|---|---|---|
| Planner | 4.7 分钟 | $0.46 |
| Build(第 1 轮) | 2 小时 7 分 | $71.08 |
| QA(第 1 轮) | 8.8 分钟 | $3.24 |
| Build(第 2 轮) | 1 小时 2 分 | $36.89 |
| QA(第 2 轮) | 6.8 分钟 | $3.09 |
| Build(第 3 轮) | 10.9 分钟 | $5.88 |
| QA(第 3 轮) | 9.6 分钟 | $4.06 |
| 合计 | 3 小时 50 分 | $124.70 |
| Agent & phase | Duration | Cost |
|---|---|---|
| Planner | 4.7 min | $0.46 |
| Build (Round 1) | 2 hr 7 min | $71.08 |
| QA (Round 1) | 8.8 min | $3.24 |
| Build (Round 2) | 1 hr 2 min | $36.89 |
| QA (Round 2) | 6.8 min | $3.09 |
| Build (Round 3) | 10.9 min | $5.88 |
| QA (Round 3) | 9.6 min | $4.06 |
| Total | 3 hr 50 min | $124.70 |
V2 的 QA 反馈也很有代表性——第一轮指出的核心问题是:「应用看起来很唬人、AI 集成也能用,但好几个 DAW 核心功能是纯展示的:片段不能在时间线上拖动,没有乐器 UI 面板(合成器旋钮、鼓垫),没有可视化效果编辑器(EQ 曲线、压缩器表头)。这些不是边界情况,这就是让 DAW 可用的核心交互,而且 spec 里明确要求了。」结论性的失败模式:generator 在无人看管时仍倾向于漏掉细节、或把功能做成存根(stub)。QA 在抓这些「最后一公里」问题上仍然有价值。
The V2 QA feedback is representative. Round 1's core finding: "the app looks impressive and the AI integration works well, [but] several core DAW features are display-only without interactive depth: clips can't be dragged/moved on the timeline, there are no instrument UI panels (synth knobs, drum pads), and no visual effect editors (EQ curves, compressor meters). These aren't edge cases — they're the core interactions that make a DAW usable, and the spec explicitly calls for them." The conclusive failure mode: left unsupervised, the generator still tends to skip details or stub features out. QA still earns its keep on these last-mile problems.
原文点名:Claude 听不见。这让 QA 反馈回路在「这段音乐好不好听」这一层完全失效。同理,你的 evaluator 只在它能感知的维度上有效——设计上要清楚这条边界在哪。
The post names it directly: Claude can't actually hear. That renders the QA feedback loop useless at the level of "does this music sound good". By the same token, your evaluator is only effective along dimensions it can perceive — know where that boundary is when you design it.
Meta-harness:把脑和手解耦Meta-harness: decoupling brain from hands
如果「假设会过期」是真的,那不断重写 harness 就是个无底洞。2026 年 4 月的《Scaling Managed Agents: Decoupling the brain from the hands》给出了另一条路:不去猜未来需要什么 harness,而是设计一组能比任何具体 harness 活得更久的接口。
If assumptions really do go stale, then endlessly rewriting the harness is a bottomless pit. Anthropic's April 2026 post "Scaling Managed Agents: Decoupling the brain from the hands" offers a different route: don't guess which harness the future needs; design a small set of interfaces meant to outlive any particular one.
几十年前,操作系统把硬件虚拟化成 process、file 这类抽象。抽象比硬件活得久:read() 这条命令,对它访问的是 1970 年代的磁盘组还是现代 SSD,是完全无感知的。上层抽象稳定,底层实现自由更换。Managed Agents 想做的就是这件事的 agent 版。
Decades ago, operating systems virtualised hardware into abstractions like process and file. The abstractions outlived the hardware: a read() call is indifferent to whether it touches a 1970s disk pack or a modern SSD. Stable abstraction on top, freely replaceable implementation underneath. Managed Agents attempts the agent-shaped version of that.
被虚拟化的是三个东西:
Three things get virtualised:
- session
- 「一切发生之事的追加式日志」——注意它不等于模型的上下文窗口,后面会展开。
- harness
- 「调用 Claude、并把 Claude 的工具调用路由到相应基础设施的循环」。
- sandbox
- 「Claude 可以跑代码、改文件的执行环境」。
- session
- "the append-only log of everything that happened" — note this is not the model's context window; more on that below.
- harness
- "the loop that calls Claude and routes Claude's tool calls to the relevant infrastructure".
- sandbox
- "an execution environment where Claude can run code and edit files".
团队的态度写得很清楚:「我们对这些接口的形状有立场,对它们背后跑的是什么没有立场。」
The team's stance is stated plainly: "We're opinionated about the shape of these interfaces, not about what runs behind them."
起点是一个失败:「别养宠物」
It starts with a failure: "don't adopt a pet"
最初的架构把所有组件塞进一个容器——session、harness、sandbox 共享环境。好处是文件编辑就是一次系统调用,不需要设计任何服务边界。坏处是三条:
The initial architecture put every component inside one container — session, harness and sandbox sharing an environment. The upside: file editing is a direct syscall, with no service boundary to design. The downsides were three:
- 服务器变成了「宠物」。容器一挂,session 就丢;容器无响应,就得手工把它「救回来」。(pets vs. cattle:宠物有名字、需要照料、丢不起;牛群是可互换的。)
- 可观测性坍缩。唯一的观察窗口是 WebSocket 事件流,但它说不出失败发生在哪里——harness 里的 bug、事件流丢包、容器下线,三者表现完全相同。要查清必须在容器内开 shell,而那个容器往往同时装着用户数据。
- harness 假设它要处理的东西和它住在同一个容器里。于是客户想把 Claude 连到自己的 VPC 时,只能要么把网络与 Anthropic 对等互联,要么在自己环境里跑 Anthropic 的 harness。
- The server became a pet. If the container died, the session was lost; if it went unresponsive, you had to nurse it back to health. (Pets vs cattle: a pet has a name, needs hand-care, and can't be lost; cattle are interchangeable.)
- Observability collapsed. The only window was the WebSocket event stream, and it couldn't say where a failure happened — a bug in the harness, a dropped event, and a container going offline all looked identical. Diagnosing meant opening a shell inside the container, which also held user data.
- The harness assumed whatever Claude needed lived in the same container. So a customer wanting Claude to work against resources in their own VPC had to either peer their network with Anthropic's, or run Anthropic's harness in their environment.
图 5 · 从「一切都在一个容器里」到 brain / hands / session 三分。接口签名为原文所列。依据 Anthropic《Scaling Managed Agents》(2026-04-08)。
Fig. 5 · From "everything in one container" to a brain / hands / session split. Interface signatures as listed in the original. Per Anthropic, "Scaling Managed Agents" (2026-04-08).
安全:一条被明确写出来的攻击链
Security: an attack chain spelled out
这一段值得逐句读:
This passage rewards a close reading:
- 耦合设计下,Claude 生成的不可信代码与凭据跑在同一个容器里。
- 于是一次 prompt injection 只需要说服 Claude 去读自己的环境变量。
- 一旦攻击者拿到这些 token,就能派生全新的、不受限的 session,并把工作委派给它们。
- In the coupled design, untrusted Claude-generated code and the credentials run in the same container.
- So a prompt injection only needs to convince Claude to read its own environment variables.
- Once an attacker holds those tokens, they can spawn brand-new, unconstrained sessions and delegate work to them.
「把 token 权限收窄」是显而易见的缓解——但注意原文的反驳:那本身又编码了一条关于「Claude 拿一个受限 token 做不到什么」的假设,而 Claude 越来越聪明。(又是那条主线。)
"Narrow the token's scope" is the obvious mitigation — but note the rebuttal: that itself encodes an assumption about what Claude can't do with a restricted token, and Claude keeps getting smarter. (That spine again.)
结构性修复是:确保 token 从 Claude 生成的代码所运行的沙箱里根本不可达。两种落地模式:
The structural fix: make the token unreachable from the sandbox where Claude-generated code runs. Two patterns:
- 凭据与资源捆绑:对 Git,在沙箱初始化时用每个 repo 的 access token 克隆 repo,并把它接进本地 git remote。这样沙箱内的
push/pull正常工作,而 agent 自始至终没碰过 token。 - 凭据存在沙箱之外的保险库:对自定义工具走 MCP,OAuth token 存进 vault。Claude 通过一个专用 proxy 调 MCP 工具;proxy 收到一个与 session 关联的 token,据此从 vault 取出真凭据向外发请求。原文:「The harness is never made aware of any credentials.」
- Bundle credentials with the resource: for Git, clone the repo at sandbox initialisation using a per-repo access token and wire it into the local git remote.
pushandpullthen work inside the sandbox, while the agent never touches the token at all. - Keep credentials in a vault outside the sandbox: for custom tools, go through MCP with OAuth tokens in a secure vault. Claude calls MCP tools through a dedicated proxy; the proxy receives a session-associated token and uses it to fetch the real credential from the vault before calling out. Verbatim: "The harness is never made aware of any credentials."
Session ≠ 上下文窗口
Session ≠ context window
这是这篇文章最有迁移价值的一个概念区分。长任务常常超出上下文窗口,而标准应对手段——compaction、memory 工具、context trimming——都涉及关于「保留什么」的不可逆决策。失败机理是:很难知道未来的回合需要哪些 token。一旦某条消息被压缩步骤转换掉,harness 就会把原消息从上下文窗口移除,而它只有被存下来才可恢复。
This is the most transferable conceptual distinction in the piece. Long tasks routinely exceed the context window, and the standard responses — compaction, a memory tool, context trimming — all involve irreversible decisions about what to keep. The failure mechanism: it's hard to know which tokens a future turn will need. Once a message has been transformed by a compaction step, the harness removes the original from the context window, and it is recoverable only if it was stored.
Managed Agents 让 session 就是那个活在上下文窗口之外的 context 对象——但它不存在沙箱或 REPL 里,而是持久存储在 session log。getEvents() 支持三种用法:从上次停止阅读的地方接着读;回退到某个具体时刻之前几个事件去看前因;在某个具体动作之前重读 context。取回的事件可以在 harness 里被任意转换后再送进上下文窗口——包括为了拉高 prompt 缓存命中率而做的组织。
Managed Agents makes the session itself the context object living outside the context window — not in a sandbox or a REPL, but durably in the session log. getEvents() supports three uses: continue from where you stopped reading; rewind to a few events before a specific moment to see what led up to it; and re-read context immediately before a specific action. Retrieved events can be arbitrarily transformed in the harness before entering the context window — including whatever shaping raises the prompt-cache hit rate.
关注点分离的理由说得很清楚:无法预测未来模型需要什么具体的 context engineering,所以接口把 context 管理推到 harness 里,而只保证 session 是持久的、可被审问的。
The rationale for the separation is explicit: you cannot predict what specific context engineering future models will need, so the interface pushes context management into the harness and guarantees only that the session is durable and interrogable.
性能上的意外收获
An unexpected performance win
brain 在容器里时,多少个 brain 就需要多少个容器,而每个 brain 在容器 provision 完成前无法开始任何推理——即使是永远不会碰 sandbox 的 session,也得先 clone repo、boot 进程、fetch 待处理事件。这段死时间直接体现在 TTFT(首 token 时间)上,是用户最直接感受到的延迟。
When the brain lives in the container, you need as many containers as brains, and no brain can start reasoning until its container is provisioned — even a session that will never touch the sandbox has to clone the repo, boot processes, and fetch pending events first. That dead time lands directly on time-to-first-token, the latency a user feels most.
因为没有任何一只手与任何一个 brain 耦合,还带来一个额外能力:brain 之间可以互相传递手。
And because no hand is coupled to any particular brain, there's an extra capability: brains can pass hands to one another.
11 系统审计的横切发现Cross-cutting findings from the eleven-system audit
把论文里最有决策价值的几条单独拎出来。注意:这些是对生产系统的观察,不是规范。
The findings with the most decision value, pulled out. Note: these are observations of production systems, not prescriptions.
| 发现 | 对你的含义 |
|---|---|
| 0/11 引入通用 agent 框架 | 手写 async 循环仍是生产选择。上框架前先问:框架替我解决了哪个我真的遇到过的问题? |
| 0/11 用向量嵌入检索代码 | 确定性检索(grep / glob / AST)在代码这个场景里赢了。给代码库做向量索引在生产里未被验证为必需。 |
| SKILL.md skills 9/11,MCP 8/11 | skills 采用率已超过 MCP。做扩展时,「一个 markdown 文件 + 渐进式披露」的形态门槛更低、传播更快。 |
| ACP 进入 6/11,出现第三种角色:harness hosting | harness 开始互相托管——「用哪个编码 agent」正在从二选一变成可组合。 |
| 行为策略从 prompt 散文迁移到配置 | 把你的规则从 CLAUDE.md 的散文段落改写成结构化配置/hooks/权限规则,能进 git、能 diff、能被 CI 引用同一份。 |
| 趋同正在变成模仿 | 差异化不在「有没有某个功能」上了。想做不一样的东西,得往子系统组合方式上想。 |
| Finding | What it means for you |
|---|---|
| 0/11 import a general-purpose agent framework | Hand-rolled async loops remain the production choice. Before adopting a framework, ask: which problem I have actually hit does it solve? |
| 0/11 retrieve code with vector embeddings | Deterministic retrieval (grep / glob / AST) wins in the code setting. Vector-indexing a codebase is not validated as necessary in production. |
| SKILL.md skills 9/11 vs MCP 8/11 | Skills adoption has overtaken MCP. For extensions, "one markdown file plus progressive disclosure" has a lower barrier and spreads faster. |
| ACP in 6/11, with a third role: harness hosting | Harnesses have begun hosting one another — "which coding agent" is shifting from either/or to composable. |
| Behavioral policy migrating from prompt prose to config | Rewriting your rules from CLAUDE.md prose into structured config / hooks / permission rules makes them git-tracked, diffable, and referenceable from CI. |
| Convergence becoming imitation | Differentiation is no longer about having a given feature. To build something different, think at the level of how subsystems combine. |
来源:arXiv:2609.00006(2026-09-02 上架 cs.SE,提交日 2026-07-15)。83 页,7 图 18 表,含 13 条横切观察、29 个设计模式、18 条设计建议和一份 90 行的最小可用 harness 脚手架。〔单篇预印本,无第三方复现〕
Source: arXiv:2609.00006 (listed in cs.SE 2026-09-02, submitted 2026-07-15). 83 pages, 7 figures, 18 tables; 13 cross-cutting observations, 29 design patterns, 18 design recommendations, and a 90-line minimum-viable-harness scaffold. [Single preprint, no third-party replication.]
怎么用:自建 harness 的检查清单How to use this: a checklist for your own harness
你不需要从零写一个 harness 才用得上这些。如果你在 Claude Code 里维护一套 CLAUDE.md + skills + hooks + 权限规则 + subagent 定义,你就已经在做 harness 工程了。下面这份清单同时适用于两种情况。
You don't need to build a harness from scratch for any of this to apply. If you maintain a CLAUDE.md plus skills, hooks, permission rules and subagent definitions in Claude Code, you are already doing harness engineering. The checklist below applies to both cases.
- 先给每个组件写下它的「假设」。逐条问:这个 hook / 这段 CLAUDE.md / 这个 subagent,是在补模型的哪个短板?写不出那个短板,这个组件就是凭感觉加的。
- 约束「交付什么」,不要预先指定「怎么做」。细粒度技术决策写死在 spec 里,一旦错了会级联进所有下游实现。
- 把「做」和「判」分开,但不要以为分开就够了。Evaluator 仍是 LLM、仍偏袒 LLM 产物;它需要被调教——读它的日志,找它和你判断分歧的地方,针对性改它的 prompt,跑好几轮。
- 给 evaluator 真正的感知手段。只看 diff 的评审抓不到「代码存在但接线断了」的 bug。浏览器自动化、真实运行、真实点击——evaluator 只在它能感知的维度上有效。
- 动手前先谈好「done 长什么样」。让它写下验收标准、你确认、再开工。这是最便宜的一条,收益却很大。
- 把状态放在 agent 之外。progress 文件、feature list JSON、git commit ——这些不是文档洁癖,它们是下一个上下文窗口唯一能读到的东西。
- 让凭据从代码运行的地方不可达。不是「收窄权限」,是「根本拿不到」。参考 git remote 预挂载 与 vault + proxy 两种模式。
- 每次模型升级,重新审视一遍 harness。剥掉不再承重的部分,加上此前不可能实现的新部分。一次只动一个组件,否则你分不清是谁在起作用。
- Write down each component's assumption first. Ask of every hook, every CLAUDE.md paragraph, every subagent: which model weakness does this compensate for? If you can't name the weakness, the component was added on vibes.
- Constrain WHAT, not HOW. Fine-grained technical decisions baked into a spec cascade into every downstream implementation once they're wrong.
- Separate doing from judging — but don't assume separation is enough. The evaluator is still an LLM and still favours LLM output; it has to be tuned: read its logs, find where its judgement diverged from yours, patch its prompt for those cases, repeat over several rounds.
- Give the evaluator real perception. A review that only sees the diff won't catch "the code exists but the wiring is broken" bugs. Browser automation, real runs, real clicks — an evaluator is only effective along dimensions it can perceive.
- Agree on "done" before any code is written. Have it write the acceptance criteria, confirm them, then let it go. The cheapest item on this list and one of the highest-yield.
- Keep state outside the agent. A progress file, a feature-list JSON, git commits — this isn't documentation fastidiousness; it's the only thing the next context window can read.
- Make credentials unreachable from where code runs. Not "narrow the scope" — unreachable. See the pre-mounted git remote and the vault + proxy patterns.
- Revisit the harness on every model upgrade. Strip what is no longer load-bearing; add what was previously impossible. Change one component at a time, or you won't be able to tell what is doing the work.
如果只做三件事:① 一个 init.sh(怎么把东西跑起来);② 一个结构化的功能清单 JSON(什么算做完,只允许改 passes 字段);③ 每次会话结尾写 git commit + 一段进展记录。这三件事覆盖了 Anthropic 那张失败模式表里的全部四条。
If you only do three things: (1) an init.sh (how to get the thing running); (2) a structured feature-list JSON (what counts as done, with only the passes field editable); (3) a git commit plus a progress note at the end of every session. Those three cover all four rows of Anthropic's failure-mode table.
对比与选型Comparison and selection
「我该自己搭 harness 吗?」这个问题有四个层次的答案,选哪个取决于你的任务落在能力边界的哪一侧、以及你能承担多少复杂度。
"Should I build my own harness?" has four tiers of answer, and which one fits depends on where your task sits relative to the capability frontier and how much complexity you can carry.
| 做法 | 适合 | 代价 |
|---|---|---|
| 直接用现成 harness(Claude Code、Codex CLI 等) | 绝大多数编码与自动化任务。这些系统已经把七个子系统的成熟实现打包好了 | 基本没有;上限受限于该 harness 的设计选择 |
| 用现成 harness + 自定义扩展(skills / hooks / MCP / subagent / CLAUDE.md) | 你有稳定重复的工作流、有团队约定要执行、有外部系统要接 | 低。且这是 11 系统审计里采用率最高的扩展路径(skills 9/11) |
| 基于 Agent SDK 自己写编排(planner / generator / evaluator) | 任务确实超出单 agent 可靠完成的范围,且你愿意为质量付 20 倍成本 | 高。要写编排、要调教 evaluator(好几轮)、要设计文件通信协议 |
| Meta-harness 路线(brain / hands / session 解耦) | 要跑很多并发 agent、要连很多异构执行环境、有真实的安全边界要求 | 最高。但它换来的是「不必每代模型都重写 harness」 |
| Approach | Fits when | Cost |
|---|---|---|
| Use an off-the-shelf harness (Claude Code, Codex CLI, …) | The vast majority of coding and automation tasks. These systems already ship mature implementations of all seven subsystems | Essentially none; your ceiling is that harness's design choices |
| Off-the-shelf plus custom extensions (skills / hooks / MCP / subagents / CLAUDE.md) | You have a repeating workflow, team conventions to enforce, or external systems to reach | Low. Also the highest-adoption extension path in the eleven-system audit (skills, 9/11) |
| Write your own orchestration on the Agent SDK (planner / generator / evaluator) | The task genuinely exceeds what a single agent completes reliably, and you'll pay ~20× for quality | High. You write the orchestration, tune the evaluator over several rounds, and design a file-based protocol |
| The meta-harness route (brain / hands / session decoupling) | Many concurrent agents, many heterogeneous execution environments, real security boundaries | Highest. What you buy is not having to rewrite the harness every model generation |
「找到可能的最简方案,只在需要时才增加复杂度。」——这句出自 Anthropic 更早的《Building effective agents》,在 2026 年的这两篇里被反复引用。配套的判断是:模型变强时,有时候你可以什么都不做,等下一个模型,某些问题自己就解决了;但另一方面,模型越好,越有空间去开发能达成「超出模型 baseline」复杂任务的 harness。
"Find the simplest solution possible, and only increase complexity when needed." — from Anthropic's earlier "Building effective agents", repeatedly cited in both 2026 posts. Its companion judgement: as models improve, sometimes you can do nothing and wait for the next model to solve the problem for you; but equally, the better the model, the more room there is to build a harness that reaches tasks beyond its baseline.
原文对这条的收尾是一句我认为值得记住的话:「有趣的 harness 组合空间并不随模型变强而缩小。它只是移动了。」
The post closes that thread with a line worth keeping: "the space of interesting harness combinations doesn't shrink as models improve. Instead, it moves."
常见坑与限制Pitfalls and limits
压缩保留连续性但不给白板,上下文焦虑照样存在。如果你观察到模型在长任务后半程草草收尾,压缩解决不了它。〔Anthropic 官方实测,针对 Sonnet 4.5〕
Compaction preserves continuity but gives no clean slate, so context anxiety persists. If you see a model wrapping up prematurely in the back half of a long task, compaction won't fix it. [Anthropic-measured, on Sonnet 4.5.]
Anthropic 第一次精简就是这么干的——无法复现原性能,且分不清哪些片段是承重的。正确做法是一次移除一个组件。
Anthropic's first trimming attempt did exactly this — the original performance couldn't be reproduced and load-bearing pieces couldn't be identified. Remove one component at a time.
Evaluator 是 LLM、偏袒 LLM 产物,而且本周的两篇论文(arXiv:2609.02925 / 2609.03422)正好在质疑多 agent 之间失效独立性这条前提。多个同源模型互相复核,可能只是把同一个错误确认了两遍。
The evaluator is an LLM and favours LLM artifacts — and this week's two papers (arXiv:2609.02925 / 2609.03422) question the very premise of failure independence across agents. Same-lineage models checking each other may just be confirming the same error twice.
Spec 里错误的细粒度技术决策会级联进所有下游实现。Planner 该管产品语境和高层设计,路径留给执行者自己找。
A wrong fine-grained technical decision in the spec cascades into every downstream implementation. The planner should own product context and high-level design; leave the path to whoever executes.
Claude 听不见,所以它评不了音乐好不好听;类似地,只读 diff 的 reviewer 评不了「点下去有没有反应」。先想清楚这个 evaluator 能感知什么,再决定让它打哪几条分。
Claude can't hear, so it can't judge whether music sounds good; likewise a diff-only reviewer can't judge whether a click does anything. Work out what the evaluator can perceive before deciding what it scores.
这份材料本身的限制
Limits of this material itself
1. 单一来源风险。arXiv:2609.00006 的所有数字(11 系统、400 万行、9/11 vs 8/11)来自单篇预印本,截至调研时无第三方复现。Anthropic 三篇的成本表和 TTFT 数字是官方自述实验,同样是单一来源。
2. HEART 的基准分数未经复现。「超过 GPT-5.4 / Claude-4.6-Sonnet / Gemini-3.1-Pro 平均 6%」「API 成本降 85%」「真实任务完成率 84% vs 22%」全部来自论文自报,应视为方向性参考。
3. 模型版本在快速移动。本文引用的实验分别跑在 Sonnet 4.5、Opus 4.5 和 Opus 4.6 上;截至 2026-09-07,前沿已经是 Fable 5.1 / Opus 5 / Mythos 5.1 一代。「哪些脚手架已经过期」这个具体判断,必须在你自己用的模型上重跑一遍——这恰恰是本文的核心论点。
4. 领域偏向。11 系统审计的对象全是编码 harness。结论对编码 agent 最可靠;迁移到科研、金融建模、内容生产等场景时需要重新验证。
1. Single-source risk. All figures from arXiv:2609.00006 (eleven systems, four million lines, 9/11 vs 8/11) come from a single preprint with no third-party replication as of this writing. The cost tables and TTFT figures in the three Anthropic posts are self-reported experiments — also single-source.
2. HEART's benchmark numbers are unreplicated. "6% above GPT-5.4 / Claude-4.6-Sonnet / Gemini-3.1-Pro on average", "up to 85% lower API cost", "84% vs 22% real-task completion" are all paper-reported and should be treated as directional.
3. Model versions are moving fast. The experiments cited here ran on Sonnet 4.5, Opus 4.5 and Opus 4.6 respectively; as of 2026-09-07 the frontier is the Fable 5.1 / Opus 5 / Mythos 5.1 generation. The specific judgement of "which scaffolding has gone stale" must be re-run on the model you actually use — which is precisely this document's central claim.
4. Domain bias. The eleven-system audit covers coding harnesses exclusively. The conclusions are most reliable for coding agents; transferring them to scientific research, financial modelling or content production needs fresh validation.
术语表Glossary
- Harness
- 把语言模型接到真实世界的运行时层:循环、工具、上下文管理、安全控制、编排、扩展面。中文没有定译,直译是「挽具」。
- Meta-harness
- 不实现某个具体 harness,而是提供一组能容纳多种 harness 的通用接口(如 brain / hands / session 三分)。
- Context anxiety(上下文焦虑)
- 模型在自认为接近上下文上限时提前草草收尾的行为。
- Compaction(压缩)
- 就地总结早期对话,同一个 agent 在缩短的历史上继续。保留连续性,不给白板。
- Context reset(上下文重置)
- 清空上下文窗口、起全新 agent,靠结构化 handoff 传递状态。给白板,但要求 handoff 足够完整。
- Sprint contract(冲刺契约)
- 动手前 generator 与 evaluator 就「这块工作的 done 长什么样」谈成的一致文件。
- Load-bearing(承重)
- 形容一个 harness 组件是否真的在支撑性能。判断方法是移除它并观察结果——一次只移除一个。
- Pets vs. cattle
- 运维术语。宠物有名字、需要照料、丢不起;牛群可互换、坏了就换新的。Harness 组件应该是后者。
- Session log
- 一切发生之事的追加式日志,持久存储在 agent 之外。它不是模型的上下文窗口。
- TTFT(time-to-first-token)
- 会话从接受工作到吐出第一个响应 token 的等待时长,是用户最直接感受到的延迟。
- Tool Primitives
- arXiv:2609.01736 提出的设计:把每个工具包一层 LLM 接口,用自然语言而非 JSON schema 作为调用界面。
- ACP
- 一种 agent 间协议,在 11 系统中出现于 6 个,并带来「harness 托管 harness」这一新角色。
- Harness
- The runtime layer coupling a language model to the world: loop, tools, context management, safety controls, orchestration, extension surfaces.
- Meta-harness
- Not an implementation of a particular harness, but a set of general interfaces that can accommodate many (e.g. the brain / hands / session split).
- Context anxiety
- A model wrapping up prematurely when it believes it is nearing the context limit.
- Compaction
- Summarising the early conversation in place; the same agent continues on a shortened history. Preserves continuity; gives no clean slate.
- Context reset
- Clearing the context window and starting a fresh agent, carrying state across in a structured handoff. Gives a clean slate but demands a complete handoff.
- Sprint contract
- A file the generator and evaluator negotiate to agreement, before code is written, on what "done" looks like for that unit of work.
- Load-bearing
- Describes whether a harness component genuinely supports performance. Test by removing it and observing — one component at a time.
- Pets vs. cattle
- An ops idiom. A pet has a name, needs hand-care and can't be lost; cattle are interchangeable and simply replaced. Harness components should be the latter.
- Session log
- The append-only record of everything that happened, stored durably outside the agent. It is not the model's context window.
- TTFT (time-to-first-token)
- How long a session waits between accepting work and emitting its first response token — the latency a user feels most directly.
- Tool Primitives
- The design proposed in arXiv:2609.01736: wrap each tool in an LLM interface and use natural language, rather than a JSON schema, as the calling surface.
- ACP
- An inter-agent protocol present in 6 of the 11 systems, bringing with it the new role of one harness hosting another.
来源清单与延伸阅读Sources and further reading
调研时间 2026-09-07。以下是本文每个关键事实的出处;标注日期为来源自身的发布日期。
Researched 2026-09-07. Below is the provenance for every key fact in this document; dates are those of the sources themselves.
- Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents — A Source-Code Study of Eleven Systems — Barbaste, Darrigol, Vu, Wiltberger. arXiv:2609.00006, cs.SE。提交 2026-07-15,列于 2026-09-02。子系统划分、两个「缺席」、9/11 vs 8/11、纵向观察。
- Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives — Jin, Wang, Yu, Luo, Wang. arXiv:2609.01736,2026-09-01。Tool Primitives / ToolFace / HEART 及全部基准数字(论文自报)。
- Effective harnesses for long-running agents — Justin Young, Anthropic Engineering, 2025-11-26。四种失败模式表、初始化 agent / 编码 agent 分工、feature list JSON、init.sh。
- Harness design for long-running application development — Prithvi Rajasekaran, Anthropic Engineering, 2026-03-24。planner / generator / evaluator、sprint contract、context anxiety、成本表($9 vs $200、V2 $124.70)、「Claude is a poor QA agent」、能力边界论。
- Scaling Managed Agents: Decoupling the brain from the hands — Martin, Cemaj, Cohen, Anthropic Engineering, 2026-04-08。brain / hands / session 三分、接口签名、凭据攻击链与两种隔离模式、session ≠ context window、TTFT 数字。
- The Illusion of Independent Quorums: Epistemic Fault Domains and Correlated Cognitive Failures in Agentic Quorums — arXiv:2609.02925, cs.DC, 2026-09-04。
- Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Models — arXiv:2609.03422, cs.LG, 2026-09-04。
- Building effective agents — Anthropic Engineering, 2024-12-19。「找到最简方案,只在需要时增加复杂度」的出处。
- Engineering at Anthropic · Claude Agent SDK 文档 — 上述实验所用的编排底座。
- Harness Engineering: Anatomy, Architecture, and Evolution of Coding Agents — A Source-Code Study of Eleven Systems — Barbaste, Darrigol, Vu, Wiltberger. arXiv:2609.00006, cs.SE. Submitted 2026-07-15, listed 2026-09-02. Subsystem breakdown, the two absences, 9/11 vs 8/11, longitudinal findings.
- Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives — Jin, Wang, Yu, Luo, Wang. arXiv:2609.01736, 2026-09-01. Tool Primitives / ToolFace / HEART and all benchmark figures (paper-reported).
- Effective harnesses for long-running agents — Justin Young, Anthropic Engineering, 2025-11-26. The four-failure-mode table, initializer/coding agent split, feature-list JSON, init.sh.
- Harness design for long-running application development — Prithvi Rajasekaran, Anthropic Engineering, 2026-03-24. Planner / generator / evaluator, sprint contracts, context anxiety, the cost tables ($9 vs $200; V2 at $124.70), "Claude is a poor QA agent", the capability-frontier argument.
- Scaling Managed Agents: Decoupling the brain from the hands — Martin, Cemaj, Cohen, Anthropic Engineering, 2026-04-08. The brain / hands / session split, interface signatures, the credential attack chain and two isolation patterns, session ≠ context window, the TTFT figures.
- The Illusion of Independent Quorums: Epistemic Fault Domains and Correlated Cognitive Failures in Agentic Quorums — arXiv:2609.02925, cs.DC, 2026-09-04.
- Inferred Generative-Process Diversity Predicts Correlated Failure Across Language Models — arXiv:2609.03422, cs.LG, 2026-09-04.
- Building effective agents — Anthropic Engineering, 2024-12-19. Origin of "find the simplest solution possible, and only increase complexity when needed".
- Engineering at Anthropic · Claude Agent SDK docs — the orchestration substrate used in the experiments above.