主页
Meta Muse Code 把「自动验证结果」当主打卖点,OpenAI 把 Auto-review 降价 10 倍,Claude Code 用 hooks 建关卡——本文讲清:模型怎么发现自己写错了,验证器(verifier)怎么设计,以及闭环在哪里会失效。
Meta's Muse Code leads with "automatically verifies results," OpenAI cut Auto-review costs 10x, and Claude Code builds gates with hooks. This doc explains how a model catches its own mistakes, how to design verifiers, and where the loop breaks down.
早期编码助手的形态是「一次生成」:你提需求,它出代码,对不对由你判断。自主验证闭环(self-verification loop)把「判断对不对」也交给系统:模型每产出一步,就有一个验证器(verifier)——测试、编译器、linter、另一个模型——给出通过/不通过的信号;不通过就带着错误信息回炉修复,直到通过或达到重试上限。
Early coding assistants were one-shot: you ask, it emits code, and correctness is your problem. A self-verification loop hands the correctness check back to the system: after each step, a verifier — tests, a compiler, a linter, or another model — returns pass/fail; on failure the model retries with the error message in context, until it passes or hits a retry cap.
Meta · Muse Code(8月6日):基于 Muse Spark 1.2 的编程助手,官方卖点明确写着「写代码、修 bug、自动验证结果、管理复杂项目」——验证从工程技巧升格为产品主打。
OpenAI · Auto-review(8月4日):ChatGPT 应用与 Codex CLI 的 Auto-review 升级到 GPT-5.6 Luna 并叠加降价,成本降约 10 倍——「让第二个模型审代码」变得几乎免费。
Anthropic · Claude Code:hooks(在工具调用前后插入确定性检查)、测试关卡、截图对比等机制早已内建,官方最佳实践长期强调「给 Claude 一个能自己跑的验证目标」。
Meta · Muse Code (Aug 6): a coding assistant on Muse Spark 1.2 whose headline pitch literally reads "writes code, fixes bugs, automatically verifies results, manages complex projects" — verification promoted from technique to flagship feature.
OpenAI · Auto-review (Aug 4): Auto-review in the ChatGPT app and Codex CLI upgraded to GPT-5.6 Luna with new pricing, roughly 10x cheaper — having a second model review your code is now nearly free.
Anthropic · Claude Code: hooks (deterministic checks before/after tool calls), test gates, and screenshot comparison have long been built in; official best practices consistently stress "give Claude a verification target it can run itself."
三条路线殊途同归:编码 Agent 的竞争焦点已经从「能不能写出来」转向「能不能自证正确」。原因也直白——模型无人值守跑得越久,人工检查每一步就越不现实,验证必须自动化。
Three routes, one destination: the competitive frontier has shifted from "can it write the code" to "can it prove the code right." The reason is plain — the longer agents run unattended, the less feasible human review of every step becomes, so verification has to be automated.
一个完整闭环有四个环节:生成 → 执行 → 验证 → 反馈。关键设计点在于:验证信号必须是「可执行获得的」——模型自己声称做完了不算数,要有外部世界(测试进程、编译器、浏览器截图)给出的证据。
A full loop has four stages: generate → execute → verify → feed back. The critical design point: the verification signal must be obtainable by execution — the model claiming it's done doesn't count; evidence must come from the outside world (a test process, a compiler, a browser screenshot).
| 类型 | 例子 | 信号质量 | 成本 |
|---|---|---|---|
| 静态检查 | lint、类型检查、编译 | 确定性,秒级,但只抓「形」的错 | 极低 |
| 动态测试 | 单元/集成测试、e2e | 确定性,抓行为错误;质量取决于测试覆盖 | 低-中 |
| 感知验证 | 截图对比、无障碍树检查 | 抓 UI/视觉错误,规则难写,常配视觉模型判断 | 中 |
| 模型评审 | Auto-review、LLM-as-Judge | 覆盖面广(风格/安全/边界情况),但本身概率性 | 中-高 |
| Type | Examples | Signal quality | Cost |
|---|---|---|---|
| Static checks | lint, type check, compile | Deterministic, seconds; catches errors of form only | Very low |
| Dynamic tests | unit / integration / e2e | Deterministic, catches behavioral bugs; only as good as coverage | Low-mid |
| Perceptual checks | screenshot diff, accessibility tree | Catches UI/visual bugs; rules are hard to write, often paired with a vision model | Mid |
| Model review | Auto-review, LLM-as-Judge | Broad coverage (style/security/edge cases) but itself probabilistic | Mid-high |
工程上把四类排成漏斗:便宜且确定的在前(lint→测试),贵且概率性的在后(截图→模型评审)。前面拦下 80% 的低级错误,后面的昂贵验证只处理漏网之鱼。OpenAI 把 Auto-review 降价 10 倍的意义就在这:漏斗最贵的一层突然便宜了,整条漏斗可以更激进地全量启用。
In practice you arrange the four as a funnel: cheap deterministic checks first (lint→tests), expensive probabilistic ones last (screenshots→model review). The front catches 80% of trivial errors; the costly layers only see what slips through. That's why OpenAI's 10x Auto-review price cut matters: the most expensive funnel layer just got cheap enough to run on everything.
# CLAUDE.md
## 验证纪律 / Verification discipline
- 每次修改后必须运行:npm run lint && npm test
- 测试不过不允许宣布任务完成
- UI 改动需截图对比 docs/design/ 下的设计稿CLAUDE.md 是「请求」,hook 是「强制」。PostToolUse hook 在每次文件修改后自动跑检查,失败信息直接回流给模型:
CLAUDE.md is a request; a hook is enforcement. A PostToolUse hook runs checks after every file edit and pipes failures straight back to the model:
// .claude/settings.json
{"hooks": {"PostToolUse": [{
"matcher": "Edit|Write",
"hooks": [{"type": "command",
"command": "npm run lint --silent || echo 'LINT FAILED — fix before proceeding'"}]
}]}}# 任务完成前派一个 subagent 独立评审
> 用一个 subagent 以评审者身份重新读这次 diff:
> 找边界条件、错误处理缺失和安全问题,输出问题清单;
> 若清单非空,先修复再重新评审。评审者不该看到实现者的推理过程——独立上下文才能避免「自己查自己」的确认偏误,这正是 subagent 隔离上下文的价值。
The reviewer shouldn't see the implementer's reasoning — an independent context avoids the confirmation bias of grading your own homework, which is exactly what subagent context isolation buys you.
坑 1 · 面向测试作弊(reward hacking):验证器成了优化目标,模型可能改测试、写死返回值来「通过」。对策:hook 禁止修改测试文件;评审模型专查「实现是否绕过了测试意图」。
坑 2 · 验证器覆盖不足产生假安全感:测试全绿不等于正确,只等于「没触发已写的断言」。对策:把「为新逻辑补测试」写进任务定义,让覆盖率关卡也进漏斗。
坑 3 · 无限重试烧钱:失败-重试循环没有上限时,一个死结能烧掉一晚上预算。对策:3-5 次重试后强制停下、汇总失败证据升级给人。
坑 4 · 把概率性验证当确定性用:模型评审会漏报误报,不能作为唯一关卡,只能作为漏斗末端的补充层。
Pitfall 1 · Gaming the verifier (reward hacking): once the verifier is the objective, the model may edit tests or hard-code returns to "pass." Counter: hooks that forbid touching test files; a reviewer model that checks whether the implementation dodges test intent.
Pitfall 2 · False confidence from thin coverage: all-green tests only mean "no written assertion fired," not "correct." Counter: make "add tests for new logic" part of the task definition and gate on coverage too.
Pitfall 3 · Unbounded retries burn money: without a cap, one deadlock can eat a night's budget. Counter: hard-stop after 3-5 retries, summarize failure evidence, escalate to a human.
Pitfall 4 · Treating probabilistic checks as deterministic: model review has false negatives and positives; it's a supplementary last funnel layer, never the only gate.