130 秒速览

1The 30-second version

一句话:当任务描述缺了关键信息时,编码 Agent 有两条路——自己猜一个最可能的意思接着干,或者停下来问你。基准测试通常只奖励“做对”,不奖励“问对”,于是越会刷分的模型越可能“大胆猜”;但真实工作里需求、约束、预算很少写全,猜错的代价由你来付。

In one sentence: when a task description is missing key information, a coding agent can either guess the most likely meaning and keep going or stop and ask you. Benchmarks usually reward “getting it right”, not “asking the right question”, so the better a model is at scoring, the more likely it is to guess boldly; but real requirements, constraints and budgets are rarely fully written down, and you pay for a wrong guess.

先说名词:歧义 / 欠规范(underspecification)指任务描述缺少完成它所需的信息;澄清提问(clarifying question)是 Agent 在动手前向人要这些信息;假设(assumption)是 Agent 没问、自己替你做的决定。

Quick vocabulary: ambiguity / underspecification means the task description lacks information needed to complete it; a clarifying question is the agent asking a person for that information before acting; an assumption is a decision the agent makes for you without asking.

2背景:一篇文章点燃的讨论

2Background: the essay that started it

作者 Mun logadan 在 2026-08-14 发文称,他和同事觉得 Opus 5 在日常使用中比 Opus 4.7、Opus 4.8 和 Fable “退了一步”,但他强调这不是能力退步——“它比 Opus 4.7 和 4.8 更强,在基准上甚至可比肩 Fable”。问题出在协作方式:

On 2026-08-14 Mun logadan wrote that he and his colleagues find Opus 5 a step back in day-to-day use compared with Opus 4.7, Opus 4.8 and Fable, while stressing this is not a capability regression: “it is a more capable model than Opus 4.7 and Opus 4.8 and even rivals Fable in benchmarks.” The problem is how it collaborates:

作者把原因归为两点,并自称“毫无根据的猜测”:一是行业追求能自我迭代的 AI;二是基准压力——好的基准往往“自成一体、无需提示或外部信息即可解”,按基准(包括可验证奖励的强化学习)选模型,会“天然选择那些在歧义面前做出大胆、通常正确的假设的模型”。他的反驳是:“现实生活不是基准”,在后果真实的场景里,“我不希望 Agent 拿它的最佳猜测去赌”。

The author offers two causes and calls them “baseless speculation”: the industry’s drive toward self-improving AI, and benchmark pressure. Good benchmarks tend to be self-contained and solvable without hints or outside information, so selecting models by benchmark (including RL with verifiable rewards) “inherently selects for models that make bold, usually-correct assumptions in the face of ambiguity.” His rebuttal: “Real life just isn’t a benchmark,” and when consequences are real, “I do not want an agent taking its best guess!”

据 Developers Digest 08-15 简报,HN 帖子约 873 分、794 条评论,讨论共识是“跑你自己的评测”。注意:原文没有给出具体对话记录或量化数据,属于用户体感;下文的研究也没有直接测 Opus 5,只说明这类行为可以被量化。

According to the Developers Digest brief of 08-15, the HN thread reached about 873 points and 794 comments, with the consensus “run your own evals”. Note that the essay gives no transcripts or measurements; it is a user impression. The research below also does not test Opus 5 directly; it shows that this behavior can be measured.

3核心概念

3Core concepts

4研究证据:三套基准怎么量化“该不该问”

4The evidence: how three benchmarks measure “should it ask”

Ambig-SWE(CMU 等,arXiv 2502.13069,2025-02 首发,v3 于 2026-02):把 SWE-bench Verified 改成欠规范版本,分三步评估——能否察觉信息不足、能否提出有针对性的问题、能否利用回答。结论:模型很难区分“说清楚了”和“没说清楚”的指令;信息不全时倾向于做无依据的假设而非提问,导致结果变差、工具误用和算力浪费;而一旦真正交互,性能比不交互最高提升 74%。

Ambig-SWE (CMU and others, arXiv 2502.13069, first posted 2025-02, v3 in 2026-02) turns SWE-bench Verified into an underspecified variant and evaluates three steps: detecting missing information, asking targeted questions, and using the answers. Findings: models struggle to tell well-specified instructions from underspecified ones; with incomplete instructions they tend to make unwarranted assumptions instead of asking, leading to worse results, tool misuse and wasted compute; when they do interact, performance improves by up to 74% over non-interactive settings.

ClarEval(吉林大学,arXiv 2603.00187):在 HumanEval 与 LiveCodeBench 上注入三类歧义,共 750 题、2,250 个实例,测 11 个代码 Agent。用“关键问题覆盖率(KQC)”“平均澄清轮数(ATC)”等指标衡量“问得全不全、快不快”。

ClarEval (Jilin University, arXiv 2603.00187) injects three kinds of ambiguity into HumanEval and LiveCodeBench tasks, 750 problems and 2,250 instances, and tests 11 code agents. Metrics such as Key Question Coverage (KQC) and Average Turns to Clarify (ATC) measure how complete and how quick the clarification is.

ClarEval 的几个要点:术语歧义最难(GPT-4o 消融中 Pass@1 仅 6.71%);GPT5-Coder 单轮 KQC 0.867、多轮 MPR 0.951、ATC 最低 1.493;Qwen2.5-Coder ATC 2.396,约多用 60% 的轮次;Aider-GPT5 单轮 KQC 排第二,多轮却掉到第十。作者认为顶尖模型已基本能察觉歧义,难点转为高效、完整地问。论文部分表格标签有不一致之处,模拟用户靠关键词触发(作者称与 LLM 裁判一致率 96.5%)。

ClarEval highlights: ambiguous terminology is hardest (Pass@1 only 6.71% in the GPT-4o ablation); GPT5-Coder has single-turn KQC 0.867, multi-turn MPR 0.951 and the lowest ATC, 1.493; Qwen2.5-Coder has ATC 2.396, about 60% more turns; Aider-GPT5 ranks second on single-turn KQC but tenth multi-turn. The authors argue top models have largely solved detecting ambiguity; the hard part is now asking efficiently and completely. Some table labels in the paper are inconsistent, and the simulated user relies on keyword triggers (reported 96.5% agreement with an LLM judge).

LHAW(Scale Labs,2026-02-17):面向长时程任务,从 MCP-Atlas、TheAgentCompany、SWE-Bench Pro 中挑出关键信息片段,用删除、模糊化、泛化等方式造出 285 个验证过的欠规范变体。发现:澄清能找回相当一部分损失但不是全部;Claude Opus 4.5 与 Gemini 3 Pro 问得更少、每问收益更高,GPT-5.2 问得多得多、绝对收益更高但单问收益低;提问成本低时 Agent 问得多、进展高,成本高时问得少、承担更多失败风险;各模型差异最大的是问题质量、过度澄清和误读回答。

LHAW (Scale Labs, 2026-02-17) targets long-horizon tasks: it picks key information segments from MCP-Atlas, TheAgentCompany and SWE-Bench Pro and deletes, vaguifies or genericizes them to build 285 validated underspecified variants. Findings: clarification recovers much but not all of the lost performance; Claude Opus 4.5 and Gemini 3 Pro ask less often and gain more per question, while GPT-5.2 asks far more, gaining more in absolute terms but less per question; when asking is cheap, agents ask more and progress further; when it is expensive, they ask less and accept more risk of failure; models differ most in question quality, over-clarification and misreading answers.

5三套基准对比

5Comparing the three benchmarks

6在 Claude Code 里怎么做

6What to do in Claude Code

Claude Code 本身提供了“问”的通道。官方工具文档(2026-10 检索)写明:AskUserQuestion 用于“收集需求或澄清歧义”的多选题,可选项之外还能通过 Other 行或备注框自由输入;问题默认一直等你回答。若设置 askUserQuestionTimeout(60s / 5m / 10m),超时后对话框会自动提交已选项,并告诉 Claude 你可能不在,让它按自己的判断继续,之后可再问;后台会话、读屏模式和 Remote Control 下计时器不启动;权限确认(包括计划审批)永远不会因空闲自动通过。

Claude Code already provides a channel for asking. The official tools reference (checked 2026-10) says AskUserQuestion asks multiple-choice questions “to gather requirements or clarify ambiguity”, and you can also type free text through the Other row or the notes field; questions stay open until you answer by default. If you set askUserQuestionTimeout (60s / 5m / 10m), an unanswered dialog closes on its own, submits any selected options and tells Claude you may be away, so it proceeds on its own judgment and can re-ask later; the timer never starts in background sessions, screen reader mode or Remote Control; permission prompts, including plan approval, never auto-resolve on idle.

以下是作者基于上述研究的建议做法,不是 Anthropic 官方指南:

The following are the author’s suggested practices based on the research above, not official Anthropic guidance:

## 澄清规则(CLAUDE.md 示例)
- 以下情形动手前必须用 AskUserQuestion 一次问全,给出 2-4 个选项:
  删除或迁移数据、修改公共 API / 数据库 schema、改 CI 或部署配置、需求里有两种以上合理理解。
- 其他情形可以做最小假设,但每次回复末尾列出「本次假设」与「待确认」。
- 不要在一轮里只问一个问题再等下一轮;相关问题合并提问。
- 不要擅自改动我给出的计划;需要改时先说明原因并征求同意。

## Clarification rules (CLAUDE.md example)
- Before any of the following, ask once with AskUserQuestion, offering 2-4 options:
  deleting or migrating data, changing a public API / database schema, changing CI or deploy config, or a request with two or more reasonable readings.
- Otherwise minimal assumptions are fine, but end every reply with "Assumptions made" and "Still to confirm".
- Don't ask one question per turn; group related questions together.
- Don't change a plan I gave you on your own; explain why and ask first.

7坑与未解问题

7Pitfalls and open questions

8资源与来源

8Resources and sources