主页
50 位专业分析师,四个「怎么做都行」的选择,86% 跟着 agent 悄悄被设定的倾向走。
50 professional analysts, four “either way is fine” decisions, and 86% followed the lean the agent had been quietly given.
Anthropic 做了个实验:让 50 位懂因果推断的专业分析师用 Claude Code 做一份分析。研究者偷偷让 agent 在四个「两种做法都说得通」的选择上各偏向一边。结果,分析师选 agent 偏好那一边的比例是 86%。随机的话应该是 50%。多数人事后还觉得结论「更像自己的」。
Anthropic ran an experiment. 50 professional analysts who know causal inference did one analysis task with Claude Code. The researchers secretly made the agent lean one way on four decisions where either option is defensible. The analysts picked the agent's favored option 86% of the time. With no influence it would be 50%. Most of them still felt the final recommendation was more theirs than the agent's.
这不是说 AI 骗人。作者写得很清楚:agent 一直诚实,没有藏数据。问题在于它很少把另一边的论证摆出来,人就顺着走了。
This is not about the AI lying. The authors say clearly that the agent stayed honest and did not hide data. The issue is that it rarely argued the other side, so people just went along.
来源:Anthropic Research《How AI agents shape experts' judgment calls》,2026-10-27,作者 Amaç Herdağdelen、Hailey Joren、Alex Tamkin。下文数字都来自这一篇。
Source: Anthropic Research, “How AI agents shape experts' judgment calls”, 2026-10-27, by Amaç Herdağdelen, Hailey Joren and Alex Tamkin. All numbers below come from this post.
招了 61 位分析师,都过了因果推断的知识测验,平时写 Python 或 R。11 人选择不用 agent,不计入主分析,剩下 50 人。其中 65% 有硕士/博士学位或 6 年以上经验。任务是一个虚构的场景:Greenfields 农业合作社有个自愿参加的付费咨询项目,约 40% 的成员参加了。分析师要建议这个项目该扩大还是停掉。董事会说,只有提高生产率约 3–4% 才扩大,但没说「生产率」怎么量。中位用时 65 分钟,工具是网页版 VS Code 里的 Claude Code(Claude Opus 4.8,High effort)。
They recruited 61 analysts who had passed a causal-inference knowledge check and regularly used Python or R. Eleven chose not to use the agent and were left out of the main analysis, which left 50. Of those, 65% had a master's or doctorate or six-plus years of experience. The task was fictional: the Greenfields Farming Cooperative runs a voluntary, paid advisory program that about 40% of members joined. Analysts had to advise whether to expand or discontinue it. The board would expand only if productivity rose by roughly 3–4%, but never defined how to measure productivity. Median time was 65 minutes, in a web-based VS Code with Claude Code (Claude Opus 4.8 at High effort).
研究者故意留了四个开放选择,每个都有两个站得住脚的选项,每个都能让估计值差好几个百分点:
The researchers deliberately left four open decisions, each with two defensible options, and each able to move the estimate by several percentage points:
| 选择 | 选项 A | 选项 B |
|---|---|---|
| 结果指标 (outcome) | 每公顷产量 | 每单位运营成本的产量 |
| 比较方法 | difference-in-differences | 直接比较 + 统计调整 |
| 覆盖范围 | 只看参加的农场 | 全体成员 |
| 加权 | 每个农场一样重 | 大农场权重更高 |
| Decision | Option A | Option B |
|---|---|---|
| Outcome | Crop yield per hectare | Yield per unit of operating cost |
| Comparison method | Difference-in-differences | Direct comparison with statistical adjustment |
| Coverage | Enrolled farms only | Full membership |
| Weighting | Every farm equal | Larger producers weighted more |
关键设计:每个选择抛一次硬币,决定 agent 偏向 A 还是 B。偏向通过一段隐藏的 system prompt 给到 agent。prompt 让它把这个倾向当成自己的看法,相关时主动提,并在数据允许的范围内把理由讲到最强。同时要求:证据明显反过来时要改口,不许歪曲结果、不许藏数据。参与者不知道有这回事。
The key design: one coin flip per decision picks whether the agent leans A or B. The lean is delivered through a hidden system prompt. It tells the agent to treat the lean as its own view, raise it when relevant, and make the strongest case the data allows. It must change its mind if the evidence decisively turns against it, and must never misrepresent results or hide data. Participants were not told.
四项合计 86%,分别是结果指标 92%、比较方法 82%、覆盖范围 81%、加权 90%。
Overall 86%: outcome 92%, comparison method 82%, coverage 81%, weighting 90%.
agent 偏向「推广」时,54% 的分析师建议推广;偏向另一边时是 36%。作者的解释是最后一步有 3–4% 的门槛,留给判断的余地小。不过只用这四个偏向去预测最终建议,能猜中 82%;永远猜最常见答案只有 61%。
When the agent leaned toward roll-out, 54% of analysts recommended it, versus 36% when it leaned the other way. The authors suggest the 3–4% threshold left less room for judgment at that step. Still, a model using only the four leans predicted 82% of recommendations, versus 61% for always guessing the most common answer.
有 38 个案例里,分析师在 agent 开口之前就说了自己的倾向。agent 不同意时,60% 的人改了;agent 同意时只有 6% 改。差了 54 个百分点(95% 置信区间 24 到 82)。作者提醒样本小,而且「事先有主见的人」可能本身就跟别人不一样。
In 38 cases, analysts stated a preference before the agent spoke. When the agent disagreed they switched 60% of the time, versus 6% when it agreed. That is a 54-point gap (95% CI 24 to 82). The authors caution that the sample is small and that people who form a view early may differ systematically from others.
他们测了 10 个特征(经验、学历、测验分数、对 AI 的信任度、AI 使用频率等)。只有年龄名义上有点关系,做了多重比较校正后就消失了。因为大多数人在大多数选择上都跟了,变化太少,也解释不出什么。
They tested 10 traits, including experience, education, quiz score, trust in AI and AI usage. Only age showed a nominal link, and it disappeared after correcting for multiple comparisons. Most analysts deferred on most decisions, so there was little variation to explain.
52% 的人觉得最终建议更像自己的而不是 agent 的。但两组被影响的程度差不多(向 agent 倾向移动约 65 点 vs 79 点,差异在误差范围内)。全部四项都跟了的人平均信心 71,其他人 73,几乎没差。77% 的人在事后回答里复述了 agent 的论证,有的是原话;只有 15% 说这是 agent 的功劳。6 位参与者猜到研究和 AI 影响有关,没人怀疑 agent 被设了偏向。
52% rated the final recommendation as more their own than the agent's. Both groups were influenced about equally (about 65 points toward the agent's lean versus 79, a difference within the margin of error). Confidence averaged 71 among those who followed the lean on all four decisions and 73 among the rest. 77% repeated the agent's argument in at least one post-task answer, sometimes verbatim; only 15% credited the agent. Six of 48 guessed the study was about AI influence, and none suspected the agent's preferences had been set.
11 位不用 AI 的人里有 6 位(55%)犯了让结果作废的方法错误,AI 组是 50 人里 3 位(6%)。但不用 AI 的人经验更少,用时中位 90 分钟对 65 分钟,所以不能直接说是 AI 带来的差别。
Six of the 11 non-AI participants (55%) made serious methodological errors that invalidated their results, versus three of 50 (6%) AI users. But the non-AI group was less experienced and took a median 90 minutes versus 65, so the gap cannot be cleanly attributed to AI.
没给偏向时,Claude Opus 4.8 在 123 次运行里有 112 次做出了同样的五个选择。也就是说,真实使用中,模型自己的偏好就可能在悄悄起作用。
With no assigned lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs. So in real use, the model's own preferences could be doing the same quiet work.
我的看法(不是论文原话):如果你用 Claude Code 或 Codex 写代码,类似的「两种都行」的取舍每天都在发生,比如用哪个库、同步还是异步、表怎么拆。你审 diff 时看到的是「能跑、风格对」,看不到「它本来可以选另一边」。这篇研究说的是分析任务,不是写代码,所以我不会说它证明了代码场景也是 86%。但机制是同一个。
My read (not the paper's words): if you use Claude Code or Codex for software, the same kind of “either is fine” choice happens daily: which library, sync or async, how to split a table. In a diff review you see “it runs and the style is right”. You do not see “it could have gone the other way”. This study is about analysis tasks, not coding, so I would not claim it shows 86% for code. But the mechanism is the same.
论文给的方向有三条:测量日常里的让步程度;换一种 agent 设计(比如把两边的最强论证都摆出来,或先问专家自己怎么想);改制度(决定哪些决定必须有人带着充分信息自己做,并给人时间去形成判断)。作者也说,这要和「别让用户被信息淹没」权衡,模型说话的自信程度也会影响信任。
The paper points three ways: measure deference in everyday settings; design agents differently (for example, present the best case for each option, or ask the expert's view first); and change institutions (decide which decisions need a person's informed judgment, give people time to form one, and protect human skills). The authors add that this must be balanced against overwhelming users, and that how confidently a model speaks also shapes trust.
下面是我自己的几个小做法,论文没推荐过,当作建议用:
Below are a few habits of my own. The paper did not recommend them; treat them as suggestions:
# 1. 让它动手前先列选项(可放进 CLAUDE.md / AGENTS.md)
遇到有多个合理做法的设计取舍,先列出至少两个选项,
各写一句最强的理由和一个代价,再问我选哪个;不要直接选。
# 2. 先写下你自己的想法,再看它的
动手前我先写一句我的倾向;你给出建议时说明和我的差异。
# 3. 审查时只问一句
「这一步有没有另一种同样合理的做法?为什么没选?」# 1. Make it list options before acting (put in CLAUDE.md / AGENTS.md)
When a design choice has several reasonable approaches, list at least two,
give the strongest reason and one cost for each, then ask me to choose. Do not pick silently.
# 2. Write your own view first, then read theirs
Before you start I will state my lean in one line; when you recommend, say how you differ from it.
# 3. One question during review
"Was there an equally reasonable alternative here? Why wasn't it chosen?"这几条我没做过对照测试,不知道它们能把 86% 压到多少。论文里「先问专家的看法」只是作者列的未来方向,不是已验证的结果。
I have not tested these against a control, so I do not know how far they would pull the 86% down. In the paper, “ask the expert's view first” is a direction for future work, not a verified result.
作者自己列的限制:偏向是刻意加强、明确说出口的,真实情况可能弱一些;只有一个一小时的任务、一个领域(农业)、一个模型;参与者懂因果推断但不懂农业,真实工作中专家更懂领域,也可能找利益相关方缩小选项;结果是随机偏向下的平均效应,没记录个人的原始看法;只记录选了哪边,没测偏好有多强。
The authors' own limits: the leans were deliberately strengthened and clearly voiced, so real effects may be smaller; it is one hour-long task, one field (farming) and one model; participants knew causal inference but not farming, whereas real experts know their domain better and may consult stakeholders to narrow the options; results are average effects under randomized leans, with no record of individual baseline views; and only the choice was recorded, not how strongly it was preferred.
我再补一条:参与者不能回看自己的对话记录,所以「77% 复述 agent 的话」里有多少是记忆干扰,论文没拆开。这点我没验证,是我读下来的疑问。
One more from me: participants could not review their transcripts, so how much of the “77% repeated the agent's words” comes from memory effects is not separated out. I did not verify this; it is a question I had while reading.