30 秒版本

The 30-second version

编码 agent 的上下文快满时,会把前面的对话压成一段摘要,后面只带摘要往下干。这叫压缩(compaction)。Compact-Bench 这篇论文(10 月 22 日,中国科大团队)想知道:压缩让本来能做完的活,有多少做不完了。

When a coding agent's context fills up, it squeezes the earlier conversation into a summary and carries on with only the summary. That is compaction. Compact-Bench (October 22, a USTC team) asks how many tasks that were finishable before compaction stop being finishable after it.

答案是:不少。在 14 个做摘要的模型上,22 个"带完整历史能做对"的修复任务里,有 5% 到 64% 压缩后稳定做不对。回头还会再用到的字面细节(路径、commit 哈希、报错类型这些),LLM 写的摘要最多只留下 16.0%,而不用模型、直接机械提取能留 79.1%。

The answer: quite a few. Across 14 summarizer models, 5% to 64% of 22 repairs that full-history runs could solve were reliably lost after compaction. For literal details that come back later (paths, commit hashes, error types), LLM-written summaries keep at most 16.0%, while plain mechanical extraction keeps 79.1%.

压缩到底在干什么

What compaction does

长任务里 agent 每走一步,上下文就长一截:读过的文件、跑过的命令、试错的过程。上限一到,就得腾地方。主流做法是让模型把前面的内容总结成一段话,然后丢掉原文,从摘要继续。Claude Code、Codex、Gemini CLI、opencode、Kimi Code 各有自己的压缩 prompt,论文测了这五个。

In a long task the context grows with every step: files read, commands run, dead ends tried. At the limit something has to go. The common approach is to have a model summarize the earlier material, throw away the original, and continue from the summary. Claude Code, Codex, Gemini CLI, opencode and Kimi Code each ship their own compaction prompt, and the paper tests all five.

以前的评测多半看三类东西:省了多少 token、摘要里保留了多少信息、任务最终成没成功。问题是这三类很难分开回答"摘要本身到底造成了多少损失"。

Earlier evaluations mostly looked at three things: tokens saved, information retained, and end-to-end success. It is hard to separate "how much did the summary itself cost us" from those.

Compact-Bench 怎么测

How Compact-Bench measures it

核心是成对对比:同一个任务、同一个压缩前的仓库状态,一条跑道带完整历史继续(PRE),一条只带摘要继续(POST)。两边用同一个 agent、同样的工具、同样的 8 步上限,最后都由仓库的 FAIL_TO_PASS 测试判分。结果分四类:保住、受损、捡回、没解出。"稳定受损"的定义是 POST 三次里至少挂两次。

The core is a paired comparison: same task, same pre-compaction repository state. One lane continues with the full history (PRE), the other with only the summary (POST). Both use the same agent, tools and eight-step limit, and both are scored by the repository's FAIL_TO_PASS tests. Outcomes fall into four labels: preserved, damaged, rescued, unsolved. "Stable damage" means POST fails in at least two of three runs.

三套评测:

Three suites:

结果一:任务做不完了

Result 1: tasks stop getting finished

用 Claude Code 的压缩 prompt、Kimi-K3 当继续干活的 agent,换 14 个模型来写摘要:稳定受损比例从 5%(Claude Opus 4.8,22 个里坏 1 个)到 64%(DeepSeek-Flash,22 个里坏 14 个)。Claude Opus 4.8 保住了 22 个里的 21 个。重复跑确认了第一轮 126 个"摘要模型-任务"受损对里的 120 个。

With the Claude Code compaction prompt and Kimi-K3 as the continuing agent, swapping in 14 summarizer models gave stable damage from 5% (Claude Opus 4.8, 1 of 22) to 64% (DeepSeek-Flash, 14 of 22). Claude Opus 4.8 preserved 21 of 22. Repeat runs confirmed 120 of the 126 damaged model-task pairs from the first round.

换压缩 prompt 也有差别(Kimi-K3 写摘要,22 个任务里 POST 能过的数量):

The compaction prompt matters too (Kimi-K3 as summarizer; POST passes out of 22):

另外,同一个任务会被很多摘要模型一起搞坏:pytest-10081 被 14 个里的 13 个损坏,只有 Claude Opus 4.8 保住;sphinx-9230 被 9 个模型损坏,五个 prompt 下全挂。也有只在某个 prompt 下才坏的(比如 pytest-7490 只在 Claude Code 的 prompt 下坏)。

Failures also cluster on specific tasks: pytest-10081 is damaged by 13 of 14 summarizers and preserved only by Claude Opus 4.8; sphinx-9230 is damaged by nine summarizers and fails under all five prompts. Some failures are prompt-specific (pytest-7490 breaks only under the Claude Code prompt).

结果二:细节几乎没留下

Result 2: details barely survive

"重复细节"指后面的请求里还会原样出现的字面值。七个模型 × 五个 prompt 的矩阵里,LLM 摘要的保留率在 5.0% 到 16.0% 之间。用 Claude Code 的 prompt 时:DeepSeek-Flash 16.0%、Kimi-K3 15.0%、GPT-5.6-Sol 14.8%、GPT-5.6-Luna 13.3%、DeepSeek-V4-Pro 11.9%、GLM-5.3 9.5%、GPT-5.6-Terra 8.2%。对照组是不用模型的机械提取(anchor_index),保留率 79.1%,比最好的 LLM 摘要高 63.1 个百分点。

"Recurring details" are literal values that reappear verbatim in later requests. Across the seven-model by five-prompt matrix, LLM summaries retain between 5.0% and 16.0%. With the Claude Code prompt: DeepSeek-Flash 16.0%, Kimi-K3 15.0%, GPT-5.6-Sol 14.8%, GPT-5.6-Luna 13.3%, DeepSeek-V4-Pro 11.9%, GLM-5.3 9.5%, GPT-5.6-Terra 8.2%. The baseline is mechanical extraction (anchor_index) with no model involved: 79.1%, 63.1 points above the best LLM summary.

生产环境里的会话也一样:第一次压缩边界后只保留了 15.7%(106 个样本),第二次 19.9%(27 个),第三次 4.9%(6 个),第四次 1.8%(1 个)。后几次样本很少,这几个数别当趋势看。

Production sessions tell the same story: 15.7% kept after the first boundary (n=106), 19.9% after the second (n=27), 4.9% after the third (n=6), 1.8% after the fourth (n=1). The later samples are tiny, so do not read them as a trend.

为什么"留了证据"还是做不完

Why keeping the evidence is not enough

论文分析了受损的几个例子。在 Kimi-K3 的四个稳定受损格子里,每份摘要都保留了目标文件;pytest-7571 的摘要甚至保留了全部五个被审计的 gold-patch token,任务还是挂了。sphinx-9230 的摘要漏掉了一个关键推理:这里的 split(None, 1) 必须改成 rsplit。POST 那一跑就在仓库里到处搜,最后没做出修改。

The paper examines some damaged cases. In the four stably damaged Kimi-K3 cells, every summary kept the target file. The pytest-7571 summary even kept all five audited gold-patch tokens, and the task still failed. The sphinx-9230 summary dropped one key piece of reasoning: split(None, 1) had to become rsplit. The POST run then searched the repository broadly and never made the edit.

我会怎么用(我的推断)

What I would do (my inference)

• 把关键状态写到文件里,别指望摘要。机械提取能留 79.1%,这提示:路径、哈希、报错原文这类"字面值"放在一份 NOTES.md 或 TODO 里,比赌摘要靠谱。Claude Code 的 CLAUDE.md、Codex 的 AGENTS.md 本来也是这类持久文件。
• 压缩前,让 agent 先写"下一步准备改什么、为什么"。sphinx 的例子里丢的就是这类推理。这是把论文的失败原因倒过来用,我没有测过效果。
• 长任务里自己决定什么时候压,别等自动触发。论文显示第一次压缩后细节就只剩约 16%。自动触发往往落在任务中途,这后半句是我的推断,不是论文的发现。
• 想换摘要模型或 prompt,自己先做个 PRE/POST 小测。挑几个你常做的任务,在压缩点前后各跑一遍,看哪些翻车。这个方法直接借自论文的设计。

• Put key state in files, not in the summary. Mechanical extraction keeps 79.1%, which suggests literal values (paths, hashes, error text) are safer in a NOTES.md or TODO than left to a summary. CLAUDE.md in Claude Code and AGENTS.md in Codex are persistent files of this kind anyway.
• Before compacting, have the agent write down what it plans to edit next and why. The sphinx case lost exactly that kind of reasoning. This inverts the paper's failure cause; I have not tested it.
• Choose when to compact in long tasks instead of waiting for the automatic trigger. The paper shows only about 16% of details survive the first compaction, and a late compaction is more likely to land mid-task. That second half is my reasoning, not a finding.
• Before swapping the summarizer model or prompt, run a small PRE/POST test of your own. Pick a few tasks you often do, run them before and after the compaction point, and see which ones flip. The method is borrowed straight from the paper's design.

这也解释了为什么 Claude Code 2.1.311 的几条修复值得留意:压缩返回没有摘要时重试一次、自动压缩后 CLAUDE.md/MEMORY.md 过期的 bug 修复。这些是 changelog 里的事实;它们和论文的数字没有直接对应关系,论文测的是各家压缩 prompt,不是这一版的行为。

It also explains why a few Claude Code 2.1.311 fixes matter: retrying once when a compaction reply has no summary, and the fix for stale CLAUDE.md/MEMORY.md after automatic compaction. These are changelog facts; they do not map directly onto the paper's numbers, because the paper tests each tool's compaction prompt, not this version's behavior.

保留意见

Reservations

• 结果依赖这批修复任务、继续干活的 agent(Kimi-K3)和 8 步预算。换个 agent 或放宽步数,结果可能变。论文自己也这么说。
• "细节保留率"只数字面值的重复,不数语义。摘要把一个路径改写成同义描述,算丢。
• 生产链路后几次压缩的样本太少(最后一格只有 1 个)。
• 私有会话数据和缓存的修复状态没有公开,公开的是评分规则、配置和结果表。想完全复现要自己重做数据。
• 成本上,35 个修复的实验用了约 8,000 次模型调用,压缩本身占不到 5% 的 token。

• Results depend on this set of repairs, the continuing agent (Kimi-K3) and the 8-step budget. A different agent or a larger budget could move them. The paper says as much.
• "Detail retention" counts literal recurrence only, not meaning. A summary that rewords a path into a description counts as a loss.
• The later production boundaries have very few samples (the last cell has one).
• Private session payloads and cached repair states are not released; scoring rules, configurations and outcome tables are. Full reproduction means rebuilding the data.
• On cost: the 35-repair study used about 8,000 model calls, with compression under 5% of tokens.

术语表

Glossary

compaction(压缩)

把前面的对话总结成摘要、丢掉原文,从摘要接着干。

PRE / POST

PRE:带完整历史继续;POST:只带摘要继续。同一任务各跑一次来比较。

FAIL_TO_PASS

修复前失败、修复后应该通过的那组测试,用来判定修复成不成功。

stable damage(稳定受损)

PRE 能过,但 POST 三次里至少挂两次。

anchor_index

论文里的机械提取基线:不用模型,直接把路径、哈希这类字面值抽出来保留。

summarizer backbone

负责写摘要的那个模型。

compaction

Summarizing earlier conversation, discarding the original, and carrying on from the summary.

PRE / POST

PRE: continue with full history. POST: continue with only the summary. Run once each on the same task to compare.

FAIL_TO_PASS

The tests that fail before a fix and should pass after it; they decide whether a repair worked.

stable damage

PRE passes but POST fails in at least two of three runs.

anchor_index

The paper's mechanical-extraction baseline: no model, just pull out literal values like paths and hashes and keep them.

summarizer backbone

The model that writes the summary.

来源

Sources

• Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents(arXiv 2610.31710v1,Sun、Cai、Zhao、Zhou、Li,2026-10-22;主要来源,通过 HTML 版摘要读取)
• Claude Code changelog(2.1.311,2026-10-26)

• Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents (arXiv 2610.31710v1, Sun, Cai, Zhao, Zhou, Li, 2026-10-22; primary source, read via a summary of the HTML version)
• Claude Code changelog (2.1.311, 2026-10-26)