主页
深度学习文档 · 2026-09-18
Deep Dive · 2026-09-18
给智能体上记忆,大多数人的默认动作是「切片 + 向量库 + 语义检索」。但在事实会被修改的场景里——地址、配置、日程,也就是绝大多数运营事实——这个默认选择不只是稍差一点,而是不可预测。本文用一份 2026 年 9 月的实证研究,把记忆从「存得准不准」重讲成「做得对不对」。
The default move when adding memory to an agent is "chunk it, embed it, retrieve it." But for facts that get revised — addresses, configs, schedules, which is to say most operational facts — that default is not merely a bit worse. It is unpredictable. This piece uses a September 2026 empirical study to reframe memory from "did it store the right thing" to "did the agent do the right thing."
先说结论,再解释为什么。如果你的 agent 依赖的事实会被更新(用户改了地址、运维改了配置、日程被挪了),那么基于向量检索的记忆是错误的默认选项——不是慢一点或者差一点,而是同一套系统换个随机种子成功率能从 0.45 跳到 0.90,换个模型能从 0.30 跳到 0.95。原因很朴素:检索会把「旧值」和「新值」并排塞进上下文,而检索本身不带时间仲裁,agent 只能靠猜。
Conclusion first, explanation after. If the facts your agent depends on get revised — the user changed their address, ops changed a config, a meeting moved — then vector-retrieval memory is the wrong default. Not slightly slower or slightly worse: the same system swings from 0.45 to 0.90 success just by changing the random seed, and from 0.30 to 0.95 across agent models. The reason is plain: retrieval puts the stale value and the fresh value side by side in context, and retrieval itself carries no recency arbitration. The agent is left guessing.
对应的正确默认是「写时覆盖」(update-on-write):在写入记忆的那一刻就把冲突解决掉,而不是留到读取时让模型去判断。结构化事实库属于这一类;有点出人意料的是,LLM 摘要也属于这一类——因为它每个 episode 都重写一遍摘要,天然保留最新值。
The right default is update-on-write: resolve the conflict at write time instead of leaving it for the model to adjudicate at read time. A structured fact store does this. Somewhat surprisingly, LLM summarization does too — because it rewrites the summary every episode, it naturally keeps the latest value.
图 1:长上下文解决「容量」,记忆解决「连续性」。两者失败方式不同,同一个分数不能同时衡量两者。
Fig. 1 — Long context solves capacity; memory solves continuity. They fail differently, and one score cannot measure both.
2026 年上下文窗口已经到了百万级别,一个很自然的想法是:那还要什么记忆系统?把历史全塞进去不就行了。这个想法有三处站不住。
Context windows reached a million tokens in 2026, so the natural thought is: why build a memory system at all, when you can just stuff the whole history in? Three things break that idea.
NIAH、RULER、BABILong、InfiniteBench、LongBench 这些常被拿来当「记忆能力」证据的基准,测的都是单次固定输入上的注意力:给你一大段文字,里面藏一根针,你能不能找出来。这里没有写入步骤、没有淘汰、没有按用户分区、没有 token 预算。而记忆系统至少是四件事:从对话流里抽取有用事实、写进外部存储、推理时取回正确的那些、以及在它们过时时更新或删除。只测检索,等于只测了四分之一。
NIAH, RULER, BABILong, InfiniteBench and LongBench — routinely cited as evidence of "memory" — all measure attention over a single fixed input: here is a long passage with a needle in it, can you find it. There is no write step, no eviction, no per-user partition, no token budget. Memory, meanwhile, is at least four capabilities: extracting useful facts from a stream of dialogue, writing them to an external store, retrieving the right ones at inference time, and updating or pruning them when they go stale. Testing retrieval alone tests one quarter of the system.
长上下文基准失败,是因为模型注意到了错误的片段。记忆基准失败,可能是因为系统存错了东西、存对了但取错了、或者存对取对却排序排坏了。混淆两者会让一个系统看起来比实际更好。
A long-context benchmark fails when the model attends to the wrong span. A memory benchmark fails when the system stores the wrong thing, stores the right thing but retrieves the wrong one, or stores and retrieves the right thing but ranks it badly. Conflating them makes a system look better than it is.
MERIT 研究做了一个很干脆的对照实验:在「泄漏检查」(leak check)保证下——即答案所需的事实确实不在本次 episode 的任何输入里、也不在世界初始状态里——把记忆整个关掉(C0 条件)。结果:九个(领域×难度)格子上,C0 全部得 0.000。这个 0 在 Claude Opus 4.8 和 Claude Sonnet 5 这两个 2026 世代的前沿模型上同样成立。而这些模型在「本 episode 内可解」的独立任务上得分 0.83–1.00,说明任务本身没问题。
The MERIT study ran a blunt control: with a leak check guaranteeing that the required fact is genuinely absent from the episode's inputs and from the initial world state, turn memory off entirely (condition C0). Result: C0 scored 0.000 in all nine (domain × difficulty) cells. That zero holds for Claude Opus 4.8 and Claude Sonnet 5, both 2026-generation frontier models. On independent tasks solvable within the episode, the same models score 0.83–1.00, confirming the tasks themselves are solvable.
一个在每次查询注入 5 万 token 记忆的系统,和一个注入 7 千 token 达到同样准确率的系统,不是同一个产品。mem0 公布的数据是:四个基准上平均每次检索 6.7K–7.0K token,而全上下文基线每次查询消耗 25,000+ token,相当于同等准确率下约 3–4 倍的 token 成本差。MERIT 用更严格的方式说了同一件事:全量回放在任何一个领域都不是经济选择,单位成本上比该领域最优方案差 2.7–3.9 倍。
A system that injects 50K tokens of memory per query is not the same product as one that hits the same accuracy with 7K. Mem0 reports a mean of 6.7K–7.0K tokens per retrieval across four benchmarks, against full-context baselines consuming 25,000+ tokens per query — roughly a 3–4× token cost difference at comparable accuracy. MERIT says the same thing more strictly: full replay is never the economical choice in any domain, running 2.7–3.9× worse per unit of success than that domain's best condition.
长上下文像是「把整个书库搬到你桌上,你自己翻」。书确实都在,但每次都要重新搬一遍(贵),而且书越多你越容易翻错页(注意力衰减)。记忆系统则是「有个图书管理员,平时帮你做卡片索引,你来了他递给你该看的那几页」。管理员会不会记卡片(写入)、卡片准不准(抽取质量)、旧卡片过期了会不会换掉(更新)——这些才是记忆系统的真问题,而它们在「把书搬上桌」这件事里根本不存在。
Long context is "wheel the entire stacks onto your desk and flip through them yourself." The books are all there, but you pay to wheel them in every single time, and the more there are the likelier you turn to the wrong page. A memory system is "a librarian who keeps index cards and hands you the few pages you need." Whether the librarian writes cards at all, whether the cards are accurate, and whether stale cards get replaced — those are memory's real problems, and none of them exist in the wheel-the-stacks-in approach.
在讨论谁好谁坏之前,先把候选项统一到一个接口下——所有记忆系统都只做两件事:write(episode)(这一轮结束,把该记的记下来)和 read(context)(这一轮开始,把该给的给出来)。MERIT 正是按这个接口把业界主流架构重新实现成五个条件,以便把架构机制和产品工程解耦比较。
Before arguing about which is better, put all candidates behind one interface. Every memory system does exactly two things: write(episode) — at the end of a turn, store what's worth storing — and read(context) — at the start of a turn, surface what's worth surfacing. MERIT reimplements the field's main architecture families as five conditions behind exactly this interface, so that the mechanism can be compared separately from the product engineering around it.
text-embedding-3-small)。关键性质:它累积——新旧值会并存。text-embedding-3-small). Key property: it accumulates — stale and fresh values coexist.图 2:差别不在「存了多少」,而在「冲突在哪一侧解决」。C2 把冲突推给读取时的模型;C3/C4 在写入时就消灭它。
Fig. 2 — The difference isn't how much is stored, but on which side the conflict is resolved. C2 defers it to the model at read time; C3/C4 kill it at write time.
下面的数字全部来自 arXiv 2609.05441(MERIT,2026 年 7 月预印本 v1.1,9 月检索到)。实验规模:两代试点(gpt-4.1-mini,9,940 个 episode)加上一份预注册的 3 模型 × 3 领域 × 3 难度 × 3 种子完整网格(13,500 个 episode),合计 23,440 个计分 episode、42.57 美元 API 成本。所有配对比较用按 arc 聚类的配对 bootstrap(10,000 次重采样)、Holm–Bonferroni 校正。
All numbers below come from arXiv 2609.05441 (MERIT, July 2026 preprint v1.1, retrieved September 2026). Scale: a two-generation pilot (gpt-4.1-mini, 9,940 episodes) plus a preregistered 3-model × 3-domain × 3-tier × 3-seed grid (13,500 episodes), totalling 23,440 scored episodes at $42.57 in API cost. Paired comparisons use an arc-clustered paired bootstrap (10,000 resamples) with Holm–Bonferroni correction.
在「hard 档」(事实先被建立、后被修改、探测时必须用最新值)上,向量检索(C2)跌到 0.35–0.70,而全量回放(C1)保持 0.95–1.00、事实库(C4)保持 0.75–1.00。这本身还只是「更差」。真正的问题在完整网格里才暴露出来:
On the hard tier (a fact is planted, later revised, and the probe requires the latest value), embedding retrieval (C2) falls to 0.35–0.70 while full replay (C1) holds 0.95–1.00 and the fact store (C4) holds 0.75–1.00. That alone is just "worse." The real problem only surfaces in the full grid:
| hard 档 · 依赖任务成功率Hard tier · dependent-task success | 领域Domain | Haiku 4.5 | GPT-4.1 | gpt-4.1-mini (±sd) |
|---|---|---|---|---|
| C2 向量检索C2 embedding retrieval | D1 | 0.90 | 0.70 | 0.70 ± 0.23 |
| D2 | 0.95 | 0.45 | 0.38 ± 0.08 | |
| D3 | 0.30 | 0.60 | 0.37 ± 0.03 | |
| C3 LLM 摘要C3 LLM summarization | D1 | 1.00 | 1.00 | 0.98 ± 0.03 |
| D2 | 1.00 | 0.90 | 0.80 ± 0.10 | |
| D3 | 1.00 | 1.00 | 1.00 ± 0.00 | |
| C4 结构化事实库C4 structured fact store | D1 | 1.00 | 1.00 | 0.93 ± 0.08 |
| D2 | 1.00 | 1.00 | 0.90 ± 0.05 | |
| D3 | 0.70 | 0.90 | 0.90 ± 0.09 |
记忆侧全部固定为 gpt-4.1-mini,唯一变动的是 agent 模型。C0 地板线为 0.00–0.05,C1 全量回放对每个模型都是 1.00(表中略去)。领域:D1 电商客服 / D2 IT 运维 / D3 个人助理。
The memory side is pinned to gpt-4.1-mini throughout; the agent model is the only varying factor. The C0 floor is 0.00–0.05 and C1 full replay is 1.00 for every model (omitted). Domains: D1 retail support / D2 IT ops / D3 personal assistant.
一个开发者用单模型、单种子的基准分数来选记忆系统,会系统性地高估或低估向量检索——同一套系统诚实地报告出来的数字可以是 0.45,也可以是 0.90。只有「多种子均值 + 离散度」才是站得住的汇报方式。
A practitioner choosing a memory system from a single-model, single-seed benchmark number will systematically over- or under-estimate embedding retrieval — the same system can honestly report anywhere from 0.45 to 0.90. Seed means reported with their spread are the only defensible summary.
C5(事实库 + 检索)在 hard 档的三个领域上是 0.50–0.80,而单独用 C4 是 0.75–1.00。原因很直白:检索那一半把事实库已经消除掉的陈旧性重新导了回来。在陈旧污染实验里,C5 也是唯一一个通过 Holm 校正的显著受害者(污染率 ρ=0.3 时陈旧记忆伤害 +0.25,Holm 校正后 p=0.030)。
C5 (fact store + retrieval) scores 0.50–0.80 across the three domains on the hard tier, while C4 alone scores 0.75–1.00. The reason is direct: the retrieval half re-imports the staleness the fact store had eliminated. In the corruption sweep, C5 is also the only condition with a Holm-significant harm (stale-memory harm of +0.25 at corruption rate ρ=0.3, Holm-adjusted p=0.030).
可迁移的教训:不要假设混合方案会继承更好那个组件的行为。它更可能继承更差那个组件的失败模式。要么实测,要么别混。
The transferable lesson: do not assume a hybrid inherits the better component's behaviour. It is more likely to inherit the worse component's failure mode. Measure it, or don't hybridize.
这是整篇研究里最容易被忽略、却最该被工程师记住的一条。MERIT 定义了一个指标叫 MUR(memory utilization rate,记忆利用率):只统计那些「正确的值确实已经出现在检索回来的记忆块里」的 episode,看 agent 在实际执行的工具调用参数里有没有用上这个值。无视率 = 1 − MUR。
This is the finding most easily skipped and most worth remembering. MERIT defines MUR (memory utilization rate): among only those episodes where every gold value was present in the retrieved memory block, the fraction where each value actually appears in the executed tool-call arguments. Ignore Rate = 1 − MUR.
结果:hard 档汇总三个领域,正确的最新值出现在 C2 的记忆块里共 55 次,agent 真正据此行动的只有 30 次——无视率 0.45。把检索质量从关键词升级到向量嵌入,这个数字几乎不动(试点一代是 0.53)。更扎心的是 C1:记忆块就是干干净净的完整逐字记录,在多事实(medium 档)episode 上无视率仍然高达 0.50。
The result: pooling domains on the hard tier, the correct latest value appeared in C2's retrieved block in 55 probe episodes; the agent acted on it in 30 — an Ignore Rate of 0.45. Upgrading retrieval from keyword overlap to embeddings barely moves it (0.53 in the starter generation). More pointedly, C1 — whose memory block is a clean full transcript — still ignores up to 0.50 of held facts on multi-fact (medium-tier) episodes.
研究者从 trace 里看出两种模式:(i) 检索条件下,新旧值同时出现,agent 会取平均、发问、或干脆选了旧的——因为没有来源和时间戳可供仲裁;(ii) 多事实组合时,agent 只对「和任务措辞最近的那几个事实」下手。
The authors identify two patterns in the traces: (i) under retrieval, stale and fresh values co-occur and the agent averages, asks, or picks the stale one — because there is no provenance or timestamp to arbitrate with; (ii) under multi-fact composition, agents act on the subset of facts nearest the task phrasing.
给记忆块加上来源标注、时间戳、显式的冲突提示(「这个键有两个值,较新的是 X」),和优化你的向量检索同样重要——很可能更重要,因为 45% 的损失发生在检索已经成功之后。
Adding provenance, recency markers, and explicit contradiction surfacing to the memory block ("this key has two values; the newer one is X") matters as much as tuning your retriever — probably more, since 45% of the loss happens after retrieval already succeeded.
图 3:记忆系统准确率会高估端到端收益,除非利用率也被测量。
Fig. 3 — Memory-system accuracy overstates end-task benefit unless utilization is measured too.
MERIT 做了一件很聪明的事:把同一个架构的「简易实现」和「真实实现」放在完全相同的网格上各跑一遍。结果是双向的:
MERIT does something clever: it runs a "starter" and a "real" implementation of each architecture on the identical grid. The swap moves things in both directions:
研究者称之为「抽取税」:结构化记忆必然要为抽取质量付费,要么付领域工程的人力,要么付计量出来的 token,但绝不会是零。
The authors call this the extraction tax: structured memory always pays for extraction quality, either in domain engineering or in metered tokens, but never zero.
| 基准Benchmark | 年份Year | 它测什么What it tests | 产出类型Output type |
|---|---|---|---|
| LoCoMo | 2024 | 超长多会话对话召回。每段对话平均约 300 轮、约 9,000 token,跨最多 35 个会话;1,540 道题,含单会话、多会话、时间线排序Very long multi-session dialogue recall. ~300 turns and ~9,000 tokens per conversation across up to 35 sessions; 1,540 questions spanning single-session, multi-session and timeline ordering | 答案Answers |
| LongMemEval | 2024 | 五种能力:信息抽取、多会话推理、时间推理、知识更新、弃权(问不存在的事时应拒答而非编造)。S 版约 115K token / 约 40 会话,M 版约 500 会话Five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention (declining on events that never happened). The S split is ~115K tokens over ~40 sessions; M reaches ~500 sessions | 答案Answers |
| BEAM | 2026 (ICLR) | 十种记忆能力,100 段对话、每段最长 1000 万 token、2,000 道探测题;分 BEAM-1M 与 BEAM-10M 两个赛道。设计上就不让任何现有架构刷满Ten memory capabilities over 100 conversations of up to 10M tokens each, with 2,000 probing questions; split into BEAM-1M and BEAM-10M tracks. Designed so no current architecture saturates it | 答案Answers |
| HaluMem | 2025 | 在记忆操作层面评幻觉:抽取、更新、问答三个环节各自的幻觉率Hallucination at the level of memory operations: extraction, updating, and QA each scored separately | 答案Answers |
| MERIT | 2026 | 行动级:成功判定是可变世界状态(SQLite)上的谓词翻转,不是回答正确。带泄漏检查、难度阶梯(单事实→多事实→更新事实)、受控记忆污染、逐 token 计费Action-level: success is a predicate flipping on a mutable world state (SQLite), not a correct answer. Includes leak checks, a difficulty ladder (single-fact → multi-fact → updated-fact), controlled corruption, and full token/dollar metering | 动作Actions |
对话召回基准问的是「你记得用户三周前说了什么吗」。生产环境要问的是「你记得之后,你做对事了吗」。这两者中间隔着上一节那 45%。MERIT 把 LongMemEval 和 HaluMem 用对话方式探测的「知识更新」维度,变成了一个行为测试:agent 有没有真的按最新值去执行那次退款、那次配置变更、那次日程创建。
Conversational recall benchmarks ask "do you remember what the user said three weeks ago." Production asks "having remembered, did you do the right thing." Between the two sits the 45% from the last section. MERIT converts the knowledge-update dimension that LongMemEval and HaluMem probe conversationally into a behavioural test: did the agent actually act on the latest value when issuing that refund, applying that config, creating that event.
公开的 LoCoMo 榜单本身就自相矛盾:Mem0 自报 92.5%、Zep 自报 94.7%、Dakera 88.2%(明确说明是标准协议、无 LLM 重排)、ByteRover 和 ZeroMemory 都自报 96.1%。而 ByteRover 自己发布的对比表把 Zep 放在 75.1%、Mem0 放在 66.9%。差异主要来自三个变量:判分用的 judge 模型、生成答案的模型、以及是否加了重排等后处理。Mem0 自己的文章直接引用 Dakera 的说法:两家最高分「用了不同模型和评测设置,不可直接比较」。看到一个记忆基准分数时,先问是哪套模型栈跑出来的、厂商有没有公开完整协议。
The public LoCoMo leaderboard contradicts itself: Mem0 self-reports 92.5%, Zep 94.7%, Dakera 88.2% (explicitly under the standard protocol with no LLM reranking), and ByteRover and ZeroMemory both report 96.1%. ByteRover's own comparison table, meanwhile, puts Zep at 75.1% and Mem0 at 66.9%. Three variables account for most of the spread: the judge model, the answering model, and whether reranking or other post-processing is applied. Mem0's own post quotes Dakera directly: the two highest claims "use different models and evaluation setups — not directly comparable." When you see a memory benchmark score, ask what model stack produced it and whether the vendor documents the exact protocol.
LangGraph 官方文档把这两件事分成两个原语,这个划分本身就值得照搬:
The LangChain docs split these into two primitives, and the split itself is worth copying:
官方的说法是:大多数应用两个都要——checkpointer 跟住当前 thread,store 跟住跨 thread 的持久信息。用户下周带着新 thread_id 回来时,checkpoint 帮不上忙,store 才行。
The docs' guidance: most applications use both — a checkpointer tracks the current thread, a store tracks durable information across threads. When the user returns next week under a new thread_id, the checkpoint cannot help; the store can.
图 4:选型决策树。分叉点不是「数据量多大」,而是「事实会不会被改写」。
Fig. 4 — The decision tree. The fork isn't "how much data" but "do the facts get rewritten."
这是针对 45% 无视率的直接对策。下面是一个说明性的写法对比——不是从某个官方文档抄来的 API,而是把研究结论翻译成的 prompt 结构:
This is the direct countermeasure to the 45% ignore rate. The contrast below is illustrative — not an API copied from any vendor's docs, but the study's conclusions translated into prompt structure:
# ❌ 常见写法:裸的检索结果,新旧并排,无仲裁信息
<memory>
- 用户同意退款金额 50 元
- 用户同意退款金额 80 元
- 用户地址:朝阳区 XX 路 1 号
- 用户地址:海淀区 YY 街 8 号
</memory>
# ✅ 带来源与时间仲裁,并显式标注冲突
<memory>
<fact key="refund_amount" value="80" unit="CNY"
set_at="2026-09-16T10:12Z" source="episode_4:user_utterance"
supersedes="50 (set_at 2026-09-12)" />
<fact key="shipping_address" value="海淀区 YY 街 8 号"
set_at="2026-09-15T09:03Z" source="episode_3:user_utterance"
supersedes="朝阳区 XX 路 1 号 (set_at 2026-08-30)" />
<conflicts>none unresolved</conflicts>
</memory>
# 并在系统提示里加一条硬规则:
# "当 <fact> 带 supersedes 属性时，只使用 value，绝不使用 supersedes 中的旧值。
# 若 <conflicts> 非 none，先向用户确认再执行任何写操作。"三个要点:(1) 每条事实带 set_at 时间戳和 source 来源,让模型有仲裁依据;(2) 用 supersedes 显式说明「这条替代了什么」,而不是让模型从两条并列记录里猜;(3) 单独一个 <conflicts> 区块,把「有未解决冲突」变成一个模型能看见的信号,而不是一个它得自己发现的问题。
Three points: (1) every fact carries a set_at timestamp and a source, so the model has something to arbitrate with; (2) supersedes states explicitly what this entry replaced, instead of leaving the model to infer it from two parallel records; (3) a separate <conflicts> block turns "there is an unresolved conflict" into a signal the model can see, rather than a problem it has to discover.
照搬 MERIT 的 arc 结构,在你自己的领域上造一套最小回归集。一个 arc 是 4–6 个共享实体的 episode,包含四类:
Copy MERIT's arc structure to build a minimal regression suite in your own domain. An arc is 4–6 episodes sharing entities, containing four kinds:
每个探测轮次的判分是一个程序化的世界状态谓词(数据库里那一行的值是不是 80),并且必须用增量计分:谓词得在这一轮之内由假翻真。此外,至少跑三个随机种子——单种子结果对检索式记忆完全不可信。
Each probe is scored by a programmatic world-state predicate (is that database row now 80?), with delta scoring: the predicate must flip from false to true within that episode. And run at least three seeds — single-seed results are simply not trustworthy for retrieval-based memory.
| 方案Approach | 更新事实上的表现On updated facts | 成本Cost | 什么时候选它When to pick it |
|---|---|---|---|
| C4 结构化事实库Structured fact store | 0.70–1.00 (稳)(stable) | $0.00066/episode,D1/D3 的最佳性价比(4,839 / 3,245 分每边际美元)$0.00066/episode; best CAMU in D1/D3 (4,839 / 3,245 points per marginal dollar) | 默认首选。事实有清晰的键(金额、地址、配置项)、领域窄到能为抽取器调模式The default. Facts have clean keys (amounts, addresses, config entries) and the domain is narrow enough to tune the extractor |
| C3 LLM 滚动摘要Rolling LLM summary | 0.80–1.00 (全网格最稳)(most robust in the grid) | $0.00095/episode | 事实是叙事性的、不好切成键值对;或者你不想承担抽取器的领域工程Facts are narrative and don't split cleanly into key–value pairs; or you don't want to own extractor domain engineering |
| C2 向量检索Embedding retrieval | 0.30–0.95 (不可预测)(unpredictable) | $0.00073/episode;在 D2 反而是最佳性价比(6,629 分每美元)$0.00073/episode; best CAMU in D2 (6,629 pts/$) | 仅当事实只追加不修改:知识库检索、历史工单查询、不变的偏好Only when facts are append-only: knowledge-base lookup, historical ticket search, immutable preferences |
| C1 全量回放Full replay | 0.95–1.00 | $0.00126/episode(2,914 token,C0 的 2.7 倍);性价比 1,222–1,741 分每美元,比各领域最优差 2.7–3.9 倍$0.00126/episode (2,914 tokens, 2.7× C0); 1,222–1,741 pts/$, i.e. 2.7–3.9× worse than the per-domain best | 仅作对照基线和调试工具。另有一个限制:D1-medium 上只有 0.60——信息全在,但组合不出来As a control baseline and debugging tool only. One further limit: 0.60 on D1-medium — it has every fact and still can't compose them |
| C5 混合Hybrid | 0.50–0.80 (比 C4 差)(worse than C4) | $0.00111/episode | 默认不要。检索半边会把事实库消除掉的陈旧性导回来,且是唯一 Holm 显著的陈旧受害者Not by default. The retrieval half re-imports the staleness the fact store removed, and it's the only Holm-significant victim of stale corruption |
MERIT 定义了 CAMU(成本感知边际效用)= 相对 C0 的成功率增量 ÷ 相对 C0 的成本增量。结论是:在三个领域里,CAMU 最优的条件和准确率最优的条件从来不是同一个。特别值得注意:在 D2(IT 运维),C4 因为 LLM 抽取器的准确率退化而失去优势,CAMU 最优反而是 C2。这说明「用哪种记忆」不是一个全局答案,而是一个逐领域的实测问题。
MERIT defines CAMU (cost-aware marginal utility) = ΔTSR vs. C0 ÷ Δcost vs. C0. The result: across all three domains, the best-CAMU condition and the best-accuracy condition are never the same one. Notably, in D2 (IT ops) C4 loses its edge to the LLM extractor's accuracy regression and C2 takes the best CAMU. Which memory to use is not a global answer but a per-domain empirical question.
关于绝对成本数字的时效提醒:上表的每 episode 美元数绑定在 2026 年年中的 API 定价上,随价格变动会漂移。作者明确指出,不随价格变化的结论只有两条:决定性因素是鲁棒性而非单次成本(在这个价位上所有条件的盈亏平衡任务价值都是不到一分钱),以及全量回放在 CAMU 上从不领先——因为这依赖的是各条件之间的成本差之比,而非绝对价格。
A timeliness caveat on the absolute figures: the per-episode dollar amounts are tied to mid-2026 API pricing and will drift. The authors are explicit that only two conclusions are price-independent: robustness rather than per-episode cost is the deciding factor (at these prices, break-even task value is a fraction of a cent for every condition), and full replay never leads on CAMU — because that rests on ratios of cost differences between conditions, not absolute prices.
抽取式截断在 hard 档得 0.00 / 0.15 / 0.00,换成 LLM 摘要后是 1.00 / 0.70 / 1.00。同一个架构,60–100 分的差距全在实现上。如果你的「摘要记忆」实际上只是掐掉前 N 轮,那它不是摘要记忆。
Extractive truncation scores 0.00 / 0.15 / 0.00 on the hard tier; LLM summarization on the same architecture scores 1.00 / 0.70 / 1.00. Same architecture, 60–100 points of difference, entirely in the implementation. If your "summary memory" is really "drop the first N turns," it is not summary memory.
在 IT 运维领域,换成通用 LLM 抽取器后 medium 档从 1.00 掉到 0.40,同时对被污染记录的吞入意愿也上升了。通用性是有代价的,而这个代价在窄领域上可能大到无法接受。
In the IT-ops domain, swapping to a generic LLM extractor dropped the medium tier from 1.00 to 0.40, while also increasing willingness to ingest corrupted records. Generality has a price, and in a narrow domain that price can be unacceptable.
C2 在最坏一格上三个种子分别是 0.45 / 0.90 / 0.75。你跑一次得到 0.90,写进技术方案,上线后发现是 0.45——这不是运气不好,是方法学问题。
C2's worst cell scores 0.45 / 0.90 / 0.75 across three seeds. You run it once, get 0.90, write it into the design doc, and ship something that behaves like 0.45. That is not bad luck; it is a methodology failure.
对话式 QA 基准没有这个问题,但行动级评测有:积极的 agent 会提前把事做了,后面的探测靠继承的世界状态「成功」。必须对世界状态做泄漏检查,并用增量计分。
Conversational QA benchmarks don't have this problem; action-level evaluation does. Eager agents act early, and later probes then "succeed" off inherited world state. Leak-check the world state, and use delta scoring.
一个值得单独想一想的开放问题:Opus 4.8 那个「引用了记住的金额,然后拒绝据此行动,要求主管确认」的行为——作者指出,MERIT 的「用户跨会话重新协商金额」这种 arc,结构上和社会工程学升级很像。安全训练过的模型可能会越来越把记忆的来源当成决策的一部分。只按任务完成度打分的基准,会系统性地低估这类模型。
One open question worth sitting with: the Opus 4.8 behaviour — quoting the remembered amounts, then declining to act and asking for supervisor confirmation. The authors note that MERIT's updated-fact arcs (a user renegotiating an amount across sessions) are structurally similar to social-engineering escalations. Safety-tuned models may increasingly treat memory provenance as part of the decision. Benchmarks that score only task completion will systematically under-credit such models.