主页
先说结论:你用 agent 写代码时,模型服务器吃的不是「一问一答」,而是一个越滚越长的会话。每一轮都要把之前的全部内容(系统提示、工具定义、历史、工具返回)再送一遍,新生成的却只有几百个 token。所以瓶颈主要在「读上下文」(prefill,预填充)和显存里的 KV cache(模型为已读内容存的中间结果),不在「写答案」。
The short version: when you run a coding agent, the model server does not see one question and one answer. It sees a session that keeps growing. Every turn resends everything so far (system prompt, tool definitions, history, tool results) and generates only a few hundred new tokens. So the bottleneck is reading context (prefill) and the KV cache (the intermediate results the model keeps for what it has already read), not writing the answer.
这件事对你有两个直接影响:自己部署模型的人,测性能时不能再用聊天基准;用 API 的人,缓存命中率直接决定账单和延迟(这个话题我们在 8 月讲过 Prompt 缓存命中率工程,今天看的是服务端那一侧)。
This matters in two ways. If you self-host, chat benchmarks will mislead you. If you use an API, cache hit rate decides your bill and latency (we covered the client side in August under prompt-cache engineering; today is the server side).
AgentPerfBench(arXiv 2609.34683,2026-09-28 提交,作者来自 Imperial College London、Cambridge、Oxford)回放 SWE-Bench、TerminalBench、OSWorld 的真实多轮轨迹,再和聊天负载对比。引擎是 vLLM v0.19.x 和 SGLang v0.5.9,硬件从 H100 到 RTX 2080Ti。下面的数字都是论文报告的,我没有自己复现。
AgentPerfBench (arXiv 2609.34683, submitted 2026-09-28, authors from Imperial College London, Cambridge and Oxford) replays real multi-turn traces from SWE-Bench, TerminalBench and OSWorld and compares them with chat load. Engines were vLLM v0.19.x and SGLang v0.5.9, on hardware from H100 down to RTX 2080Ti. All numbers below are as reported in the paper; I did not reproduce them.
| 场景 | 论文报告 |
|---|---|
| LLaMA-3.1-70B,H100,饱和时 | TTFT 239 ms → 1,139 ms(4.8×);TPOT 19 → 28 ms(1.5×) |
| SWE-Bench 会话每轮中位 TTFT | 433 ms,聊天是 62 ms(7.0×) |
| LLaMA-3.1-8B | 中位 TTFT 11,719 ms 对 343 ms;吞吐 13.5 → 2.4 req/s |
| 并发 C=200 → 320 | 聊天吞吐中位 +11%;agent 负载在代表性案例里掉 43–76% |
| 算术强度(H100) | coding agent 2,409 FLOP/byte,聊天 181;H100 拐点 295 |
| KV cache 容量 | coding agent 在 C=40 时超过 80 GB 显存,C=80 时约 155 GB(解析模型) |
| Setting | Reported |
|---|---|
| LLaMA-3.1-70B, H100, at saturation | TTFT 239 ms → 1,139 ms (4.8×); TPOT 19 → 28 ms (1.5×) |
| Median per-turn TTFT, SWE-Bench sessions | 433 ms vs 62 ms for chat (7.0×) |
| LLaMA-3.1-8B | Median TTFT 11,719 ms vs 343 ms; throughput 13.5 → 2.4 req/s |
| Concurrency C=200 → 320 | Chat throughput median +11%; agentic profiles lose 43–76% in representative cases |
| Operational intensity (H100) | Coding agent 2,409 FLOP/byte vs 181 for chat; H100 ridge point 295 |
| KV cache capacity | Coding agent exceeds 80 GB HBM by C=40 and reaches about 155 GB at C=80 (analytical model) |
TTFT 是首 token 延迟,TPOT 是每个输出 token 的耗时。读表的要点:agent 的输出速度(TPOT)变化不大,变慢的几乎全是首 token,也就是在读上下文。另一个值得记住的点是「过了饱和点还加并发」:聊天负载吞吐还会涨,agent 负载反而大幅下跌。
TTFT is time to first token; TPOT is time per output token. The pattern: output speed (TPOT) barely moves, while time to first token blows up, which means the server is busy reading context. The other thing worth remembering is what happens past saturation: chat throughput keeps growing, agent throughput drops sharply.
vLLM 在 2026-09-08 的博客里讲了它和 AgentX 合作的结果。AgentX 的轨迹中位数是 43 轮、142K 输入 token、444 输出 token,前缀缓存命中率超过 96%。vLLM 的改动包括:混合 KV 缓存管理器(多种注意力共用一个块池)、用 Mooncake Store 把 KV 放到 CPU 和磁盘分层、限制单次长 prefill 的 token 数(示例是 512)避免它堵住短请求、以及按「先测饱和、再扫组合」的方法定 P/D 比例。
vLLM's 2026-09-08 post describes its work with AgentX. The AgentX traces had a median of 43 turns, 142K input tokens and 444 output tokens, with prefix-cache hit rates above 96%. vLLM's changes include a hybrid KV cache manager (one block pool shared across attention types), tiered KV storage on CPU and disk through Mooncake Store, a cap on long-prefill tokens (512 in the example) so one long prompt does not block short turns, and a two-phase method for the P/D ratio: profile saturation first, then sweep combinations.
它给出的一条经验很实在:会话粘性路由(session-sticky routing)比各种负载均衡策略都强,因为缓存位置比负载均匀更重要。llm-d 的 agentic serving 指南也是同一个思路:前缀和负载感知路由打底,再叠加分层 KV 卸载、精确的前缀缓存路由、P/D 分离。
One lesson from that post is practical: session-sticky routing beat the load-balancing policies, because cache locality matters more than even load. The llm-d agentic serving guide follows the same idea: prefix- and load-aware routing as the baseline, then tiered KV offloading, precise prefix-cache routing and P/D disaggregation on top.
| 你的情况 | 做法 |
|---|---|
| 只用 API | 把稳定内容放前面、保持前缀不变;能传缓存提示就传(Anthropic 的 cache_control,OpenAI 的 prompt_cache_key,llm-d 文档提到的就是这两个);看账单里的缓存命中字段 |
| 自己部署开源模型 | 按 agent 轨迹压测,不要用聊天数据;先打开前缀缓存;路由做会话粘性;盯 KV cache 占用,超显存前就要有分层或限并发的方案 |
| 在选型、做容量规划 | 用 request throughput 和饱和点定容量;别按聊天 benchmark 的并发数外推 |
| Your situation | What to do |
|---|---|
| API user only | Keep stable content first and the prefix unchanged; pass cache hints when available (Anthropic cache_control, OpenAI prompt_cache_key, both named in the llm-d docs); watch the cache-hit fields on your bill |
| Self-hosting open models | Load-test with agent traces, not chat data; turn on prefix caching; route sticky by session; watch KV cache usage and have tiering or a concurrency limit before you run out of HBM |
| Capacity planning | Size on request throughput and the saturation point; do not extrapolate from chat-benchmark concurrency |
AgentPerfBench 只测了单机多卡(最多 TP 8),没有测多机和流水线并行;长会话被上限截断,所以「滚雪球」效应是下限;前缀缓存结果是逻辑估算,没有直接测服务端的 KV 淘汰和上下文压缩;合成负载用随机 token,只在 coding agent 一种情况下验证过。另外 AgentPerfBench 是 9 月底的 arXiv 预印本,我没查到它是否已经过同行评审。vLLM 的数字来自厂商自己的博客。
AgentPerfBench ran single-node multi-GPU servers only (up to TP 8), with no multi-node or pipeline-parallel overhead; long sessions were truncated by per-run context caps, so the snowball effect is a lower bound; prefix-cache results are logical estimates and do not directly measure server-side KV eviction or context compaction; synthetic profiles use random tokens and were validated only for the coding-agent case. It is also a late-September arXiv preprint, and I did not find whether it has been peer reviewed. The vLLM numbers come from the vendor's own blog.