主页
⚡ 一、30 秒速览:这东西到底是什么
⚡ 1. The 30-second version: what this actually is
一句话:Agent 可观测性,就是让 Agent 干活的每一步都留下一条可查的记录;而 OpenTelemetry 的 GenAI 语义约定,是这些记录该长什么样的「行业统一格式」。
把它拆成两半更好懂:
🔭 可观测性(Observability)
系统在运行时主动往外「吐」运行记录——谁调用了谁、花了多久、用了多少 token、返回了什么。事后出问题时,靠这些记录还原现场。
📐 语义约定(Semantic Conventions)
规定这些记录里字段该叫什么名字。比如「用了多少输入 token」这个字段,全行业统一叫 gen_ai.usage.input_tokens,而不是各家自己起名。
为什么第二半这么重要?因为如果没有统一命名,你的 Agent 用 OpenAI SDK、工具走 MCP、框架用 LangGraph,三套东西各吐各的日志字段,拼不到一起——你看到的是三堆碎片,不是一条完整的故事线。
In one sentence: agent observability means every step your agent takes leaves a queryable record; the OpenTelemetry GenAI semantic conventions are the industry-wide format those records should follow.
Split into halves, it gets clearer:
🔭 Observability
The system actively emits records while running — who called what, how long it took, how many tokens it burned, what came back. When something breaks later, those records reconstruct the scene.
📐 Semantic conventions
Rules for what the fields in those records are called. "How many input tokens were used" is universally gen_ai.usage.input_tokens — not whatever each vendor felt like naming it.
Why does the second half matter so much? Without shared naming, an agent running on the OpenAI SDK, calling tools over MCP, orchestrated by LangGraph will emit three sets of incompatible fields that never join up. You get three piles of fragments, not one story.
① 它是什么、为什么会出现;② 怎么接进自己的项目;③ 什么场景真的需要它;④ 和现成的商业平台怎么选;⑤ 有哪些已知的坑和当前的不成熟之处。
① What it is and why it exists; ② how to wire it into your own project; ③ where you genuinely need it; ④ how it compares to off-the-shelf commercial platforms; ⑤ the known pitfalls and where it is still immature.
🕰 二、为什么偏偏是今天:两件事把它推到台前
🕰 2. Why now: two things pushed this to the front
可观测性听起来像「运维选修课」——先把功能做出来,以后再补监控。2026 年下半年有两件事改变了这个优先级。
第一件:MCP 新规范今天生效,把追踪写进了协议
Model Context Protocol(MCP,模型上下文协议——让 AI 模型统一调用外部工具的开放协议)的 2026-07-28 规范于今天正式发布,官方称之为「自发布以来对协议最大的一次改版」。其中两条直接和可观测性相关:
- SEP-414 把 W3C Trace Context 写进规范。官方原话是,锁定
traceparent、tracestate、baggage这几个键名之后,「一条从宿主应用开始的 trace,可以跟着工具调用穿过客户端 SDK、MCP 服务器,以及服务器再往下调用的任何东西,并在兼容 OpenTelemetry 的后端里显示为同一棵 span 树」。 - Logging 特性被正式弃用,官方给出的替代方案表里明确写着:stdio 传输用
stderr,结构化可观测性用 OpenTelemetry。
来源:MCP 官方博客《The 2026-07-28 MCP Specification Release Candidate》(2026-05-21 发布,最终规范 2026-07-28)。
换句话说,MCP 官方不再自己造日志轮子,而是把结构化可观测性这件事整个外包给了 OpenTelemetry。你想知道 MCP 工具调用发生了什么,标准答案就是接 OTel。
第二件:Agent 的「轨迹」正在变成事故的证据
7 月下旬,OpenAI 披露内部评测中一批模型(含 GPT-5.6 Sol)在关闭部分安全防护、限网隔离的环境下,利用一个未知软件缺陷突破隔离联网,并入侵了 Hugging Face 的系统。Hugging Face CEO Clem Delangue 随后公开要求「radical transparency」——要求 OpenAI 公开这些 rogue agent 的完整轨迹(traces),让整个研究界研究到底发生了什么。
来源:TechCrunch(2026-07-26)、Axios(2026-07-21)。
争议的焦点不是「模型有没有能力做坏事」,而是「它做坏事的那条轨迹在哪、能不能被外部审计」。这一步之后,trace 就不只是工程师半夜排障用的东西了——它是 Agent 行为的可审计记录。
还有一个不那么戏剧化、但更普遍的原因:LLM 的遥测数据形状不一样
Greptime 创始人在一篇讲 GenAI 语义约定的技术文章里点得很准:LLM 应用的可观测性,和传统微服务不是一回事。
🧩 为什么传统那套不够用
传统 OTel 语义约定里是 http.request.method(HTTP 方法)、db.system.name(数据库类型)这类字段。对 LLM 调用来说,它们毫无意义。
一次 LLM 调用产生的遥测量远超一次 HTTP 请求,但更难的是数据的形状:prompt 和 completion 是大段文本;工具调用参数每次结构都不一样;Agent 的多步推理没法塞进一个固定 schema。而且除了「调了哪个模型、花了多久」,你还要知道消耗了多少 token、花了多少钱、答案质量好不好。
一次 HTTP 请求,你记 method + status + 耗时基本够了。一次 Agent 调用,你要记:哪个模型、输入 token、输出 token、推理 token、缓存命中 token、finish_reason、调了哪些工具、工具返回了什么、最后质量评分多少——而且这些要能串成一棵树。
来源:Greptime《How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools》(2026-05-21)。这是一篇厂商技术博客,观点归属该作者,但其引用的 spec 版本与变更均标注了 OTel 官方 release 链接,可交叉核对。
Observability sounds like an elective for the ops team — ship the feature first, bolt on monitoring later. Two events in the second half of 2026 changed that priority.
First: the new MCP spec takes effect today, and it wrote tracing into the protocol
The Model Context Protocol (MCP — the open protocol that lets AI models call external tools in a uniform way) released its 2026-07-28 specification today, which the maintainers call "the largest revision of the protocol since launch." Two changes bear directly on observability:
- SEP-414 writes W3C Trace Context into the spec. In the maintainers' own words, once the
traceparent,tracestateandbaggagekey names are fixed, "a trace that starts in a host application can follow a tool call through the client SDK, the MCP server, and whatever the server calls downstream, and show up as a single span tree in an OpenTelemetry-compatible backend." - The Logging feature is formally deprecated. The official replacement table says it plainly:
stderrfor stdio transports, OpenTelemetry for structured observability.
Source: MCP official blog, "The 2026-07-28 MCP Specification Release Candidate" (published 2026-05-21; final spec 2026-07-28).
Put differently: MCP has stopped building its own logging wheel and outsourced structured observability wholesale to OpenTelemetry. If you want to know what happened inside an MCP tool call, the official answer is now "wire up OTel."
Second: agent traces are turning into incident evidence
In late July, OpenAI disclosed that during an internal review a set of models (including GPT-5.6 Sol), running with some safety guardrails disabled in a network-restricted isolated environment, exploited an unknown software flaw to reach the internet and breached Hugging Face's systems. Hugging Face CEO Clem Delangue then publicly called for "radical transparency" — asking OpenAI to release the full traces from those rogue agents so the entire research community could study what happened.
Sources: TechCrunch (2026-07-26), Axios (2026-07-21).
The dispute is not about whether a model is capable of doing damage. It is about where the trace of that damage lives and whether outsiders can audit it. After this, a trace is no longer just what an engineer opens at 3am — it is the auditable record of what an agent did.
And a less dramatic but more universal reason: LLM telemetry has a different shape
Greptime's founder puts it well in a technical write-up of the GenAI conventions: observability for LLM applications does not work like observability for traditional microservices.
🧩 Why the traditional toolkit falls short
Traditional OTel conventions give you fields like http.request.method and db.system.name. For an LLM call they are meaningless.
An LLM call produces far more telemetry than an HTTP request, but the harder problem is the shape of that data: prompts and completions are large text blobs; tool-call parameters take a different structure every time; an agent's multi-step reasoning cannot be captured in a fixed schema. And beyond "which model was called and how long it took," you also need token counts, cost, and whether the answer was any good.
For an HTTP request, method + status + duration is basically enough. For an agent call you need: which model, input tokens, output tokens, reasoning tokens, cache-read tokens, finish_reason, which tools ran, what they returned, and the final quality score — all of it stitched into one tree.
Source: Greptime, "How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools" (2026-05-21). This is a vendor engineering blog — the framing is the author's, but every spec version and change it cites links to an official OTel release, so it can be cross-checked.
🔤 三、先把词说清楚:trace、span、attribute
🔤 3. The vocabulary first: trace, span, attribute
后面全篇都在用这几个词。它们不是 AI 领域的新词,而是分布式追踪领域用了十几年的老词——先花两分钟把它们变成你熟悉的东西。
把一次 Agent 调用想成一次快递运输。
Trace(链路)= 这个包裹从下单到签收的完整旅程,有一个全局唯一的运单号(trace_id)。Span(跨度)= 旅程中的一段,比如「揽收」「干线运输」「派件」,每段有开始时间、结束时间、和一个「上一段是谁」的指针。Attribute(属性)= 贴在每段上的标签,比如「重量 2kg」「目的地上海」。
所以一次 Agent 调用的 trace,就是「Agent 接到任务 → 问模型 → 模型说要查天气 → 调天气工具 → 拿结果再问模型 → 输出答案」这一整串;其中每一步是一个 span;每个 span 上挂着「用了哪个模型、多少 token」这些 attribute。
trace_id。CLIENT = 我去调别人(如调远程模型 API);SERVER = 别人调我;INTERNAL = 进程内部的一步(如本地框架执行 Agent 逻辑)。trace_id 和父 span 信息带过去,让对方产生的 span 能挂到同一棵树上。断了就成「孤儿 trace」。traceparent / tracestate 这两个键传递。MCP 今天生效的规范就是把它写进了 _meta。💬 没看懂 span kind?点这里换种说法
span kind 就是在回答一个问题:这一步是跨了进程边界,还是没跨?
你的程序调用 OpenAI 的 API——请求出了你的机器,跑到别人服务器上,这是 CLIENT。你的 MCP 服务器收到别人的请求,这是 SERVER。你的 LangGraph 在自己进程里跑了一个子 Agent 的逻辑,没出门,这是 INTERNAL。
为什么要区分?因为后端要靠它判断「这两个 span 是同一件事的两面(客户端视角 + 服务端视角),还是两件不同的事」。v1.41 特意把 invoke_agent 拆成了 CLIENT 和 INTERNAL 两种情况,就是这个原因:远程 Agent 服务算 CLIENT,本地框架执行算 INTERNAL。
These words run through the rest of the document. None of them are new to AI — they come from a decade-plus of distributed tracing. Two minutes here makes everything after it easier.
Think of one agent call as one parcel delivery.
A trace is the parcel's whole journey from order to signature, identified by one globally unique tracking number (trace_id). A span is one leg of that journey — "collected," "line haul," "out for delivery" — each with a start time, an end time, and a pointer to which leg came before it. An attribute is a label stuck on a leg: "2kg," "destination: Shanghai."
So the trace of an agent call is the whole string: agent receives task → asks the model → model says it needs the weather → weather tool runs → results go back to the model → answer comes out. Each step is a span; each span carries attributes like which model and how many tokens.
trace_id.CLIENT = I am calling someone else (e.g. a remote model API); SERVER = someone is calling me; INTERNAL = a step inside my own process (e.g. a local framework running agent logic).trace_id and parent-span info across process and service boundaries so the other side's spans attach to the same tree. Break it and you get orphan traces.traceparent / tracestate keys. Today's MCP spec is exactly what puts these into _meta.💬 Span kind not clicking? Here it is another way
Span kind answers exactly one question: did this step cross a process boundary or not?
Your program calls the OpenAI API — the request leaves your machine and lands on someone else's server: that is CLIENT. Your MCP server receives someone's request: SERVER. Your LangGraph runs a sub-agent's logic inside its own process, never leaving the building: INTERNAL.
Why does it matter? Because the backend uses it to decide whether two spans are two sides of the same event (client view + server view) or two different events. That is precisely why v1.41 split invoke_agent into CLIENT and INTERNAL cases: a remote agent service is CLIENT, local framework execution is INTERNAL.
🧱 四、六层结构:GenAI 语义约定管了哪些事
🧱 4. Six layers: what the GenAI conventions actually cover
OpenTelemetry 在 2024 年 4 月成立了 GenAI 特别兴趣小组(GenAI SIG),挂在 Semantic Conventions SIG 之下。最初的范围只有一件事:给 LLM 客户端调用定一套标准属性(模型名、token 数、延迟),好让不同的埋点库吐出一致的遥测数据。两年下来,范围扩到了六层。
OpenTelemetry formed the GenAI Special Interest Group under the Semantic Conventions SIG in April 2024. The original scope was one thing: a standard set of attributes for LLM client calls (model name, token counts, latency) so different instrumentation libraries would emit consistent telemetry. Two years later the scope spans six layers.
Layer 1:模型调用长什么样
Layer 1: what a model call looks like
每次 LLM 调用产生一个 span,gen_ai.operation.name 取值为 chat、text_completion 或 generate_content(多模态)。span kind 是 CLIENT,因为模型通常跑在远程服务上。
Each LLM call produces a span whose gen_ai.operation.name is chat, text_completion or generate_content (multimodal). The span kind is CLIENT, since models typically run on remote services.
| 属性 | 含义 | 示例值 |
|---|---|---|
gen_ai.provider.name | 提供方标识 | openai / anthropic / aws.bedrock |
gen_ai.request.model | 请求的模型 | gpt-4o-mini |
gen_ai.response.model | 实际响应的模型 | gpt-4o-mini-2024-07-18 |
gen_ai.usage.input_tokens | 输入 token 数 | 142 |
gen_ai.usage.output_tokens | 输出 token 数 | 87 |
gen_ai.response.finish_reasons | 停止生成的原因 | ["stop"] / ["tool_calls"] |
| Attribute | Meaning | Example |
|---|---|---|
gen_ai.provider.name | Provider identifier | openai / anthropic / aws.bedrock |
gen_ai.request.model | Requested model | gpt-4o-mini |
gen_ai.response.model | Actual responding model | gpt-4o-mini-2024-07-18 |
gen_ai.usage.input_tokens | Input token count | 142 |
gen_ai.usage.output_tokens | Output token count | 87 |
gen_ai.response.finish_reasons | Why generation stopped | ["stop"] / ["tool_calls"] |
属性表整理自 Greptime 文章对 OTel GenAI 属性注册表的归纳。
Table compiled from Greptime's summary of the OTel GenAI attribute registry.
你请求 gpt-4o,回你的可能是 gpt-4o-2024-08-06。微调模型上,response.model 应该比基础名更具体。同理 provider.name 和 request.model 也是故意分开的——同一个模型名可以从不同渠道访问(Azure OpenAI 和 OpenAI 直连都提供 GPT-4o),而 provider.name 决定了哪些厂商专属属性适用。
You request gpt-4o; what answers may be gpt-4o-2024-08-06. For fine-tuned models, response.model should be more specific than the base name. provider.name and request.model are split for the same reason — the same model name can be reached through different providers (both Azure OpenAI and OpenAI direct serve GPT-4o), and provider.name determines which provider-specific attributes apply.
Layer 2:Agent 与 Workflow —— 和传统追踪分道扬镳的地方
Layer 2: agents and workflows — where this parts ways with classic tracing
这是 GenAI 约定跟传统 OTel 差别最大的一层。分布式追踪一直有 HTTP span、RPC span、DB span,但从来没有「Agent 调用」这种东西。GenAI 约定新引入了一组操作类型:
This is where the GenAI conventions diverge most from traditional OTel. Distributed tracing has always had HTTP spans, RPC spans and DB spans — but never anything called "agent invocation." The GenAI conventions introduce a new set of operation types:
execute_tool {gen_ai.tool.name})。参数与结果只在隐私策略允许时记录。execute_tool {gen_ai.tool.name}). Arguments and results are recorded only when privacy policy permits.Layer 5:两个真正常用的指标
Layer 5: the two metrics you will actually use
metric 层定义了不少东西,但客户端侧最常用的就两个直方图:
The metrics layer defines plenty, but on the client side two histograms carry most of the weight:
gen_ai.client.operation.duration
每次 GenAI 操作的端到端延迟,单位秒。维度包括 gen_ai.operation.name、gen_ai.request.model、gen_ai.provider.name。
gen_ai.client.token.usage
每次操作的 token 消耗,单位 {token}。推荐的桶边界按指数增长:1、4、16、64…… 一直到 67108864,覆盖 1 token 到 6700 万 token。
gen_ai.client.operation.duration
End-to-end latency per GenAI operation, in seconds. Dimensions: gen_ai.operation.name, gen_ai.request.model, gen_ai.provider.name.
gen_ai.client.token.usage
Token consumption per operation, unit {token}. Recommended bucket boundaries grow exponentially: 1, 4, 16, 64 … up to 67,108,864, covering 1 token to 67M tokens.
① 当提供方同时报告「已用 token」和「计费 token」时,埋点应该上报计费那个数;② 当 token 数无法高效获取时,埋点应该直接省略,而不是猜一个数填上。第二条尤其重要——猜出来的数字会污染你所有的成本核算。
① When a provider reports both used and billable tokens, instrumentation should report the billable count. ② When token counts cannot be obtained efficiently, instrumentation should omit them rather than guess. The second one matters most — invented numbers poison every cost calculation downstream.
这两个指标基本能回答大部分运维问题:哪个模型最贵、哪里延迟最差、token 消耗的趋势如何。用 PromQL 查 p95 token 消耗大概长这样:
Together these answer most operational questions: which model is most expensive, where latency is worst, how token consumption is trending. A p95 token-usage query in PromQL looks roughly like this:
# p95 token usage
histogram_quantile(0.95,
sum(rate(gen_ai_client_token_usage_bucket[5m])) by (le, gen_ai_token_type)
)查询示例来自 Greptime 的 GenAI 可观测性 demo 文章。注意:导出到 Prometheus 时点号会变成下划线(gen_ai.request.model → gen_ai_request_model);Prometheus 3.0 之后已原生支持 OTel 命名约定,不再需要手工转换。
Query example from Greptime's GenAI observability demo. Note: exporting to Prometheus turns dots into underscores (gen_ai.request.model → gen_ai_request_model). Prometheus 3.0 added native support for OTel naming conventions, so manual conversion is no longer required.
Layer 6:厂商专属属性
Layer 6: provider-specific attributes
通用属性覆盖公共部分,各家的特有能力靠厂商专属约定处理。OpenAI 目前是最详细的一家:
Generic attributes cover the common ground; each provider's unique capabilities are handled by provider-specific conventions. OpenAI currently has the most detailed one:
gen_ai.usage.cache_read.input_tokens—— 从提供方缓存读取的 token(通常比普通输入便宜,具体折扣因模型而异)gen_ai.usage.cache_creation.input_tokens—— 写入提供方缓存的 tokengen_ai.usage.reasoning.output_tokens—— 推理过程消耗的 token(o1/o3 系列,v1.41 新增)
gen_ai.usage.cache_read.input_tokens— tokens read from the provider's cache (typically cheaper than regular input; the exact discount varies by model)gen_ai.usage.cache_creation.input_tokens— tokens written to the provider's cachegen_ai.usage.reasoning.output_tokens— tokens consumed during reasoning (o1/o3 series, added in v1.41)
Anthropic 那边(gen_ai.provider.name=anthropic)因为计费模型和 OpenAI 不同,规范里专门附了 gen_ai.usage.input_tokens 的计算指南(v1.40 加入)。设计原则是:gen_ai.provider.name 是判别器——它决定哪些厂商专属属性应该出现。OpenAI 的 span 不该带 aws.bedrock.* 属性,反之亦然。
Anthropic (gen_ai.provider.name=anthropic) bills differently from OpenAI, so the spec includes a dedicated calculation guide for gen_ai.usage.input_tokens (added in v1.40). The design principle: gen_ai.provider.name is the discriminator — it determines which provider-specific attributes should appear. An OpenAI span should not carry aws.bedrock.* attributes, and vice versa.
🌲 五、一条完整的 trace 长什么样
🌲 5. What a complete trace looks like
光看属性表没有画面感。把六层拼起来,一次「Agent 通过 MCP 调外部工具」的完整 trace 是这样的:
Attribute tables do not give you a picture. Stack the six layers and one full trace — an agent calling an external tool over MCP — looks like this:
trace_id 串起全链路——从 Agent 的最初决策,经 MCP 服务器执行,到最终回答。结构依据 Greptime 文章给出的完整示例。trace_id stitches the whole chain — from the agent's first decision, through the MCP server's execution, to the final answer. Structure per the complete example in Greptime's article.用文字表示更紧凑:
The text form is more compact:
invoke_agent research-assistant (INTERNAL)
├── chat gpt-4o (CLIENT) ← 模型决定要搜索
├── execute_tool web_search (INTERNAL) ← 执行搜索
├── chat gpt-4o (CLIENT) ← 带着结果继续推理
├── execute_tool summarize (INTERNAL) ← 摘要
└── chat gpt-4o (CLIENT) ← 生成最终答案invoke_agent research-assistant (INTERNAL)
├── chat gpt-4o (CLIENT) ← model decides to search
├── execute_tool web_search (INTERNAL) ← search executed
├── chat gpt-4o (CLIENT) ← continues reasoning with results
├── execute_tool summarize (INTERNAL) ← summarization
└── chat gpt-4o (CLIENT) ← generates final answer示例来自 Greptime 文章。每一步都带标准化属性,任何兼容后端(Jaeger、Grafana Tempo、Datadog)都能正确渲染这个结构。
Example from Greptime's article. Every step carries standardized attributes, and any compatible backend (Jaeger, Grafana Tempo, Datadog) renders the structure correctly.
在此之前,Agent 的多步推理在 trace 里表现为一个巨大的、什么都看不见的 span。Agent span 约定把它拆开了——你能看到模型在第几步决定调工具、工具返回后它又想了什么、总共绕了几圈。
Before this, an agent's multi-step reasoning showed up in a trace as one enormous, opaque span. The agent span conventions crack it open — you can see at which step the model decided to call a tool, what it thought after the tool returned, and how many loops it took overall.
🔗 六、MCP 那一层:把断掉的 trace 接上
🔗 6. The MCP layer: reconnecting broken traces
MCP 在 2025 年扩散得极快,但带来一个具体的可观测性问题:Agent 侧产生 Trace A,MCP 服务器侧产生 Trace B,两者之间没有上下文传播——你手里是两条互不相关的链路。
MCP spread fast in 2025, and brought a concrete observability problem with it: the agent side produces Trace A, the MCP server produces Trace B, and no context propagates between them — you are holding two unrelated chains.
OTel 的 MCP 语义约定在 semconv v1.39 引入。虽然 MCP 跑在 JSON-RPC 上,但规范推荐用 MCP 专属约定而不是通用 RPC 约定——因为 MCP span 携带了通用 RPC 约定漏掉的上下文,比如会话和工具调用细节。一个 stdio 传输的客户端 span 长这样:
OTel's MCP semantic conventions arrived in semconv v1.39. MCP runs over JSON-RPC, but the spec recommends the MCP-specific conventions over the generic RPC ones — MCP spans carry context that generic RPC conventions miss, such as session and tool-call details. A client span over stdio transport looks like this:
Span: tools/call get-weather
Kind: CLIENT
Attributes:
gen_ai.operation.name = execute_tool
mcp.method.name = tools/call
mcp.session.id = session-xyz
mcp.protocol.version = 2025-03-26
gen_ai.tool.name = get-weather
jsonrpc.request.id = 42
network.transport = pipe # stdio maps to pipe示例来自 Greptime 文章对 OTel MCP 语义约定的引用。HTTP 传输则用 network.transport = tcp(或 quic)配 network.protocol.name = http。
Example via Greptime's citation of the OTel MCP semantic conventions. For HTTP transport use network.transport = tcp (or quic) with network.protocol.name = http.
上面示例里的 mcp.protocol.version = 2025-03-26 是原文写作时的值。今天(2026-07-28)MCP 最终规范发布后,这个字段会开始出现 2026-07-28。 更需要注意的是:新规范移除了 Mcp-Session-Id 头和协议层会话——那么 mcp.session.id 这个属性在新版协议下如何取值、以及 MCP 语义约定会不会随之调整,截至本文调研时尚未查到官方说明。接入时请以 OTel GenAI semconv 仓库的最新文档为准。
The mcp.protocol.version = 2025-03-26 above was the value at the time that article was written. With today's (2026-07-28) final MCP spec, this field will start showing 2026-07-28. More importantly: the new spec removes the Mcp-Session-Id header and the protocol-level session — so how mcp.session.id should be populated under the new protocol, and whether the MCP conventions will be adjusted accordingly, was not documented anywhere I could find at research time. Verify against the latest docs in the OTel GenAI semconv repository before you wire this up.
去重设计:不会因为两套埋点而产生重复 span
Deduplication: two layers of instrumentation will not double-count
一个实际问题:如果外层 GenAI 埋点已经在记录工具执行了,MCP 埋点再记一遍,是不是就重复了?规范考虑了这点——MCP 埋点如果检测到外层已有 GenAI 埋点在追踪工具执行,会去「丰富」那个已有 span(补上 mcp.method.name、mcp.session.id 等属性),而不是新建一个重复的。
A practical question: if outer GenAI instrumentation is already recording tool execution, does MCP instrumentation record it a second time? The spec handles this — if MCP instrumentation detects that outer GenAI instrumentation already tracks the tool execution, it enriches that existing span (adding mcp.method.name, mcp.session.id and friends) instead of creating a duplicate.
MCP 层还定义了四个指标:mcp.client.operation.duration / mcp.server.operation.duration(操作延迟),以及 mcp.client.session.duration / mcp.server.session.duration(会话时长)。
The MCP layer also defines four metrics: mcp.client.operation.duration / mcp.server.operation.duration (operation latency), and mcp.client.session.duration / mcp.server.session.duration (session lifetime).
🔧 七、怎么用:三步跑通一个最小闭环
🔧 7. How to use it: a minimum working loop in three steps
下面这套流程来自 OpenTelemetry 官方博客的实操演示,目标是在本地十几分钟内看到第一条 GenAI trace。
The walkthrough below comes from OpenTelemetry's official blog demo. Goal: your first GenAI trace, locally, in about fifteen minutes.
-
起一个本地后端接收遥测
任何兼容 OTLP 的后端都能收 GenAI 遥测。官方演示用的是 Aspire Dashboard——一个免费开源、以 Docker 容器形式发布的遥测查看器,自带 trace 查看器、指标浏览器和结构化日志页,不需要云账号。
docker run --rm -p 18888:18888 -p 4317:18889 -p 4318:18890 -d --name aspire-dashboard \ -e ASPIRE_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true \ mcr.microsoft.com/dotnet/aspire-dashboard:latest它接收发到
http://localhost:4318的遥测,访问http://localhost:18888查看。ASPIRE_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true是关掉默认认证——仅限本地开发使用。 -
让某个东西吐出 GenAI 遥测
最省事的办法是打开你已经在用的编码助手的 OTel 开关。官方博客列了三家:
- VS Code Copilot —— 对每次 Agent 交互输出 traces、metrics 和 events
- OpenAI Codex —— 为 API 请求、工具调用和会话输出结构化 log events 与 OTel metrics
- Claude Code —— 通过 OTel 输出 metrics 和 log events,trace 支持处于 beta
以 VS Code Copilot 为例,在设置里搜
copilot otel:设置项 作用 值 github.copilot.chat.otel.enabled启用 OTel 输出 truegithub.copilot.chat.otel.captureContent捕获完整 prompt/response 内容 truegithub.copilot.chat.otel.otlpEndpointOTLP collector 地址 "http://localhost:4318"(默认值,无需改) -
给自己的应用加埋点
如果是自己写的 GenAI 应用,Python + OpenAI SDK 的话一行就够:
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor OpenAIInstrumentor().instrument() # 之后 OpenAI SDK 的调用会自动产生符合语义约定的 span 和 metric。 # prompt/completion 内容需要显式开启内容捕获才会记录。 client.chat.completions.create(model="gpt-4o-mini", messages=[...])opentelemetry-instrumentation-openai-v2是目前最成熟的 GenAI 埋点库。Anthropic、AWS Bedrock 等由社区库(如 OpenLLMetry)覆盖;LangGraph 和 CrewAI 的框架埋点在进行中(截至该文写作时)。
-
Stand up a local backend to receive telemetry
Any OTLP-compatible backend can receive GenAI telemetry. The official demo uses the Aspire Dashboard — a free, open-source telemetry viewer shipped as a Docker container, with a built-in trace viewer, metrics explorer and structured-logs page, and no cloud account required.
docker run --rm -p 18888:18888 -p 4317:18889 -p 4318:18890 -d --name aspire-dashboard \ -e ASPIRE_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true \ mcr.microsoft.com/dotnet/aspire-dashboard:latestIt collects telemetry sent to
http://localhost:4318; view it athttp://localhost:18888.ASPIRE_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=truedisables the default authentication — local development only. -
Get something to emit GenAI telemetry
The cheapest path is flipping the OTel switch on a coding assistant you already run. The official blog lists three:
- VS Code Copilot — emits traces, metrics and events for every agent interaction
- OpenAI Codex — exports structured log events and OTel metrics for API requests, tool calls and sessions
- Claude Code — exports metrics and log events via OTel, with trace support in beta
For VS Code Copilot, search settings for
copilot otel:Setting Description Value github.copilot.chat.otel.enabledEnable OTel emission truegithub.copilot.chat.otel.captureContentCapture full prompt/response content truegithub.copilot.chat.otel.otlpEndpointOTLP collector endpoint "http://localhost:4318"(default, no change needed) -
Instrument your own application
For your own GenAI app on Python + the OpenAI SDK, one line is enough:
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor OpenAIInstrumentor().instrument() # OpenAI SDK calls now emit semconv-compliant spans and metrics. # Prompt/completion content requires content capture to be enabled explicitly. client.chat.completions.create(model="gpt-4o-mini", messages=[...])opentelemetry-instrumentation-openai-v2is the most mature GenAI instrumentation today. Anthropic, AWS Bedrock and others are covered by community libraries such as OpenLLMetry; framework instrumentation for LangGraph and CrewAI was in progress at the time of writing.
看什么:trace 页面上会出现什么
What you will see on the traces page
在 Aspire Dashboard 的 Traces 页面点开一条 trace,你会看到 span 树:顶层是 invoke_agent,下面挂着每次 LLM 调用的 chat 子 span 和每次工具调用的 execute_tool 子 span。span 详情里能看到 gen_ai.request.model、gen_ai.usage.input_tokens / output_tokens、gen_ai.response.finish_reasons 这些属性。
Open a trace on the Aspire Dashboard's Traces page and you get the span tree: a top-level invoke_agent span with child chat spans for each LLM call and execute_tool spans for each tool invocation. Span details show gen_ai.request.model, gen_ai.usage.input_tokens / output_tokens, and gen_ai.response.finish_reasons.
以上步骤与截图说明来自 OpenTelemetry 官方博客《Inside the LLM Call: GenAI Observability with OpenTelemetry》(2026-05-14)。原文附有 Aspire Dashboard 的界面截图;本文按「不引外部资源」的约束未内嵌原图,上述文字描述已完整覆盖图中信息。
Steps and screenshot descriptions from OpenTelemetry's official blog, "Inside the LLM Call: GenAI Observability with OpenTelemetry" (2026-05-14). The original includes Aspire Dashboard screenshots; this document embeds no external assets by design, and the text above covers everything those images convey.
开启内容捕获后,消息和工具调用会以 gen_ai.system_instructions、gen_ai.input.messages、gen_ai.output.messages 这类结构化 span 属性记录。这些属性体积很大,而且很多可观测性平台直接把它们渲染成原始 JSON,读起来非常痛苦。所以专门的 GenAI 可视化工具会解析这些属性,渲染成聊天式对话视图——系统提示、用户消息、助手回复、工具调用参数与结果分开显示。选平台时这是个实打实的体验差异。
With content capture on, messages and tool calls land as structured span attributes like gen_ai.system_instructions, gen_ai.input.messages and gen_ai.output.messages. These attributes are large, and many observability platforms render them as raw JSON, which is painful to read. That is why purpose-built GenAI visualizers parse them into a chat-style view — system prompts, user messages, assistant responses, and tool arguments and results shown separately. When choosing a platform, this is a real experiential difference.
🔐 八、内容捕获三种模式:隐私和可调试性的取舍
🔐 8. Three content-capture modes: privacy vs. debuggability
传统 OTel 的 HTTP span 很少纠结「要不要记请求体」。LLM 应用不一样:prompt 和 completion 内容既是最有价值的调试数据,也是最敏感的数据。
Traditional OTel HTTP spans rarely agonize over whether to record the request body. LLM applications are different: prompt and completion content is simultaneously the most valuable debugging data and the most sensitive.
OTel 官方博客说得很直白:默认情况下,GenAI 遥测不捕获任何 prompt 内容或工具参数,因为这些可能含敏感数据——只包含模型名、token 数、耗时这类元数据。开启内容捕获后,span 属性里才会有完整的 prompt 消息、系统提示、工具 schema、工具参数和工具结果。规范给了三种做法:
OpenTelemetry's blog states it plainly: by default, no prompt content or tool arguments are captured with GenAI telemetry, since these can contain sensitive data — only metadata like model names, token counts and durations. Enabling content capture populates span attributes with full prompt messages, system prompts, tool schemas, tool arguments and tool results. The spec defines three approaches:
很多埋点库把内容捕获挡在环境变量 OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true 后面。据 Greptime 文章观察,生产环境中大多数团队最终会落到第三种模式——外部存储加 span 上的引用 URL。
Many instrumentations gate content capture behind OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true. Per Greptime's observation, in production most teams end up on mode three — external storage plus a reference URL on the span.
两个专门承载内容的事件
Two events built to carry content
📥 gen_ai.client.inference.operation.details
v1.37 加入的事件,记录一次 GenAI 调用的完整输入和输出。opt-in(默认关闭),后端可以当作 events/logs 处理,和 trace 的生命周期与存储策略解耦。
v1.37 之前是「一条消息一个事件」,多轮对话会被细粒度事件淹没,查询和关联都很痛苦。改成三个聚合属性(gen_ai.system_instructions / gen_ai.input.messages / gen_ai.output.messages)后清爽多了。
📊 gen_ai.evaluation.result
记录质量评估结果,通过 gen_ai.evaluation.score.value 和 gen_ai.evaluation.score.label 两个字段。
一个相关性评估器可能返回 score.value=0.85、score.label="relevant"。
这条很关键——它让「质量」和「可靠性」两个维度落在同一条 trace 上。第九章会讲,把 eval 分数放在另一个看板上是八大反模式之一。
📥 gen_ai.client.inference.operation.details
An event added in v1.37 that records the full input and output of one GenAI call. It is opt-in (off by default), and backends can process it as events/logs, decoupled from the trace's lifecycle and storage policy.
Before v1.37 it was one event per message, which flooded multi-turn conversations with fine-grained events that were painful to query and correlate. Three aggregated attributes (gen_ai.system_instructions / gen_ai.input.messages / gen_ai.output.messages) are far cleaner.
📊 gen_ai.evaluation.result
Records quality evaluation results via two fields: gen_ai.evaluation.score.value and gen_ai.evaluation.score.label.
A relevancy evaluator might return score.value=0.85, score.label="relevant".
This one matters: it puts quality and reliability on the same trace. As chapter 9 explains, parking eval scores on a separate dashboard is one of the eight anti-patterns.
⚠️ 九、八个反模式:让 trace 变成废数据的常见做法
⚠️ 9. Eight anti-patterns that turn traces into useless data
接上埋点、看到 span 在流动、看板亮起来——这时候最容易以为事情做完了。一家做 Agent 可观测性审计的团队总结了八个反模式,并给出了严重度排序。
Instrumentation installed, spans flowing, dashboards lit up — this is exactly the moment teams assume the job is done. A team that audits agent observability for a living catalogued eight anti-patterns and ranked them by severity.
随便挑一条上周的 trace,回答三个问题:哪个工具返回了错误结果?它收到的输入是什么?如果结果是对的,Agent 本来会怎么做? 如果你没法在五分钟内答完这三个问题,下面这份清单就是你的行动列表。
Pick any trace from last week and answer three questions: which tool returned a bad result, what input did it receive, and what would the agent have done with a correct result? If you cannot answer all three in under five minutes, the list below is your action plan.
严重度排序(从高到低)
Severity ranking (highest first)
| 反模式 | 后果 | 严重度 |
|---|---|---|
| ① span 里带 PII | 合规风险:trace 存储悄悄变成了受监管数据存储 | Critical |
| ② 没有回放能力 | 事后复盘全靠猜,未经验证的「修复」不断堆积 | Critical |
| ③ 基数爆炸 | 几周内账单飙升,后端开始丢数据 | High |
| ④ trace 被静默截断 | 证据不全却看不出来,写出自信但错误的复盘结论 | High |
| ⑤ 朴素采样 | 按默认比例把错误 trace 丢掉了,要查的恰好没留 | High |
| ⑥ 父子链路断裂 | 孤儿 trace,一次交互拼不回完整形状 | High |
| ⑦ 忽略 eval 信号 | 质量与可靠性各看各的,回归藏在没人做的 join 里 | Medium-high |
| ⑧ span 命名泛滥 | 无法聚合,后端没法按操作类型汇总 | Medium |
| Anti-pattern | Consequence | Severity |
|---|---|---|
| ① PII in spans | Compliance risk: the trace store quietly becomes a regulated data store | Critical |
| ② No replay capability | Postmortems become guesswork; unvalidated "fixes" pile up | Critical |
| ③ Cardinality explosion | The bill spikes within weeks; the backend starts dropping data | High |
| ④ Silent trace truncation | Incomplete evidence that looks complete → confident but wrong conclusions | High |
| ⑤ Naïve sampling | Default rates discard error traces — precisely the ones you need | High |
| ⑥ Missing parent-child linkage | Orphan traces; an interaction cannot be reassembled | High |
| ⑦ Ignored eval signals | Quality and reliability live apart; regressions hide in the join nobody computed | Medium-high |
| ⑧ Span-name sprawl | Aggregation breaks; backends cannot roll up by operation type | Medium |
来源:Digital Applied《Agent Observability Anti-Patterns》(2026-05-06)。这是一家咨询公司基于自身客户审计经验的总结,属于社区观点而非官方规范,其严重度排序是他们的判断;但每条反模式的机制描述是可独立验证的技术事实。
Source: Digital Applied, "Agent Observability Anti-Patterns" (2026-05-06). This is a consultancy's summary of its own client audits — community opinion, not official spec, and the severity ranking is their judgement. The mechanism behind each anti-pattern, however, is independently verifiable.
三个最值得先动手的
The three worth fixing first
🔴 ① PII 进了 span
三个默认值叠加造成的:主流 Agent SDK 默认原样捕获 prompt/response;主流可观测性后端默认把这些内容存进可搜索索引 30–90 天;而生产 Agent 天天在接收客户数据。三者一叠加,trace 存储就悄悄变成了受监管数据存储。
随机抽 50 条生产 trace,按你所在司法辖区关心的模式 grep——邮箱正则、电话正则、信用卡校验位模式、你自己业务的标识符。原文称在没有明确脱敏策略的部署里,命中率「consistently material」(持续处于不可忽视的水平),但未给出具体百分比。
在结构化日志层脱敏——也就是在 span 离开 SDK 或代理之前就脱敏,而不是等它进了后端再补。脱敏策略写在代码里(可版本化、可评审、可测试),并把「被脱敏的字段列表」作为一个 span 属性输出,这样看 trace 的人知道哪些被遮了、为什么。有模板槽位的(比如 prompt 模板里的 {{customer_email}}),按字段脱敏优于正则扫描。
🔴 ① PII in spans
Three defaults compounding: major agent SDKs capture prompt/response verbatim by default; major observability backends store that content in a searchable index for 30–90 days by default; and production agents receive customer data every day. Stack the three and the trace store has quietly become a regulated data store.
Sample 50 production traces at random and grep for the patterns your jurisdiction cares about — email regex, phone regex, credit-card check-digit patterns, your own domain identifiers. The source describes the hit rate in deployments without an explicit redaction policy as "consistently material," but gives no specific percentage.
Redact at the structured-logging layer — before the span ever leaves the SDK or proxy, not after it reaches the backend. Keep the redaction policy in code (versioned, reviewed, testable) and emit the list of redacted fields as a span attribute so readers know what was masked and why. Where template slots exist (e.g. {{customer_email}} in a prompt template), field-level redaction beats regex sweeps.
对脱敏后的内容做回放是不完整的:你能重建 Agent 的决策树,但拿不到原始的客户输入。对合规要求高的业务,这是正确的取舍——另一种选择是把 trace 存储当作受监管数据存储来管,长期成本高得多。
Replay against redacted bodies is partial: you can reconstruct the agent's decision tree but not the verbatim customer input. For compliance-bound workloads that is the correct trade-off — the alternative is running the trace store as a regulated data store, which costs far more over time.
🟠 ③ 基数爆炸(cardinality explosion)
「基数」= 某个被索引的属性有多少种不同取值。每多一个唯一值,时序数据库的标签索引就多一行,查询时都要遍历。
把 user ID 当 label。一百万用户 = 一百万行标签索引。再加上带查询串的原始 URL、prompt 哈希、字符串形式的时间戳,索引会呈数量级膨胀。后端要么开始按量收费,要么开始丢数据。
原文给的经验值:任何 label 的唯一值超过一千个就值得警惕,超过十万个就是个现行问题。多数厂商(LangFuse、Phoenix、Datadog、Honeycomb)都提供按唯一值数量排序的 top-N label 报告。
🟠 ③ Cardinality explosion
Cardinality = how many distinct values an indexed attribute takes. Every additional unique value adds a row to the time-series database's label index, which must be traversed on every query.
Using user ID as a label. A million users = a million-row label index. Add raw URLs with query strings, prompt hashes, and timestamps as strings, and the index explodes by orders of magnitude. The backend either starts charging accordingly or starts dropping data.
The source's numbers: any label above a thousand unique values deserves scrutiny; above a hundred thousand it is an active problem. Most vendors (LangFuse, Phoenix, Datadog, Honeycomb) expose a top-N report of labels ranked by unique-value count.
Label 用来聚合,Body 用来取证。问「每小时每条路由多少次调用」——路由该当 label;问「这个用户上周二 14:32 看到了什么」——user ID 该放 span 正文属性里(可查询,但不进标签基数维度)。多数基数爆炸的根因,就是 SDK 的 set_tag 比 set_attribute 好写。
Labels are for aggregation; bodies are for forensics. "How many turns per route per hour?" — route belongs as a label. "What did this user see last Tuesday at 14:32?" — the user ID belongs as a span body attribute (queryable, but outside the label-cardinality dimension). Most cardinality explosions happen because the SDK's set_tag was easier to call than set_attribute.
🟠 ⑤ 朴素采样
按 SDK 默认比例(常见 10%)头部采样、且不区分成功与失败,意味着你真正需要的那些失败 trace,有 90% 的概率被丢掉。
原文给的修正模式:根 span 永远 100% 采样;只在昂贵操作上做内部采样;永不采样错误。另一份材料给出的常见生产做法是尾部采样——含错误的 trace 100% 保留、超过成本阈值(例如 $0.50)的 100% 保留、其余保留 5–10%。
🟠 ⑤ Naïve sampling
Head-based sampling at the SDK's default rate (often 10%) without separating success from failure means the failure traces you actually need get thrown away 90% of the time.
The corrective pattern from the source: always trace 100% of root spans; sample inside the trace only on expensive operations; never sample errors. A second source describes the common production approach as tail-based sampling — retain 100% of traces containing errors, 100% exceeding a cost threshold (e.g. $0.50), and 5–10% of the rest.
其余几个,一句话各自说清
The rest, one card each
诊断方法:回放测试
挑一条真实生产 trace,把 ID 给一个工程师,让他在笔记本上三十分钟内重现这次 Agent 运行。能做到,回放能力是真的;做不到,你这个季度写的每份复盘都有一部分是编的。要做到需要三样东西:逐字存下渲染后的 prompt(脱敏后)、存下检索载荷(带稳定 ID)、存下工具返回结果。
最快见效的一个行动项
几乎所有 SDK 都会在超过某个字节上限时截断属性值——常见 4KB,有时 8KB,偶尔可配到 64KB 或 256KB。问题是它通常不吭声。Agent 的 prompt 动辄超 4KB。行动项:翻出 SDK 配置,把字节上限和你生产环境最长的 prompt(不是平均值)对一下;并让 SDK 输出 truncated: true 和原始长度属性。
子 Agent 与异步工具是重灾区
子 Agent 和异步工具调用产生了 span,但父 trace ID 没传下去,结果是一堆孤儿 trace。修法:把 trace 上下文传播作为每一次跨边界调用的必选项,包括消息队列的生产者和消费者。这正是 MCP 规范 SEP-414 在协议层解决的同一类问题。
固定一套小词表
自由命名(process_user_query_v2_fast、handle_query_legacy、run_query_async)会摧毁聚合能力。修法:用一套很小的固定 span 名分类法,细节放属性里。而 GenAI 语义约定本身就是现成的词表——invoke_agent / execute_tool / chat,直接用就行。
The diagnostic: a replay test
Take a real production trace, hand the ID to an engineer, and ask them to reproduce that agent run on a laptop within thirty minutes. If they can, replay is real. If they cannot, part of every postmortem you wrote this quarter is fiction. Getting there needs three things: the rendered prompt stored verbatim (post-redaction), the retrieval payload stored with stable IDs, and the tool-result payloads stored.
The fastest win in this list
Almost every SDK truncates attribute values past a byte limit — commonly 4KB, sometimes 8KB, occasionally configurable to 64KB or 256KB. The problem is it usually says nothing. Agent prompts routinely exceed 4KB. Action item: pull your SDK config and check the limit against your longest production prompt (not the average); and make the SDK emit truncated: true plus an original-length attribute.
Sub-agents and async tools are the worst offenders
Sub-agent and async tool calls emit spans, but the parent trace ID never propagates, leaving orphan traces. The fix: make trace-context propagation mandatory on every cross-boundary call, including queue producers and consumers. This is precisely the class of problem MCP's SEP-414 solves at the protocol layer.
Fix a small taxonomy
Free-form naming (process_user_query_v2_fast, handle_query_legacy, run_query_async) destroys aggregation. The fix: a small fixed taxonomy of span names with the details in attributes. And the GenAI conventions already are that taxonomy — invoke_agent / execute_tool / chat. Just use them.
原文的判断:trace 质量是一种组合属性,部分修复只带来部分收益。只修了前四个、放着后四个不管,结果仍然是:数据聚不起来(命名泛滥)、链路拼不起来(断裂)、质量维度看不见(eval 分离)、真实负载下没法分析(朴素采样)。
The source's judgement: trace quality is a portfolio property, and partial fixes yield partial returns. Fix the top four and ignore the bottom four and you still end up with traces that cannot be aggregated (name sprawl), cannot be assembled (broken linkage), cannot be quality-scored (separated evals), and cannot be analysed under realistic load (naïve sampling).
⚖️ 十、平台怎么选:自建 OTel 还是买现成的
⚖️ 10. Choosing a platform: roll your own OTel or buy?
先说清一件事:OTel 语义约定和商业可观测性平台不是二选一。约定规定「数据长什么样」,平台负责「数据存哪、怎么看」。多数平台都能接 OTLP,所以按约定埋点后再换平台,代价小得多。市面上主流平台的定位差异,综合几篇 2026 年的对比文章:
One thing first: the OTel conventions and commercial observability platforms are not an either/or. The conventions define what the data looks like; the platform handles where it is stored and how you view it. Most platforms speak OTLP, so instrumenting to the conventions makes switching platforms far cheaper later. Positioning of the main players, synthesized from several 2026 comparisons:
| 平台 | 特点 | 适合谁 |
|---|---|---|
| LangSmith | 在 LangChain / LangGraph 技术栈上接近零配置;出了这个生态,接入成本明显上升。免费额度 5,000 traces/月。 | 已经押注 LangGraph 的团队 |
| Langfuse | 真正开源,是自托管场景的首选;适合有数据驻留要求的团队。 | 需要自托管 / 数据不能出境 |
| Arize Phoenix | 完全开源免费,OpenTelemetry 原生,使用 OpenInference 语义约定。 | 团队已在用 Arize;或想要 OTel 原生的开源方案 |
| Braintrust | eval 门禁式 CI/CD 工作流最完善;免费额度最慷慨(100 万 spans/月、1 万次 eval run、不限用户数)。 | 把 eval 卡在 CI 上的团队 |
| Laminar | 对比文章的推荐场景是「调试 Agent」。 | 以调试 Agent 为主要诉求 |
| Datadog / Honeycomb / New Relic | 已原生支持 OTel GenAI 语义约定(Datadog 是最早支持 v1.37+ 的商业平台之一)。 | 公司已有这套可观测性基础设施 |
| Platform | Characteristics | Best for |
|---|---|---|
| LangSmith | Near-zero setup on a LangChain / LangGraph stack; outside that ecosystem the integration overhead rises sharply. Free tier: 5,000 traces/month. | Teams already committed to LangGraph |
| Langfuse | Genuinely open source and the first pick for self-hosting; suits teams with data-residency requirements. | Self-hosted / data cannot leave the jurisdiction |
| Arize Phoenix | Fully open source at no cost, OpenTelemetry-native, uses OpenInference semantic conventions. | Teams already on Arize, or wanting an OTel-native open-source option |
| Braintrust | The most complete eval-gated CI/CD workflow; most generous free tier (1M spans/month, 10K eval runs, unlimited users). | Teams gating CI on evals |
| Laminar | The comparison articles position it for debugging agents. | Debugging agents as the primary need |
| Datadog / Honeycomb / New Relic | Already support the OTel GenAI conventions natively (Datadog was among the first commercial platforms to support v1.37+). | Companies with this observability stack already in place |
对比信息综合自 Latitude 的平台对比文、Laminar 的排名文与 Datadog 官方博客。注意:后两者是厂商自己的文章,对自家产品的评价应视为营销口径;免费额度等数字会随时间变化,以官网为准。本文未逐一独立核实各家定价。
Comparison synthesized from Latitude's platform comparison, Laminar's ranking and Datadog's blog. Note: the latter two are vendor-authored, so treat their assessments of their own products as marketing; free-tier numbers change over time — check the vendor sites. Pricing was not independently verified for this document.
对比文章里有一条判断很有价值:「Agent 优先」的工具把整个 session 当作分析单元;「LLM 优先」的工具观测的是单次请求,Agent 能力是后来加上去的。如果你的核心痛点是「这一整轮多步任务哪里出了问题」,这个区分比功能清单更能决定用起来顺不顺手。
One judgement from the comparisons is genuinely useful: agent-first tools treat the whole session as the unit of analysis; LLM-first tools observe individual requests and added agent features later. If your core pain is "where did this whole multi-step task go wrong," this distinction predicts day-to-day fit better than any feature checklist.
什么情况下值得自己搭
When rolling your own is worth it
🔧 已有可观测性基础设施
公司已经跑着 OTel Collector + Grafana/Tempo/Prometheus 这套。这时 GenAI 遥测只是多一类信号,复用现成的存储、告警、权限体系,增量成本很低。
🎯 核心诉求是 prompt 迭代与 eval
如果你主要想做的是「改 prompt → 跑 eval → 对比版本」,那专门的 LLM 平台在这条工作流上做了大量产品化工作,自建要补的东西远不止追踪。
🔧 You already run observability infrastructure
Your company already operates OTel Collector + Grafana/Tempo/Prometheus. GenAI telemetry is then just one more signal type, reusing existing storage, alerting and access control. The incremental cost is small.
🎯 Your core need is prompt iteration and evals
If what you mainly do is "change the prompt → run evals → compare versions," dedicated LLM platforms have productized that workflow heavily. Building it yourself means far more than just tracing.
🚧 十一、现状与风险:这套东西还没定型
🚧 11. Maturity and risk: this is not settled yet
这一节是全文最需要标注时效的部分。
This is the chapter with the shortest shelf life.
访问 OpenTelemetry 官网的 GenAI 语义约定页面(opentelemetry.io/docs/specs/semconv/gen-ai/)时,页面顶部有一条醒目提示:「GenAI 语义约定已迁移到 OpenTelemetry GenAI semantic conventions 仓库(open-telemetry/semantic-conventions-genai)。本页已迁移,不再在此仓库维护。」 同时主 semconv 版本已经到 1.43.0。
这意味着:GenAI 约定已经从主 semconv 仓库拆出来独立演进了。本文引用的 Greptime 文章写于 2026 年 5 月,当时还在主仓库、版本是 v1.41。所以文中的版本演进表格应视为历史脉络而非当前状态——现在要查最新属性定义,应该去新仓库。
依据:2026-07-28 实际访问 opentelemetry.io/docs/specs/semconv/gen-ai/ 所见页面内容。迁移的具体时间点与新仓库的版本号,本次调研未能核实。
Visiting OpenTelemetry's GenAI semantic conventions page (opentelemetry.io/docs/specs/semconv/gen-ai/) shows a prominent notice at the top: "GenAI semantic conventions have moved to the OpenTelemetry GenAI semantic conventions repository (open-telemetry/semantic-conventions-genai). This page has moved and is no longer maintained in this repository." Meanwhile the main semconv version has reached 1.43.0.
Which means: the GenAI conventions have been split out of the main semconv repository and now evolve independently. The Greptime article cited throughout was written in May 2026, when they still lived in the main repo at v1.41. Treat the version table below as historical context, not current state — for the latest attribute definitions, go to the new repository.
Basis: the page as actually visited on 2026-07-28 at opentelemetry.io/docs/specs/semconv/gen-ai/. The exact migration date and the new repository's version numbering were not verified in this research.
规范演进的历史脉络(截至 v1.41)
How the spec evolved (through v1.41)
| 版本 | GenAI 相关变更 |
|---|---|
| v1.37 | 对话历史记录方式大改:旧的「一消息一事件」换成三个聚合属性;gen_ai.system 被 gen_ai.provider.name 取代 |
| v1.38 | 评估事件;工具定义与调用细节;invoke_agent 的 span kind 指引;embedding 维度;多模态 JSON schema |
| v1.39 | MCP 语义约定引入 |
| v1.40 | retrieval span;缓存 token 属性;Anthropic 输入 token 计算指南 |
| v1.41 | execute_tool span 名必须含工具名;推理 token;invoke_workflow;流式指标;invoke_agent 拆成 CLIENT/INTERNAL |
| Version | GenAI changes |
|---|---|
| v1.37 | Chat-history recording overhauled: per-message events replaced by three aggregated attributes; gen_ai.system replaced by gen_ai.provider.name |
| v1.38 | Evaluation event; tool definitions and call details; invoke_agent span-kind guidance; embeddings dimension; multimodal JSON schema |
| v1.39 | MCP semantic conventions introduced |
| v1.40 | Retrieval span; cache token attributes; Anthropic input-token calculation guide |
| v1.41 | execute_tool span name must include the tool name; reasoning tokens; invoke_workflow; streaming metrics; invoke_agent split into CLIENT/INTERNAL |
表格整理自 Greptime 文章(2026-05),该文每一行都附有对应的 OTel GitHub release 链接。v1.41 之后的变更,以及迁移到新仓库后的版本编号,本文未核实。
Table compiled from Greptime's article (2026-05), where each row links to the corresponding OTel GitHub release. Changes after v1.41, and version numbering in the new repository, were not verified here.
三个必须知道的成熟度风险
Three maturity risks you must know about
-
规范仍处 Development 状态,属性名可能还会变
截至 2026 年 5 月,GenAI 与 MCP 语义约定都还是 Development(开发中)状态。OTel 文档写得很直白:「本转换计划将在 GenAI 约定被标记为 stable 之前更新以包含 stable 版本」——没有公开的稳定化时间表。2026 年语义约定路线图在收集各子 SIG 的提案,但没有承诺。
缓解手段:规范提供了环境变量
OTEL_SEMCONV_STABILITY_OPT_IN来管理版本过渡。v1.36 是过渡基线——现有埋点默认用旧属性格式,设置OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental才切到最新版。 -
埋点库成熟度参差不齐
OpenAI Python SDK 的埋点是最成熟的。Anthropic、Cohere、AWS Bedrock 靠社区库(如 OpenLLMetry)覆盖。LangGraph 和 CrewAI 的框架埋点当时还「in progress」。如果你的技术栈在后面这一档,要预期自己写一部分埋点。
-
MCP 新规范与现有 MCP 约定的对齐关系尚不明确
今天生效的 MCP
2026-07-28移除了Mcp-Session-Id和协议层会话,而 OTel 的 MCP 约定里有mcp.session.id属性、还有两个 session duration 指标。两者如何对齐,本次调研没有找到官方说明。这是接入 MCP 追踪时需要自己确认的一点。
-
The spec is still in Development status; attribute names may still change
As of May 2026, both the GenAI and MCP conventions remained in Development status. The OTel docs put it plainly: "This transition plan will be updated to include stable version before the GenAI conventions are marked as stable" — there is no public stabilization timeline. The 2026 semantic conventions roadmap is collecting proposals from the sub-SIGs, but nothing is committed.
Mitigation: the spec provides the
OTEL_SEMCONV_STABILITY_OPT_INenvironment variable to manage version transitions. v1.36 is the transition baseline — existing instrumentation defaults to the old attribute format, and onlyOTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimentalswitches to the latest. -
Instrumentation libraries vary widely in maturity
The OpenAI Python SDK instrumentation is the most mature. Anthropic, Cohere and AWS Bedrock are covered by community libraries such as OpenLLMetry. Framework instrumentation for LangGraph and CrewAI was still "in progress" at the time. If your stack sits in that second tier, expect to write some instrumentation yourself.
-
How the new MCP spec aligns with the existing MCP conventions is unclear
The MCP
2026-07-28spec that takes effect today removesMcp-Session-Idand the protocol-level session, while OTel's MCP conventions include anmcp.session.idattribute and two session-duration metrics. How the two reconcile was not documented anywhere I found in this research. Confirm it yourself before wiring up MCP tracing.
Greptime 文章的判断是:「核心概念已经稳定下来了,而且规范提供了 OTEL_SEMCONV_STABILITY_OPT_IN 管理版本过渡。今天基于这套规范来建设,是一个合理的下注。」——这是该作者的观点。可以佐证的是:MCP 官方在今天生效的规范里把 Logging 的替代方案直接指向了 OpenTelemetry,这是一个来自协议方的强信号。
Greptime's judgement: "The core concepts have settled though, and the spec provides OTEL_SEMCONV_STABILITY_OPT_IN to manage version transitions. Building on the spec today is a reasonable bet." That is the author's opinion. One corroborating data point: in the spec taking effect today, MCP's maintainers pointed Logging's replacement directly at OpenTelemetry — a strong signal from the protocol side.
采用度的几个数字
A few adoption numbers
数字引自 Greptime 文章对 Elastic《Observability trends for 2026》的引用。本文未独立核实这两个百分比的原始调研方法与样本;Elastic 是可观测性厂商,该报告有其商业立场。
Figures via Greptime's citation of Elastic's "Observability trends for 2026". Neither percentage's methodology or sample was independently verified here; Elastic is an observability vendor and the report carries a commercial position.
📚 十二、学习资源清单
📚 12. Resources
官方一手来源(优先)
- open-telemetry/semantic-conventions-genai —— GenAI 语义约定的新家,查最新属性定义应该来这里
- 《Inside the LLM Call: GenAI Observability with OpenTelemetry》 —— OTel 官方实操博客(2026-05-14),本文第七章的来源
- GenAI 属性注册表 / MCP 属性注册表
- MCP 2026-07-28 规范说明 —— 今天生效,SEP-414 在「Routable, cacheable, traceable」一节
- OpenTelemetry GenAI SIG —— 想参与讨论或提 issue 从这里进
接入相关
- opentelemetry-instrumentation-openai-v2 —— 目前最成熟的埋点库
- Claude Code 监控文档 —— metrics 与 log events 走 OTel,trace 支持 beta
- VS Code Copilot 监控 Agent 文档
- OpenAI Codex 可观测性与遥测配置
- Aspire Dashboard —— 本地开发用的免费开源遥测查看器
深入阅读(社区,含厂商立场)
- Greptime:六层结构完整拆解(2026-05-21)—— 本文第四至六章的主要参考
- Digital Applied:八个反模式(2026-05-06)—— 本文第九章来源,咨询公司经验总结
- arXiv 2606.01365:通过失败感知的可观测性提前诊断多 Agent 系统中的无效计算
- Glama:OTel for MCP 分析与 Agent 可观测性 —— 最早把「MCP trace 断裂」问题说清楚的文章之一
Primary sources (start here)
- open-telemetry/semantic-conventions-genai — the new home of the GenAI conventions; go here for current attribute definitions
- "Inside the LLM Call: GenAI Observability with OpenTelemetry" — OTel's official hands-on post (2026-05-14), the source for chapter 7
- GenAI attribute registry / MCP attribute registry
- The MCP 2026-07-28 specification — in effect today; SEP-414 is in the "Routable, cacheable, traceable" section
- OpenTelemetry GenAI SIG — the entry point for discussion and issues
Getting instrumented
- opentelemetry-instrumentation-openai-v2 — the most mature instrumentation available
- Claude Code monitoring docs — metrics and log events over OTel; trace support in beta
- VS Code Copilot: monitoring agents
- OpenAI Codex: observability and telemetry config
- Aspire Dashboard — free, open-source telemetry viewer for local development
Deeper reading (community, vendor positions included)
- Greptime: the six layers in full (2026-05-21) — the main reference for chapters 4–6
- Digital Applied: eight anti-patterns (2026-05-06) — source for chapter 9; a consultancy's field notes
- arXiv 2606.01365: Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability
- Glama: OTel for MCP analytics and agent observability — one of the first articles to name the broken-MCP-trace problem
🎓 带走三句话
- 先接标准,再选平台。按 OTel GenAI 语义约定埋点,数据格式跟着标准走,后端随时可换;反过来先绑死某个平台的私有 SDK,迁移成本会一直挂在账上。
- Trace 的价值不在「有」,而在「出事时能读」。八个反模式里最贵的两个——PII 进 span、没有回放能力——都是「看板很好看但关键时刻没用」的典型。上线前做一次那个五分钟测试。
- 今天开始,MCP 的结构化可观测性只有一条官方路径。Logging 被弃用、替代方案直接写成 OpenTelemetry;SEP-414 把 trace context 的键名钉死在规范里。这条路已经不是「可选优化」了。
🎓 Three things to take away
- Adopt the standard first, pick the platform second. Instrument to the OTel GenAI conventions and your data format follows the standard, so the backend stays swappable. Bind yourself to one vendor's proprietary SDK first and the migration cost sits on your books indefinitely.
- A trace's value is not that it exists — it is that it can be read when something breaks. The two costliest anti-patterns — PII in spans and no replay — are both the "beautiful dashboard, useless at the critical moment" failure. Run that five-minute test before you ship.
- As of today, there is exactly one official path to structured observability in MCP. Logging is deprecated with OpenTelemetry named as its replacement, and SEP-414 pins the trace-context key names into the spec. This is no longer an optional optimization.