1. MoE 是什么:一句话与一张图

1. What MoE is: one sentence and one diagram

一句话:稀疏 Mixture-of-Experts(混合专家)把 Transformer 里的前馈层(FFN)拆成很多个并列的「专家」,每个 token 经过一个路由器(router / gate),只被送到其中少数几个专家去算,其余专家这一步完全不参与计算。

One sentence: A sparse Mixture-of-Experts (MoE) model replaces the feed-forward layer (FFN) of a Transformer with many parallel "experts." A router (gate) sends each token to only a few of them; the rest do no work on that token at all.

因此一个 MoE 模型有两个必须分开记的数字:总参数(total parameters,决定你要装多少权重进内存)和激活参数(active parameters,决定单 token 前向的算力开销 FLOPs)。DeepSeek-V3 是最经典的说明:6710 亿总参,单次推理只激活约 370 亿,却达到当时前沿闭源模型级别的能力,而计算成本只是其一部分。

So an MoE model has two numbers you must never conflate: total parameters (how much weight you have to hold in memory) and active parameters (the FLOPs cost of a single-token forward pass). DeepSeek-V3 is the canonical illustration: 671B total parameters, roughly 37B active per inference step, reaching frontier-class capability at a fraction of the compute cost.

2. 最贵的直觉陷阱:激活参数 ≠ 显存

2. The expensive intuition trap: active params ≠ VRAM

这是自托管 MoE 最常见、也最烧钱的误判。看到「49B 激活」,很多人下意识按 49B 稠密模型去估算硬件——大约 100GB 显存(BF16),两张 80GB 卡绰绰有余。但推理时任何一个专家都可能被路由到,所以所有专家权重都必须驻留。1.6T 总参在 BF16 下就是 3TB 量级的权重,即使 FP8 也在 1.6TB 量级,再叠加 KV cache 与激活值,这是一个几十张卡的集群,不是一台工作站。

This is the most common and most expensive misjudgment in self-hosting MoE. Seeing "49B active," people instinctively size hardware like a 49B dense model — roughly 100 GB in BF16, comfortably two 80 GB cards. But at inference any expert may be routed to, so every expert's weights must stay resident. 1.6T total parameters is on the order of 3 TB of weights in BF16, still ~1.6 TB in FP8, before KV cache and activations. That is a multi-node cluster, not a workstation.

这带来一个反直觉的结论:MoE 让「同样的钱买到更强的模型」在云端 API 上成立,但在自托管上不一定成立。API 厂商用高并发把庞大的权重摊薄到成千上万个请求上;你自己跑一个 QPS 个位数的内部服务,那 3TB 权重就是纯粹的固定成本。

The counter-intuitive consequence: MoE's "more capability per dollar" holds for cloud APIs but not automatically for self-hosting. API providers amortize those enormous weights across thousands of concurrent requests. If you run an internal service at single-digit QPS, those 3 TB of weights are pure fixed cost.

怎么算一笔粗账

A back-of-envelope sizing formula

权重显存 ≈ 总参数 × 每参数字节数
   BF16 = 2 B/param  ·  FP8 = 1 B/param  ·  INT4 ≈ 0.5 B/param

例:1.6T 总参
   BF16 → ≈ 3.2 TB      FP8 → ≈ 1.6 TB      INT4 → ≈ 0.8 TB
再加 KV cache(随 context 长度与并发数线性增长)+ 激活值 + 碎片
   实际预留 ≈ 权重 × 1.2 ~ 1.5

对比同一模型的「算力侧」:单 token FLOPs 只按 49B 激活参数算

weight VRAM ≈ total_params × bytes_per_param
   BF16 = 2 B/param  ·  FP8 = 1 B/param  ·  INT4 ≈ 0.5 B/param

e.g. 1.6T total params
   BF16 → ≈ 3.2 TB      FP8 → ≈ 1.6 TB      INT4 → ≈ 0.8 TB
plus KV cache (linear in context length × concurrency) + activations + fragmentation
   provision ≈ weights × 1.2 – 1.5

Meanwhile the compute side scales with 49B active params only, not 1.6T.

⚠️ 上面是量级估算,不是精确容量规划。真实数字受专家分片策略、KV cache 量化、张量并行切分方式影响很大,务必以你实际使用的推理引擎实测为准。

⚠️ These are order-of-magnitude estimates, not a capacity plan. Real numbers depend heavily on expert sharding strategy, KV-cache quantization, and tensor-parallel layout — always measure on your actual serving engine.

3. 2026 年 8 月的开源格局

3. The open-weight landscape as of August 2026

2026 年的一个结构性事实:几乎每一个旗舰开源模型都是稀疏 MoE。第三方汇总的参数规格如下(总参 / 激活参):

A structural fact about 2026: nearly every flagship open-weight model is a sparse MoE. Specs as compiled by third-party trackers (total / active):

被反复引用的一句判断是:开源与闭源的能力差距「现在是以单个 benchmark 分点、而不是以代际来衡量的」。这句话在编码这一个维度上有数据支撑(SWE-bench Verified 上 80.6% vs 80.8%),但把它推广到长上下文推理、工具调用可靠性、多语种、安全对齐等维度是没有依据的。选型时要问的是「在我的任务上差多少」。

The widely quoted line is that the open-versus-closed gap "is now measured in single benchmark points, not generations." That is defensible on coding specifically (80.6% vs 80.8% on SWE-bench Verified), but generalizing it to long-context reasoning, tool-calling reliability, multilingual quality, or safety alignment is unsupported. The question that matters is: how large is the gap on your task.

4. 怎么部署:专家并行与服务栈

4. How to serve it: expert parallelism and the stack

通用的 LLM 服务方式无法高效处理专家路由,因此 vLLM 与 TensorRT-LLM 都在 2025 年起陆续加入了 MoE 专用优化:融合的 MoE kernel、FlashInfer 集成与 kernel autotuning、以及专家并行(expert parallelism, EP)——把不同专家分散到不同 GPU 上,而不是像张量并行那样把每一层横切。

General-purpose LLM serving cannot handle expert routing efficiently, so both vLLM and TensorRT-LLM have been adding MoE-specific work since 2025: fused MoE kernels, FlashInfer integration with kernel autotuning, and expert parallelism (EP) — distributing distinct experts across GPUs rather than slicing every layer as tensor parallelism does.

三种并行方式的分工

How the three parallelism axes divide the work

• 张量并行 TP:把单层的矩阵切开,多卡协同算一层。通信频繁,适合高速互联(NVLink)内部。
• 流水并行 PP:把不同层放到不同卡,像流水线一样传。跨节点友好,但会引入气泡(bubble)。
• 专家并行 EP:MoE 特有。不同专家放不同卡,路由后用 all-to-all 把 token 发到对应卡再收回。负载均衡是核心难点——若路由长期偏爱少数专家,那几张卡会成为瓶颈,其余卡空转。

• Tensor parallelism (TP): split a layer's matrices across GPUs. Communication-heavy; wants fast interconnect (NVLink) within a node.
• Pipeline parallelism (PP): put different layers on different GPUs and stream through. Cross-node friendly, but introduces pipeline bubbles.
• Expert parallelism (EP): MoE-specific. Different experts live on different GPUs; after routing, an all-to-all shuffles tokens to the right device and back. Load balancing is the hard part — if routing persistently favors a few experts, those GPUs become the bottleneck while the rest idle.

低显存路线:把 MoE 当「工作集」问题

The low-memory route: treat MoE as a working-set problem

如果你就是没有集群,学术界给出了几条真实可用的方向,共同思路是:既然每步只用少数专家,就不必让全部专家常驻 GPU。

If you simply do not have a cluster, the research literature offers usable directions, all sharing one idea: since only a few experts fire per step, not every expert needs to sit in GPU memory.

• FloE(arXiv:2505.05950):显存受限 GPU 上的 on-the-fly MoE 推理。
• eMoE(arXiv:2503.06823):任务感知的显存高效 MoE 推理,利用路由的任务级可预测性预取专家。
• WiSP(arXiv:2606.21868):把专家调度建模为「工作集」,面向极低资源硬件的 MoE 服务。
• 专家内激活稀疏(arXiv:2605.08575):在 vLLM 的融合 MoE 执行流水线上进一步利用专家内部的激活稀疏。

• FloE (arXiv:2505.05950): on-the-fly MoE inference on memory-constrained GPUs.
• eMoE (arXiv:2503.06823): task-aware memory-efficient MoE inference, exploiting task-level predictability of routing to prefetch experts.
• WiSP (arXiv:2606.21868): models expert scheduling as a working-set problem for extremely low-resource hardware.
• Intra-expert activation sparsity (arXiv:2605.08575): extends vLLM's fused MoE pipeline to exploit sparsity inside each expert.

5. 什么场景该自托管:一棵决策树

5. When self-hosting wins: a decision tree

三类站得住脚的自托管场景

Three self-hosting cases that hold up

1. 合规驱动:数据不能出边界。此时不是成本问题,而是可行性问题——按 FP8 全参预算硬件,并接受它。
2. 高并发同质流量:比如批量文档处理、日志摘要、代码索引。利用率高,权重成本被摊薄,MoE 的低激活参数带来的吞吐优势在这里真正兑现。
3. 可控性与稳定性:模型版本自己钉死,不会被上游静默升级改变行为;这对已经建了大量提示词与评测的团队价值很高。

1. Compliance-driven: data cannot cross a boundary. Then it is not a cost question but a feasibility one — budget for full FP8 weights and accept it.
2. High-volume homogeneous traffic: bulk document processing, log summarization, code indexing. Utilization is high, weight cost amortizes, and MoE's low active-parameter throughput advantage actually materializes.
3. Control and stability: you pin the model version, so upstream silent upgrades cannot change behavior — valuable for teams with a large investment in prompts and evals.

6. 常见坑与反模式

6. Common pitfalls and anti-patterns

7. 术语表与来源

7. Glossary and sources

术语表

Glossary

• MoE / Mixture-of-Experts(混合专家):把 FFN 拆成多个并列专家、按 token 路由的架构。
• router / gate(路由器):决定每个 token 送往哪几个专家的小网络。
• total / active parameters(总参 / 激活参):前者决定内存,后者决定算力。
• expert parallelism, EP(专家并行):把不同专家分布到不同 GPU 的并行方式。
• all-to-all:EP 中把 token 分发到目标 GPU 再收回的集合通信原语。
• KV cache:缓存已生成 token 的键值向量,随上下文长度与并发线性增长。
• working set(工作集):一段时间内实际被访问到的专家集合,低显存部署的核心概念。

• MoE / Mixture-of-Experts: an architecture that replaces the FFN with many parallel experts and routes tokens among them.
• router / gate: the small network deciding which experts each token goes to.
• total / active parameters: the former sets memory, the latter sets compute.
• expert parallelism (EP): distributing distinct experts across GPUs.
• all-to-all: the collective communication primitive that ships tokens to their target GPU and back under EP.
• KV cache: cached key/value vectors for generated tokens; grows linearly with context length and concurrency.
• working set: the set of experts actually touched over a window of time — the central concept for low-memory serving.

来源

Sources

• Spheron — MoE Model Inference on GPU Cloud: Expert Parallelism, Memory, and Cost (2026)(显存倍数、EP、成本)
• TensorOps — LLM Mixture of Experts Explained: A 2026 Field Guide
• Introl — Mixture of Experts Infrastructure(vLLM / TensorRT-LLM 的 MoE 优化)
• Open-Source LLMs Landscape 2026(各模型总参/激活参、SWE-bench 汇总,第三方口径)
• LLM Reference — DeepSeek V4 Pro(许可、上下文、GA 时间,第三方口径)
• WiSP: A Working-Set View of MoE Serving on Extremely Low-Resource Hardware(arXiv:2606.21868)
• FloE: On-the-Fly MoE Inference on Memory-constrained GPU(arXiv:2505.05950)
• eMoE: Task-aware Memory Efficient MoE Inference(arXiv:2503.06823)
• Uncovering Intra-expert Activation Sparsity for Efficient MoE Execution(arXiv:2605.08575)

• Spheron — MoE Model Inference on GPU Cloud: Expert Parallelism, Memory, and Cost (2026) (memory multiples, EP, cost)
• TensorOps — LLM Mixture of Experts Explained: A 2026 Field Guide
• Introl — Mixture of Experts Infrastructure (vLLM / TensorRT-LLM MoE optimizations)
• Open-Source LLMs Landscape 2026 (per-model total/active params and SWE-bench figures; third-party compilation)
• LLM Reference — DeepSeek V4 Pro (license, context, GA dates; third-party compilation)
• WiSP: A Working-Set View of MoE Serving on Extremely Low-Resource Hardware (arXiv:2606.21868)
• FloE: On-the-Fly MoE Inference on Memory-constrained GPU (arXiv:2505.05950)
• eMoE: Task-aware Memory Efficient MoE Inference (arXiv:2503.06823)
• Uncovering Intra-expert Activation Sparsity for Efficient MoE Execution (arXiv:2605.08575)