130 秒速览

130-Second Overview

量化(quantization)就是把模型里的数字用更少的比特来存:训练时常用的 BF16 每个数占 16 bit,量化到 4 bit 后,同一个模型的显存占用约为原来的 1/4。MXFP4 是开放组织 OCP(Open Compute Project)制定的 4-bit 浮点微缩放格式:每 32 个数为一块,块内共享一个 8-bit 缩放因子,平均每个数只花约 4.25 bit。NVIDIA Blackwell 一代 GPU 和 AMD MI355X 的张量核心可以原生按这种格式计算——不只是省显存,算得也更快。

Quantization means storing a model's numbers with fewer bits: BF16, common in training, uses 16 bits per value; at 4 bits, the same model needs roughly 1/4 of the memory. MXFP4 is the 4-bit floating-point microscaling format standardized by OCP (Open Compute Project): values are grouped into blocks of 32 that share one 8-bit scale factor, averaging about 4.25 bits per value. NVIDIA Blackwell-generation GPUs and AMD's MI355X execute this format natively in their tensor cores — so it's not just smaller, it also computes faster.

2背景:为什么非要压到 4 bit

2Why Squeeze All the Way to 4 Bits

先看痛点。大模型推理的瓶颈往往不是「算不动」,而是「搬不动」:生成每个 token 都要把全部权重从显存搬进计算单元,权重越大、搬运越慢、显卡越贵。一个 70B 参数的模型,BF16 存储要约 140 GB——单卡放不下,就得多卡并行,成本翻倍。把每个数从 16 bit 压到 4 bit,显存和搬运量都变成约 1/4:原来要 4 张卡的,现在 1 张卡可能就够。

Start with the pain point. LLM inference is usually bottlenecked not by compute but by data movement: generating every token requires streaming all the weights from GPU memory into the compute units. Bigger weights mean slower transfers and pricier GPUs. A 70B-parameter model needs ~140 GB in BF16 — too big for one card, so you shard across several and multiply cost. Compressing each value from 16 bits to 4 cuts both memory and traffic to ~1/4: what took four GPUs may now fit on one.

行业于是沿着「精度阶梯」一路往下走:FP32(科学计算)→ FP16/BF16(训练标配)→ FP8(Hopper 一代推理主流)→ FP4(Blackwell 一代开始原生支持)。每下一级,单位算力翻倍、显存减半。但天下没有免费午餐——4 bit 只能表示 16 个不同的数值,直接硬压会把模型压坏。让 4 bit 变得可用的关键发明,就是下一节的块缩放(block scaling)。

The industry has walked down a "precision ladder": FP32 (scientific computing) → FP16/BF16 (the training standard) → FP8 (mainstream inference on Hopper) → FP4 (natively supported starting with Blackwell). Each step down doubles effective compute and halves memory. But there's no free lunch — 4 bits can only represent 16 distinct values, and naive compression would wreck the model. The invention that makes 4 bits workable is block scaling, covered next.

3核心概念:块缩放让 4 bit 变得可用

3Core Idea: Block Scaling Makes 4 Bits Usable

4 bit 只有 16 个可表示的值,而神经网络权重的数值范围可能横跨好几个数量级。传统 INT4 给整个张量(可理解为一大片数字表格)定一个固定范围,遇到离群值(outlier)——个别特别大的数——就会崩:为了装下那一个大数,其他所有小数都被压得面目全非。

Four bits give you only 16 representable values, while neural-network weights can span several orders of magnitude. Traditional INT4 assigns one fixed range to an entire tensor (think: one huge table of numbers), and outliers — the occasional very large value — break it: to fit that one big number, every small number gets crushed into mush.

微缩放(microscaling,MX 的含义)的解法很朴素:别给整张表定一个范围,给每一小块单独定。MXFP4 把数据切成每 32 个一块,每块配一个共享的 8-bit 缩放因子,把这块里最大的数映射到 FP4 能表示的最大值。这样每块的 4-bit 编码都「量身定制」,离群值最多祸害同块的 31 个邻居,而不是整个张量。

Microscaling (the "MX") fixes this simply: don't set one range for the whole table — set one per small block. MXFP4 slices data into blocks of 32, each with a shared 8-bit scale factor that maps the block's largest magnitude to the largest FP4-representable value. Every block's 4-bit encoding is thus tailor-made; an outlier can distort at most its 31 neighbors, not the entire tensor.

还有一个关键概念:量化感知(quantization-aware)。事后把训好的模型压到 4 bit 叫 PTQ(post-training quantization,训练后量化),省事但可能掉精度;而在训练/后训练阶段就让模型「知道」自己将来要以 4 bit 运行,叫 QAT(quantization-aware training,量化感知训练),掉点更少。Kimi K3 和 OpenAI 的 gpt-oss 都属于后者——官方直接发布量化感知的 MXFP4 权重,拿到手就是「原生 4 bit」。

One more key concept: quantization-aware. Compressing an already-trained model down to 4 bits afterward is PTQ (post-training quantization) — convenient, but it can cost accuracy. Letting the model "know" during training/post-training that it will run in 4 bits is QAT (quantization-aware training), which loses less. Kimi K3 and OpenAI's gpt-oss are both in the latter camp — the official weights ship quantization-aware in MXFP4, natively 4-bit out of the box.

4两大格式:MXFP4 vs NVFP4

4The Two Formats: MXFP4 vs NVFP4

实际选型时你会遇到两个名字很像的格式。两者每个数都用 E2M1 编码,差别在块大小和缩放因子精度——别小看这两处,它们决定了你用哪套工具链、checkpoint 能在哪些硬件上跑、以及推理质量掉多少。

In practice you'll meet two similar-sounding formats. Both encode each value as E2M1; they differ in block size and scale precision — and those two details determine your toolchain, which hardware your checkpoint runs on, and how much quality you lose.

直觉解释:NVFP4 的块更小(16 vs 32),意味着每个缩放因子只需迁就 16 个数的分布,离群值「祸害面」减半;缩放因子带尾数也让映射更贴合。代价是缩放存储开销翻倍。据 Spheron 的对比指南,在同等校准质量下 NVFP4 一般精度更好;但 MXFP4 配上 MR-GPTQ(ICLR 2026 提出的校准方法,量化前先做分块 Hadamard 旋转,把离群值「摊平」到块内所有通道)后,差距大部分被抹平。

Intuition: NVFP4's smaller blocks (16 vs 32) mean each scale only has to accommodate 16 values, halving an outlier's blast radius; a mantissa-bearing scale also fits distributions more tightly. The cost is double the scale-storage overhead. Per Spheron's comparison guide, NVFP4 is generally more accurate at equal calibration quality — but MXFP4 with MR-GPTQ (an ICLR 2026 calibration method that applies block-wise Hadamard rotations before quantizing, spreading outliers across all channels in a block) closes most of the gap.

5硬件与框架支持一览

5Hardware & Framework Support

FP4 是「硬件绑定」的技术:没有 FP4 张量核心的 GPU 享受不到加速。这是选型时最先要查的一条。

FP4 is hardware-bound: without FP4 tensor cores, a GPU gets none of the speedup. Check this first.

框架侧:TensorRT-LLM(v0.17+)对两种格式全支持,数据中心追求极限吞吐首选;vLLM 全支持 MXFP4,NVFP4 支持加载预量化 checkpoint(MoE 模型需设 VLLM_USE_FLASHINFER_MOE_FP4=1);SGLang 两者全支持;量化工具用 nvidia-modelopt(两种格式的 PTQ 都支持)或更轻量的 llm-compressor(vLLM 官方配套)。注意 bitsandbytes 的 NF4 是另一种无关格式,别混淆。(来源:Spheron 框架支持表,2026-06)

On the framework side: TensorRT-LLM (v0.17+) fully supports both formats — first choice for max datacenter throughput; vLLM fully supports MXFP4 and loads pre-quantized NVFP4 checkpoints (MoE models need VLLM_USE_FLASHINFER_MOE_FP4=1); SGLang supports both; for quantizing, use nvidia-modelopt (PTQ for both formats) or the lighter llm-compressor (vLLM's companion tool). Note that bitsandbytes NF4 is an unrelated format — don't confuse them. (Source: Spheron framework support table, June 2026)

6怎么用:从零到上线的五步

6Hands-On: Five Steps from Zero to Serving

以下流程与代码整理自 Spheron 部署指南与 vLLM/llm-compressor 官方文档,适用于「拿一个开源模型,量化后用 vLLM 提供服务」的典型场景。

The workflow and code below are adapted from the Spheron deployment guide and the vLLM/llm-compressor docs, for the typical "take an open model, quantize it, serve with vLLM" scenario.

pip install "nvidia-modelopt[all]>=0.17"

import modelopt.torch.quantization as mtq
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-70B-Instruct", torch_dtype="auto")

# NVFP4(纯 N 卡推荐 / NVIDIA-only):
mtq.quantize(model, mtq.NVFP4_DEFAULT_CFG,
             forward_loop=calibration_dataloader)
# MXFP4 OCP(跨平台 / cross-platform):
# mtq.quantize(model, mtq.MXFP4_DEFAULT_CFG,
#              forward_loop=calibration_dataloader)

mtq.export_hf_checkpoint(model, output_dir="./quantized-nvfp4")

# 稠密 / dense:
vllm serve nvidia/Llama-3.3-70B-Instruct-NVFP4 \
  --tensor-parallel-size 1 --gpu-memory-utilization 0.90

# MoE(Llama 4 Scout、DeepSeek R1 等):
VLLM_USE_FLASHINFER_MOE_FP4=1 vllm serve \
  nvidia/Llama-4-Scout-17B-16E-Instruct-FP4 \
  --tensor-parallel-size 1 --gpu-memory-utilization 0.90

7实战案例:谁在用原生 4 bit

7Real-World Cases: Who Ships Native 4-Bit

Kimi K3:第一个原生 MXFP4 的前沿开源旗舰

Kimi K3: The First Frontier Open Model Native in MXFP4

2026 年 7 月 27 日,Moonshot 开源 Kimi K3:MoE 架构,2.8 万亿总参数、每 token 激活 1040 亿(896 个路由专家取 16 个 + 2 个共享专家),上下文 1,048,576 token。关键在发布形态:量化感知的 MXFP4 权重 + MXFP8 激活,并在发布当天就带上了 vLLM 的生产级支持(含其 Kimi Delta Attention 混合线性注意力)。据 Northflank 与 Hugging Face 社区分析,这让「自托管一个 2.8T 旗舰」从不可想象变成一件贵但可执行的事——权重体积和显存需求都因 4-bit 大约砍到 BF16 方案的四分之一量级。

On July 27, 2026, Moonshot open-sourced Kimi K3: an MoE with 2.8 trillion total parameters, 104 billion active per token (16 of 896 routed experts plus 2 shared), and a 1,048,576-token context window. The release format is the point: quantization-aware MXFP4 weights with MXFP8 activations, with production-ready vLLM support (including its Kimi Delta Attention hybrid linear attention) on day one. Per analyses from Northflank and the Hugging Face community, this turned "self-hosting a 2.8T flagship" from unthinkable into expensive-but-doable — weight size and VRAM needs land at roughly a quarter of what BF16 would demand.

gpt-oss 与 NVIDIA 官方 checkpoint 生态

gpt-oss and NVIDIA's Official Checkpoint Ecosystem

OpenAI 2025 年发布的开源模型 gpt-oss 同样采用 MXFP4(社区分析称其显存占用较 BF16 减少过半),是 MXFP4 走向主流的早期标志。另一条线是 NVIDIA 在 Hugging Face nvidia/ 命名空间维护的预量化 NVFP4 checkpoint(Llama 3.x、DeepSeek-R1、Llama 4 Scout 等),Mistral Large 3 的 NVFP4 checkpoint 则由 llm-compressor 产出——「官方压好、直接部署」正在成为大模型分发的新常态。

OpenAI's 2025 open-weight gpt-oss models also used MXFP4 (community analyses report a memory cut of more than half vs BF16), an early landmark in MXFP4 going mainstream. The other track is NVIDIA's pre-quantized NVFP4 checkpoints on Hugging Face under nvidia/ (Llama 3.x, DeepSeek-R1, Llama 4 Scout, etc.), with Mistral Large 3's NVFP4 checkpoint produced via llm-compressor — "officially quantized, deploy as-is" is becoming the new normal for model distribution.

8常见坑与限制

8Pitfalls & Limits

9学习资源清单

9Learning Resources

• NVFP4 vs MXFP4 决策指南(Spheron,2026-06)——本文格式对比与部署流程的主要来源,含完整硬件/框架矩阵。
• MXFP4 微缩放量化深入(Spheron)——MR-GPTQ 校准工作流详解。
• MXFP4 主题页(EmergentMind)——追踪相关论文的入口。
• llm-compressor FP4 量化示例(vLLM 官方文档)——W4A4 NVFP4 的官方可运行示例。
• Kimi K3 模型卡(Hugging Face) 与 Kimi K3 自托管指南(Northflank)——原生 MXFP4 发布的第一手案例。
• gpt-oss 的 MXFP4 解析(Medium)——早期主流化案例的社区拆解。

• NVFP4 vs MXFP4 Decision Guide (Spheron, June 2026) — primary source for this document's format comparison and deployment workflow, with full hardware/framework matrices.
• MXFP4 Microscaling Deep Dive (Spheron) — detailed MR-GPTQ calibration workflow.
• MXFP4 topic page (EmergentMind) — entry point for tracking related papers.
• llm-compressor FP4 example (official vLLM docs) — runnable official W4A4 NVFP4 example.
• Kimi K3 model card (Hugging Face) and the Kimi K3 self-hosting guide (Northflank) — the first-hand case of a native MXFP4 release.
• gpt-oss MXFP4 explainer (Medium) — community breakdown of the early mainstream case.