Cloudflare Workers AI 发布了针对 GPU 推理的三项关键优化技术，用于高效运行 Moonshot Kimi K-series 和 Z.ai GLM 等大规模长上下文混合专家模型[1]。这些优化包括 KV 缓存量化、模型权重压缩和共享 KV 缓存完整性检查，在保持模型准确性的前提下，使单个 GPU 能承载更多并发请求并降低服务成本[1]。
KV 缓存量化是首项重点优化。通过将缓存从 BF16 精度降低至 FP8，Kimi K2.6 在内存中能容纳的上下文长度从约 686,000 tokens 增至约 1.37 million tokens，提高约 2 倍[1]。在并发能力上，BF16 缓存支持最多 32 个并发请求，而 FP8 可达 64 个，使总吞吐量达到 2,192 tokens/秒，相比 BF16 峰值高出 41%，最终成本降低约 30%[1]。
模型权重压缩进一步优化了存储和计算效率。GLM 5.2 的权重从 8 位浮点压缩至 4 位整数（INT4），使检查点大小从 705 GB 减少至 421 GB，降幅约 40%[1]。在 8 路张量并行部署中，单个 GPU 的内存占用从 88 GB 降至 52 GB[1]。INT4 权重同时加快了解码阶段的速度，GLM 在 FP8 下的 prefill 吞吐约 10,160 tokens/秒，INT4 下约 8,660 tokens/秒[1]。
为确保推理过程的可靠性，共享 KV 缓存完整性检查被纳入优化方案，其成本低于 1% 的吞吐量和尾延迟，95% 置信区间上界接近 1%[1]。所有实验和生产流量均基于开源推理框架 SGLang 运行和基准测试[1]。
Cloudflare Workers AI has introduced three optimization techniques designed to enable efficient large-scale execution of long-context mixture-of-experts models such as Moonshot's Kimi K-series and Z.ai's GLM on GPU infrastructure [1]. The approach combines KV cache quantization—reducing precision from BF16 to FP8—with model weight compression from 8-bit floating point to 4-bit integers, alongside shared KV cache integrity verification [1]. These methods maintain model accuracy while allowing individual GPUs to handle more concurrent requests and reduce operational costs [1].
Testing on Kimi K2.6 demonstrated that FP8 quantization more than doubled the in-memory context capacity from approximately 686,000 tokens to around 1.37 million tokens [1]. Under BF16 precision, the system supported up to 32 concurrent requests, whereas FP8 enabled 64 concurrent requests with a peak throughput of 2,192 tokens per second—representing a 41 percent increase over BF16's maximum performance and reducing costs by approximately 30 percent [1]. For GLM 5.2, compressing weights to INT4 reduced checkpoint size from 705 GB to 421 GB—a 40 percent reduction—while in an 8-way tensor parallelism deployment, per-GPU memory usage dropped from 88 GB to 52 GB [1]. The integrity verification mechanism for shared KV caches imposed minimal overhead, consuming less than 1 percent of throughput and tail latency, with the 95 percent confidence interval upper bound approaching 1 percent [1]. All experiments and production traffic were executed and benchmarked using SGLang, an open-source inference framework [1].