推测解码加速大模型推理的真实机制

推测解码(Speculative Decoding)技术能够显著加快大语言模型的推理速度，但其加速原理与直观认知不同1。虽然这一技术会增加总的浮点运算量而非减少，但它通过改变GPU的计算负载类型实现了性能提升1。传统的大模型推理属于内存密集型工作，需要为每个生成的token流式传输整个模型参数——以Qwen3.5-27B为例，每次都要访问27B个参数1。推测解码将单个token的解码过程改造成批处理模式，使GPU从内存密集型的瓶颈转向计算密集型的工作状态1。

在小批量大小的场景下，推测解码能够充分利用GPU的计算资源加速序列生成1。这种机制允许系统在批量大小远小于最优token数的条件下，通过额外计算换取整体吞吐量提升1。然而，这一优势在高吞吐量场景中可能带来计算浪费1。因此vLLM等推理框架提供了--speculative-disable-by-batch-size参数，支持根据批量大小动态调整是否启用推测解码，以在不同工作负载下获得最优性能1。

Speculative decoding accelerates large language model inference not by reducing computational workload, but rather by altering the nature of GPU operations.1 While the total floating-point operations actually increase rather than decrease with this technique,1 it fundamentally transforms the processing bottleneck from memory-bound to compute-bound.1

Traditional LLM decoding is memory-intensive, requiring the streaming of all model parameters for each token—such as the 27 billion parameters in Qwen 3.5-27B.1 Speculative decoding converts single-token decoding into batch processing, shifting GPU utilization from memory-bound constraints toward compute-bound efficiency.1 This approach enables acceleration in small batch size scenarios where batch size remains significantly below the optimal token count, effectively leveraging otherwise unused computational resources to expedite sequence completion.1

However, the technique presents trade-offs in high-throughput environments. The computational overhead that proves beneficial at small batch sizes can become wasteful when throughput is high.1 To address this, vLLM offers a --speculative-disable-by-batch-size parameter that dynamically disables speculative decoding in high-throughput scenarios to prevent computational waste.1