What Is Speculative Decoding, and How Does It Speed Up Inference?

Hello, this is the Qualiteg Research Team.

What Is Speculative Decoding?

Speculative decoding is a technique for accelerating inference in large language models (LLMs).

It has been reported to speed up most models by roughly 1.4x to 2.0x.

In this approach, a small model (the draft model) produces initial predictions, and a larger model (the target model) verifies them, which speeds up the inference process as a whole.

Put simply,

large models are computationally expensive and slow, so the idea is to first run a fast computation with a small model, then make clever use of that result to reduce the computational load on the large model and improve overall speed.

As a basic premise, the large model and the small model are assumed to share essentially the same network architecture, differing only in size.

For example, picture combining a 70B Llama 3 with an 8B Llama 3.

Naturally, inference with the 70B Llama 3 is heavier than with the 8B version, so the technique lets the small 8B Llama 3 run ahead and do inference computation in advance to achieve the speedup.

How Speculative Decoding Works

In speculative decoding, deciding whether the large model (target model) should use the predictions from the small model (draft model) mainly involves the following steps and considerations:

1. Generation by the draft model: In the initial prediction stage, the draft model rapidly generates multiple candidate tokens. Because this model is far smaller than the target model, it can produce predictions quickly.
   
2. Verification by the target model: The target model verifies the tokens generated by the draft model and determines whether they are acceptable. In this process, the draft model's output is compared with the target model's predictions, and only the tokens that match are adopted as the final output.
   
3. Computing the TAR (Token Acceptance Rate): The TAR indicates the proportion of tokens generated by the draft model that the target model accepts. The higher this rate, the better the draft model's predictions conform to the target model's standards, which contributes to higher throughput.
   
4. The throughput-latency trade-off: The main purpose of using a draft model is to increase the throughput of the inference process. When the draft model's latency is sufficiently low and the TAR is high, this approach can shorten overall inference time and improve efficiency.
   
5. Performance benchmarking: When actually using a draft model with a target model, you need to run multiple experiments with different draft model configurations and sizes to find the optimal setup. This makes it possible to determine scientifically which draft model is the most effective.

Through the steps and considerations above, you can determine whether the small model's predictions can be put to practical use by the large model.

When the target model evaluates the draft model's results, inference on the target model still has to run, so does this really reduce the amount of computation?

Does a question like that come to mind?

If the target model has to do the computation anyway, it is understandable to wonder why a small model is used at all.

The advantage of speculative decoding (or rather, its core idea) lies in managing the target model's computational load efficiently. Here, we explain concretely the mechanism by which computation is reduced.

The Basic Process of Speculative Decoding

1. Using the draft model:
   The draft model generates a large number of candidate tokens with low latency. This happens far more quickly than it would with the target model.
   
2. Batch processing
   The target model verifies the multiple tokens generated by the draft model all at once. Compared with ordinary autoregressive generation (generating tokens one at a time), the model can process much more data in a single pass, which makes efficient use of computational resources such as GPUs.
   
3. The prefill approach:
   The target model makes predictions based on the multiple tokens generated by the draft model, using them as a kind of prefill. Instead of the target model generating every candidate independently, one at a time, it checks and accepts a set of valid tokens, thereby skipping computation.

Where the Actual Computation Savings Come From

• Parallel processing
  By having the target model process the batch of tokens supplied by the draft model, efficient parallel processing becomes possible instead of token-by-token generation.
  
• Selective verification
  The target model accepts only the tokens it judges to be valid. This reduces the number of steps in the overall generation process and eliminates wasted computation.
  
• Efficient data processing
  By using the input from the draft model, the target model leverages it as part of its own input, reducing the overall computational load.

Summary

We hope you found this useful. In this article, we explained speculative decoding while keeping mathematical formulas to a minimum.

In speculative decoding, the final computation is indeed performed by the target model, but by leveraging the draft model's output to process things efficiently, the overall computational cost and latency can be reduced. This method lightens the computational burden on the target model and enables faster, more efficient data processing.

References

https://arxiv.org/pdf/2211.17192
https://arxiv.org/pdf/2302.01318

According to the papers "2402.01528v2" and "2211.17192v2", the effectiveness of speculative decoding appears to depend heavily on the choice of draft model.

These studies examine how different draft models affect the performance of the target model, and in particularthe token acceptance rate (TAR), that is,how many of the tokens generated by the draft model the target model accepts, appears to hold the key to improving throughput. That is only natural, of course: it comes down to how many good tokens (logits) the draft model can produce.