Accelerating LLM Inference with Lossless Speculative Decoding Methods
New lossless speculative decoding methods accelerate LLM inference without retraining.
Accelerating inference for large language models (LLMs) poses a significant challenge in generative AI. Speculative decoding (SD) methods provide notable efficiency improvements by generating multiple tokens in a single forward pass. However, traditional SD approaches require both drafter and target models to share the same vocabulary, limiting the selection of drafter models and often necessitating training from scratch. This work introduces three new SD methods that eliminate the shared-vocabulary constraint, ensuring lossless performance and compatibility with off-the-shelf models without additional training. Empirical results show speedups of up to 2.8x in summarization, programming, and long-context tasks compared to standard autoregressive decoding.
This synthesis was produced from its source by AI; there is no human editor or manual review step. How we work