Co-Designing AI Models Using Speculative Decoding

This entry is the third in a series on AI model co-design, focusing on accelerating LLM inference through speculative decoding. This technique enhances the autoregressive decoding phase by predicting multiple tokens simultaneously, thus reducing the total number of decoding iterations while increasing the arithmetic intensity of the target model. The article outlines five guidelines for optimizing draft length and acceptance mechanisms.