github-actions[bot]andnilbuild
1288e97c42
Co-authored-by: nilbuild <4921183+nilbuild@users.noreply.github.com>
1.1 KiB
1.1 KiB
Speculative Decoding
Speculative decoding takes advantage of spare compute during memory-bound decode by generating multiple draft tokens and validating them in a single forward pass. If the target model accepts N draft tokens, the forward pass produces N+1 tokens instead of one. The performance gain depends on draft token cost, draft sequence length, and acceptance rate, which decreases at higher temperatures and larger batch sizes.
Visit the following resources to learn more:
- @article@An Introduction to Speculative Decoding for Reducing Latency in AI Inference
- @article@Speculative Decoding - vLLM
- @article@Fast Inference from Transformers via Speculative Decoding
- @video@Faster LLMs: Accelerate Inference with Speculative Decoding
- @video@Speculative Decoding: When Two LLMs are Faster than One