The SM70 DFlash2 draft quantizes its four large decoder projections to
channel FP8 for M=8, one drafting request. With 2-4 requests the decoder
sees M=16..32 query rows and fell back to an FP16 torch.mm over the
original matrices, which reads twice the bytes. fp8_qpn8_gemm_sm70_out
already covers M<=32, so route M=8..32 in steps of eight there, using
split 16 where M>16 needs split 12 or 16 (gate_up). Each request's rows
now see the same FP8 weights it uses when drafting alone; with an
unchanged split they are bit-identical to its M=8 result. Prefill,
context KV, partial blocks and M>32 keep the FP16 matrices.
Draft projection, 4x V100 TP4 shard, M32: gate_up 156 -> 77 us,
down 77 -> 40 us, qkv 41 -> 41 us, o 20 -> 23 us.
Qwen3.8-27B NVFP4 + DFlash2, 4x V100-SXM2-16GB, packaged profile,
official-style tasks (GSM8K / MT-Bench / HumanEval, T=1.0, thinking,
16 prompts each) with 4 concurrent clients, aggregate tok/s:
282 -> 303, 337 -> 414, 501 -> 587. Acceptance length at c4
4.78/3.14/4.73 -> 4.95/3.15/4.41, against 4.90/3.23/4.62 for the
same tasks at c1 (single-request path, unchanged and bit-identical).
c1 is unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
