A fresh engine paid about 1 s of TTFT on the first batch at each new
concurrency: several Triton kernels specialize on the request count, a
decode step that also admits a prefill takes its own route, and a full
4096-token chunk reaches long-row and divisible-by-16 variants the short
warmup prompts never did. Warmup ran only the max_num_seqs batch.
Warmup now also runs, for every smaller request count, one cycle with a
pure decode step and one whose decode step admits a new prefill, plus a
single-request prefill of a full chunk. Each cycle reuses the existing
prefill -> one step -> finish pattern.
Qwen3.8-27B NVFP4 + DFlash2, 4xV100, first batch after start (T=0):
c2 2.0 -> 0.8 s, c8 1.9-3.2 -> 1.33 s (steady state 0.47 / 1.34 s);
kernels compiled while serving 7 -> 1. Startup grows 115 -> 129 s.
Co-authored-by: mac <mac@MacBook-xinyi.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
