python-build-standalone CPython, which the Studio runtimes use, is built
against old kernel headers and has no fcntl.F_ADD_SEALS or F_SEAL_*, so
DeepSeek-V4.1's shared expert arena failed at startup with AttributeError
even though the kernel supports sealing. Fall back to the Linux ABI values.
Co-authored-by: mac <mac@MacBook-xinyi.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A fresh engine paid about 1 s of TTFT on the first batch at each new
concurrency: several Triton kernels specialize on the request count, a
decode step that also admits a prefill takes its own route, and a full
4096-token chunk reaches long-row and divisible-by-16 variants the short
warmup prompts never did. Warmup ran only the max_num_seqs batch.
Warmup now also runs, for every smaller request count, one cycle with a
pure decode step and one whose decode step admits a new prefill, plus a
single-request prefill of a full chunk. Each cycle reuses the existing
prefill -> one step -> finish pattern.
Qwen3.8-27B NVFP4 + DFlash2, 4xV100, first batch after start (T=0):
c2 2.0 -> 0.8 s, c8 1.9-3.2 -> 1.33 s (steady state 0.47 / 1.34 s);
kernels compiled while serving 7 -> 1. Startup grows 115 -> 129 s.
Co-authored-by: mac <mac@MacBook-xinyi.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* [Bugfix][Core] Keep sliding-window runs at retained Mamba checkpoints
With --prefix-cache-retention-interval, aligned Mamba groups keep a state
every interval tokens, so a prompt that shares only an earlier part of a
cached one can resume there. A sliding-window group beside them (the
DFlash2 draft on Qwen3.8-27B) released every block its window had passed
with reuse_first, except the run at the prompt's last alignment boundary.
The prefill itself then reused those blocks, the window run before each
retained checkpoint lost its hash, and the hybrid hit collapsed to zero:
a 32K prompt edited after 90% recomputed from scratch.
Sliding-window managers now also hold the run ending at each retained
checkpoint. The held blocks keep normal eviction priority, as the
last-boundary run already did, so admission reservations are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [Release] Keep a 27B GDN state every 8K prompt tokens
The 27B recipe kept only the prompt's last Mamba checkpoint, so an edit
anywhere before the last 2K tokens of a long prompt recomputed it all.
With prefix_cache_retention_interval 8192 (and the sliding-window fix
before this commit), a 32K/64K prompt with its last 10% replaced resumes
at 24576/57344 tokens, as 1.5.1 did: TTFT 7.4 -> 2.0 s and 16.3 -> 2.3 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [Release] Keep a Flash-Next GDN state every 8K prompt tokens
The same retention as the 27B recipe for Flash-Next. With the last 10%
of a 32K/64K prompt replaced, prefill resumes at 24576/49152 tokens
(TTFT 1.5/3.1 s against 5.5/11.4 s cold); MTP's lookbehind costs one
alignment block at 64K. Both the bundled launcher and the Docker image.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [Misc] Fix line length and formatting in the Flash-Next launcher test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: mac <mac@MacBook-xinyi.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* [Bugfix][SM70] GDN prefill: use the prebuilt kernel when TileLang has no nvcc
The original FlashQLA prefill is the default on SM70 and JIT-compiles
with TileLang on first use. A bare wheel install has no CUDA toolkit, so
the first prefill failed with "No CUDA or HIP or MPS available on this
system" (or later, when /usr/local/cuda holds only the runtime).
When neither VLLM_SM70_FLASHQLA_ORIGINAL_PREFILL nor its legacy alias is
set and TileLang would find no nvcc, the default now falls back to the
prebuilt VLK prefill and logs once. An explicit request still selects the
original path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [Doc] Regenerate the runtime architecture reference for gdn.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: mac <mac@MacBook-xinyi.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A request rejected by validation, such as a prompt longer than
max_model_len, got its 400 response and then a full "Exception in ASGI
application" traceback in the server log: the only handler that matched
it was registered for Exception, and Starlette runs that one in
ServerErrorMiddleware, which re-raises after responding. The 1.5.2
release matrix logged 44 of them.
VLLMValidationError and VLLMNotFoundError now have their own
registrations, so they are answered and not re-raised. Other exceptions
still reach the server log. Handler registration moves to
register_exception_handlers() so tests can build the same app.
Co-authored-by: mac <mac@MacBook-xinyi.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* [Perf][SM70] DFlash2: keep compact target rejection with penalties
Any presence, frequency or repetition penalty sent DFlash2 verification
off the compact top-k rejection path and its sampler graph, onto the
dense full-vocabulary sampler. On Qwen3.8-27B NVFP4 that cost about
3.5 ms per round at c1 (12.7 -> 16.2 ms) with unchanged acceptance.
These penalties never raise a logit when presence >= 0, frequency >= 0
and repetition >= 1. The compact path now applies them to the 64-wide
probe with the dense kernel's arithmetic, including the counts of
earlier draft rows, then re-sorts it. The cutoff guard compares the
penalized top-20 boundary with the probe's pre-penalty floor: when an
unprobed token could reach the top 20, the request takes the existing
full-vocabulary reference. Other penalties keep the dense sampler.
The sampler graph keys on whether any request uses penalties, so
batches without them launch no extra kernel.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [Misc] DFlash2 sampler graph: use torch.accelerator memory counters
The repository's torch.cuda check flags the headroom probe now that this
file changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: mac <mac@MacBook-xinyi.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
serve_flash_next_nvfp4_v100.sh still launched the old 128K, single-sequence
recipe without host KV. It now starts the 1.5.2 release configuration (256K
with FP8 KV history in host memory, eight sequences, utilization 0.89) and
accepts --draft for a separate MTP checkpoint. The 27B launcher's help text
said four sequences; its profile serves eight.
Co-authored-by: yangzhuxinyzx <Susannah_Carterect@gmx.ca>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The Phase 3 YAML shipped a literal api-key. Pass the key at launch instead,
with --api-key or VLLM_API_KEY. The old key is in git history and must be
treated as exposed.
Co-authored-by: yangzhuxinyzx <Susannah_Carterect@gmx.ca>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
FlashQLA's TileLang GDN prefill kernels compile with nvcc at first use, and
pip's nvidia-cuda-nvcc-cu12 only ships ptxas. In a clean container both release
models failed warm-up with 'No CUDA or HIP or MPS available on this system'.
Install cuda-nvcc-12-8 and cuda-cudart-dev-12-8 from NVIDIA's repository and
set CUDA_HOME.
Co-authored-by: yangzhuxinyzx <Susannah_Carterect@gmx.ca>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* [Bugfix][SM70] Stop compiling prefill kernels for every new prompt length
_qk_norm_rope and _sm70_qwen35_gdn_split_kernel took the token count as a
constexpr, so each prefill chunk length not seen before compiled two Triton
kernels while the request waited: about 0.34 s of TTFT per new length, and
worker RSS that kept growing while serving.
Pass the length-dependent values as runtime arguments instead. Floating-point
behaviour does not change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [SM70] Leave Qwen3.8-27B about 1 GiB more VRAM headroom
With vision enabled and maximum-size images the 27B engine peaked at 32.1 GiB of
32.48 GiB. KV at 0.86 holds 5.85x of 256K, more than eight requests need at
usual context lengths, so trade about 1 GiB of KV for headroom.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: yangzhuxinyzx <Susannah_Carterect@gmx.ca>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The device-side compact/reference dispatch only captured single-request
rounds, so concurrent sampled rounds paid the Python path. Capture it for up to
four uniform requests: request slots stay live GPU inputs, and the reference
branch copies only the active proposal slots.
Graphs live in private pools outside the KV budget, so bound them: c5-c8 cost
2.7 GiB in total, ran slower than the host-decision path and ended in an OOM at
capture. Capture also needs an all-rank vote that 2.5 GiB stays free.
Co-authored-by: yangzhuxinyzx <Susannah_Carterect@gmx.ca>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* [Bugfix][SM70] Keep host-KV Flash-Next off the hybrid PLE default
The device shards of a hybrid PLE table left the state pool 0.87 GiB against the
1.05 GiB it needs for a 65536-token request, so host KV with the #1340 default
failed to start. Host KV already uses pinned memory for history, so it now keeps
the disk-backed table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [Docker] Add the SM70 release image and a launcher for both release models
One image serves Qwen3.8-27B NVFP4 + DFlash2 and Flash-Next NVFP4 + MTP4, chosen
with MODEL. The arguments match the Studio presets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [SM70] Serve eight concurrent requests in the Qwen3.8-27B profile
c8 runs without errors at 38.3 ms per pure decode round with 8K contexts and
465 tok/s in aggregate; KV capacity is 5.85x of 256K.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [Bugfix][SM70] Fit the host-KV default within the line limit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: yangzhuxinyzx <Susannah_Carterect@gmx.ca>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Inserting _prefill_graph_reserve between the decorator and
determine_available_memory moved @torch.inference_mode() off the profiling
entry point. The V1 runner's profile_run is not decorated itself, so it ran the
vision tower with autograd on and kept every block's activations alive: 27B
NVFP4 without a draft profiled a 17.89 GiB activation peak and refused to start
at 256K for lack of KV memory.
Co-authored-by: yangzhuxinyzx <Susannah_Carterect@gmx.ca>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Size the MTP draft MoE, shared gate and push all-reduce for up to 16 requests,
leave GDN and router batches above their row limits to cuBLAS instead of pieces,
add a fused dense MoE permute, and run HCX only for one request's rows. Also fix
the host-KV startup crash (slice output_gate to num_tokens) and copy every mapped
host KV run on the copy engine.
Co-authored-by: yangzhuxinyzx <Susannah_Carterect@gmx.ca>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Above eight rows the GDN input fell back to a cuBLAS GEMM with its reduction,
plus a zero fill and a tail copy: three to four extra launches per layer. Appended
CTAs of the QKVZ launch now compute b/a over the K16 activations and write qkv, z,
b and a directly, as the eight-row route already does.
Co-authored-by: yangzhuxinyzx <Susannah_Carterect@gmx.ca>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A verify round launches pack_k16_input before almost every M>8 projection. Each
launch moves under 300 KB but costs a launch and a gap. The all-reduce norm and
gate/up kernels now also write the K16 layout, and the next projection reads it
instead of launching a pack.
Co-authored-by: yangzhuxinyzx <Susannah_Carterect@gmx.ca>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
RGB PNGs can carry tRNS and palette or LA images can carry alpha, so test for
transparency before the same-mode shortcut and composite onto the configured
background.
Carries #1311 onto main, resolving its import conflict with #1310.
Co-authored-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The dual-compile lane auto-enabled disk-mmap PLE. Every MTP4 decode round
then waited about 1 ms for the CPU worker's disk lookup before the target
graph could continue (and the automatic default also blocked the PLE
cascade as an "existing explicit placement").
When physical host memory, less an 8 GiB reserve, holds three quarters of the
checkpoint's PLE n-gram table, the default now selects the existing hybrid
placement instead: decode gathers PLE rows from local device and pinned-host
shards on the GPU, prefill keeps the CPU/disk path. The per-rank host budget
is fixed from physical memory ((total - PLE host reserve) / TP - 2 GiB, capped
by the table share). Smaller hosts keep the disk default.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
#1297 stopped caching the output blocks of requests that start in reasoning
mode, assuming chat templates drop past reasoning. Qwen3.8 renders past
`reasoning` back verbatim by default, and keeps it inside a tool-call loop
even with preserve_thinking=false, so a client that returns it sends the
previous output back as the next prompt prefix.
The rule only applied in KVCacheManager.allocate_slots; the async
scheduler, used by the Qwen3.8 release profile, caches every finalized
token anyway. Remove it so both schedulers cache finalized output blocks.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The route report, acceleration report and target storage dump were logged at INFO on every rank, up to about 1 MB per line. Keep a one-line summary at INFO and move the full dumps to DEBUG.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Partial prefix entries are keyed by the parent block hash and the tail tokens. Image placeholder tokens are the same for every image of a given size, so a new image of that size resumed from the previous image's KV state and the model answered about the wrong image.
Add the multimodal items overlapping the tail to the key, and compare them on lookup.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The native PLE request lane only accepted a CPU-owned GGUF IQ4_NL table, so
the Qwen3.8-Flash-Next NVFP4 checkpoint (raw E4M3 n-gram rows in 128 mapped
safetensors shards) went through the Python owner: socket, msgpack and Python
scheduling on every decode request, about 1.7 ms per MTP4 round.
The lane now addresses rows through a list of equally sized mapped shards
(one shard is the old contiguous table) and validates rows only for IQ4_NL.
The worker admits whole-table FP8 disk layers and hands the lane their shard
pointers; results stay raw rows in n-gram id order, as the Python owner
writes them. The existing hashing, row cache, parallel cold-page reads and
draft-token page prefetch apply unchanged. The op gains three defaulted
arguments; GGUF callers are unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Compact grouped NVFP4 MoE stages dispatch with dispatch_num_override=1, so the
split-K count is chosen for one group. The launch then checks it against the
actual descriptor (one group per routed expert), whose partials need more
workspace; with the engine-bounded TurboMind scratch the check fails and
Gemm::Run returns -1. Qwen3.8-Flash-Next NVFP4 startup warmup failed this way
for 4..8 tokens (40..80 routed rows) and the engine did not start.
The override path already clamps the swizzle to the actual descriptor; clamp
the split count the same way (at least one). Launches that fit are unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Enable --enable-prompt-tokens-details so responses carry
usage.prompt_tokens_details.cached_tokens and agent clients can see their
prefix-cache hits.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A prefix-cache hit leaves a short chunk over a long prefix, which had too
few query tiles to fill the GPU, and chunks below Q=1024, below 32 tokens or
with unaligned sequence lengths fell back to the slow paged BM32 kernel.
- FA2 patch: split-KV3 takes a runtime split count (N-way merge; the
three-split entry is bit-identical).
- Route prefix chunks below the 79T range to it with a wave-based split
count and a 24 MiB workspace reserved at profiling.
- E4M3 bridge pads unaligned sequences with masked trailing rows and admits
chunks under 32 tokens at 8K+ keys.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* [Perf][Core][SM70] Partial prefix-cache hits for hybrid Mamba models; agent turns resume at the last message
Aligned Mamba prefix caching resumes only at block boundaries, where a
recurrent (GDN) state was saved. With 2048-token blocks an agent turn
recomputed up to a whole block of history, and in thinking mode the next
turn diverges a token before the end of the previous prompt (the
generation prompt renders differently once the turn is finished), so the
block boundary was the best possible hit.
Partial entries (vllm/v1/core/partial_prefix_cache.py) keep, for one exact
position, cache-owned copies of the blocks a request needs to resume
there: the partially filled full-attention page, the recurrent state
block and every sliding-window block of the window before it. They are
keyed by the hash of the last full block plus the tail tokens, so they
extend ordinary full-block hits. A step that ends at a requested boundary
registers an entry; the copies run on the worker before the next forward
pass and their sources are held until then. A request that hits an entry
continues on private copy-on-write blocks. Entries are LRU, bounded to a
quarter of the pool, and released first when running requests need
blocks.
The chat API derives the boundaries (end of the last message before the
generation prompt, end of the system prompt) by rendering small probes
with the same template and passes them as sampling extra_args. They are
rounded down to the 64-token recurrent chunk so a resumed prefill chunks
exactly like a cold one. Requests that start in reasoning mode no longer
cache their output blocks. Enabled with --partial-prefix-cache, on in the
Qwen3.8-27B profile.
Qwen3.8-27B NVFP4 + DFlash2, 4x V100-SXM2-16GB, eight-turn agent
conversations (shared 2.6K system prompt, ~1.3K new tokens per turn):
- hit / reusable prefix: 83% -> 99%, extra recomputed tokens per turn
~1,060 -> 43 (thinking: 88% -> 99.5%, 914 -> 31).
- four concurrent conversations: TTFT median 1.07 -> 0.71 s, p90
2.99 -> 1.52 s, decode speed per request 31 -> 64 tok/s, wall 27.6 ->
22.7 s.
- with FP16 KV, partial hits are bit-identical to cold prefill (23/23
greedy continuations); with E4M3 KV the prefix is read quantized, as
for any prefix hit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Type partial prefix lookup tokens as a sequence
Request.all_token_ids is a ConstantList; accept any int sequence.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The profile runs prefix caching in mamba "align" mode, which can resume a
cached prefix only where a GDN state was saved, at multiples of
mamba_block_size. With 8192-token blocks a conversation shorter than 8K
never hit the cache, and longer ones recomputed up to 8K tokens on every
turn. Use 2048, the KV page size and the smallest valid value.
Qwen3.8-27B NVFP4 + DFlash2, 4x V100-SXM2-16GB, packaged profile:
- KV cache size unchanged (425,473 tokens); 8K decode rounds identical
(c1 13.88, c4 25.96 ms) and acceptance unchanged.
- Uncached prefill: 8K 1.92 -> 1.95 s, 32K 8.07 -> 8.20 s, 128K 43.0 ->
43.6 s.
- Eight-turn agent conversation (shared 2.6K system prompt, ~1.3K new
tokens per turn): prefix-cache hit rate 28% -> 68%, 14.8 -> 9.5 s.
- Four such conversations at once: 57.7 -> 36.4 s, hit rate 29% -> 69%,
TTFT median 2.34 -> 1.34 s and p90 4.36 -> 2.93 s, median decode
speed 22 -> 56 tok/s.
- A 1.1K-token turn on a 7K cached prefix: TTFT 2.13 -> 0.58 s.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
On SM70 every mixed prefill+decode step pays a fixed cost of about
130 ms, mostly launch-bound recurrent layers. With the default 250 ms
mixed-prefill target the budget settled near 112 tokens, so a prompt
arriving beside decoding requests was prefilled at under 1000 tokens/s,
and the resident requests advanced one round per slow step for the whole
prefill.
Disable the target in the packaged profile
(mixed_prefill_step_latency_ms=0). Prefill then takes the full 4096-token
budget, finishes at near-peak speed and returns the batch to pure
decode. With prefix caching most new prefills are an uncached
conversation tail, so the decode pause stays short. A positive value
still bounds mixed steps for latency-sensitive services.
Qwen3.8-27B NVFP4 + DFlash2, 4x V100-SXM2-16GB, warm server:
- 2 decoding requests + 2x4K prompts arriving together: TTFT
4.8 / 9.6 s -> 2.0 / 2.2 s (prefill ~3800 tok/s, standalone peak
~4000); decoders pause up to 1.0 s once instead of crawling for 9.6 s.
- 2 decoding requests + 4x1K prompts: TTFT 1.4-4.9 s -> 0.6-1.1 s.
- 3 decoders + a new ~120-token prompt every 0.5 s: decode rounds in
30 s 681 -> 836.
- Official-style tasks with 4 clients: unchanged (agg tok/s
560/436/571 -> 567/431/593).
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Only the eight-row (one DFlash2 request) residual norm used the fused
TP4 push all-reduce + Gemma RMSNorm kernel. Two to four requests ran a
separate push all-reduce, or the pull path for 24 rows, followed by a
norm kernel, so every verifier norm paid an extra launch and a second
pass over the reduced rows.
Add sm70_push_allreduce_gemma_rms_norm_batch<Rows> for 16, 24 and 32
rows. It keeps the 40-CTA residency of the eight-row kernel (each CTA
owns Rows / 8 rows), pushes all of its rows before polling, and gives
every row its own generation and epoch. The rows use a separate
2.6 MB packet region, so they never share epochs with the eight-row
route. The reduction order and FP32 residual add match the eight-row
kernel.
TP4 V100 microbenchmark (CUDA graph, per call), separate -> fused:
16 rows 17.2 -> 13.3 us, 24 rows 27.3 -> 17.6 us, 32 rows 24.2 -> 22.0 us.
Qwen3.8-27B NVFP4 + DFlash2 8K-context verifier rounds: c2 19.77 ->
19.37 ms, c3 25.74 -> 24.74 ms, c4 26.02 -> 25.85 ms; c1 unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* [Perf][SM70][Scheduler] Piecewise graphs for short mixed prefills; keep the mixed budget off the graph regime
On SM70 a mixed step that joins a new prompt to resident speculative
verifiers ran eagerly, because the speculative capture sizes stop at the
verifier rows (32 tokens for four DFlash2 requests). A ~100-token prompt
then cost 170-190 ms of launch-bound eager work while every resident
request stalled, about 7 decode rounds.
Add piecewise capture sizes 64..256 to the SM70 speculative shapes, so
short mixed prefills replay graphs outside attention and GDN. FULL
verifier graphs are unchanged.
MixedPrefillBudget fitted one fixed+linear cost to all mixed steps. Graph
steps are far cheaper than eager steps of the same size, which made
prefill tokens look expensive and cut 1K prompts into many small chunks.
Graph-replayed samples (MixedPrefillTiming.cuda_graph, set by both model
runners) no longer enter the fit but can still move the budget toward
it. The budget is also never below the prefill that costs as much as the
fixed part: with a fixed cost just under the target the old rule left a
budget of a few tokens, each step spending almost all of its time on
overhead.
Qwen3.8-27B NVFP4 + DFlash2, 4x V100-SXM2-16GB, packaged profile, two
rounds with the GPU groups swapped:
- 3 decoders + a new ~120-token prompt every 0.5 s: mixed step
186/180 -> 135/134 ms, decode rounds in 30 s 777/795 -> 896/893.
- Same with 1K-token prompts: decode rounds 125/297 -> 201/318.
- Official-style tasks with 4 clients, agg tok/s: GSM8K 472 -> 539,
HumanEval 604 -> 634, MT-Bench 437 -> 418; acceptance unchanged.
- Pure decode rounds, KV cache size (425,473 tokens) unchanged; graph
memory +0.09 GiB.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Narrow the legacy per-token budget for mypy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Keep the mixed capture-size constant platform-neutral for check-layering
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The q8 verifier's fused sigmoid-gating delta-rule update launches
NV * N * HV one-warp CTAs. Batches of four or more requests already
switch BV32 to BV8. Two and three requests kept the generic BV16 and
left the GPU under-filled. BV4 exposes four times the V tiles with the
same K reduction and recurrence; output and every stored state snapshot
stay bitwise identical.
TP4 shard (H4, HV12, K=V=128, eight tokens per request), V100, per call:
N2 29.3 -> 20.5 us, N3 36.5 -> 30.9 us; N1 and N4 unchanged. With 48
GDN layers that is about 0.4 ms (c2) and 0.3 ms (c3) per verifier round
of Qwen3.8-27B.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The SM70 DFlash2 draft quantizes its four large decoder projections to
channel FP8 for M=8, one drafting request. With 2-4 requests the decoder
sees M=16..32 query rows and fell back to an FP16 torch.mm over the
original matrices, which reads twice the bytes. fp8_qpn8_gemm_sm70_out
already covers M<=32, so route M=8..32 in steps of eight there, using
split 16 where M>16 needs split 12 or 16 (gate_up). Each request's rows
now see the same FP8 weights it uses when drafting alone; with an
unchanged split they are bit-identical to its M=8 result. Prefill,
context KV, partial blocks and M>32 keep the FP16 matrices.
Draft projection, 4x V100 TP4 shard, M32: gate_up 156 -> 77 us,
down 77 -> 40 us, qkv 41 -> 41 us, o 20 -> 23 us.
Qwen3.8-27B NVFP4 + DFlash2, 4x V100-SXM2-16GB, packaged profile,
official-style tasks (GSM8K / MT-Bench / HumanEval, T=1.0, thinking,
16 prompts each) with 4 concurrent clients, aggregate tok/s:
282 -> 303, 337 -> 414, 501 -> 587. Acceptance length at c4
4.78/3.14/4.73 -> 4.95/3.15/4.41, against 4.90/3.23/4.62 for the
same tasks at c1 (single-request path, unchanged and bit-identical).
c1 is unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The mixed-prefill budget divided a whole mixed step's GPU time by its
prefill tokens. A mixed step also pays a fixed cost for the resident
decode rows and for launches that do not scale with prefill. On SM70,
mixed steps run without CUDA graphs and that fixed part alone exceeds the
250 ms target, so every sample shrank the budget. It then stuck at the
16-token quantum, because 1.25x of 16 rounds back down to 16. Prompts
were prefilled 32 to 144 tokens at a time over dozens of eager steps,
which slowed every decoding request in the batch.
Fit elapsed = fixed + per_token * tokens over decayed samples once chunk
sizes differ. Budget (target - fixed) / per_token when the fixed cost is
below the target. Otherwise budget fixed / per_token: smaller chunks
cannot meet the target and only add mixed steps. Grow by at least one
quantum, or up to 2x per step once fitted. Behaviour with a single chunk
size, the floor and the maximum is unchanged.
Qwen3.8-27B NVFP4 + DFlash2, 4x V100-SXM2-16GB, 1K prompts, warm server:
c2 36.98 -> 21.56, c3 49.73 -> 30.39, c4 83.42 -> 35.02 ms per round;
c4 aggregate 107 -> 316 tokens/s. 8K-prompt concurrency is unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* [Perf][SM70] 4096-token prefill chunks: native Q4096 79T attention and reserved long-prefix workspaces
The 79T Q8000/Q8192 prefill attention needs budgets of 8000+, so 4096-token
chunks fell back to Split-D at ~47 TFLOPS for long prefixes (250K TTFT +22%).
Add a native Q4096 specialization of the same FP32-accumulated architecture
(256-token tail, own namespace, workspace slots and a half-height score
buffer), dispatched for Q4000..Q4096 once KV reaches 8192 keys; Q4000..Q4095
are leading-padded. Operator medians at KV128K/248K are ~62 TFLOPS.
Reserve the E4M3 prefix bridge (for max_model_len) and the bridge tail
during memory profiling instead of growing them by doubling at the first
long prompt, which ran a 128K request out of memory at 4096-token chunks.
The Qwen3.8-27B NVFP4 + DFlash2 release profile now uses 4096-token chunks
and 0.86 memory utilization and supports V100-SXM2 16GB: 4.07 GiB of KV per
GPU against 3.33 GiB for 8192/0.92, TTFT within 1% to 32K and +3.9% at 250K,
decode and speed-fixture rounds unchanged, 250K + four concurrent 64K
requests without OOM.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [Style] Format the 79T owner dispatch and type the Q8000 prefill reason
clang-format reflows PrefillOwner::run; mypy needs the optional reason
declared before its branches.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* [Bugfix] Keep the pynccl all-reduce out-of-place by default
#1277 made pynccl reduce every all-reduce of 4194304 or more elements in
place. The vllm.all_reduce op is registered as out-of-place, so compiled
graphs may read its input again after the call; reducing in place hands
those readers the summed tensor instead of the rank's own partial.
Qwen3.8-27B NVFP4 + DFlash2 on TP4 V100 crosses the threshold on every
prefill of about 820 tokens or more. Greedy target output is unchanged
(identical tokens on all eight 1K/8K fixtures, identical logprobs at 1K),
but the draft context changes: speed-fixture tokens per round fell from
2.875/2.974 to 2.655/2.613 at 1K/8K.
Unset VLLM_INPLACE_AR_MIN now means out-of-place, except with
VLLM_V41_PACKED_KV=1, where the DeepSeek-V4.1 packed-KV prefill keeps the
in-place reduction it was qualified with. An explicit value still wins.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [Bugfix] Resolve the in-place all-reduce default in envs.py
Keep the communicator free of env and model coupling (check-layering):
envs.py now resolves the default, so the communicator reads
envs.VLLM_INPLACE_AR_MIN. Unset still means out-of-place, except with
VLLM_V41_PACKED_KV=1.
Also drop the model names that #1265 added to two comments in
SpeculativeConfig, which pushed its model coupling over the baseline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* [Doc] Regenerate the architecture runtime reference
#1265 added five lines to vllm/config/speculative.py, which moved the
SpeculativeConfig line anchors in runtime_reference.md and left
check-architecture-reference failing on main.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Runs DeepSeek-V4.1-Flash-1017 (GSQ-RCO IQ2_XXS GGUF + Engram IQ4_NL +
DSpark5 draft) on 4x V100-32G (TP4, one x16 PCIe uplink, 62 GB host RAM)
with expert offloading. All new paths are lossless and switchable.
Prefill (G1-G7):
- G1 activation memory: per-tile MoE scratch + FP32 accumulator, in-place
expand_hc / large all-reduce, windowed hc_transition, pynccl all_gather,
empty_cache before big passes; chunks up to 16K.
- G2 cold-expert staging ring issued ahead across tiles and layers.
- G3 routed down projection split by output rows (no duplicated TP
boundary blocks, -2.25 GiB/GPU) with an IPC peer gather for decode.
- G5 Engram readers, adaptive indexer tile, persistent activation grid.
- G6 routed-only staging for short prompts.
- G7 same-session prefix cache (scheduler guard + worker restore_prefix).
Long context (G4):
- Window layers keep 256-row private rings.
- Packed KV (VLLM_V41_PACKED_KV): FP4 + E4M3 compressed latents in a
shared host memfd plus a 32K GPU hot ring, FP4 index keys on GPU, CUDA
decode index scores and segment-parallel exact top-k. Enables 512K.
- RoPE rotates on the fly past its table; C2/Engram/guard limits and
the DSpark draft length cap are lifted to the target length.
Results vs phase 1: TTFT short 5.6 -> 0.8-1.1 s, 8K 27.7 -> 5.8 s,
15K 44.0 -> 11.1 s; 128K 110 s and 500K 651 s (needle passes); follow-up
turns hit the prefix cache in ~1-1.7 s; fresh-text decode 57.8 -> 53.5
ms/round. Tokens at <=16K are identical with packed KV.
Tooling: benchmarks/deepseek_v41/ gains the v41k SM70 extension source
and build script, serve configs (phase 2: 16K, phase 3: 512K + prefix
cache + tools + thinking), launch/validation tools, GPU micro-tests and
HANDOFF.md. V4.1 switches are registered in vllm/envs.py and read through
vllm.envs.
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1258 made a fitting divisor only usable when it fills at least half its
padded page. The 1648/1664 pairing is the case that motivated it, and no
existing test reached it: every other hybrid fixture happens to give the
draft page the same byte size as the target page, so the non-integral
branch never ran.
Assert the block size the round-up path produces and that the draft page
carries its payload rather than a fraction of it.
`_compact_sm70_hybrid_sliding_window_pages` picks the largest divisor of the
shared grid whose page still fits the target page, then pads that page to the
common size. Fitting is not enough: the padded page is charged per block, so
a divisor that fills half the page halves the cache for the same memory.
On the documented 1648-token draft page with a 1664-token target the grid is
16, so block_size=16 was chosen and each padded 3407872-byte page carried only
65536 bytes of payload - 212992 B/token for a 4096 B/token cache. Requiring
the divisor to fill at least half its page leaves such specs untouched, which
lets the round-up path below size the block from the byte budget instead.
The SDPA fallback of `_continuation_prefill` built one dense score matrix
for the whole continuation chunk. Split the query rows into the same
128-row blocks the decode path already uses, so peak score memory is
bounded by the block size instead of the chunk length.
The current chunk keeps its original precision and storage: no
requantization, no copy of K/V.
`page_size_padded` is a byte count, but `copy_with_new_block_size()` copied
it verbatim while the payload underneath it grew with the block size. Once
the new payload exceeded the padding, the copy's own `page_size_bytes`
property failed its `page_size_padded >= real_page_size` assert.
Raise the padding to the new payload when the original no longer fits, and
keep it byte-identical otherwise so callers that rely on an exact common
page still get it.
`layout_cache_dtype` looks its own mode up in a fixed table. `KVQuantMode`
gained `NVFP4` after that table was written, so an NVFP4 spec raised
KeyError as soon as the engine-wide dtype differed from it - which is the
normal case, since `kv_cache_dtype` defaults to `auto`.
Add the missing spelling and assert a round trip for every member so a
future mode cannot regress the same way.
Record the actual Flash-Next MTP4 routes, expert reuse, loaded storage
and selected sparse KV operands so a bandwidth/instruction account can
be compared against profiled service, including pipelining between HC
exchange phases.
Everything is gated behind `kernel_config.sm70_round_cost_diagnostics`,
which defaults to False: no recorder is attached and no extra kernel is
launched unless it is enabled, because diagnostic kernels invalidate
latency measurements.
With device_transcode disabled, expert loading decoded each complete
projection on every TP rank and sliced the canonical result afterwards.
When the TP boundary keeps whole source blocks, slice the packed rows
first and decode only this rank's share; when the boundary cuts an
original quantization block, keep the decode-then-slice order because
canonical groups are smaller than source blocks.
Quantization codes, coefficients and prepared layouts are unchanged.
A cache type whose scale plane makes its physical page larger than the
target page could previously only be compacted by halving the block size
repeatedly, and the search stopped at the first integer ratio. When the
ratio is non-integral the compaction skipped the type, leaving the
unification to pad it, which wastes a whole page per block.
Search the divisors of the shared logical grid instead and pick the
largest block size whose physical page still fits the target page, so a
type keeps its logical block when possible and only shrinks when the
padding would be larger than the page itself.
The hybrid page-size unification raised NotImplementedError whenever two
cache types had a non-integral page ratio. An INT8 target page of
1,064,960 B against a 1,048,576 B FP16 draft page is exactly 65/64, which
no integer block-size multiplication can express; the compaction step
silently skipped such ratios, so the failure surfaced as an exception
before allocation.
Keep the logical block size and pad the physical page instead: carry
page_size_padded through unify_kv_cache_spec_page_size, and bind the
carrier dtype spelling to the spec's declared quantization layout so a
draft spec no longer inherits the target's per-token-head scale bytes.
The retained test uses the legacy int8_per_token_head spelling because
main deliberately does not register an int8 codec; the scale plane, not
the codec name, is what creates the 65/64 ratio.
Bind existing packed PLE result and pinned transport metadata under the declared language-model scope while retaining native adapter names for language weight loading.
Assisted-by: GPT-6 (Codex)
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Reuse the native GGUF language adapter and prepared weights inside the conditional wrapper, validate a complete local unquantized tower checkpoint, and retain the Qwen3-VL encoder family. Add an optional image-mode launcher without a practical per-request count cap and preserve the existing accelerated text configuration.
Assisted-by: GPT-6 (Codex)
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Record the one-boot 52-request ordinary run, cold-prefix prefill, long-context quality, exact capacity and sampled per-device margins. Keep loading-cache pressure and the unmeasured verifier target explicit; profiling was deferred at the user request to restore the public API.
Assisted-by: GPT-6 (Codex)
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Reuse the target MTP buffer as the HC gather destination while preserving padded tails and qualified fallbacks. Qualify shared workspace ownership and FP16 draft staging at page-rounded 256K boundaries, retain the measured 2048-token Q8 prefill throughput, and version the full acceleration profile with a 512 MiB allocator reserve.
Assisted-by: GPT-6 (Codex)
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Pass the actual expert count through the existing MMQ family instead of rejecting IDs above 255. Keep expert pointer arithmetic wide enough for larger banks. Cover expert-count boundaries, upper-half routing and changed-input graph replay against dequantized references.
Assisted-by: GPT-6 (Codex)
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Keep complete causal history and state commits while bounding TP4 output storage. Preserve original unpartitioned operations and qualified fallbacks. Exercise actual metadata dispatch and padding through graph replay.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Keep the complete query count explicit for packed result decoding and remove the full hidden gather and duplicate shard before PLE. Preserve the full convolution transaction and existing fallback.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Communicate every HC prefill partial in FP32 and preserve the established process GEMM policy so MTP packed decode remains admitted. Qualify existing true-reduction state/graph cases without relaxing the failed FP16 drift limit; full-model resource and verifier gates remain pending.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Expose a typed queue budget without changing runtime placement, weights or computation hashes. Preserve the default and qualify bounded ordered loading; record host-pressure failures and pending model acceptance.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Distribute long score rows across existing exact local selectors and merge their canonical candidate union without changing FP32 scores or tie ordering. Keep typed opt-in admission and original fallbacks; record exact GPU replay/guard checks and pending model gates.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Add an explicit startup resource policy after all kernel and graph warmup. Preserve active buffers and private capture pools, and retain configuration/consumer and changed-input GPU replay checks. Record the ordinary long-context verifier curve and remaining deployment gates.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Reuse the qualified peer HC stages with separate MoE partial dependencies and padded row ownership. Restrict deployment admission to unreduced GGUF and retain existing decode/capture fallbacks. Record passed CPU gates and the remaining destination acceptance work.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Port the peer TP4 QSA row partition and bounded PLE prefill convolution, retaining tile lifetime repairs, explicit admission and ordinary decode paths. Record destination operator gates and the complete 15.217 ms verifier window.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Release consumed HC and QSA score temporaries. Keep serial layer prefetch fenced on the existing stream, with typed one/two-slot policy and 256K boundary tests.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Use the existing 32-row workspace for row-independent tiled prefill. Retain both reference and native capacity in either initialization order, and verify tiled output and changed-input CUDA graph replay bitwise.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Bound long single-prefill PLE temporaries using the existing channel chunks. Record matched host/device indexer evidence and validate state equality and device headroom.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Expose the existing host-indexer cache setting in the launcher while preserving its default. Record destination-native GPU checks, memory qualification, near-256K task completion, matched prefill, prefix reuse and public authentication/stream evidence.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Record measured prefill and pure decode, near-256K task completion, prefix hit counters and public API validation. Retain the measured prefix admission limitation and transparent proxy routing correction.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Combine the qualified prefill and decode dependencies in an owned local deployment with MTP4, prefix caching, tools, encoded target hot KV and independent draft capacity. Record the model revision, direct mirror download contract and focused GPU qualification.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Add opt-in E4M3 target hot storage with native and protected-reader support, preserving FP16 attention operands and the default cache policy. Qualify 32K mean verifier bounds, ordinary decode, quality and near-256K capacity on TP4 V100 16GB.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Share the existing projection launch with squared-norm partials for small K splits and improve output-mixing CTA coverage through eight tokens. Retain larger-shape fallbacks, FP32 arithmetic, FP16 saturation and caller-owned buffers.
Validation: 87 GPU regressions on V100-SXM2 with Torch 2.10/CUDA 12.8, bitwise own-before/after operator checks, graph replay, installed-component dependency/route audit and complete pre-commit checks. No model performance claim.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
