The SM70 DFlash2 draft quantizes its four large decoder projections to
channel FP8 for M=8, one drafting request. With 2-4 requests the decoder
sees M=16..32 query rows and fell back to an FP16 torch.mm over the
original matrices, which reads twice the bytes. fp8_qpn8_gemm_sm70_out
already covers M<=32, so route M=8..32 in steps of eight there, using
split 16 where M>16 needs split 12 or 16 (gate_up). Each request's rows
now see the same FP8 weights it uses when drafting alone; with an
unchanged split they are bit-identical to its M=8 result. Prefill,
context KV, partial blocks and M>32 keep the FP16 matrices.
Draft projection, 4x V100 TP4 shard, M32: gate_up 156 -> 77 us,
down 77 -> 40 us, qkv 41 -> 41 us, o 20 -> 23 us.
Qwen3.8-27B NVFP4 + DFlash2, 4x V100-SXM2-16GB, packaged profile,
official-style tasks (GSM8K / MT-Bench / HumanEval, T=1.0, thinking,
16 prompts each) with 4 concurrent clients, aggregate tok/s:
282 -> 303, 337 -> 414, 501 -> 587. Acceptance length at c4
4.78/3.14/4.73 -> 4.95/3.15/4.41, against 4.90/3.23/4.62 for the
same tasks at c1 (single-request path, unchanged and bit-identical).
c1 is unchanged.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Runs DeepSeek-V4.1-Flash-1017 (GSQ-RCO IQ2_XXS GGUF + Engram IQ4_NL +
DSpark5 draft) on 4x V100-32G (TP4, one x16 PCIe uplink, 62 GB host RAM)
with expert offloading. All new paths are lossless and switchable.
Prefill (G1-G7):
- G1 activation memory: per-tile MoE scratch + FP32 accumulator, in-place
expand_hc / large all-reduce, windowed hc_transition, pynccl all_gather,
empty_cache before big passes; chunks up to 16K.
- G2 cold-expert staging ring issued ahead across tiles and layers.
- G3 routed down projection split by output rows (no duplicated TP
boundary blocks, -2.25 GiB/GPU) with an IPC peer gather for decode.
- G5 Engram readers, adaptive indexer tile, persistent activation grid.
- G6 routed-only staging for short prompts.
- G7 same-session prefix cache (scheduler guard + worker restore_prefix).
Long context (G4):
- Window layers keep 256-row private rings.
- Packed KV (VLLM_V41_PACKED_KV): FP4 + E4M3 compressed latents in a
shared host memfd plus a 32K GPU hot ring, FP4 index keys on GPU, CUDA
decode index scores and segment-parallel exact top-k. Enables 512K.
- RoPE rotates on the fly past its table; C2/Engram/guard limits and
the DSpark draft length cap are lifted to the target length.
Results vs phase 1: TTFT short 5.6 -> 0.8-1.1 s, 8K 27.7 -> 5.8 s,
15K 44.0 -> 11.1 s; 128K 110 s and 500K 651 s (needle passes); follow-up
turns hit the prefix cache in ~1-1.7 s; fresh-text decode 57.8 -> 53.5
ms/round. Tokens at <=16K are identical with packed KV.
Tooling: benchmarks/deepseek_v41/ gains the v41k SM70 extension source
and build script, serve configs (phase 2: 16K, phase 3: 512K + prefix
cache + tools + thinking), launch/validation tools, GPU micro-tests and
HANDOFF.md. V4.1 switches are registered in vllm/envs.py and read through
vllm.envs.
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With device_transcode disabled, expert loading decoded each complete
projection on every TP rank and sliced the canonical result afterwards.
When the TP boundary keeps whole source blocks, slice the packed rows
first and decode only this rank's share; when the boundary cuts an
original quantization block, keep the decode-then-slice order because
canonical groups are smaller than source blocks.
Quantization codes, coefficients and prepared layouts are unchanged.
Concurrent verification (2-8 requests x 5 rows) now keeps the single-request
fast paths or a grouped equivalent; every change is bitwise equal to the
per-request result.
- QSA shared-key indexer scores several requests in one launch.
- Router, shared expert and PLE spec conv run 5..40 rows in one launch.
- Exact gated RMSNorm covers 64 tokens (was a 9-kernel torch chain at c4).
- Routed gate/up decodes each expert once per few of its routes (IQ3_S,
IQ4_NL, IQ4_XS) from 10 rows.
- QSA fused side projection chooses its path per call: the decode compile
range spans 1..40 rows and had frozen the unfused branch.
- Expert-cache lookups publish hits without system-scope fences.
- GDN replay helpers move to a neutral module; the commit-only verifier call
lives with the native verifier.
54304 GPUs 4-7, MAX_NUM_SEQS=8, verify ms/round:
c1 4K 18.96, 32K 19.56; c2 4K 26.2, 32K 30.9; c4 4K 36.4, 32K 45.2;
c8 4K 62.9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Imports the uncommitted 2026-10-15 state of
codex/v100-flashnext-prefill-6k-9683-20261013-160548: predictive and
resident GGUF expert preparation, batched expert swap transport, the
256-thread original GDN prefill option, extra affine BLAS thresholds,
ID-only tokenizer encode and the QSA physical block-table fix. The MTP
expert-capacity fix keeps this branch's earlier version (898db479d).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The NumPy decode/pack of the fused side projections built whole-array
uint32 temporaries and spent ~49 s per worker at every startup. A C++ op in
_C does decode and pack in one pass per 32-row tile, tiles in parallel;
it is byte-identical to the NumPy path (36 synthetic cases over all six
GGUF types incl. row/K padding, and 15 real Flash-Next tensors), 22x
faster on real weights. The NumPy path remains the fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reuse the target MTP buffer as the HC gather destination while preserving padded tails and qualified fallbacks. Qualify shared workspace ownership and FP16 draft staging at page-rounded 256K boundaries, retain the measured 2048-token Q8 prefill throughput, and version the full acceleration profile with a 512 MiB allocator reserve.
Assisted-by: GPT-6 (Codex)
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Pass the actual expert count through the existing MMQ family instead of rejecting IDs above 255. Keep expert pointer arithmetic wide enough for larger banks. Cover expert-count boundaries, upper-half routing and changed-input graph replay against dequantized references.
Assisted-by: GPT-6 (Codex)
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
- GGUF affine BLAS scratch is sized by the projections actually admitted
(12.5 MiB) instead of the largest calibrated shape of any model (170 MiB).
- The speculator reads the target's persistent MTP hidden buffer in place
instead of copying it into a second 8192 x 10240 buffer (160 MiB).
- The MTP layer borrows the target's serially reused QSA top-k workspace
(64 MiB) and identical RoPE tables (32 MiB).
- MTP Q8 MoE prefill chunk 2048 -> 1024 tokens (its temporaries sit at the
measured prefill peak).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Register the new VLLM_* switches in vllm/envs.py and read them through
envs; torch.accelerator instead of torch.cuda; regex instead of re; read
the hybrid PLE flag through ple_policy; move the IQ4_NL packed PLE result
helpers to kernels/ple/gguf_packed_result.py; type annotations for lazily
started expert-cache state; ruff/clang-format/markdown fixes. The layering
baseline accepts the opt-in SM70 sparse target sampling hooks in the EAGLE
speculator and its greedy-draft rejection kernel.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Prefill was launch-bound. The batched MMQ expert kernel was admitted only
for exactly 512 tokens of a calibrated source list, so every other chunk
ran the per-token MMVQ path (tens of thousands of launches per layer).
Admit MMQ from 64 tokens up for any expert bank the installed operator
supports.
Host-history QSA resolved and attended prefill rows in 8-row slices bounded
by the Triton staging rows. The native reader needs no staging: resolve
and attend 32 rows per launch (rows are independent, outputs unchanged).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use common weight/scale descriptors and dense launch plumbing for AWQ, FP8, MXFP4 and NVFP4. Keep runtime caches and workspace ownership behind a native interface without changing operators or numerical kernels.
Co-authored-by: Codex <codex@openai.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
