Runs DeepSeek-V4.1-Flash-1017 (GSQ-RCO IQ2_XXS GGUF + Engram IQ4_NL +
DSpark5 draft) on 4x V100-32G (TP4, one x16 PCIe uplink, 62 GB host RAM)
with expert offloading. All new paths are lossless and switchable.
Prefill (G1-G7):
- G1 activation memory: per-tile MoE scratch + FP32 accumulator, in-place
expand_hc / large all-reduce, windowed hc_transition, pynccl all_gather,
empty_cache before big passes; chunks up to 16K.
- G2 cold-expert staging ring issued ahead across tiles and layers.
- G3 routed down projection split by output rows (no duplicated TP
boundary blocks, -2.25 GiB/GPU) with an IPC peer gather for decode.
- G5 Engram readers, adaptive indexer tile, persistent activation grid.
- G6 routed-only staging for short prompts.
- G7 same-session prefix cache (scheduler guard + worker restore_prefix).
Long context (G4):
- Window layers keep 256-row private rings.
- Packed KV (VLLM_V41_PACKED_KV): FP4 + E4M3 compressed latents in a
shared host memfd plus a 32K GPU hot ring, FP4 index keys on GPU, CUDA
decode index scores and segment-parallel exact top-k. Enables 512K.
- RoPE rotates on the fly past its table; C2/Engram/guard limits and
the DSpark draft length cap are lifted to the target length.
Results vs phase 1: TTFT short 5.6 -> 0.8-1.1 s, 8K 27.7 -> 5.8 s,
15K 44.0 -> 11.1 s; 128K 110 s and 500K 651 s (needle passes); follow-up
turns hit the prefix cache in ~1-1.7 s; fresh-text decode 57.8 -> 53.5
ms/round. Tokens at <=16K are identical with packed KV.
Tooling: benchmarks/deepseek_v41/ gains the v41k SM70 extension source
and build script, serve configs (phase 2: 16K, phase 3: 512K + prefix
cache + tools + thinking), launch/validation tools, GPU micro-tests and
HANDOFF.md. V4.1 switches are registered in vllm/envs.py and read through
vllm.envs.
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
