Retain full crossing gaps separately and support deliberate prefix-reuse probes. Correct snapshot scalar argument positions for shared and separate NVFP4 dispatch.
Co-authored-by: OpenAI Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Use the measured 256-row crossover, preserve FP32 accumulation on the new small-prefill path, and cover QPN2 and TurboMind packed layouts with real kernel basis tests.
Co-authored-by: OpenAI Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Keep the common scaled-MM weight orientation at the loader boundary, retain the original activation dtype for Marlin scale preparation, and record Turing NVFP4 fallback reasons for startup diagnostics.
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Use per-engine policy, native layout checks and registered linear kernels. Own prefill scratch per invocation and resolve workspace addresses inside opaque runtime operations. Preserve existing Volta execution and retain the original QPN arithmetic.
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
The compiled SM70 ops carry the QPN2 (NVFP4) and QPN8 (FP8) decode kernels
and the QPN8 dense prefill, none of which depend on the TurboMind GEMM
registry, but every gate admitted exact SM70 only and the registry itself
is Arch<700, 750>. A Turing worker therefore took Marlin for NVFP4 and,
for the mixed ModelOpt checkpoints, did not load at all (minimum
capability 89; with the gate lowered the Marlin FP8 path fails on
orig_dtype).
Turing now prepares NVFP4 linears as the QPN2 prepack only and runs the
QPN2 kernels for M <= 32 and a Triton dequantization into a transient
fp16 buffer plus cuBLAS above that, split inside one registered op so the
compiled decode graph keeps the kernel path; FP8 per-tensor linears become
QPN8 codes with channel scales and run the existing QPN8 dispatch. Gates
read the worker's own device. Volta keeps its TurboMind path unchanged.
The Turing FP8 QPN8 route registers its dense workspace with
sm70_layer_workspaces and dispatches through vllm::sm70_fp8_qpn8_dispatch
with the layer prefix, instead of keeping workspace.data_ptr() in the
layer state: an address passed from apply() is baked into AOT artifacts,
and every warm start of a Turing FP8 model then failed with "The
specified pointer resides on host memory".
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Peuqui <peuqui@github.com>
(cherry picked from commit 2d0796310150ee52b799c08bc636c0ed55580217)
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
The SM70 route's kernels are FP16 Triton with software FP8 and do not
depend on Volta. One predicate, is_device_capability_family(70), now
replaces exact (7, 0) at the route's gates: impl selection and
_use_sm70_path, the backend capability, the FlashMLA tile-scheduler skip,
the C128A fixed row stride, the FP16 indexer, the cuBLAS indexer decode,
the compressor kernel choice, DSpark's main_proj scale and context KV
insert, the --dtype half check and the compressed-index decode graph
buckets. Volta and Turing stages of one pipeline therefore declare the
same backend.
Turing takes Marlin for block FP8, which cannot serve the grouped matmul
of an is_bmm layer; Fp8LinearMethod now dequantizes such a weight to fp16
once at load and applies it per group, with the contract of the TurboMind
grouped path (DeepSeek-V4 wo_a).
The indexer test takes vLLM's current_stream() so that leaving its
capture contexts does not record torch's default stream as vLLM's current
stream and break later graph captures in the same process.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
process_weights_after_loading prepared every expert into a list and
stacked the list at the end, while the checkpoint tensors stayed alive
until both projections were stacked. Post-load processing runs with every
layer of the stage resident, so on DeepSeek-V4-Flash with eight layers per
V100 stage the stack no longer fit.
Copy each prepared expert straight into a preallocated stack and release
the w13 checkpoint before repacking w2. The prepared tensors are
byte-identical; only the order of allocation and release changes.
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Peuqui <peuqui@github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Several SM70 linear paths passed a scratch workspace address from apply()
as an int: Fp8LinearMethod to fp8_qpn8_dispatch_sm70_out (three sites) and
fp8_gemm_sm70_prefill_dispatch_out (two sites), qwen3_5.py to
fp8_qpn8_dispatch_ba_split_sm70_out, and the TurboMind NVFP4 QPN4 state to
nvfp4_qpn4_dispatch_sm70_out (two sites). TorchDynamo records the integer
as a constant, so the AOT artifact carries the compiling process's
address. The next process loads the artifact, the kernel writes through a
stale address, and the first forward fails with "The specified pointer
resides on host memory and is not registered with any CUDA device". #672
fixed the same pattern for compressed-tensors channel FP8 only.
New module quantization/utils/sm70_layer_workspaces.py: prepared layers
register their workspace under the layer prefix, which is identical in
every process, and apply() passes the prefix to four opaque custom ops
(vllm::sm70_fp8_qpn8_dispatch, vllm::sm70_fp8_prefill_dispatch,
vllm::sm70_fp8_qpn8_dispatch_ba_split, vllm::sm70_nvfp4_qpn4_dispatch)
that resolve the address at run time. A lookup by weight size like #672's
would not do here: depending on the route a layer holds the prefill-dense,
the PP2 x TP4 or the QPN4 workspace. Registering a second workspace for one
prefix, or a layer without a prefix, raises. fp8.py keeps
sm70_fp8_prefill_exact_dense_workspace_ptr as the "workspace present"
marker other gates read; the QPN4 state drops its pointer field.
Tests: export, save, re-register a new workspace and reload for the
dispatch ops; the existing dispatch and adapter tests register a workspace
instead of a fake pointer and run the opaque ops through a CPU kernel for
the test's duration. That fixture also registers #672's op, whose
CPU-tensor test fails on current main with "Could not run
'vllm::sm70_ct_fp8_qpn8_dispatch' with arguments from the 'CPU' backend".
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Peuqui <peuqui@github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
These fail on main 357d07bc for reasons in the tests, not in the code they
cover:
- test_scheduler: the scheduler strips reasoning tokens through
structured_output_manager.trim_reasoning_for_advance before the grammar
sees them; the Mock returned a Mock, which the grammar rejected. Pass the
token ids through.
- test_spec_token_aliasing (2): the scheduler output carries
ddtree_payloads_by_req_id and scheduled_ddtree_payloads now; the
hand-built outputs lacked them.
- test_sm70_compressed_tensors_fastpaths: an SM70 VllmConfig built earlier
in the process exports VLLM_SM70_BATCH_GEMM_LAYOUTS=1, which sends the
QPN8 prepare test into the batch-GEMM layout and the stubbed
fp8_sm70_prepare. Pin that layout off in the test; alone the test passed.
- test_sm70_flash_v100_multihead: the built-in long and grouped E4M3
cases call SM70-only kernels and failed on Turing ("E4M3 grouped FP32
supports SM70 only"); skip them off SM70. The XQA cases keep running
there.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Peuqui <peuqui@github.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Keep separate HC transport channels and explicit FP32 concurrent versus existing MTP numerical policies. Preserve both dense/GDN dispatch contracts.
Assisted-by: OpenAI Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Reuse the PR #692 engine measurement harness with full-precision control/candidate contracts. Native HC mode never loads research extensions.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Warm prepared compressed batch layouts before coordinated cache import,
prefetch weight/K panels, share merge work, and reduce sampling overhead.
Keep the draft context projection's concurrent reduction tree stable so
startup autotuning cannot erase the serving gain through acceptance drift.
Matched 32K/256 service medians improve C4/C8 pure decode by 15.44%/15.78%,
with identical output tokens and acceptance counters at C1/C2/C4/C8.
Assisted-by: OpenAI Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Preserve both migration ledgers. The 23.85 ms measurement retains its original source and extension hashes; record post-integration validation separately.
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Reuse grouped MTP5 experts, fuse native gated RMSNorm, and pack GDN input projections. Record matched 23.85 ms rounds with token and acceptance parity and the remaining under-20 ms target.
Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
ModelOpt NVFP4 MoE on exact SM70 binds to ModelOptNvFp4SM70MoEMethod before
the backend oracle runs, and that method owns routing and the TurboMind
expert GEMMs -- it never reads --moe-backend. Its tuned prefill and decode
paths are gated on the TP4 shapes (num_experts 512, hidden 2560, w13 n=320),
in Python and again in the kernels' own TORCH_CHECKs. At other topologies
every fast path falls through to the per-expert dense stage, which re-reads
an expert's weights per call: on Qwen3.8-Flash-Next with TP2 that is about
147k launches of ~31 rows each per 18k-token prefill, 20.6 s of the 28 s.
Honour the user's explicit pick instead. Without --moe-backend nothing
changes. Measured on 2x Quadro RTX 8000 + 2x Tesla V100-PCIE-32GB (TP2 PP2),
18k prefill, needle tests at 30k and 124k 4/4 in all three arms:
TurboMind (before) prefill 26.6-27.2 s, step 53-56 ms
--moe-backend sm70_skinny prefill 19.3-19.4 s, step 52-55 ms
--moe-backend marlin prefill 17.7-17.8 s, step 56-59 ms
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Peuqui <peuqui@github.com>
Accept aligned local projections independently of TP size and retain checks rejecting partial packing rows and scale groups. The former TP4-only assertion predates the generalized source route.
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Older extensions expose the original compact-scale operator but retain one allocation per captured layer. Keep persistent FP16 scales until the rebuilt reuse capability is available.
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
test_mxfp4_sm70_platform_gate_is_exact replaces
current_platform.is_device_capability with a lambda that only accepts
`capability`. Now that is_exact_sm70_cuda_platform passes the worker's device
through, that stub raises TypeError. Widen it the way
tests/config/test_sm70_gates_any_visible_device.py already does; the test's
intent is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
