171acceded
* [Bugfix][SM70] GDN prefill: use the prebuilt kernel when TileLang has no nvcc The original FlashQLA prefill is the default on SM70 and JIT-compiles with TileLang on first use. A bare wheel install has no CUDA toolkit, so the first prefill failed with "No CUDA or HIP or MPS available on this system" (or later, when /usr/local/cuda holds only the runtime). When neither VLLM_SM70_FLASHQLA_ORIGINAL_PREFILL nor its legacy alias is set and TileLang would find no nvcc, the default now falls back to the prebuilt VLK prefill and logs once. An explicit request still selects the original path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Doc] Regenerate the runtime architecture reference for gdn.py Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: mac <mac@MacBook-xinyi.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
61 KiB
61 KiB
Runtime architecture reference
Generated by .venv/bin/python -m tools.generate_architecture_reference --write.
Do not edit this file by hand. This describes the accompanying source,
not environment-derived effective settings or observed execution.
Configuration owners
Reachable types are derived from VllmConfig annotations. Method links
identify initialization/hash code; they do not evaluate its semantics.
| Type | Owner paths | Lifecycle / hash methods | Source |
|---|---|---|---|
vllm.config.attention.AttentionConfig |
attention_config | compute_hash: vllm/config/attention.py | vllm/config/attention.py |
vllm.config.cache.CacheConfig |
cache_config | compute_hash: vllm/config/cache.py | vllm/config/cache.py |
vllm.config.compilation.CUDAGraphMode |
compilation_config.cudagraph_mode | vllm/config/compilation.py | |
vllm.config.compilation.CompilationConfig |
compilation_config | compute_hash: vllm/config/compilation.py; post_init: vllm/config/compilation.py | vllm/config/compilation.py |
vllm.config.compilation.CompilationMode |
compilation_config.mode | vllm/config/compilation.py | |
vllm.config.compilation.DynamicShapesConfig |
compilation_config.dynamic_shapes_config | compute_hash: vllm/config/compilation.py | vllm/config/compilation.py |
vllm.config.compilation.DynamicShapesType |
compilation_config.dynamic_shapes_config.type | vllm/config/compilation.py | |
vllm.config.compilation.PassConfig |
compilation_config.pass_config | compute_hash: vllm/config/compilation.py; post_init: vllm/config/compilation.py | vllm/config/compilation.py |
vllm.config.device.DeviceConfig |
device_config | compute_hash: vllm/config/device.py; post_init: vllm/config/device.py | vllm/config/device.py |
vllm.config.diagnostic_dump.TensorDiagnosticsConfig |
observability_config.runtime_trace.dumps | post_init: vllm/config/diagnostic_dump.py | vllm/config/diagnostic_dump.py |
vllm.config.diagnostic_dump.TensorDumpConfig |
kernel_config.sm70_moe.awq.diagnostics.compare_policy, kernel_config.sm70_moe.awq.diagnostics.dump_policy, kernel_config.sm70_moe.fp8.diagnostics.compare_policy, kernel_config.sm70_moe.fp8.diagnostics.dump_policy, observability_config.runtime_trace.dflash.pp_aux, observability_config.runtime_trace.dflash.tensors, observability_config.runtime_trace.dumps.awq_buffers, observability_config.runtime_trace.dumps.awq_compare, observability_config.runtime_trace.dumps.compile_inputs, observability_config.runtime_trace.dumps.fp8_compare, observability_config.runtime_trace.dumps.gdn_compare, observability_config.runtime_trace.dumps.gdn_core, observability_config.runtime_trace.dumps.gdn_graph, observability_config.runtime_trace.dumps.gdn_projection, observability_config.runtime_trace.dumps.mtp_step, observability_config.runtime_trace.dumps.qsa_calibration, observability_config.runtime_trace.dumps.qwen_layer, observability_config.runtime_trace.dumps.sample, observability_config.runtime_trace.dumps.sample_sync, observability_config.runtime_trace.dumps.sampler_logits, observability_config.runtime_trace.dumps.top1_sync, observability_config.runtime_trace.dumps.top_token_margin | resolve: vllm/config/diagnostic_dump.py; value: vllm/config/diagnostic_dump.py | vllm/config/diagnostic_dump.py |
vllm.config.diagnostic_sampling.SamplingDiagnosticsConfig |
observability_config.runtime_trace.sampling | post_init: vllm/config/diagnostic_sampling.py; value: vllm/config/diagnostic_sampling.py | vllm/config/diagnostic_sampling.py |
vllm.config.ec_transfer.ECTransferConfig |
ec_transfer_config | compute_hash: vllm/config/ec_transfer.py; post_init: vllm/config/ec_transfer.py | vllm/config/ec_transfer.py |
vllm.config.execution_policy.CommunicationPolicy |
parallel_config.communication, speculative_config.draft_parallel_config.communication, speculative_config.target_parallel_config.communication | value: vllm/config/execution_policy.py; resolve: vllm/config/execution_policy.py; compute_hash: vllm/config/execution_policy.py | vllm/config/execution_policy.py |
vllm.config.execution_policy.GraphPolicy |
compilation_config.runtime | resolve: vllm/config/execution_policy.py; compute_hash: vllm/config/execution_policy.py | vllm/config/execution_policy.py |
vllm.config.execution_policy.LayerExecutionPolicy |
kernel_config.layer_execution | value: vllm/config/execution_policy.py; resolve: vllm/config/execution_policy.py; compute_hash: vllm/config/execution_policy.py | vllm/config/execution_policy.py |
vllm.config.execution_policy.PlePlacementPolicy |
offload_config.ple | resolve: vllm/config/execution_policy.py; gib_bytes: vllm/config/execution_policy.py; compute_hash: vllm/config/execution_policy_base.py | vllm/config/execution_policy.py |
vllm.config.flash_v100.FlashV100Diagnostics |
observability_config.runtime_trace.flash_v100 | resolve: vllm/config/flash_v100.py; value: vllm/config/flash_v100.py; compile_ignored_aliases: vllm/config/flash_v100.py | vllm/config/flash_v100.py |
vllm.config.flash_v100.FlashV100Options |
attention_config.flash_v100.options | resolve: vllm/config/flash_v100.py; value: vllm/config/flash_v100.py; compile_ignored_aliases: vllm/config/flash_v100.py | vllm/config/flash_v100.py |
vllm.config.flash_v100.FlashV100Policy |
attention_config.flash_v100 | bind_consumers: vllm/config/flash_v100.py; compile_ignored_aliases: vllm/config/flash_v100.py; resolve: vllm/config/flash_v100.py; compute_hash: vllm/config/flash_v100.py | vllm/config/flash_v100.py |
vllm.config.gdn.GdnConfig |
kernel_config.gdn | resolve: vllm/config/gdn.py; compile_ignored_aliases: vllm/config/gdn.py; compute_hash: vllm/config/gdn.py; graph_options: vllm/config/gdn.py | vllm/config/gdn.py |
vllm.config.gdn.GdnProfileConfig |
observability_config.gdn_profile | resolve: vllm/config/gdn.py; compile_ignored_aliases: vllm/config/gdn.py | vllm/config/gdn.py |
vllm.config.gdn_projection.GdnProjectionConfig |
kernel_config.gdn.projection | value: vllm/config/gdn_projection.py; resolve: vllm/config/gdn_projection.py; compute_hash: vllm/config/execution_policy_base.py | vllm/config/gdn_projection.py |
vllm.config.gdn_schedule.GdnScheduleConfig |
kernel_config.gdn.schedule | resolve: vllm/config/gdn_schedule.py; graph_options: vllm/config/gdn_schedule.py | vllm/config/gdn_schedule.py |
vllm.config.gdn_state.GdnStateConfig |
kernel_config.gdn.state | resolve: vllm/config/gdn_state.py; graph_options: vllm/config/gdn_state.py | vllm/config/gdn_state.py |
vllm.config.gdn_state.GdnStateTraceConfig |
observability_config.gdn_state | resolve: vllm/config/gdn_state.py; compile_ignored_aliases: vllm/config/gdn_state.py | vllm/config/gdn_state.py |
vllm.config.kernel.IrOpPriorityConfig |
kernel_config.ir_op_priority | compute_hash: vllm/config/kernel.py | vllm/config/kernel.py |
vllm.config.kernel.KernelConfig |
kernel_config | compute_hash: vllm/config/kernel.py | vllm/config/kernel.py |
vllm.config.kernel.Sm70AwqConfig |
kernel_config.sm70_awq | resolve: vllm/config/kernel.py; capture_inputs: vllm/config/kernel.py | vllm/config/kernel.py |
vllm.config.kernel.Sm70Fp8Config |
kernel_config.sm70_fp8 | resolve: vllm/config/kernel.py; capture_inputs: vllm/config/kernel.py | vllm/config/kernel.py |
vllm.config.kernel.Sm70GgufConfig |
kernel_config.sm70_gguf | vllm/config/kernel.py | |
vllm.config.kernel.Sm70NvFp4Config |
kernel_config.sm70_nvfp4 | resolve: vllm/config/kernel.py; capture_inputs: vllm/config/kernel.py | vllm/config/kernel.py |
vllm.config.kernel.Sm70RingConfig |
kernel_config.sm70_ring | vllm/config/kernel.py | |
vllm.config.kv_events.KVEventsConfig |
kv_events_config | post_init: vllm/config/kv_events.py | vllm/config/kv_events.py |
vllm.config.kv_transfer.KVTransferConfig |
kv_transfer_config | compute_hash: vllm/config/kv_transfer.py; post_init: vllm/config/kv_transfer.py | vllm/config/kv_transfer.py |
vllm.config.legacy_inputs.LegacyInputs |
kernel_config.sm70_awq.legacy, kernel_config.sm70_fp8.legacy, kernel_config.sm70_moe.awq.legacy, kernel_config.sm70_moe.fp8.legacy, kernel_config.sm70_moe.mxfp4.legacy, kernel_config.sm70_moe.nvfp4.legacy, kernel_config.sm70_nvfp4.legacy, scheduler_config.sm70_inputs | value: vllm/config/legacy_inputs.py | vllm/config/legacy_inputs.py |
vllm.config.load.LoadConfig |
load_config | compute_hash: vllm/config/load.py | vllm/config/load.py |
vllm.config.lora.LoRAConfig |
lora_config | compute_hash: vllm/config/lora.py | vllm/config/lora.py |
vllm.config.mamba.MambaBackendEnum |
mamba_config.backend | vllm/config/mamba.py | |
vllm.config.mamba.MambaConfig |
mamba_config | post_init: vllm/config/mamba.py | vllm/config/mamba.py |
vllm.config.model.ModelConfig |
model_config, speculative_config.draft_model_config, speculative_config.target_model_config | compute_hash: vllm/config/model.py; post_init: vllm/config/model.py | vllm/config/model.py |
vllm.config.multimodal.MultiModalConfig |
model_config.multimodal_config, speculative_config.draft_model_config.multimodal_config, speculative_config.target_model_config.multimodal_config | compute_hash: vllm/config/multimodal.py | vllm/config/multimodal.py |
vllm.config.observability.ObservabilityConfig |
observability_config | compute_hash: vllm/config/observability.py | vllm/config/observability.py |
vllm.config.offload.OffloadConfig |
offload_config | compute_hash: vllm/config/offload.py | vllm/config/offload.py |
vllm.config.offload.PrefetchOffloadConfig |
offload_config.prefetch | vllm/config/offload.py | |
vllm.config.offload.UVAOffloadConfig |
offload_config.uva | vllm/config/offload.py | |
vllm.config.parallel.EPLBConfig |
parallel_config.eplb_config, speculative_config.draft_parallel_config.eplb_config, speculative_config.target_parallel_config.eplb_config | vllm/config/parallel.py | |
vllm.config.parallel.ParallelConfig |
parallel_config, speculative_config.draft_parallel_config, speculative_config.target_parallel_config | compute_hash: vllm/config/parallel.py; post_init: vllm/config/parallel.py | vllm/config/parallel.py |
vllm.config.pooler.PoolerConfig |
model_config.pooler_config, speculative_config.draft_model_config.pooler_config, speculative_config.target_model_config.pooler_config | post_init: vllm/config/pooler.py; compute_hash: vllm/config/pooler.py | vllm/config/pooler.py |
vllm.config.profiler.ProfilerConfig |
profiler_config | compute_hash: vllm/config/profiler.py | vllm/config/profiler.py |
vllm.config.quantization.QuantSpec |
model_config.quantization_config.linear, model_config.quantization_config.moe, speculative_config.draft_model_config.quantization_config.linear, speculative_config.draft_model_config.quantization_config.moe, speculative_config.target_model_config.quantization_config.linear, speculative_config.target_model_config.quantization_config.moe | vllm/config/quantization.py | |
vllm.config.quantization.QuantizationConfigArgs |
model_config.quantization_config, speculative_config.draft_model_config.quantization_config, speculative_config.target_model_config.quantization_config | vllm/config/quantization.py | |
vllm.config.reasoning.ReasoningConfig |
reasoning_config | vllm/config/reasoning.py | |
vllm.config.scheduler.SchedulerConfig |
scheduler_config | compute_hash: vllm/config/scheduler.py; post_init: vllm/config/scheduler.py | vllm/config/scheduler.py |
vllm.config.sm70_dflash2.DFlashDiagnosticsConfig |
observability_config.runtime_trace.dflash | compile_ignored_aliases: vllm/config/sm70_dflash2.py; post_init: vllm/config/sm70_dflash2.py; value: vllm/config/sm70_dflash2.py; resolve: vllm/config/execution_policy_base.py; compute_hash: vllm/config/execution_policy_base.py | vllm/config/sm70_dflash2.py |
vllm.config.sm70_dflash2.DFlashLookupPolicy |
speculative_config.sm70_dflash2.lookup | resolve: vllm/config/execution_policy_base.py; compute_hash: vllm/config/execution_policy_base.py | vllm/config/sm70_dflash2.py |
vllm.config.sm70_dflash2.Sm70DFlash2Config |
speculative_config.sm70_dflash2 | resolve: vllm/config/sm70_dflash2.py; graph_options: vllm/config/sm70_dflash2.py | vllm/config/sm70_dflash2.py |
vllm.config.sm70_moe.Sm70MoEConfig |
kernel_config.sm70_moe | capture_inputs: vllm/config/sm70_moe.py; compute_hash: vllm/config/sm70_moe.py | vllm/config/sm70_moe.py |
vllm.config.sm70_moe.Sm70MoEDiagnostics |
kernel_config.sm70_moe.awq.diagnostics, kernel_config.sm70_moe.fp8.diagnostics | resolve: vllm/config/sm70_moe.py | vllm/config/sm70_moe.py |
vllm.config.sm70_moe.Sm70MoEFormatConfig |
kernel_config.sm70_moe.awq, kernel_config.sm70_moe.fp8 | capture_inputs: vllm/config/sm70_moe.py; resolve: vllm/config/sm70_moe.py; hash_options: vllm/config/sm70_moe.py | vllm/config/sm70_moe.py |
vllm.config.sm70_moe.Sm70MoERoutingPolicy |
kernel_config.sm70_moe.routing | resolve: vllm/config/execution_policy_base.py; value: vllm/config/execution_policy_base.py; compute_hash: vllm/config/execution_policy_base.py | vllm/config/sm70_moe.py |
vllm.config.sm70_moe.Sm70MxFp4MoEConfig |
kernel_config.sm70_moe.mxfp4 | resolve: vllm/config/sm70_moe.py; capture_inputs: vllm/config/sm70_moe.py; hash_options: vllm/config/sm70_moe.py | vllm/config/sm70_moe.py |
vllm.config.sm70_moe.Sm70NvFp4MoEConfig |
kernel_config.sm70_moe.nvfp4 | resolve: vllm/config/sm70_moe.py; capture_inputs: vllm/config/sm70_moe.py; hash_options: vllm/config/sm70_moe.py | vllm/config/sm70_moe.py |
vllm.config.sm70_moe.Sm70UnquantizedMoEConfig |
kernel_config.sm70_moe.unquantized | resolve: vllm/config/execution_policy_base.py; value: vllm/config/execution_policy_base.py; compute_hash: vllm/config/execution_policy_base.py | vllm/config/sm70_moe.py |
vllm.config.sm70_native.CollectiveNativeConfig |
parallel_config.communication.native, speculative_config.draft_parallel_config.communication.native, speculative_config.target_parallel_config.communication.native | resolve: vllm/config/sm70_native.py; finalize_hash: vllm/config/sm70_native.py; compute_hash: vllm/config/sm70_native.py | vllm/config/sm70_native.py |
vllm.config.sm70_native.Sm70NativeConfig |
kernel_config.layer_execution.native, kernel_config.sm70_awq.native, kernel_config.sm70_fp8.native, kernel_config.sm70_gguf.native, kernel_config.sm70_marlin, kernel_config.sm70_moe.awq.native, kernel_config.sm70_moe.fp8.native, kernel_config.sm70_moe.mxfp4.native, kernel_config.sm70_moe.nvfp4.native, kernel_config.sm70_mxfp4, kernel_config.sm70_nvfp4.native | capture_inputs: vllm/config/sm70_native.py; resolve: vllm/config/sm70_native.py; hash_options: vllm/config/sm70_native.py | vllm/config/sm70_native.py |
vllm.config.sm70_runtime.RuntimeTraceConfig |
observability_config.runtime_trace | value: vllm/config/sm70_runtime.py; post_init: vllm/config/sm70_runtime.py; compile_ignored_aliases: vllm/config/sm70_runtime.py | vllm/config/sm70_runtime.py |
vllm.config.sm70_runtime.Sm70RuntimeConfig |
kernel_config.sm70_runtime | value: vllm/config/sm70_runtime.py; post_init: vllm/config/sm70_runtime.py; compile_ignored_aliases: vllm/config/sm70_runtime.py | vllm/config/sm70_runtime.py |
vllm.config.sm70_runtime.SpecDecodeTraceConfig |
observability_config.spec_decode_trace | post_init: vllm/config/sm70_runtime.py; resolve: vllm/config/sm70_runtime.py; compile_ignored_aliases: vllm/config/sm70_runtime.py | vllm/config/sm70_runtime.py |
vllm.config.sm70_runtime.StepProfilerConfig |
observability_config.step_profiler | post_init: vllm/config/sm70_runtime.py; compile_ignored_aliases: vllm/config/sm70_runtime.py | vllm/config/sm70_runtime.py |
vllm.config.sm70_sparse.Sm70SparseConfig |
kernel_config.sm70_sparse | compute_hash: vllm/config/sm70_sparse.py; resolve: vllm/config/execution_policy_base.py; value: vllm/config/execution_policy_base.py | vllm/config/sm70_sparse.py |
vllm.config.sm70_triton_attention.Sm70TritonAttentionPolicy |
attention_config.sm70_triton | resolve: vllm/config/sm70_triton_attention.py; compute_hash: vllm/config/sm70_triton_attention.py; value: vllm/config/execution_policy_base.py | vllm/config/sm70_triton_attention.py |
vllm.config.speculative.SpeculativeConfig |
speculative_config | compute_hash: vllm/config/speculative.py; post_init: vllm/config/speculative.py | vllm/config/speculative.py |
vllm.config.speculative_sampling.SpeculativeSamplingPolicy |
speculative_config.sampling_policy | resolve_fields: vllm/config/speculative_sampling.py; resolve: vllm/config/speculative_sampling.py; compute_hash: vllm/config/speculative_sampling.py | vllm/config/speculative_sampling.py |
vllm.config.structured_outputs.StructuredOutputsConfig |
structured_outputs_config | compute_hash: vllm/config/structured_outputs.py | vllm/config/structured_outputs.py |
vllm.config.turboquant_runtime.TurboQuantDiagnostics |
observability_config.runtime_trace.turboquant | compile_ignored_aliases: vllm/config/turboquant_runtime.py; resolve: vllm/config/execution_policy_base.py; compute_hash: vllm/config/execution_policy_base.py | vllm/config/turboquant_runtime.py |
vllm.config.turboquant_runtime.TurboQuantRuntimePolicy |
attention_config.flash_v100.turboquant | resolve: vllm/config/turboquant_runtime.py; compute_hash: vllm/config/execution_policy_base.py | vllm/config/turboquant_runtime.py |
vllm.config.utils.SupportsHash |
additional_config | compute_hash: vllm/config/utils.py | vllm/config/utils.py |
vllm.config.vllm.OptimizationLevel |
optimization_level | vllm/config/vllm.py | |
vllm.config.vllm.VllmConfig |
(root) | compute_hash: vllm/config/vllm.py; post_init: vllm/config/vllm.py | vllm/config/vllm.py |
vllm.config.weight_transfer.WeightTransferConfig |
weight_transfer_config | vllm/config/weight_transfer.py |
KV codecs
A registered storage descriptor does not establish native support.
| Symbol | Canonical / aliases | Storage / element | Quantized | Source |
|---|---|---|---|---|
| FP16 | auto / auto, float16 | torch.float16 / None | False | vllm/v1/attention/kv_codecs.py |
| BF16 | bfloat16 / bfloat16 | torch.bfloat16 / None | False | vllm/v1/attention/kv_codecs.py |
| FP8_E4M3 | fp8_e4m3 / fp8, fp8_e4m3 | torch.uint8 / torch.float8_e4m3fn | True | vllm/v1/attention/kv_codecs.py |
| FP8_E5M2 | fp8_e5m2 / fp8_e5m2 | torch.uint8 / torch.float8_e5m2 | True | vllm/v1/attention/kv_codecs.py |
Attention route declarations
Sorted by name, not execution priority. The executor owns candidate order. These structural constraints also require native/ABI, metadata and policy admission. Observer and fallback labels preserve historical accounting.
| Route | Stage | Codecs | Declared shape / flags | Source |
|---|---|---|---|---|
decode_dense_cache |
decode | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
decode_dense_reference |
decode | BF16, FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
decode_e4m3_compact_scalar_tail |
decode | FP8_E4M3 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 16, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
decode_paged_prefill |
decode | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
decode_scalar_paged |
decode | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": true, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
decode_strategy_legacy_revision |
decode | FP8_E4M3 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": true, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": true, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
decode_triton_no_flash_decode |
decode | BF16, FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": true, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
decode_triton_scalar_disabled |
decode | BF16, FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": true, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
decode_xqa_paged |
decode | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [4, 6, 8], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": true} | vllm/v1/attention/backends/flash_v100/routing.py |
dflash_draft_triton_fallback |
any | BF16, FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": true, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
flashinfer_sm70_fixed_entry |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": true, "page_alignment": 16, "page_sizes": [784], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
flashinfer_sm70_splitkv3_fast_visible |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": true, "page_alignment": 16, "page_sizes": [784], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
metadata_none_zero_output |
metadata | BF16, FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": true, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_capture_dflash_noncausal_paged |
metadata | BF16, FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": true, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_capture_smallq |
metadata | BF16, FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": true, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_capture_smallq_ddtree_metadata |
metadata | BF16, FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": true, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_capture_smallq_no_ddtree_metadata |
metadata | BF16, FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": true, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_ddtree_dense |
verify | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_ddtree_triton |
verify | BF16, FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_dense_d256_gqa_79t_fp32 |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": 8192, "min_chunk": 8000, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_dense_d256_gqa_79t_fp32_fringe_fallback |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": 8192, "min_chunk": 8001, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_dense_d256_gqa_79t_fp32_q4096 |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": 4096, "min_chunk": 4000, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_dense_d256_gqa_79t_fp32_q4096_pad |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": 4095, "min_chunk": 4000, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_dense_d256_gqa_79t_fp32_q8192 |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": 8192, "min_chunk": 8001, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_dense_d256_gqa_79t_fp32_q8192_pad |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": 8191, "min_chunk": 8001, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_dense_d256_gqa_arch_long |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": 8192, "min_chunk": 64, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_dense_d256_gqa_v37 |
prefill | FP16 | {"chunk_alignment": 64, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": 8192, "min_chunk": 64, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_dense_splitd_d256 |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_dense_splitd_d256_splitkv3_kernel |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [4096, 8000], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_dense_splitd_d256_splitkv_kernel |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_no_prefix_dense_flash |
prefill | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_no_prefix_paged_cache_flash |
prefill | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_bfla |
prefill | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_contig_dense |
prefill | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_contig_dense_bhmd |
prefill | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_contig_dense_fa2_d256 |
prefill | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_contig_splitd_d256 |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_decode_rows_e4m3_grouped_fp32 |
mixed_decode | FP8_E4M3 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 16, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_decode_rows_scalar |
mixed_decode | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": true, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_decode_rows_xqa |
mixed_decode | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [4, 6, 8], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": true} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_dflash_noncausal_batch |
prefill | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_flash |
prefill | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_fp8_bridge_exact_d256 |
prefill | FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_fp8_bridge_exact_d256_tailpad |
prefill | FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_fp8_bridge_exact_dense_d256 |
prefill | FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_fp8_bridge_exact_dense_d256_tailpad |
prefill | FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_fp8_e4m3_bridge |
prefill | FP8_E4M3 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_fp8_e5m2_bridge |
prefill | FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_gather_splitd_d256 |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_paged_anchored |
prefill | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_paged_splitd_d256 |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_splitd_d256 |
prefill | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_prefix_splitkv |
prefill | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_smallq_decode_scalar |
verify | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": true, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_smallq_decode_xqa |
verify | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6, 8], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": true} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_smallq_dflash2_grouped_verify |
verify | FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_smallq_e4m3_grouped_fp32 |
verify | FP8_E4M3 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 16, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_smallq_fp16_grouped_fp32 |
verify | FP16 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": false, "gqa_ratios": [6], "head_dims": [256], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 16, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
prefill_triton_safe |
prefill | BF16, FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": true, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
unsupported_triton_fallback |
any | BF16, FP16, FP8_E4M3, FP8_E5M2 | {"chunk_alignment": 1, "chunk_sizes": [], "fallback": true, "gqa_ratios": [], "head_dims": [], "max_chunk": null, "min_chunk": 0, "observer": false, "page_alignment": 1, "page_sizes": [], "xqa": false} | vllm/v1/attention/backends/flash_v100/routing.py |
MoE stage bindings
AWQ/FP8 rows retain the declared suffix; native_binding constructs the
native entry name. FP4 rows declare full native names. These tables are
not a Cartesian product of supported model/shape combinations. Existing
selectors enforce supported plans; GGUF and skinny retain their own adapters.
| Family | Stage / mode | Declared binding | Covers | Layout | Arithmetic | Source |
|---|---|---|---|---|---|---|
| awq/fp8 | w13 / active_dense | single_token_dense_w13 | input+W13 | active | TM FP16 | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| awq/fp8 | w13 / active_grouped | active_dense_stage | W13 | active expert offsets | TM FP16 | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| awq/fp8 | w13 / batched | gemm | W13 | dense expert offsets | TM grouped | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| awq/fp8 | w13 / compact | single_token_compact_dense_w13 | input+W13 | compact active | TM FP16 | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| awq/fp8 | w13 / dense | dense_stage | W13 | dense expert offsets | TM FP16 | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| awq/fp8 | w13 / indexed | single_token_indexed_dense_w13 | input+W13 | indexed active | TM FP16 | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| awq/fp8 | w13 / indexed_prefill | indexed_dense_w13 | W13 | indexed input | TM FP16 | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| awq/fp8 | w13 / per_expert_dispatch | gemm | W13 | dense expert offsets | TM per expert dispatch | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| awq/fp8 | w2 / active_dense | single_token_dense_stage | W2 | active | TM FP16 | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| awq/fp8 | w2 / active_grouped | active_dense_stage | W2 | active expert offsets | TM FP16 | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| awq/fp8 | w2 / batched | gemm | W2 | dense expert offsets | TM grouped | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| awq/fp8 | w2 / dense | dense_stage | W2 | dense expert offsets | TM FP16 | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| awq/fp8 | w2 / indexed | single_token_indexed_dense_stage | W2 | indexed active | TM FP16 | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| awq/fp8 | w2 / per_expert_dispatch | gemm | W2 | dense expert offsets | TM per expert dispatch | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| mxfp4 | w13 / dense | mxfp4_moe_dense_stage_sm70_out | w13 | expert offsets | TurboMind FP16 boundary | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| mxfp4 | w13 / prepare_w13 | mxfp4_moe_single_token_prepare_w13_sm70_out | route+w13 | direct top6 | TurboMind FP16 boundary | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| mxfp4 | w13 / qpn | mxfp4_moe_qpn_m1_sm70_out | w13 | route slots | FP32 accumulation; FP16 boundary | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| mxfp4 | w2 / dense | mxfp4_moe_dense_stage_sm70_out | w2 | expert offsets | TurboMind FP16 boundary | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| mxfp4 | w2 / qpn | mxfp4_moe_qpn_m1_sm70_out | w2 | route slots | FP32 accumulation; FP16 boundary | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w13 / active_grouped | nvfp4_grouped_w13_sm70_out | w13+activation | shared compact route groups | split4/8; original fused SwiGLU rounding | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w13 / dense | nvfp4_moe_dense_stage_sm70_out | w13 | expert offsets | TurboMind FP16 boundary | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w13 / fused_batch_qpn | nvfp4_moe_qpn_w13_swiglu_batch_sm70_out | w13+activation | M4/8/16 route slots | preserves FP16 intermediate rounding | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w13 / fused_batch_qpn_raw | nvfp4_moe_qpn_raw_w13_swiglu_batch_sm70_out | w13+activation | M4/8/16 raw scales | preserves FP16 intermediate rounding | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w13 / fused_qpn | nvfp4_qwen38_w13_fused_swiglu_out | w13+activation | interleaved M1 | preserves FP16 intermediate rounding | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w13 / glm_qpn | nvfp4_glm53_moe_q8_qpn_sm70_out | w13 | GLM TP8 M8 sorted rows | preserves FP16 intermediate rounding | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w13 / indexed_fused | nvfp4_moe_indexed_fused_swiglu_sm70_out | w13+activation | indexed interleaved | preserves FP16 intermediate rounding | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w13 / indexed_prefill | nvfp4_moe_indexed_dense_stage_sm70_out | w13 | indexed input | preserves FP16 intermediate rounding | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w13 / indexed_split_fused | nvfp4_moe_indexed_fused_swiglu_sm70_out | w13+activation | N256 + N64 indexed slices | preserves FP16 intermediate rounding | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w13 / qpn | nvfp4_moe_qpn_m1_sm70_out | w13 | route slots | FP32 accumulation; FP16 boundary | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w13 / qpn_mtp | nvfp4_moe_qpn_mtp5_sm70_out | w13 | M5 route slots | explicit split-K; FP16 boundary | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w13 / qpn_raw | nvfp4_moe_qpn_raw_scale_sm70_out | w13 | raw E4M3 + global scale | explicit split-K; FP16 boundary | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w2 / active_grouped | nvfp4_grouped_w2_sm70_out | w2+reduce | shared compact route groups | original grouped W2 reduction | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w2 / batch_reduce | nvfp4_moe_qpn_w2_reduce_sm70_out | w2+reduce | Qwen batch | fused weighted reduction; original rounding | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w2 / batch_reduce_raw | nvfp4_moe_qpn_raw_w2_reduce_sm70_out | w2+reduce | Qwen batch raw scales | fused weighted reduction; original rounding | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w2 / dense | nvfp4_moe_dense_stage_sm70_out | w2 | expert offsets | TurboMind FP16 boundary | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w2 / direct_reduce | nvfp4_qwen38_w2_direct_reduce_out | w2+reduce | Qwen M1 | fused weighted reduction; original rounding | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w2 / grouped_batch_reduce | nvfp4_grouped_w2_batch_reduce_sm70_out | w2+reduce | shared MTP5 route groups | original grouped batch reduction | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w2 / qpn | nvfp4_moe_qpn_m1_sm70_out | w2 | route slots | FP32 accumulation; FP16 boundary | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w2 / qpn_mtp | nvfp4_moe_qpn_mtp5_sm70_out | w2 | M5 route slots | explicit split-K; FP16 boundary | vllm/model_executor/layers/fused_moe/sm70/declarations.py |
| nvfp4 | w2 / qpn_raw | nvfp4_moe_qpn_raw_scale_sm70_out | w2 | raw E4M3 + global scale | explicit split-K; FP16 boundary | vllm/model_executor/layers/fused_moe/sm70/declarations.py |