diff --git a/benchmarks/flashnext_prefill/DECODE_PARITY_20261015.md b/benchmarks/flashnext_prefill/DECODE_PARITY_20261015.md new file mode 100644 index 000000000..5d1484c5c --- /dev/null +++ b/benchmarks/flashnext_prefill/DECODE_PARITY_20261015.md @@ -0,0 +1,85 @@ +# Flash-Next TP4 verifier and memory qualification + +The destination is four V100-SXM2-16GB GPUs with all-pair NV2 links, CUDA +12.8, driver 570.211.01 and Torch 2.10.0+cu128. The model is +SC117/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-GGUF IQ3_S with stock MTP4. +The total request capacity is 262144 tokens. Sampling is temperature 0.6, +top-p 0.95 and top-k 20; measured requests use seed 20261014 and at most +192 output tokens. C1, FULL M5, greedy draft sampling, prefix caching and +tool calling stay enabled. Expert placement remains 80% device / 20% host. +Authoritative QSA KV is host E4M3; compressed indexer history is on GPU. + +## Acceptance metric + +Verifier cost starts at the earliest target graph GPU start and ends at the +latest target state-commit completion across all four ranks. It includes +the target graph and following state work. Acceptance length and emitted +tokens/s are separate metrics. Kernel service sums and node-trace times +are attribution evidence, not accepted wall latency. + +The ordinary-worker control has all per-step CUDA fences disabled. Low +overhead graph-envelope traces preserve every complete committed round. +Pipeline-end work is separately inventoried from graph and commit counts; +missing target starts are retained as conservative bounds, never omitted. + +| Actual input tokens | Committed rounds | Mean ms | p50 ms | p90 ms | p99 ms | +| ---: | ---: | ---: | ---: | ---: | ---: | +| 32774 | 520 | 15.894 | 14.647 | 18.675 | 31.449 | +| 65604 | 576 | 16.259 | 15.119 | 18.562 | 31.459 | +| 131124 | 680 | 16.366 | 15.593 | 17.068 | 30.765 | +| 261044 | 704 | 17.545 | 17.028 | 18.283 | 28.707 | + +Control source is 760dcbcb8b. Every one of 2480 windows is exact. Mean cost +grows 10.39% across this sweep, so an all-context 16 ms claim is not yet +supported. The following changes still need matched complete-model curves +and a fresh ordinary-worker production restart before API promotion. + +## Bounded resources and exact selector candidate + +- HC prefill residuals belong to their TP row owners. Destination operator, + recurrent reference and real model-adapter padding/graph checks pass. + At 8192 rows the stage's temporary peak falls from 502.28 to 170.02 MiB. + The optional peer-normalization bank stays disabled. +- A single prefill staging slot reuses the existing 128 MiB E4M3 budget for + 262144 tokens. Its lifetime ends after the owning attention use; side + prefetch completion is preserved. FP16 draft capacity remains 131072. +- `qsa_host_kv_trim_warmup_cache=true` returns unused ordinary allocator + pages once after all model, graph and auxiliary warmups. Actual host-QSA + consumers are required; active tensors and graph pools remain owned. + No inference-time garbage collection is added. +- `sm70_sparse.qsa_decode_partition_topk=true` reuses the normal exact + FP32 selector over 8192-column partitions, then selects their canonical + candidate union. Admission is M5/M10, one request, SM70 and score capacity + 32768..69632 compressed columns. Other inputs retain the qualified + selector. Exact FP32 values, lower-index tie ordering, output guards and + changed-input graph replay pass the destination operator suite. + +All candidate flags default off in the library. No private kernel DSO, +preload or shared-venv modification is required. Native sources and the +qualified native artifact are unchanged by these changes. + +## Startup host pressure + +The first partition/trim candidate launch was killed by systemd-oomd during +expert loading: parent user-service memory PSI exceeded 50% for 20 seconds. +The recorded first unit peak was 40.2 GiB plus 2.4 GiB swap. Another attempt +also exceeded the pressure threshold. These were host-pressure failures, +not GPU allocation failures. The separate 5 GiB MemAvailable guard did not +fire; its threshold and the system OOM policy remain unchanged. + +`sm70_gguf.expert_load_prefetch_banks=1` bounds queued pinned expert banks +during initialization. The validated range is 1..4 and the library default +remains 4. It changes neither weights nor placement nor runtime computation, +and therefore does not perturb the computation hash. The loader may also +hold the bank currently being consumed and a whole-bank unpacking scratch; +this setting is a queue bound, not a total pinned-memory quota. Tests cover +queue bounds, ordered values, configuration delivery and hash invariance. +Fresh startup and complete-model memory qualification remain required. + +## Rejected screens + +Warp-aggregated histograms regress random inputs. Warp-local emission +improves too little to justify a native change. An exact integer-key final +merge is slower than the existing FP32 selector. These standalone research +screens are not production providers. The partitioned FP32 candidate is +the only promoted selector experiment; model promotion remains pending. diff --git a/docs/design/architecture/runtime_reference.md b/docs/design/architecture/runtime_reference.md index d7c4b635a..c2ab0bc9b 100644 --- a/docs/design/architecture/runtime_reference.md +++ b/docs/design/architecture/runtime_reference.md @@ -38,12 +38,12 @@ identify initialization/hash code; they do not evaluate its semantics. | `vllm.config.gdn_state.GdnStateConfig` | kernel_config.gdn.state | resolve: [vllm/config/gdn_state.py](../../../vllm/config/gdn_state.py#L39); graph_options: [vllm/config/gdn_state.py](../../../vllm/config/gdn_state.py#L52) | [vllm/config/gdn_state.py](../../../vllm/config/gdn_state.py#L23) | | `vllm.config.gdn_state.GdnStateTraceConfig` | observability_config.gdn_state | resolve: [vllm/config/gdn_state.py](../../../vllm/config/gdn_state.py#L99); compile_ignored_aliases: [vllm/config/gdn_state.py](../../../vllm/config/gdn_state.py#L142) | [vllm/config/gdn_state.py](../../../vllm/config/gdn_state.py#L59) | | `vllm.config.kernel.IrOpPriorityConfig` | kernel_config.ir_op_priority | compute_hash: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L84) | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L67) | -| `vllm.config.kernel.KernelConfig` | kernel_config | compute_hash: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L1058) | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L619) | +| `vllm.config.kernel.KernelConfig` | kernel_config | compute_hash: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L1062) | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L623) | | `vllm.config.kernel.Sm70AwqConfig` | kernel_config.sm70_awq | resolve: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L338); capture_inputs: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L216) | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L317) | | `vllm.config.kernel.Sm70Fp8Config` | kernel_config.sm70_fp8 | resolve: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L433); capture_inputs: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L216) | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L369) | | `vllm.config.kernel.Sm70GgufConfig` | kernel_config.sm70_gguf | | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L498) | | `vllm.config.kernel.Sm70NvFp4Config` | kernel_config.sm70_nvfp4 | resolve: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L277); capture_inputs: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L216) | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L236) | -| `vllm.config.kernel.Sm70RingConfig` | kernel_config.sm70_ring | | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L608) | +| `vllm.config.kernel.Sm70RingConfig` | kernel_config.sm70_ring | | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L612) | | `vllm.config.kv_events.KVEventsConfig` | kv_events_config | __post_init__: [vllm/config/kv_events.py](../../../vllm/config/kv_events.py#L50) | [vllm/config/kv_events.py](../../../vllm/config/kv_events.py#L11) | | `vllm.config.kv_transfer.KVTransferConfig` | kv_transfer_config | compute_hash: [vllm/config/kv_transfer.py](../../../vllm/config/kv_transfer.py#L75); __post_init__: [vllm/config/kv_transfer.py](../../../vllm/config/kv_transfer.py#L93) | [vllm/config/kv_transfer.py](../../../vllm/config/kv_transfer.py#L23) | | `vllm.config.legacy_inputs.LegacyInputs` | kernel_config.sm70_awq.legacy, kernel_config.sm70_fp8.legacy, kernel_config.sm70_moe.awq.legacy, kernel_config.sm70_moe.fp8.legacy, kernel_config.sm70_moe.mxfp4.legacy, kernel_config.sm70_moe.nvfp4.legacy, kernel_config.sm70_nvfp4.legacy, scheduler_config.sm70_inputs | value: [vllm/config/legacy_inputs.py](../../../vllm/config/legacy_inputs.py#L39) | [vllm/config/legacy_inputs.py](../../../vllm/config/legacy_inputs.py#L15) | diff --git a/tests/models/gguf/test_expert_prefetch_budget.py b/tests/models/gguf/test_expert_prefetch_budget.py new file mode 100644 index 000000000..c0cf45e0f --- /dev/null +++ b/tests/models/gguf/test_expert_prefetch_budget.py @@ -0,0 +1,50 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project +from types import SimpleNamespace + +import pytest + +from vllm.config import KernelConfig +from vllm.model_executor.layers.quantization import gguf_expert_storage as storage + + +@pytest.mark.parametrize("depth", [1, 4]) +def test_bank_budget_preserves_loading_order_and_bounded_queue(monkeypatch, depth): + jobs = [(str(i), i, "w1", (0, 1), None) for i in range(9)] + monkeypatch.setattr(storage, "_local_slice", lambda tensor, *args: tensor) + loader = storage._Prefetcher(jobs, depth=depth) + try: + assert len(loader.futures) <= depth + assert loader.take("not-a-bank") is None + for i in range(9): + assert loader.take(str(i)) == i + assert len(loader.futures) <= depth + assert not loader.futures + finally: + loader.pool.shutdown(wait=True) + + +@pytest.mark.parametrize("depth", [1, 4]) +def test_selected_bank_budget_reaches_loader_owner(monkeypatch, depth): + calls = [] + cfg = SimpleNamespace(expert_load_prefetch_banks=depth) + monkeypatch.setattr(storage, "_expert_cache_config", lambda: cfg) + monkeypatch.setattr(storage, "_slice_spec", lambda *args: ((0, 1), None)) + monkeypatch.setattr( + storage, "_Prefetcher", lambda jobs, depth: calls.append((jobs, depth)) + ) + monkeypatch.setattr(storage, "_PREFETCH", None) + storage.start_prefetch({"bank": (None, None, "w1")}, {"bank": object()}) + assert calls[0][1] == depth + + +def test_startup_bank_policy_keeps_computation_hash_and_rejects_bad_budget(): + a = KernelConfig() + b = KernelConfig() + assert a.sm70_gguf.expert_load_prefetch_banks == 4 + a.sm70_gguf.active = b.sm70_gguf.active = True + b.sm70_gguf.expert_load_prefetch_banks = 1 + assert a.compute_hash() == b.compute_hash() + for value in (0, 5): + with pytest.raises(ValueError): + KernelConfig(sm70_gguf={"expert_load_prefetch_banks": value}) diff --git a/vllm/config/kernel.py b/vllm/config/kernel.py index 8a6e2ed20..ba2ff69e9 100644 --- a/vllm/config/kernel.py +++ b/vllm/config/kernel.py @@ -549,6 +549,10 @@ class Sm70GgufConfig: """Tokens per TurboMind MoE sub-batch within a prefill chunk. The layer's banks are transcoded once per chunk; sub-batches bound the routed activations (about 50 KiB per token at top-10).""" + expert_load_prefetch_banks: int = Field(default=4, ge=1, le=4) + """Banks queued in pinned host memory during original expert loading. + One reduces startup transient reservation; runtime expert placement, + weights and computation remain unchanged.""" dense_storage: Literal["canonical", "original"] = "canonical" """Keep calibrated canonical projections or dispatch original packed rows.""" @@ -1142,6 +1146,8 @@ class KernelConfig: entries.pop("active", None) entries.pop("sources", None) entries.pop("legacy", None) + if family == "gguf": + entries.pop("expert_load_prefetch_banks", None) for field, alias in getattr( getattr(self, name), "loader_aliases", {} ).items(): diff --git a/vllm/model_executor/layers/quantization/gguf_expert_storage.py b/vllm/model_executor/layers/quantization/gguf_expert_storage.py index 75a0bf8cb..fffaba923 100644 --- a/vllm/model_executor/layers/quantization/gguf_expert_storage.py +++ b/vllm/model_executor/layers/quantization/gguf_expert_storage.py @@ -268,7 +268,11 @@ def start_prefetch(targets, tensors) -> None: tensor = tensors[raw] rows, span = _slice_spec(layer, method, shard, tensor) jobs.append((raw, tensor, shard, rows, span)) - _PREFETCH = _Prefetcher(jobs) if jobs else None + config = _expert_cache_config() + depth = config.expert_load_prefetch_banks if config is not None else 4 + if jobs: + logger.info("GGUF expert loading queues at most %d pinned banks.", depth) + _PREFETCH = _Prefetcher(jobs, depth=depth) if jobs else None def load_expert_bank(layer, method, shard, tensor, raw=None) -> None: