[Loader] Bound queued pinned GGUF expert banks at startup

Expose a typed queue budget without changing runtime placement, weights or computation hashes. Preserve the default and qualify bounded ordered loading; record host-pressure failures and pending model acceptance.

Assisted-by: Codex
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>

benchmarks/flashnext_prefill/DECODE_PARITY_20261015.md

@@ -0,0 +1,85 @@
+# Flash-Next TP4 verifier and memory qualification
+
+The destination is four V100-SXM2-16GB GPUs with all-pair NV2 links, CUDA
+12.8, driver 570.211.01 and Torch 2.10.0+cu128. The model is
+SC117/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-GGUF IQ3_S with stock MTP4.
+The total request capacity is 262144 tokens. Sampling is temperature 0.6,
+top-p 0.95 and top-k 20; measured requests use seed 20261014 and at most
+192 output tokens. C1, FULL M5, greedy draft sampling, prefix caching and
+tool calling stay enabled. Expert placement remains 80% device / 20% host.
+Authoritative QSA KV is host E4M3; compressed indexer history is on GPU.
+
+## Acceptance metric
+
+Verifier cost starts at the earliest target graph GPU start and ends at the
+latest target state-commit completion across all four ranks. It includes
+the target graph and following state work. Acceptance length and emitted
+tokens/s are separate metrics. Kernel service sums and node-trace times
+are attribution evidence, not accepted wall latency.
+
+The ordinary-worker control has all per-step CUDA fences disabled. Low
+overhead graph-envelope traces preserve every complete committed round.
+Pipeline-end work is separately inventoried from graph and commit counts;
+missing target starts are retained as conservative bounds, never omitted.
+
+| Actual input tokens | Committed rounds | Mean ms | p50 ms | p90 ms | p99 ms |
+| ---: | ---: | ---: | ---: | ---: | ---: |
+| 32774 | 520 | 15.894 | 14.647 | 18.675 | 31.449 |
+| 65604 | 576 | 16.259 | 15.119 | 18.562 | 31.459 |
+| 131124 | 680 | 16.366 | 15.593 | 17.068 | 30.765 |
+| 261044 | 704 | 17.545 | 17.028 | 18.283 | 28.707 |
+
+Control source is 760dcbcb8b. Every one of 2480 windows is exact. Mean cost
+grows 10.39% across this sweep, so an all-context 16 ms claim is not yet
+supported. The following changes still need matched complete-model curves
+and a fresh ordinary-worker production restart before API promotion.
+
+## Bounded resources and exact selector candidate
+
+- HC prefill residuals belong to their TP row owners. Destination operator,
+  recurrent reference and real model-adapter padding/graph checks pass.
+  At 8192 rows the stage's temporary peak falls from 502.28 to 170.02 MiB.
+  The optional peer-normalization bank stays disabled.
+- A single prefill staging slot reuses the existing 128 MiB E4M3 budget for
+  262144 tokens. Its lifetime ends after the owning attention use; side
+  prefetch completion is preserved. FP16 draft capacity remains 131072.
+- `qsa_host_kv_trim_warmup_cache=true` returns unused ordinary allocator
+  pages once after all model, graph and auxiliary warmups. Actual host-QSA
+  consumers are required; active tensors and graph pools remain owned.
+  No inference-time garbage collection is added.
+- `sm70_sparse.qsa_decode_partition_topk=true` reuses the normal exact
+  FP32 selector over 8192-column partitions, then selects their canonical
+  candidate union. Admission is M5/M10, one request, SM70 and score capacity
+  32768..69632 compressed columns. Other inputs retain the qualified
+  selector. Exact FP32 values, lower-index tie ordering, output guards and
+  changed-input graph replay pass the destination operator suite.
+
+All candidate flags default off in the library. No private kernel DSO,
+preload or shared-venv modification is required. Native sources and the
+qualified native artifact are unchanged by these changes.
+
+## Startup host pressure
+
+The first partition/trim candidate launch was killed by systemd-oomd during
+expert loading: parent user-service memory PSI exceeded 50% for 20 seconds.
+The recorded first unit peak was 40.2 GiB plus 2.4 GiB swap. Another attempt
+also exceeded the pressure threshold. These were host-pressure failures,
+not GPU allocation failures. The separate 5 GiB MemAvailable guard did not
+fire; its threshold and the system OOM policy remain unchanged.
+
+`sm70_gguf.expert_load_prefetch_banks=1` bounds queued pinned expert banks
+during initialization. The validated range is 1..4 and the library default
+remains 4. It changes neither weights nor placement nor runtime computation,
+and therefore does not perturb the computation hash. The loader may also
+hold the bank currently being consumed and a whole-bank unpacking scratch;
+this setting is a queue bound, not a total pinned-memory quota. Tests cover
+queue bounds, ordered values, configuration delivery and hash invariance.
+Fresh startup and complete-model memory qualification remain required.
+
+## Rejected screens
+
+Warp-aggregated histograms regress random inputs. Warp-local emission
+improves too little to justify a native change. An exact integer-key final
+merge is slower than the existing FP32 selector. These standalone research
+screens are not production providers. The partitioned FP32 candidate is
+the only promoted selector experiment; model promotion remains pending.

docs/design/architecture/runtime_reference.md

@@ -38,12 +38,12 @@ identify initialization/hash code; they do not evaluate its semantics.
 | `vllm.config.gdn_state.GdnStateConfig` | kernel_config.gdn.state | resolve: [vllm/config/gdn_state.py](../../../vllm/config/gdn_state.py#L39); graph_options: [vllm/config/gdn_state.py](../../../vllm/config/gdn_state.py#L52) | [vllm/config/gdn_state.py](../../../vllm/config/gdn_state.py#L23) |
 | `vllm.config.gdn_state.GdnStateTraceConfig` | observability_config.gdn_state | resolve: [vllm/config/gdn_state.py](../../../vllm/config/gdn_state.py#L99); compile_ignored_aliases: [vllm/config/gdn_state.py](../../../vllm/config/gdn_state.py#L142) | [vllm/config/gdn_state.py](../../../vllm/config/gdn_state.py#L59) |
 | `vllm.config.kernel.IrOpPriorityConfig` | kernel_config.ir_op_priority | compute_hash: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L84) | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L67) |
-| `vllm.config.kernel.KernelConfig` | kernel_config | compute_hash: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L1058) | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L619) |
+| `vllm.config.kernel.KernelConfig` | kernel_config | compute_hash: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L1062) | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L623) |
 | `vllm.config.kernel.Sm70AwqConfig` | kernel_config.sm70_awq | resolve: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L338); capture_inputs: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L216) | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L317) |
 | `vllm.config.kernel.Sm70Fp8Config` | kernel_config.sm70_fp8 | resolve: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L433); capture_inputs: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L216) | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L369) |
 | `vllm.config.kernel.Sm70GgufConfig` | kernel_config.sm70_gguf |  | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L498) |
 | `vllm.config.kernel.Sm70NvFp4Config` | kernel_config.sm70_nvfp4 | resolve: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L277); capture_inputs: [vllm/config/kernel.py](../../../vllm/config/kernel.py#L216) | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L236) |
-| `vllm.config.kernel.Sm70RingConfig` | kernel_config.sm70_ring |  | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L608) |
+| `vllm.config.kernel.Sm70RingConfig` | kernel_config.sm70_ring |  | [vllm/config/kernel.py](../../../vllm/config/kernel.py#L612) |
 | `vllm.config.kv_events.KVEventsConfig` | kv_events_config | __post_init__: [vllm/config/kv_events.py](../../../vllm/config/kv_events.py#L50) | [vllm/config/kv_events.py](../../../vllm/config/kv_events.py#L11) |
 | `vllm.config.kv_transfer.KVTransferConfig` | kv_transfer_config | compute_hash: [vllm/config/kv_transfer.py](../../../vllm/config/kv_transfer.py#L75); __post_init__: [vllm/config/kv_transfer.py](../../../vllm/config/kv_transfer.py#L93) | [vllm/config/kv_transfer.py](../../../vllm/config/kv_transfer.py#L23) |
 | `vllm.config.legacy_inputs.LegacyInputs` | kernel_config.sm70_awq.legacy, kernel_config.sm70_fp8.legacy, kernel_config.sm70_moe.awq.legacy, kernel_config.sm70_moe.fp8.legacy, kernel_config.sm70_moe.mxfp4.legacy, kernel_config.sm70_moe.nvfp4.legacy, kernel_config.sm70_nvfp4.legacy, scheduler_config.sm70_inputs | value: [vllm/config/legacy_inputs.py](../../../vllm/config/legacy_inputs.py#L39) | [vllm/config/legacy_inputs.py](../../../vllm/config/legacy_inputs.py#L15) |

tests/models/gguf/test_expert_prefetch_budget.py

@@ -0,0 +1,50 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+from types import SimpleNamespace
+
+import pytest
+
+from vllm.config import KernelConfig
+from vllm.model_executor.layers.quantization import gguf_expert_storage as storage
+
+
+@pytest.mark.parametrize("depth", [1, 4])
+def test_bank_budget_preserves_loading_order_and_bounded_queue(monkeypatch, depth):
+    jobs = [(str(i), i, "w1", (0, 1), None) for i in range(9)]
+    monkeypatch.setattr(storage, "_local_slice", lambda tensor, *args: tensor)
+    loader = storage._Prefetcher(jobs, depth=depth)
+    try:
+        assert len(loader.futures) <= depth
+        assert loader.take("not-a-bank") is None
+        for i in range(9):
+            assert loader.take(str(i)) == i
+            assert len(loader.futures) <= depth
+        assert not loader.futures
+    finally:
+        loader.pool.shutdown(wait=True)
+
+
+@pytest.mark.parametrize("depth", [1, 4])
+def test_selected_bank_budget_reaches_loader_owner(monkeypatch, depth):
+    calls = []
+    cfg = SimpleNamespace(expert_load_prefetch_banks=depth)
+    monkeypatch.setattr(storage, "_expert_cache_config", lambda: cfg)
+    monkeypatch.setattr(storage, "_slice_spec", lambda *args: ((0, 1), None))
+    monkeypatch.setattr(
+        storage, "_Prefetcher", lambda jobs, depth: calls.append((jobs, depth))
+    )
+    monkeypatch.setattr(storage, "_PREFETCH", None)
+    storage.start_prefetch({"bank": (None, None, "w1")}, {"bank": object()})
+    assert calls[0][1] == depth
+
+
+def test_startup_bank_policy_keeps_computation_hash_and_rejects_bad_budget():
+    a = KernelConfig()
+    b = KernelConfig()
+    assert a.sm70_gguf.expert_load_prefetch_banks == 4
+    a.sm70_gguf.active = b.sm70_gguf.active = True
+    b.sm70_gguf.expert_load_prefetch_banks = 1
+    assert a.compute_hash() == b.compute_hash()
+    for value in (0, 5):
+        with pytest.raises(ValueError):
+            KernelConfig(sm70_gguf={"expert_load_prefetch_banks": value})

vllm/config/kernel.py

@@ -549,6 +549,10 @@ class Sm70GgufConfig:
     """Tokens per TurboMind MoE sub-batch within a prefill chunk. The layer's
     banks are transcoded once per chunk; sub-batches bound the routed
     activations (about 50 KiB per token at top-10)."""
+    expert_load_prefetch_banks: int = Field(default=4, ge=1, le=4)
+    """Banks queued in pinned host memory during original expert loading.
+    One reduces startup transient reservation; runtime expert placement,
+    weights and computation remain unchanged."""
 
     dense_storage: Literal["canonical", "original"] = "canonical"
     """Keep calibrated canonical projections or dispatch original packed rows."""
@@ -1142,6 +1146,8 @@ class KernelConfig:
                 entries.pop("active", None)
                 entries.pop("sources", None)
                 entries.pop("legacy", None)
+                if family == "gguf":
+                    entries.pop("expert_load_prefetch_banks", None)
                 for field, alias in getattr(
                     getattr(self, name), "loader_aliases", {}
                 ).items():

vllm/model_executor/layers/quantization/gguf_expert_storage.py

@@ -268,7 +268,11 @@ def start_prefetch(targets, tensors) -> None:
         tensor = tensors[raw]
         rows, span = _slice_spec(layer, method, shard, tensor)
         jobs.append((raw, tensor, shard, rows, span))
-    _PREFETCH = _Prefetcher(jobs) if jobs else None
+    config = _expert_cache_config()
+    depth = config.expert_load_prefetch_banks if config is not None else 4
+    if jobs:
+        logger.info("GGUF expert loading queues at most %d pinned banks.", depth)
+    _PREFETCH = _Prefetcher(jobs, depth=depth) if jobs else None
 
 
 def load_expert_bank(layer, method, shard, tensor, raw=None) -> None: