1Cat-vLLM/docs/design/gguf_v100_control.md

GGUF on V100: implementation and acceptance control

Scope and integration

Target: extensible standalone GGUF loading on four SM70 V100s, then recover prefill and C1/C4/C8/C16 throughput against matching native quantization paths. Integration is onecat/main; never change versioned source snapshots for this campaign. Kernel admission and startup reporting must use KernelConfig and vllm/sm70_profiles/acceleration.py. No new environment switches.

A model generation smoke is not a logits, quality, or throughput acceptance. Every performance comparison must freeze checkpoint and quantization, GPU UUIDs/topology, TP/EP, CUDA/Torch/build SHA, context/input/output length, sampling, MTP, attention backend, graphs, and KV format. Separate prefill, TTFT, steady decode, and emitted-token throughput. The existing 35B AWQ/FP8 migration gates remain mandatory and are not replaced by a GGUF smoke.

Ordered review scopes

1. Standalone metadata/tokenizer and qwen35 dense adapter. Preserve GGUF vocabulary/token types/chat template; explicit HF config/tokenizer wins. Restore GDN negative exp(A_log), convolution dimension, norm offsets, tiled value heads, and correctly shard the tiled out projection. Reject unknown tensors, missing shards, duplicate tensors, and invalid split counts. Mixed FP16/quantized projection shards retain distinct storage and dispatch.
2. qwen35moe adapter: stacked expert types, mixed gate/up expert quantization, MTP nextn adapter; verify full logits and greedy output against llama.cpp.
3. qwen4exp adapter: HC, QSA/indexer, and PLE hash constants/table. Use llama.cpp conversion/qwen4exp.py as format oracle. Preserve 64-bit hash constants; PLE layer IDs differ between GGUF (zero-based) and HF (one-based).
4. llama-family, glm4moe, deepseek2 (including MLA kv_b split), gpt-oss and minimax-m2 adapters. Handle architecture-specific RoPE permutations.
5. Modern native kernel closure, with upstream licenses and pinned sources: Q1_0, Q2_0, MXFP4, NVFP4, TQ1/TQ2; numerical and route capability gates.
6. Q4_0/Q4_1/Q4_K/Q8_0 load-time repack and rounding report; TurboMind/QPN decode, batched GEMM, expert grouping and tensor-core prefill.
7. Remaining types: MMVQ decode, dequant + FP16 GEMM prefill; measured Volta fused dequant HMMA for intermediate M; grouped MoE token reuse.

TP/EP decision

Quantization block alignment is an operator/storage constraint, independent of model names. K-quant has block K=256: local intermediate K=4352 is aligned; Flash-Next expert K=640 divided over TP4 gives K=160 and is not aligned. Do not slice packed bytes at that boundary. Prefer EP with complete experts when the installed GGUF expert method supports EP; otherwise convert the unaligned expert tensor at load time to a format with an admitted K boundary. EP is not considered implemented by setting enable_expert_parallel alone. Coordinate the final decision with the Flash-Next decode work; its observed GPU topology differs from the historical full-NVLink baseline.

References and provenance

• vllm-project/vllm-gguf-plugin main e2b8ad532b8b5ea175100202c30430c1d2b5e6a8 (Apache-2.0): adapter structure and GGUFHeadTilingLayout. Its current config parser still reads HF config; copying that parser does not implement standalone GGUF metadata parsing.
• Plugin PR 141, open at inspection, head aa09d6522f29325d64d999d7d7c794f79836de07: kernel integration pins llama.cpp 002a12ad25503a93501b2e188c360029830a241a. Published throughput is external evidence, not acceptance data from this campaign.
• ggml-org/llama.cpp reference bed0a856606ee4a24a164066f73d2379447033f5 (MIT). PR 27742 is merged; its original head is eaf93765572e794b8e3754fe45adbe12d381e997.

Retain the original SPDX headers in adapted plugin layout code. No llama.cpp CUDA code is included by the first review scope.

Current evidence / rejected paths

Initial integration base 3a147164833e58df2f80203cd169cbd71cea41e1. Owned branch codex/v100-gguf-loader-20261002-230454.

• Synthetic actual GGUF files: 17 native metadata/split-validation tests passed.
• Unsloth Qwen3.5-0.8B Q4_K_M revision 6ab461498e2023f6e3c1baea90a8f0fe38ab64d0: native metadata recovered 24 layers, hidden=1024, attention head_dim=256, SSM 16 key/16 value heads, inner=2048, and interleaved MRoPE [11,11,10], without HF config.
• Six tokenization/decode samples matched HF token IDs, including Chinese, combining accents, multilingual scripts, digits, emoji and tool tags. GGUF chat template is authoritative; it differs from current HF template.
• Rejected tokenizer shortcut: treating qwen35 pretokenization as qwen2 mismatches combining marks. qwen35 requires NFC and the mark-aware regex.
• First TP4 initialization attempt stopped before model loading because the new environment lacked torchvision. This is not GPU route/quality evidence.
• CPU llama.cpp oracle built from the pinned reference; raw first-token logits and greedy token sequences are retained with the task artifacts.

No 27B/35B performance acceptance or full GGUF support is claimed yet.

Runtime and artifact hygiene

Models and reference builds are under /mnt/nvme3/ymzx-data; the volume is mounted by its existing fstab UUID. GPU work must hold /tmp/gpu0-3.lock for its entire process lifetime and inspect existing GPU processes first. Use owned compiler caches. Do not use private kernel DSOs/preloads as PR evidence.

Large raw output is retained outside Git, in the owned worktree's .artifacts.

Repack admission finding: Q4_K affine coefficients

27B UD-Q4_K_M: 212,992 sampled group coefficients, 6,815,744 weights, stratified over all Q4_K tensors (up to 256 superblocks per tensor). Keeping codes identical and rounding scale and additive min/bias to FP16 gives maximum absolute weight error 9.9182e-5, RMSE 9.5210e-6 and relative L2 6.7039e-4. 68.2262% of reconstructed FP16 weights differ from direct GGUF dequantization. This is a CPU coefficient-rounding study, not GPU/model performance or a quality gate. Q4_K repack is therefore approximate; require full logits and quality evidence before making it the default. Avoid representing additive bias as a rounded zero-point ratio: the existing prepare function computes -zero*scale, which adds another rounding step and fails for zero scale. A direct affine scale+bias preparation interface is needed for Q4_1/Q4_K. +### Primary acceptance checkpoint: Flash-Next GSQ-RCO IQ3_XXS

The requested primary checkpoint is ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF, IQ3_XXS (not IQ3_S). GSQ-RCO here assigns precision per tensor; it does not introduce an additional weight rotation.

The first shard is 47,039,860,096 bytes, SHA256 219ea929900dfa9ef091f3aa473fdba6874b65fcb36526d7d851ac9e95856d15. The second is 28,800,138,432 bytes, SHA256 316b46f3a2dbd68c900f43136ab9449f9dcc3725dfd8c794847c204bc161e113. ModelScope file revisions are respectively e3a513b7830d32c0d87b07d54ca650cbc422cc26 and 26dfa70d5c61d623f472c234462b8205ef3448e2. Downloads use these immutable revisions, resume into .part files, and require size/SHA256 validation before rename.

The real first-shard directory has 1,223 tensors and GGML type counts: F32=292, F16=1, BF16=484, Q4_K=45, Q5_K=15, Q6_K=71, IQ2_XXS=18, IQ2_XS=20, IQ2_S=20, IQ3_XXS=12, IQ3_S=82, IQ4_NL=58, IQ4_XS=67, Q2_0=38. The remaining tensor is the IQ4_NL PLE table (320,001,536 rows of dimension 160), requiring packed sparse lookup. GPU vocabulary partitions or host mmap must retain the quantized rows; never materialize the complete table in FP16.

Actual metadata: qwen4exp, 48 layers, H=2560, 24 Q / 2 KV attention heads, head_dim=256, 512 experts, top-k=10, expert K=640, 16 GDN key / 48 value heads, 128-dimensional GDN heads, HC count=4/rank=320. PLE keys include exact int64 multipliers, offsets and prime vocabulary sizes. GGUF PLE layer index 1 maps to HF layer ID 2. Q2_0 uses K blocks of 64, so expert down local K=160 still cannot be byte-sliced under TP4; IQ4_NL K blocks of 32 are aligned. Flash-Next retains TP4. Canonical group32 reblocking supports Q2_0 K=160 shards in the TurboMind operator layer; the canonical MoE model connection remains pending. The adapter fallback currently converts each 64-value Q2_0 block into two Q4_1 blocks using the same integer values, scale and additive offset. This preserves decoded values but adds 2.014 GiB per rank for the 30 affected expert down projections. Replace this temporary storage with canonical u2 when integrating grouped TurboMind experts.

Three CPU tests cover decoded equality, all four K=160 boundaries and independent projection storage; three GPU FFN checks compare summed TP partials with the full expert reference. All 1,224 checkpoint tensors have adapter names. Full-model generation and quality are still pending.

Loading validation and open gates

17 metadata/tokenizer/split tests and 13 dense adapter/TP/mixed-storage tests pass on CPU. Earlier failures localized missing runtime dependencies, unaligned K-quant FFN storage, missing hybrid state traits, and KV-head replication when TP exceeds KV heads. Those have source fixes; alignment and replication have focused CPU coverage. A later run completed engine initialization and GPU prefill warmup, then rejected the text-only class's missing MRoPE position interface. That interface is now implemented and tested on CPU. Inverse A_log loading preserves its FP32 parameter contract even with FP16 model dtype.

The corrected TP4 tiny Qwen3.5-0.8B Q4_K_M run completed GPU inference on 2026-10-03 with FP16, eager execution, max length 2048, no MTP, FP16 KV, Flash-V100 attention and FlashQLA GDN. Three raw prompts generated 64 greedy tokens each. The English and arithmetic sequences matched llama.cpp for all 64 tokens; the Chinese sequence matched its first 13 tokens and then diverged. First-token full-vocabulary logits have RMSE 0.1243/0.1408/0.1656, maximum absolute error 0.6073/0.8044/0.8116 and relative L2 0.02949/0.04647/0.06349. All three top-1 tokens match. The reference is CPU llama.cpp at bed0a856606ee4a24a164066f73d2379447033f5, using identical token IDs. These numeric differences and the Chinese divergence remain an open quality gate; a completed inference smoke does not establish adapter correctness. The following chat harness failed because Transformers 5 returns a dict from apply_chat_template by default; it now requests return_dict=False.

These are implementation-localization results, not primary-model quality or speed acceptance. Hold the shared GPU flock and do not terminate other tasks. No accepted whole-model throughput is recorded.

Chinese greedy divergence localization

A controlled 20-token run uses the same tiny checkpoint, prompt token IDs, TP4, FP16, eager execution and no MTP. Single-request legacy inference chooses token 96617 at step 13 and matches all 20 CPU llama.cpp tokens. The original three-request batch is reproducible: it chooses token 103911 at step 13. The competing-logit margin (96617 minus 103911) changes from +0.09375 in the single request to -0.0625 in the batch.

The CPU reference also matches all 20 tokens after exact F32 reconstruction of every GGUF weight, so CPU activation quantization alone does not explain the divergence. Replacing the batch's quantized linear calls with the packaged FP16 dequantization plus GEMM reference restores all 20 reference tokens; its step-13 margin is +0.203125. Worker dispatch logs confirm this replacement for Q4_K, Q8_0 and Q6_K. This localizes the correction to the linear numerical path; a finer separation of activation quantization, weight rounding and accumulation remains pending. It supports the FP16-activation TurboMind direction without changing loader mappings.

The first precision-hook attempt patched the Python implementation after the custom operator had already been registered, so it did not replace the actual caller. Its outputs are excluded. A separate launch used system CUDA 12.0 for Tilelang and failed before generation; subsequent runs explicitly use CUDA 12.8. Neither failure is quality evidence. The wider quality suite and primary-model comparisons are still pending.

Loader synchronization and installed artifact

After integration at 1d1d1c9d80a762ec6d3c7fd46fb85954f730967f, the Qwen3.5 adapter's GDN layout and mixed-projection storage compose with the packaged native fallback policy. The 19 metadata and 13 adapter contracts pass both from source and from a fresh ordinary wheel installation. All 210 installed dependencies pass compatibility checking. This Python-only loader artifact uses the complete normal precompiled operator wheel for that base; no private extension override is required. Core and native extension hashes match their packaged and installed copies, without RPATH/RUNPATH.

Loader source: afad28789341c204ccf3046621124f8ec74b7e39. Wheel SHA256: 177bcd84f73eb63e46cbba602f70d515df472551e7630758599f0b60d9665f61. Core SHA256: 20ac310a9a80ac4075719cfd1c75a9e70f9649eb714ff21b7be12e584fc2a2cf. Native reference SHA256: 74ed944b8abb0f8679757a4e1bf0acef453f4a9803002f2c47b35b47f39d163f.

Installed inference uses the same tiny checkpoint and CPU oracle, V100 32GB GPU0–3, TP4, Python 3.12.3, CUDA 12.8, Torch 2.10.0+cu128, FP16, maxlen 2048, maxbatch 256, maxseqs 4, 0.1 GPU memory utilization, eager execution, no MTP and FP16 KV. English/arithmetic still match 64/64 greedy tokens each. Chinese matches 23 tokens before diverging in this run; this is not a resolved quality result. A preceding one-token full-logit generation and new packaged fallback implementation make this a distinct run from the earlier 13-token localization baseline. Its first-logit RMSE/relative-L2 values remain 0.1243/0.02949, 0.1408/0.04647 and 0.1656/0.06349, with matching top-1.

Embedded chat-template generation with thinking disabled returns Paris, 4 and 你好 for the three fixed short prompts. These checks establish installed loader/tokenizer operation; broader distribution/quality checks and primary-model speed remain pending. No model-level TurboMind speed is claimed.

Current-main loader check

The loader was synchronized with main 48a66d2838 without changing other SM70 routes. All 19 metadata and 13 adapter tests pass from source and a fresh ordinary wheel installation. The installed tiny-model run uses the same TP4, FP16/eager/no-MTP contract above: English and arithmetic match 64/64 greedy tokens; Chinese matches 23 tokens before divergence. First-logit RMSE is 0.128859/0.148466/0.165342, relative L2 is 0.030586/0.049015/0.063375, and all first-logit top-1 values match. Embedded chat results remain Paris, 4, and 你好. Broader quality work remains pending.

The normal packaged _C matches the main operator artifact, 4910c47ab1aaed253001d5950bf44dd40a350b2b087202a8ea2b13f2c5457782. Wheel 1cat_vllm-1.5.2.dev406+g7285a28e6.precompiled-cp312-cp312-linux_x86_64.whl has SHA256 17a06f4c462608c07006aaf1fb71d902dc1362380a7df0b7d7339d310873345e; all 210 installed dependencies are compatible. The subsequent main sync adds the independently validated IQ3_XXS grouped-vector codebook layout; that operator is not used by the tiny dense model in this check.

Canonical dense model integration and coalescing

The installed dense GGUF integration prepares affine, LUT4 and lattice projections through the shared TurboMind lifecycle. Adjacent compatible shards coalesce before packing; mixed types retain separate ordered dispatch. Real projection sweeps select FP16 caching for merged Q8 alpha/beta rows and canonical DQ+FP32 cuBLAS for calibrated large affine shapes. FP16 activations and FP32 accumulation are preserved. Twelve targeted GPU checks pass from source and from a fresh normal wheel; the core fingerprint is unchanged.

Qwen3.8-27B UD-Q4_K_M, TP4 on V100 x4, FP16 activation/KV, no MTP, FULL_AND_PIECEWISE graphs, I1024/O128 and two complete-cohort repeats: C1/C4/C8/C16 aggregate pure decode is 58.88/216.73/396.87/671.47 tok/s. These improve the preceding GGUF implementation by 21–24%, while still trailing the matched NVFP4 control by 12–19%. Prefill at 8K/32K is 2.423/10.263 s, 3.4%/2.0% faster than that control. All four natural greedy sequences and EOS remain identical to the preceding GGUF run and its llama.cpp reference.

Full operator grids, graph attribution and model workload details are in gguf_turbomind_model.md. The fixed 36-case quality comparison completes with 34/36 for both GGUF and NVFP4: all code, arithmetic and Chinese cases pass, as do the 8K/32K needles. Both routes fail the 128K/258048-token needles with repeated !. An eager NVFP4 audit localizes the first nonfinite output to layer 15 attention in the second 8192-token prefill chunk. The captured Q/K/V are finite; an isolated Q8192 operator reproduces 34 positive infinities. The sampled tail maximum misses the actual peak by about 17. Tail intermediate range recovery is tracked in #875, separately from GGUF weight preparation. The tiny-model 23-token Chinese difference is still open. Flash-Next remains TP4; its model integration, canonical MoE preparation and packed PLE offloading remain separate unfinished work.

Corrected long-context model check

PR #875 is merged. Both GGUF and NVFP4 now pass all four frozen needle cases, including the previous 128K/258048 failures, with unchanged prompt hashes, sampling and natural EOS. The 32 preceding short code/math/Chinese cases were already passing on unchanged attention routes. Updated GGUF decode is 59.49/207.22/394.61/674.61 tok/s; the lower C4 full-sweep mean includes long intervals despite a faster median. A matched four-repeat C4-only confirmation is 215.44 tok/s, within 0.6% of the preceding run. 8K/32K engine prefill is 2.4323/10.3453 seconds versus the updated NVFP4 2.4666/10.4106. All four greedy sequences and EOS remain identical. Full raw measurements and remaining gaps are recorded in gguf_turbomind_model.md; Flash-Next integration remains TP4.