macandClaude Opus 5.5
d836d87d2d
Register the new VLLM_* switches in vllm/envs.py and read them through envs; torch.accelerator instead of torch.cuda; regex instead of re; read the hybrid PLE flag through ple_policy; move the IQ4_NL packed PLE result helpers to kernels/ple/gguf_packed_result.py; type annotations for lazily started expert-cache state; ruff/clang-format/markdown fixes. The layering baseline accepts the opt-in SM70 sparse target sampling hooks in the EAGLE speculator and its greedy-draft rejection kernel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
94 KiB
94 KiB
Environment variable reference
Defaults describe an unset environment. The declaration is the public getter contract. Effective defaults include per-engine policy and the defaults of legacy direct consumers. Automatic conditions name the policy or consumer that makes the decision. An enabled option still needs its kernel's dtype, layout and native-operator checks. Runtime routing depends on those checks.
Categories distinguish configuration, tuning, experimental, debug and deprecated controls. Prefer engine configuration and VLLM_DISABLED_KERNELS for migrated acceleration paths.
This is the user reference. Internal tuning, diagnostics and compatibility aliases remain in the metadata inventory; they are not daily configuration options. Developers can render that inventory with --include-internal --output /tmp/environment-internals.md. See SM70 environment surface for compatibility and removal plans.
| Variable | Category | Description | Declared default | Effective default | Automatic conditions / legacy consumer defaults | Acceleration paths |
|---|---|---|---|---|---|---|
CMAKE_BUILD_TYPE |
configuration | CMake build type If not set, defaults to "Debug" or "RelWithDebInfo" Available options: "Debug", "Release", "RelWithDebInfo" | None | None | None | No SM70 path association |
CUDA_HOME |
configuration | path to cudatoolkit home directory, under which should be bin, include, and lib directories. | None | None | None | No SM70 path association |
CUDA_VISIBLE_DEVICES |
configuration | used to control the visible devices in the distributed setting | None | None | None | No SM70 path association |
K_SCALE_CONSTANT |
configuration | Divisor for dynamic key scale factor calculation for FP8 KV Cache | 200 | 200 | None | No SM70 path association |
LD_LIBRARY_PATH |
configuration | when VLLM_NCCL_SO_PATH is not set, vllm will try to find the nccl library file in the locations specified by LD_LIBRARY_PATH |
None | None | None | No SM70 path association |
LOCAL_RANK |
configuration | local rank of the process in the distributed setting, used to determine the GPU device id | 0 | 0 | None | No SM70 path association |
MAX_JOBS |
tuning | Maximum number of compilation jobs to run in parallel. By default this is the number of CPUs | None | None | None | No SM70 path association |
MOONCAKE_PREFERRED_SEGMENT |
configuration | Pin this rank to a specific owner segment ("host:port"). | None | None | None | No SM70 path association |
MOONCAKE_REQUESTER_LOCAL_HOSTNAME |
configuration | Override the hostname the rank registers as a Mooncake requester. | None | None | None | No SM70 path association |
NO_COLOR |
configuration | Standard unix flag for disabling ANSI color codes | False | False | None | No SM70 path association |
NVCC_THREADS |
tuning | Number of threads to use for nvcc By default this is 1. If set, MAX_JOBS will be reduced to avoid oversubscribing the CPU. |
None | None | None | No SM70 path association |
Q_SCALE_CONSTANT |
configuration | Divisor for dynamic query scale factor calculation for FP8 KV Cache | 200 | 200 | None | No SM70 path association |
S3_ACCESS_KEY_ID |
configuration | S3 access information, used for tensorizer to load model from S3 | None | None | None | No SM70 path association |
S3_ENDPOINT_URL |
configuration | Override the endpoint URL for the S3 model storage connection. | None | None | None | No SM70 path association |
S3_SECRET_ACCESS_KEY |
configuration | Secret key for the S3 model storage connection; values are never included in this reference. | None | None | None | No SM70 path association |
VERBOSE |
configuration | If set, vllm will print verbose logs during installation | False | False | None | No SM70 path association |
VLLM_ALLOW_CHUNKED_LOCAL_ATTN_WITH_HYBRID_KV_CACHE |
configuration | Allow chunked local attention with hybrid kv cache manager. Currently using the Hybrid KV cache manager with chunked local attention in the Llama4 models (the only models currently using chunked local attn) causes a latency regression. For this reason, we disable it by default. This flag is used to allow users to enable it if they want to (to save on kv-cache memory usage and enable longer contexts) TODO(lucas): Remove this flag once latency regression is resolved. | True | True | None | No SM70 path association |
VLLM_ALLOW_INSECURE_SERIALIZATION |
configuration | If set, allow insecure serialization using pickle. This is useful for environments where it is deemed safe to use the insecure method and it is needed for some reason. | False | False | None | No SM70 path association |
VLLM_ALLOW_LONG_MAX_MODEL_LEN |
tuning | If the env var VLLM_ALLOW_LONG_MAX_MODEL_LEN is set, it allows the user to specify a max sequence length greater than the max length derived from the model's config.json. To enable this, set VLLM_ALLOW_LONG_MAX_MODEL_LEN=1. | False | False | None | No SM70 path association |
VLLM_ALLOW_RUNTIME_LORA_UPDATING |
configuration | If set, allow loading or unloading lora adapters in runtime, | False | False | None | No SM70 path association |
VLLM_ALLREDUCE_USE_FLASHINFER |
configuration | Whether to use FlashInfer allreduce | False | False | None | No SM70 path association |
VLLM_ALLREDUCE_USE_SYMM_MEM |
configuration | Whether to use pytorch symmetric memory for allreduce | True | True | None | No SM70 path association |
VLLM_API_KEY |
configuration | API key for vLLM API server | None | None | None | No SM70 path association |
VLLM_ASSETS_CACHE |
configuration | Path to the cache for storing downloaded assets | os.path.join(VLLM_CACHE_ROOT, 'assets') | Computed when unset: os.path.expanduser(os.path.join(get_default_cache_root(), 'vllm', 'assets')) | None | No SM70 path association |
VLLM_ASSETS_CACHE_MODEL_CLEAN |
configuration | If the env var is set, we will clean model file in this path $VLLM_ASSETS_CACHE/model_streamer/$model_name | False | False | None | No SM70 path association |
VLLM_AUDIO_FETCH_TIMEOUT |
configuration | Timeout for fetching audio when serving multimodal models Default is 10 seconds | 10 | 10 | None | No SM70 path association |
VLLM_BALANCED_PREFILL_CHUNKS |
tuning | Split a prompt that needs n prefill chunks into n near-equal block- aligned chunks instead of full chunks and a short tail. Helps when every prefill step has a fixed cost (e.g. host-resident experts streamed over PCIe). | False | False | None | No SM70 path association |
VLLM_BATCH_INVARIANT |
configuration | Enable batch-invariant mode: deterministic results regardless of batch composition. Requires NVIDIA GPU with compute capability >= 9.0. | False | False | None | No SM70 path association |
VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER |
tuning | Allow use of FlashInfer FP8 block-scale GEMM for linear layers. This uses TensorRT-LLM kernels and requires SM90+ (Hopper). | True | True | None | No SM70 path association |
VLLM_CACHE_ROOT |
configuration | ================== Runtime Env Vars ================== Root directory for vLLM cache files Defaults to ~/.cache/vllm unless XDG_CACHE_HOME is set |
os.path.expanduser('~/.cache/vllm') | Computed when unset: os.path.expanduser(os.path.join(get_default_cache_root(), 'vllm')) | None | No SM70 path association |
VLLM_CI_USE_S3 |
configuration | Whether to use S3 path for model loading in CI via RunAI Streamer | False | False | None | No SM70 path association |
VLLM_COMPILE_CACHE_SAVE_FORMAT |
configuration | Format for saving torch.compile cache artifacts - "binary": saves as binary file Safe for multiple vllm serve processes accessing the same torch compile cache. - "unpacked": saves as directory structure (for inspection/debugging) NOT multiprocess safe - race conditions may occur with multiple processes. Allows viewing and setting breakpoints in Inductor's code output files. | 'binary' | 'binary' | None | No SM70 path association |
VLLM_COMPUTE_NANS_IN_LOGITS |
configuration | Enable checking whether the generated logits contain NaNs, indicating corrupted output. Useful for debugging low level bugs or bad hardware but it may add compute overhead. | False | False | None | No SM70 path association |
VLLM_CONFIGURE_LOGGING |
configuration | Logging configuration If set to 0, vllm will not configure logging If set to 1, vllm will configure logging using the default configuration or the configuration file specified by VLLM_LOGGING_CONFIG_PATH | True | True | None | No SM70 path association |
VLLM_CONFIG_ROOT |
configuration | Root directory for vLLM configuration files Defaults to ~/.config/vllm unless XDG_CONFIG_HOME is set Note that this not only affects how vllm finds its configuration files during runtime, but also affects how vllm installs its configuration files during **installation**. |
os.path.expanduser('~/.config/vllm') | Computed when unset: os.path.expanduser(os.path.join(get_default_config_root(), 'vllm')) | None | No SM70 path association |
VLLM_CPU_ATTN_SPLIT_KV |
tuning | (CPU backend only) whether to enable attention spilt KV. | True | True | None | No SM70 path association |
VLLM_CPU_CI_ENV |
configuration | Select the CPU CI platform emulation environment. | None | None | '0' at vllm/platforms/cpu.py:check_and_update_config | No SM70 path association |
VLLM_CPU_INT4_W4A8 |
configuration | (CPU backend only) whether to use SGLang INT4 W4A8 kernels for AWQ. | True | True | None | No SM70 path association |
VLLM_CPU_KVCACHE_SPACE |
configuration | (CPU backend only) CPU key-value cache space. default is None and will be set as 4 GB | 0 | Computed when unset: int('0') if 'VLLM_CPU_KVCACHE_SPACE' in os.environ else None | None | No SM70 path association |
VLLM_CPU_NUM_OF_RESERVED_CPU |
configuration | (CPU backend only) CPU cores not used by OMP threads . Those CPU cores will not be used by OMP threads of a rank. | None | Computed when unset: int('0') if 'VLLM_CPU_NUM_OF_RESERVED_CPU' in os.environ else None | None | No SM70 path association |
VLLM_CPU_OMP_THREADS_BIND |
tuning | (CPU backend only) CPU core ids bound by OpenMP threads, e.g., "0-31", "0,1,2", "0-31,33". CPU cores of different ranks are separated by '|'. | 'auto' | 'auto' | None | No SM70 path association |
VLLM_CPU_SGL_KERNEL |
configuration | (CPU backend only) whether to use SGL kernels, optimized for small batch. | False | False | None | No SM70 path association |
VLLM_CPU_SIM_MULTI_NUMA |
configuration | Simulate multiple NUMA nodes in CPU platform checks. | None | None | '0' at vllm/utils/ompmultiprocessing.py:__init__ | No SM70 path association |
VLLM_CUDAGRAPH_INPUT_ADDR_DEBUG |
debug | Debug CUDA graph replay input tensor address stability without enabling global DEBUG logging. | False | False | None | No SM70 path association |
VLLM_CUDART_SO_PATH |
configuration | In some system, find_loaded_library() may not work. So we allow users to specify the path through environment variable VLLM_CUDART_SO_PATH. | None | None | None | No SM70 path association |
VLLM_CUDA_COMPATIBILITY_PATH |
configuration | Path to the CUDA compatibility libraries when CUDA compatibility is enabled. | None | None | None | No SM70 path association |
VLLM_CUSTOM_ALLREDUCE_BLOCK_LIMIT |
configuration | vLLM: custom allreduce block limit. The consumer locations and unset defaults are listed below. | None | Computed when unset: int(os.environ['VLLM_CUSTOM_ALLREDUCE_BLOCK_LIMIT']) if 'VLLM_CUSTOM_ALLREDUCE_BLOCK_LIMIT' in os.environ else None | None | No SM70 path association |
VLLM_CUSTOM_SCOPES_FOR_PROFILING |
configuration | Add optional custom scopes for profiling, disable to avoid overheads | False | False | None | No SM70 path association |
VLLM_CUTLASS_SRC_DIR |
configuration | Override the CUTLASS source directory used by the local build. | None | None | None at vllm/model_executor/layers/sm70_sparse_attention.py:sparse_extension None at vllm/model_executor/layers/sm70_attention.py:flashattn_extension |
No SM70 path association |
VLLM_DBO_COMM_SMS |
configuration | The number of SMs/CUs to allocate for communication kernels when running DBO; the rest will be allocated to compute. Default: 20 on CUDA (SMs), 64 on ROCm (CUs). | 20 | Computed when unset: int('64' if hasattr(__import__('torch').version, 'hip') and __import__('torch').version.hip is not None else '20') | None | No SM70 path association |
VLLM_DEBUG_DUMP_PATH |
debug | Dump fx graphs to the given directory. It will override CompilationConfig.debug_dump_path if set. | None | None | None | No SM70 path association |
VLLM_DEBUG_LOG_API_SERVER_RESPONSE |
debug | Whether to log responses from API Server for debugging | False | False | None | No SM70 path association |
VLLM_DEBUG_MFU_METRICS |
debug | Debug logging for --enable-mfu-metrics | False | False | None | No SM70 path association |
VLLM_DEBUG_WORKSPACE |
debug | Debug workspace allocations. logging of workspace resize operations. | False | False | None | No SM70 path association |
VLLM_DEEPEPLL_NVFP4_DISPATCH |
configuration | Whether to use DeepEPLL kernels for NVFP4 quantization and dispatch method only supported on Blackwell GPUs and with https://github.com/deepseek-ai/DeepEP/pull/341 | False | False | None | No SM70 path association |
VLLM_DEEPEP_BUFFER_SIZE_MB |
configuration | The size in MB of the buffers (NVL and RDMA) used by DeepEP | 1024 | 1024 | None | No SM70 path association |
VLLM_DEEPEP_HIGH_THROUGHPUT_FORCE_INTRA_NODE |
configuration | Force DeepEP to use intranode kernel for inter-node communication in high throughput mode. This is useful archive higher prefill throughput on system supports multi-node nvlink (e.g GB200). | False | False | None | No SM70 path association |
VLLM_DEEPEP_LOW_LATENCY_USE_MNNVL |
configuration | Allow DeepEP to use MNNVL (multi-node nvlink) for internode_ll kernel, turn this for better latency on GB200 like system | False | False | None | No SM70 path association |
VLLM_DEEP_GEMM_WARMUP |
configuration | DeepGemm JITs the kernels on-demand. The warmup attempts to make DeepGemm JIT all the required kernels before model execution so there is no JIT'ing in the hot-path. However, this warmup increases the engine startup time by a couple of minutes. Available options: - "skip" : Skip warmup. - "full" : Warmup deepgemm by running all possible gemm shapes the engine could encounter. - "relax" : Select gemm shapes to run based on some heuristics. The heuristic aims to have the same effect as running all possible gemm shapes, but provides no guarantees. | 'relax' | 'relax' | None | No SM70 path association |
VLLM_DISABLED_KERNELS |
configuration | List of quantization kernels that should be disabled, used for testing and performance comparisons. Currently only affects MPLinearKernel selection (kernels: MacheteLinearKernel, MarlinLinearKernel, ExllamaLinearKernel) | [] | Computed when unset: [] if 'VLLM_DISABLED_KERNELS' not in os.environ else os.environ['VLLM_DISABLED_KERNELS'].split(',') | None | No SM70 path association |
VLLM_DISABLE_COMPILE_CACHE |
configuration | vLLM: disable compile cache. The consumer locations and unset defaults are listed below. | False | Computed by disable_compile_cache | None | No SM70 path association |
VLLM_DISABLE_LOG_LOGO |
configuration | Disable logging of vLLM logo at server startup time. | False | False | None | No SM70 path association |
VLLM_DISABLE_PYNCCL |
configuration | Disable pynccl (using torch.distributed instead) | False | False | None | No SM70 path association |
VLLM_DISABLE_REQUEST_ID_RANDOMIZATION |
configuration | Temporary: skip adding random suffix to internal request IDs. May be needed for KV connectors that match request IDs across instances. | False | False | None | No SM70 path association |
VLLM_DISABLE_SHARED_EXPERTS_STREAM |
configuration | Disables parallel execution of shared_experts via separate cuda stream | False | False | None | No SM70 path association |
VLLM_DIST_IDENT |
configuration | Identify the distributed instance when deriving communication names. | None | None | required at vllm/distributed/device_communicators/cpu_communicator.py:make_group_name | No SM70 path association |
VLLM_DOCKER_BUILD_CONTEXT |
configuration | Used to mark that setup.py is running in a Docker build context, in order to force the use of precompiled binaries. | False | False | None | No SM70 path association |
VLLM_DO_NOT_TRACK |
configuration | vLLM: do not track. The consumer locations and unset defaults are listed below. | False | False | None | No SM70 path association |
VLLM_DP_MASTER_IP |
configuration | IP address of the master node in the data parallel setting | '127.0.0.1' | '127.0.0.1' | None | No SM70 path association |
VLLM_DP_MASTER_PORT |
configuration | Port of the master node in the data parallel setting | 0 | 0 | None | No SM70 path association |
VLLM_DP_RANK |
configuration | Rank of the process in the data parallel setting | 0 | 0 | None | No SM70 path association |
VLLM_DP_RANK_LOCAL |
configuration | Rank of the process in the data parallel setting. Defaults to VLLM_DP_RANK when not set. | -1 | When unset: int(sys.modules[__name__].VLLM_DP_RANK) | None | No SM70 path association |
VLLM_DP_SIZE |
configuration | World size of the data parallel setting | 1 | 1 | None | No SM70 path association |
VLLM_ELASTIC_EP_DRAIN_REQUESTS |
configuration | Whether to wait for all requests to drain before sending the scaling command in elastic EP. | False | False | None | No SM70 path association |
VLLM_ELASTIC_EP_SCALE_UP_LAUNCH |
configuration | Whether it is a scale up launch engine for elastic EP, Should only be set by EngineCoreClient. | False | False | None | No SM70 path association |
VLLM_ENABLE_CUDAGRAPH_GC |
configuration | Controls garbage collection during CUDA graph capture. If set to 0 (default), enables GC freezing to speed up capture time. If set to 1, allows GC to run during capture. | False | False | None | No SM70 path association |
VLLM_ENABLE_CUDA_COMPATIBILITY |
configuration | Enable CUDA compatibility mode for datacenter GPUs with older driver versions than the CUDA toolkit major version of vLLM. | False | False | None | No SM70 path association |
VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE |
configuration | vLLM: enable fla packed recurrent decode. The consumer locations and unset defaults are listed below. | True | True; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:__post_init__ sets '1' when sm70_flash_v100_baseline and env_name not in os.environ; automatic defaults apply only when the environment override is absent. | No SM70 path association |
VLLM_ENABLE_FUSED_MOE_ACTIVATION_CHUNKING |
configuration | vLLM: enable fused moe activation chunking. The consumer locations and unset defaults are listed below. | False | False | None | No SM70 path association |
VLLM_ENABLE_INDUCTOR_COORDINATE_DESCENT_TUNING |
configuration | If set to 1, enable coordinate_descent_tuning; By default, this is enabled (1) | True | True | None | No SM70 path association |
VLLM_ENABLE_INDUCTOR_MAX_AUTOTUNE |
tuning | Enable max_autotune & coordinate_descent_tuning in inductor_config to compile static shapes passed from compile_sizes in compilation_config If set to 1, enable max_autotune; By default, this is enabled (1) | True | True | None | No SM70 path association |
VLLM_ENABLE_MAMBA_PREFIX_ASYNC |
configuration | Legacy 0.0.3 compatibility knob. Latest vLLM no longer has the old Mamba-prefix async-scheduling hard block; see VllmConfig for the no-op compatibility warning when this is explicitly set. | False | False | None | No SM70 path association |
VLLM_ENABLE_MOE_DP_CHUNK |
configuration | vLLM: enable moe dp chunk. The consumer locations and unset defaults are listed below. | False | False | None | No SM70 path association |
VLLM_ENABLE_PREGRAD_PASSES |
configuration | Inductor's pre-grad passes don't do anything for vLLM. The pre-grad passes get run even on cache-hit and negatively impact vllm cold compile times by O(1s) Can remove this after the following issue gets fixed TODO(luka): maybe_inplace requires this https://github.com/pytorch/pytorch/issues/174502 | True | True | None | No SM70 path association |
VLLM_ENABLE_RESPONSES_API_STORE |
configuration | Enables support for the "store" option in the OpenAI Responses API. When set to 1, vLLM's OpenAI server will retain the input and output messages for those requests in memory. By default, this is disabled (0), and the "store" option is ignored. NOTE/WARNING: 1. Messages are kept in memory only (not persisted to disk) and will be lost when the vLLM server shuts down. 2. Enabling this option will cause a memory leak, as stored messages are never removed from memory until the server terminates. | False | False | None | No SM70 path association |
VLLM_ENABLE_V1_MULTIPROCESSING |
configuration | If set, enable multiprocessing in LLM for the V1 code path. | True | True | None | No SM70 path association |
VLLM_ENFORCE_STRICT_TOOL_CALLING |
configuration | When 1,the model structural tags will be used to enforce the model output conforming to the model's tool-calling format and schema. Default 0 (off). | False | False | None | No SM70 path association |
VLLM_ENGINE_ITERATION_TIMEOUT_S |
configuration | timeout for each iteration in the engine | 60 | 60 | None | No SM70 path association |
VLLM_ENGINE_READY_TIMEOUT_S |
configuration | Timeout in seconds for waiting for engine cores to become ready during startup. Default is 600 seconds (10 minutes). | 600 | 600 | None | No SM70 path association |
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS |
configuration | Timeout in seconds for execute_model RPC calls in multiprocessing executor (only applies when TP > 1). | 300 | 300 | None | No SM70 path association |
VLLM_FLASHINFER_ALLREDUCE_BACKEND |
configuration | Flashinfer fused allreduce backend. | 'auto' | 'auto' | None | No SM70 path association |
VLLM_FLASHINFER_ALLREDUCE_FUSION_THRESHOLDS_MB |
tuning | Specifies the thresholds of the communicated tensor sizes under which vllm should use flashinfer fused allreduce. The variable should be a JSON with the following format: { : } Unspecified world sizes will fall back to { 2: 64, 4: 1, : 0.5 } | Computed when unset: json.loads('{}') | Computed when unset: json.loads('{}') | None | No SM70 path association |
VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR |
configuration | Override the directory for the FlashInfer autotune config cache. | None | None | None | No SM70 path association |
VLLM_FLASHINFER_MOE_BACKEND |
deprecated | Flashinfer MoE backend for vLLM's fused Mixture-of-Experts support. Both require compute capability 10.0 or above. Available options: - "throughput": [default] Uses CUTLASS kernels optimized for high-throughput batch inference. - "latency": Uses TensorRT-LLM kernels optimized for low-latency inference. Deprecated: pass --moe-backend flashinfer_{trtllm,cutlass,cutedsl} directly. | 'latency' | 'latency' | None | No SM70 path association |
VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE |
configuration | Control the workspace buffer size for the FlashInfer backend. | 394 * 1024 * 1024 | 413138944 | None | No SM70 path association |
VLLM_FLASH_V100_E4M3_GROUPED_FP32 |
experimental | Controls grouped E4M3 decode with FP32 accumulation. Default on for supported native layouts to retain qualified numerical behavior. Set 0 to compare the original grouped-attention route during diagnosis. | True | True | None | FlashAttentionV100Backend |
VLLM_FLASH_V100_PREFILL_D256_GQA_V37 |
configuration | Selects the legacy v37 D256/GQA prefill implementation. Default off because the qualified Q8000 FP32 route is preferred. Set 1 only for a matched rollback or a prefill regression investigation. | False | False | None | FlashAttentionV100Backend |
VLLM_FLOAT32_MATMUL_PRECISION |
configuration | Controls PyTorch float32 matmul precision mode within vLLM workers. Valid options mirror torch.set_float32_matmul_precision | 'highest' | 'highest' | None | No SM70 path association |
VLLM_FORCE_AOT_LOAD |
configuration | Force vllm to always load AOT compiled models from disk. Failure to load will result in a hard error when this is enabled. Will be ignored when VLLM_USE_AOT_COMPILE is disabled. | False | False | None | No SM70 path association |
VLLM_FUN_AUDIOCHAT_DEBUG |
debug | Legacy FunAudioChat multimodal embedding diagnostics from 0.0.3. Default-off; when enabled it prints tensor shapes/norms and can dump one-sample audio embeddings for offline quality debugging. | False | False | None | No SM70 path association |
VLLM_FUN_AUDIOCHAT_DUMP_PATH |
debug | vLLM: fun audiochat dump path. The consumer locations and unset defaults are listed below. | '' | '' | None | No SM70 path association |
VLLM_FUSED_MOE_CHUNK_SIZE |
tuning | Legacy 0.0.3 FusedMoE activation chunk controls. Latest keeps this explicit/default-off until model-level memory, route, and quality gates prove the chunked path for the target backend. | 16 * 1024 | Computed when unset: max(1, int(str(16 * 1024))) | None | No SM70 path association |
VLLM_GC_DEBUG |
debug | GC debug config - VLLM_GC_DEBUG=0: disable GC debugger - VLLM_GC_DEBUG=1: enable GC debugger with gc.collect elpased times - VLLM_GC_DEBUG='{"top_objects":5}': enable GC debugger with top 5 collected objects | '' | '' | None | No SM70 path association |
VLLM_GLM53_PP_MHC_MATERIALIZE |
configuration | vLLM: glm53 pp mhc materialize. The consumer locations and unset defaults are listed below. | False | False | None | No SM70 path association |
VLLM_GPT_OSS_HARMONY_SYSTEM_INSTRUCTIONS |
configuration | Allows harmony instructions to be injected on system messages | False | False | None | No SM70 path association |
VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS |
configuration | Valid values are container,code_interpreter,web_search_preview ex VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS=container,code_interpreter If the server_label of your mcp tool is not in this list it will be completely ignored. | set() | Computed by env_set_with_choices('VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS', default=[], choices=['container', 'code_interpreter', 'web_search_preview']) | None | No SM70 path association |
VLLM_GPU_NIC_PCIE_MAPPING |
configuration | Comma-separated GPU_BDF=NIC_BDF pairs for RDMA NIC selection. Must be set together with VLLM_NIC_SELECTION_VARS. | '' | '' | None | No SM70 path association |
VLLM_HAS_FLASHINFER_CUBIN |
configuration | If set, it means we pre-downloaded cubin files and flashinfer will read the cubin files directly. | False | False | None | No SM70 path association |
VLLM_HOST_IP |
configuration | used in distributed environment to determine the ip address of the current node, when the node has multiple network interfaces. If you are using multi-node inference, you should set this differently on each node. | '' | '' | None | No SM70 path association |
VLLM_HTTP_TIMEOUT_KEEP_ALIVE |
configuration | Timeout in seconds for keeping HTTP connections alive in API server | 5 | 5 | None | No SM70 path association |
VLLM_HUMMING_INPUT_QUANT_CONFIG |
configuration | The activation dtype config for humming kernel | None | Computed when unset: maybe_convert_json_str_or_file(None) | None | No SM70 path association |
VLLM_HUMMING_MOE_GEMM_TYPE |
configuration | Whether to use indexed gemm for humming moe if 1, force use indexed gemm if 0, force use grouped gemm if None, choose better gemm type automatically | None | None | None | No SM70 path association |
VLLM_HUMMING_ONLINE_QUANT_CONFIG |
configuration | The online quantization dtype for humming kernel | None | Computed when unset: maybe_convert_json_str_or_file(None) | None | No SM70 path association |
VLLM_HUMMING_USE_F16_ACCUM |
configuration | Whether to use fp16 accumulator mma | False | Computed when unset: maybe_convert_bool('0') | None | No SM70 path association |
VLLM_IMAGE_FETCH_TIMEOUT |
configuration | Timeout for fetching images when serving multimodal models Default is 5 seconds | 5 | 5 | None | No SM70 path association |
VLLM_KEEP_ALIVE_ON_ENGINE_DEATH |
configuration | If set, the OpenAI API server will stay alive even after the underlying AsyncLLMEngine errors and stops serving requests | False | False | None | No SM70 path association |
VLLM_KPOOL_SKIP_DECODE_WRITE |
configuration | vLLM: kpool skip decode write. The consumer locations and unset defaults are listed below. | None | None | None at vllm/model_executor/layers/sparse_attn_indexer_kpool.py:sparse_attn_indexer_kpool | No SM70 path association |
VLLM_KPOOL_SKIP_TAIL_CACHE |
configuration | vLLM: kpool skip tail cache. The consumer locations and unset defaults are listed below. | None | None | None at vllm/model_executor/layers/sparse_attn_indexer_kpool.py:sparse_attn_indexer_kpool | No SM70 path association |
VLLM_KV_CACHE_LAYOUT |
configuration | KV Cache layout used throughout vllm. Some common values are: - NHD - HND Where N=num_blocks, H=num_heads and D=head_size. The default value will leave the layout choice to the backend. Mind that backends may only implement and support a subset of all possible layouts. | None | None | None | No SM70 path association |
VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES |
configuration | Represent block hashes in KV cache events as 64-bit integers instead of raw bytes. Defaults to True for backward compatibility. | True | True | None | No SM70 path association |
VLLM_LOGGING_COLOR |
configuration | Controls colored logging output. Options: "auto" (default, colors when terminal), "1" (always use colors), "0" (never use colors) | 'auto' | 'auto' | None | No SM70 path association |
VLLM_LOGGING_CONFIG_PATH |
configuration | vLLM: logging config path. The consumer locations and unset defaults are listed below. | None | None | None | No SM70 path association |
VLLM_LOGGING_LEVEL |
configuration | this is used for configuring the default logging level | 'INFO' | 'INFO' | None | No SM70 path association |
VLLM_LOGGING_PREFIX |
configuration | if set, VLLM_LOGGING_PREFIX will be prepended to all log messages | '' | '' | None | No SM70 path association |
VLLM_LOGGING_STREAM |
configuration | this is used for configuring the default logging stream | 'ext://sys.stdout' | 'ext://sys.stdout' | None | No SM70 path association |
VLLM_LOG_BATCHSIZE_INTERVAL |
configuration | vLLM: log batchsize interval. The consumer locations and unset defaults are listed below. | -1 | -1.0 | None | No SM70 path association |
VLLM_LOG_MODEL_INSPECTION |
configuration | Log model inspection after loading. If enabled, logs a transformers-style hierarchical view of the model with quantization methods and attention backends. | False | False | None | No SM70 path association |
VLLM_LOG_STATS_INTERVAL |
configuration | If set, vllm will log stats at this interval in seconds If not set, vllm will log stats every 10 seconds. | 10.0 | 10.0 | None | No SM70 path association |
VLLM_LOOPBACK_IP |
configuration | Used to force set up loopback IP | '' | '' | None | No SM70 path association |
VLLM_LORA_DISABLE_PDL |
configuration | Disable PDL for LoRA, as enabling PDL with LoRA on SM100 causes Triton compilation to fail. | False | False | None | No SM70 path association |
VLLM_LORA_ENABLE_DUAL_STREAM |
configuration | Whether to enable dual cuda streams for LoRA computation (used by both BaseLinearLayerWithLoRA and FusedMoEWithLoRA to overlap the base layer compute with the LoRA fast path). | False | False | None | No SM70 path association |
VLLM_LORA_RESOLVER_CACHE_DIR |
configuration | a local directory to look in for unrecognized LoRA adapters. only works if plugins are enabled and VLLM_ALLOW_RUNTIME_LORA_UPDATING is enabled. | None | None | None | No SM70 path association |
VLLM_LORA_RESOLVER_HF_REPO_LIST |
configuration | A remote HF repo(s) containing one or more LoRA adapters, which may be downloaded and leveraged as needed. Only works if plugins are enabled and VLLM_ALLOW_RUNTIME_LORA_UPDATING is enabled. Values should be comma separated. | None | None | None | No SM70 path association |
VLLM_MAIN_CUDA_VERSION |
configuration | Main CUDA version of vLLM. This follows PyTorch but can be overridden. | '13.0' | '13.0' | None | No SM70 path association |
VLLM_MAMBA_ALIGN_CPU_POSTPROCESS |
configuration | vLLM: mamba align cpu postprocess. The consumer locations and unset defaults are listed below. | False | False | None | No SM70 path association |
VLLM_MAMBA_ALIGN_DEBUG |
debug | Legacy 0.0.3 Mamba/GDN alignment diagnostics. Default-off because it logs per-request state movement and accepted-token correction data. | False | False | None | No SM70 path association |
VLLM_MARLIN_INPUT_DTYPE |
configuration | The activation dtype for marlin kernel | None | None | None | No SM70 path association |
VLLM_MARLIN_USE_ATOMIC_ADD |
configuration | Whether to use atomicAdd reduce in gptq/awq marlin kernel. | False | False | None | No SM70 path association |
VLLM_MAX_AUDIO_CLIP_FILESIZE_MB |
tuning | Maximum filesize in MB for a single audio file when processing speech-to-text requests. Files larger than this will be rejected. Default is 25 MB | 25 | 25 | None | No SM70 path association |
VLLM_MAX_N_SEQUENCES |
tuning | Maximum allowed value for the n sampling parameter (number of output sequences per request). Limits resource consumption to prevent denial-of-service via excessively large fan-out. Default: 16384. |
16384 | 16384 | None | No SM70 path association |
VLLM_MAX_TOKENS_PER_EXPERT_FP4_MOE |
tuning | Control the maximum number of tokens per expert supported by the NVFP4 MoE CUTLASS Kernel. This value is used to create a buffer for the blockscale tensor of activations NVFP4 Quantization. This is used to prevent the kernel from running out of memory. | 163840 | 163840 | None | No SM70 path association |
VLLM_MEDIA_CACHE |
configuration | Directory for caching media downloads (images, video, audio fetched from URLs during inference). Empty string disables caching. | '' | '' | None | No SM70 path association |
VLLM_MEDIA_CACHE_MAX_SIZE_MB |
tuning | Maximum cache size in MB. When exceeded, least-recently-used entries are evicted. Default is 5120 (5 GB). | 5120 | 5120 | None | No SM70 path association |
VLLM_MEDIA_CACHE_TTL_HOURS |
configuration | Time-to-live in hours for cached media files. Entries older than this are evicted regardless of cache size. Default is 24 hours. | 24 | 24.0 | None | No SM70 path association |
VLLM_MEDIA_CONNECTOR |
configuration | Media connector implementation. - "http": Default connector that supports fetching media via HTTP. Custom implementations can be registered via @MEDIA_CONNECTOR_REGISTRY.register("my_custom_media_connector") and imported at runtime. If a non-existing backend is used, an AssertionError will be thrown. |
'http' | 'http' | None | No SM70 path association |
VLLM_MEDIA_FETCH_MAX_RETRIES |
tuning | Maximum number of retries for fetching media (images, audio, video) from URLs. Each retry quadruples the timeout. Default is 3. | 3 | 3 | None | No SM70 path association |
VLLM_MEDIA_LOADING_THREAD_COUNT |
configuration | Max number of workers for the thread pool handling media bytes loading. Set to 1 to disable parallel processing. Default is 8 | 8 | 8 | None | No SM70 path association |
VLLM_MEDIA_URL_ALLOW_REDIRECTS |
configuration | Whether to allow HTTP redirects when fetching from media URLs. Default to True | True | True | None | No SM70 path association |
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS |
debug | If set to 1, enable CUDA graph memory estimation during memory profiling. This profiles CUDA graph memory usage to provide more accurate KV cache memory allocation. Enabled by default as of v0.21.0 | True | True; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:__post_init__ sets '0' when sm70_flash_0dot3_compile_graph and sm70_compile_disabled_by_user and current_platform.is_cuda() and _any_participating_device_is_capability(self, (7, 0)) and envs.VLLM_SM70_FLASH_ATTN_V100 and not self.use_v2_model_runner and 'VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS' not in os.environ; automatic defaults apply only when the environment override is absent. | No SM70 path association |
VLLM_MLA_DISABLE |
configuration | If set, vLLM will disable the MLA attention optimizations. | False | False | None | No SM70 path association |
VLLM_MM_HASHER_ALGORITHM |
configuration | Hash algorithm for multimodal content hashing. - "blake3": Default, fast cryptographic hash (not FIPS 140-3 compliant) - "sha256": FIPS 140-3 compliant, widely supported - "sha512": FIPS 140-3 compliant, faster on 64-bit systems Use sha256 or sha512 for FIPS compliance in government/enterprise deployments | 'blake3' | 'blake3' | None | No SM70 path association |
VLLM_MODEL_REDIRECT_PATH |
configuration | Use model_redirect to redirect the model name to a local folder. model_redirect can be a json file mapping the model between repo_id and local folder: {"meta-llama/Llama-3.2-1B": "/tmp/Llama-3.2-1B"} or a space separated values table file: meta-llama/Llama-3.2-1B /tmp/Llama-3.2-1B |
None | None | None | No SM70 path association |
VLLM_MOE_DP_CHUNK_SIZE |
tuning | Legacy 0.0.3 DP all-to-all MoE chunk controls. The latest MoE runner has been refactored around custom-op entry points, so the old FusedMoE.forward_impl_chunked loop is not restored here. Keep these visible as pending compatibility knobs until a latest-runner chunk loop is ported and validated. | 256 | 256 | None | No SM70 path association |
VLLM_MOE_ROUTING_SIMULATION_STRATEGY |
configuration | MoE routing strategy selector. See RoutingSimulator.get_available_strategies() # for available strategies. Custom routing strategies can be registered by RoutingSimulator.register_strategy() Note: custom strategies may not produce correct model outputs |
'' | '' | None | No SM70 path association |
VLLM_MOE_USE_DEEP_GEMM |
configuration | Allow use of DeepGemm specifically for MoE fused ops (overrides only MoE). | True | True | None | No SM70 path association |
VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT |
configuration | Timeout (in seconds) for MooncakeConnector in PD disaggregated setup. | 480 | 480 | None | No SM70 path association |
VLLM_MOONCAKE_BOOTSTRAP_PORT |
configuration | Port used for Mooncake handshake between remote agents. | 8998 | 8998 | None | No SM70 path association |
VLLM_MOONCAKE_DISK_STAGING_USABLE_RATIO |
configuration | Fraction of the owner's DirectIO staging buffer to fill per GET batch. | 0.9 | 0.9 | None | No SM70 path association |
VLLM_MOONCAKE_STORE_TIER_LOG |
configuration | Log per-batch memory/disk tier breakdown on external GETs. | False | False | None | No SM70 path association |
VLLM_MQ_BROADCASTER_MAX_CHUNKS |
tuning | vLLM: mq broadcaster max chunks. The consumer locations and unset defaults are listed below. | 6 | 6; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:__post_init__ sets '64' when sm70_flash_0dot3_compile_graph and sm70_compile_disabled_by_user and current_platform.is_cuda() and _any_participating_device_is_capability(self, (7, 0)) and envs.VLLM_SM70_FLASH_ATTN_V100 and 'VLLM_MQ_BROADCASTER_MAX_CHUNKS' not in os.environ; automatic defaults apply only when the environment override is absent. | No SM70 path association |
VLLM_MQ_MAX_CHUNKS |
tuning | Shared-memory message queue chunk counts. Defaults preserve upstream behavior; increasing these can absorb compile/warmup bursts where worker readers do not consume scheduler broadcasts for a long time. | 10 | 10 | None | No SM70 path association |
VLLM_MQ_MAX_CHUNK_BYTES_MB |
tuning | Control the max chunk bytes (in MB) for the rpc message queue. Object larger than this threshold will be broadcast to worker processes via zmq. | 16 | 16 | None | No SM70 path association |
VLLM_MSGPACK_ZERO_COPY_THRESHOLD |
tuning | Control the threshold for msgspec to use 'zero copy' for serialization/deserialization of tensors. Tensors below this limit will be encoded into the msgpack buffer, and tensors above will instead be sent via a separate message. While the sending side still actually copies the tensor in all cases, on the receiving side, tensors above this limit will actually be zero-copy decoded. | 256 | 256 | None | No SM70 path association |
VLLM_MULTI_STREAM_GEMM_TOKEN_THRESHOLD |
tuning | Token-count cutoff for multi-stream overlap of the attention input GEMM with auxiliary GEMMs (e.g. fused_wqa_wkv overlapped with indexer weights / kv-score projections in DeepSeek-V4). At or below this many tokens the FP8 main GEMM has idle SMs to share with the bf16 aux GEMMs and overlap is a 5-45% win; above it the FP8 GEMM saturates the device and the cross-stream sync becomes pure overhead. Set to 0 to disable the multi-stream path entirely. See #PR 41526 for the empirical result for the default value of 1024 tokens. | 1024 | 1024 | None | No SM70 path association |
VLLM_MXFP4_USE_MARLIN |
deprecated | Whether to use marlin kernel in mxfp4 quantization method Deprecated: use --moe-backend marlin (MoE) or --linear-backend marlin (linear) instead. | None | Computed by deprecated_env('VLLM_MXFP4_USE_MARLIN', 'v0.23', 'Use --moe-backend marlin or --linear-backend marlin.', lambda: maybe_convert_bool(os.environ.get('VLLM_MXFP4_USE_MARLIN', None))) | None | No SM70 path association |
VLLM_NCCL_INCLUDE_PATH |
configuration | NCCL header path | None | None | None | No SM70 path association |
VLLM_NCCL_SO_PATH |
configuration | Path to the NCCL library file. It is needed because nccl>=2.19 brought by PyTorch contains a bug: https://github.com/NVIDIA/nccl/issues/1234 | None | None | None | No SM70 path association |
VLLM_NIC_SELECTION_VARS |
configuration | Comma-separated list of env vars to set from the GPU-NIC mapping. Each entry is VAR_NAME or VAR_NAME: (suffix appended to RDMA device name). Must be set together with VLLM_GPU_NIC_PCIE_MAPPING. | '' | '' | None | No SM70 path association |
VLLM_NIXL_EP_MAX_NUM_RANKS |
tuning | NIXL EP environment variables | 32 | 32 | None | No SM70 path association |
VLLM_NIXL_SIDE_CHANNEL_HOST |
configuration | IP address used for NIXL handshake between remote agents. | 'localhost' | 'localhost' | None | No SM70 path association |
VLLM_NIXL_SIDE_CHANNEL_PORT |
configuration | Port used for NIXL handshake between remote agents. | 5600 | 5600 | None | No SM70 path association |
VLLM_NO_USAGE_STATS |
configuration | vLLM: no usage stats. The consumer locations and unset defaults are listed below. | False | False | None | No SM70 path association |
VLLM_NVFP4_GEMM_BACKEND |
deprecated | Supported options: - "flashinfer-cudnn": use flashinfer cudnn GEMM backend - "flashinfer-trtllm": use flashinfer trtllm GEMM backend - "flashinfer-cutlass": use flashinfer cutlass GEMM backend - "marlin": use marlin GEMM backend (for GPUs without native FP4 support) - "emulation": use BF16/FP16 GEMM, dequantizing weights and running QDQ on activations. This is only meant for research purposes to run on devices where NVFP4 GEMM kernels are not available. - |
None | None | None | No SM70 path association |
VLLM_NVTX_SCOPES_FOR_PROFILING |
debug | Add optional nvtx scopes for profiling, disable to avoid overheads | False | False | None | No SM70 path association |
VLLM_OBJECT_STORAGE_SHM_BUFFER_NAME |
configuration | Name of the shared memory buffer used for object storage. Only effective when mm_config.mm_processor_cache_type == "shm". Automatically generates a unique UUID-based name per process tree if not explicitly set. | 'VLLM_OBJECT_STORAGE_SHM_BUFFER' | Computed by get_env_or_set_default('VLLM_OBJECT_STORAGE_SHM_BUFFER_NAME', lambda: f'VLLM_OBJECT_STORAGE_SHM_BUFFER_{uuid.uuid4().hex}') | None | No SM70 path association |
VLLM_PATTERN_MATCH_DEBUG |
debug | Debug pattern matching inside custom passes. Should be set to the fx.Node name (e.g. 'getitem_34' or 'scaled_mm_3'). | None | None | None | No SM70 path association |
VLLM_PLE_CPU_OFFLOAD |
configuration | Run n-gram PLE lookup in a dedicated CPU offload worker. The initial implementation supports ModelRunner V1/V2 and node-local MP DP/TP. | False | False; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:_apply_sm70_qwen38_hybrid_ple_defaults sets '1' when owner policy; automatic defaults apply only when the environment override is absent. | No SM70 path association |
VLLM_PLE_DISK_OFFLOAD |
configuration | Retain Qwen4Exp PLE safetensor shards as file-backed mappings instead of copying the complete learned n-gram table into anonymous host memory. | False | False; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:_apply_sm70_qwen38_hybrid_ple_defaults sets '1' when owner policy; automatic defaults apply only when the environment override is absent. | No SM70 path association |
VLLM_PLE_DISK_OFFLOAD_NUM_THREADS |
tuning | Number of cross-shard mmap gather workers. Zero selects a bounded hardware-aware default. | 0 | 0 | None | No SM70 path association |
VLLM_PLE_DISK_OFFLOAD_PROFILE |
debug | vLLM: ple disk offload profile. The consumer locations and unset defaults are listed below. | False | False | None | No SM70 path association |
VLLM_PLE_OFFLOAD_AUTO_NUMA |
configuration | Keep the latency-critical PLE lookup process on the NUMA node local to its first visible GPU. This changes CPU placement only; allocations use a local-first policy with fallback so large tables are not forced into a single NUMA node and swapped out. | True | True | None | No SM70 path association |
VLLM_PLE_OFFLOAD_PREFAULT |
configuration | Fault PLE table pages back into RAM after GPU workers finish loading. Concurrent checkpoint loading can otherwise leave anonymous table pages in swap while reclaimable checkpoint page cache occupies host memory. | True | True | None | No SM70 path association |
VLLM_PLE_OFFLOAD_READY_TIMEOUT |
configuration | Timeout for PLE weight loading and TP worker registration. | 600.0 | 600.0 | None | No SM70 path association |
VLLM_PLUGINS |
configuration | a list of plugin names to load, separated by commas. if this is not set, it means all plugins will be loaded if this is set to an empty string, no plugins will be loaded | None | Computed when unset: None if 'VLLM_PLUGINS' not in os.environ else os.environ['VLLM_PLUGINS'].split(',') | None | No SM70 path association |
VLLM_PORT |
configuration | used in distributed environment to manually set the communication port Note: if VLLM_PORT is set, and some code asks for multiple ports, the VLLM_PORT will be used as the first port, and the rest will be generated by incrementing the VLLM_PORT value. | None | Computed by get_vllm_port | None | No SM70 path association |
VLLM_PP_LAYER_PARTITION |
configuration | Pipeline stage partition strategy | None | None | None | No SM70 path association |
VLLM_PROCESS_NAME_PREFIX |
configuration | Used to set the process name prefix for vLLM processes. This is useful for debugging and monitoring purposes. The default value is "VLLM". | 'VLLM' | 'VLLM' | None | No SM70 path association |
VLLM_QWEN3NEXT_ENABLE_SHARED_MOE_OVERLAP |
configuration | Allows shared-expert stream overlap on eligible Qwen3Next/Qwen4Exp layers. Default enabled by the Qwen4Exp policy, otherwise the standalone getter is off. Set 0 to diagnose stream ordering; placement and DBO guards still decide eligibility. | False | False; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:_apply_sm70_qwen38_decode_defaults sets '1' when name not in os.environ; automatic defaults apply only when the environment override is absent. | SM70 runtime/kernel policy |
VLLM_QWEN3_NEXT_FUSED_SIGMOID_GATING |
configuration | Legacy 0.0.3 coarse Qwen3Next fused-sigmoid gate. Latest splits this into recurrent schedule and mixed-QKV controls; keep the old name visible for command compatibility and route-ledger diagnostics. | True | True | None | No SM70 path association |
VLLM_QWEN4EXP_PLE_HOST_GIB |
configuration | Sets pinned-host PLE memory in GiB per TP rank. Default automatic from actual table size and available host memory, avoiding a checkpoint-specific manual budget. Set an explicit limit when sharing host RAM or reproducing a fixed-memory deployment. | None | None | None | SM70 runtime/kernel policy |
VLLM_QWEN4EXP_PLE_HOST_RESERVE_GIB |
configuration | Host memory in GiB the automatic PLE placement leaves untouched for the engine processes, checkpoint loading and other tenants; the rest is shared equally by the tensor-parallel ranks that pin the table. Unset: 25 % of the physical host memory. | None | None | None | No SM70 path association |
VLLM_QWEN4EXP_PLE_VRAM_RESERVE_GIB |
configuration | Device memory in GiB the automatic PLE placement keeps free for the activation peak and the CUDA graph pool. Unset: 8 % of the device, at most 4 GiB. | None | None | None | No SM70 path association |
VLLM_RANDOMIZE_DP_DUMMY_INPUTS |
configuration | Randomize inputs during dummy runs when using Data Parallel | False | False | None | No SM70 path association |
VLLM_RAY_BUNDLE_INDICES |
configuration | Bundle indices for Ray, if it is set, it can control precisely which indices are used for the Ray bundle, for every worker. Format: comma-separated list of integers, e.g. "0,1,2,3" | '' | '' | None | No SM70 path association |
VLLM_RAY_DP_PACK_STRATEGY |
configuration | Strategy to pack the data parallel ranks for Ray. Available options: - "fill": for DP master node, allocate exactly data-parallel-size-local DP ranks, for non-master nodes, allocate as many DP ranks as can fit; - "strict": allocate exactly data-parallel-size-local DP ranks to each picked node; - "span": Should be used only when a single DP rank requires multiple nodes. allocate one DP rank over as many nodes as required for set world_size; This environment variable is ignored if data-parallel-backend is not Ray. | 'strict' | 'strict' | None | No SM70 path association |
VLLM_RAY_EXTRA_ENV_VARS_TO_COPY |
configuration | Comma-separated *additional* individual env var names to copy from the driver to Ray workers. Merged with the built-in defaults defined in vllm.ray.ray\_env (PYTHONHASHSEED). Example: "MY_SECRET,MY_FLAG" |
'' | '' | None | No SM70 path association |
VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY |
configuration | Comma-separated *additional* prefixes of env vars to copy from the driver to Ray workers. These are merged with the built-in defaults defined in vllm.ray.ray\_env (VLLM_, etc.). Example: "MYLIB_,OTHER_" |
'' | '' | None | No SM70 path association |
VLLM_RAY_PER_WORKER_GPUS |
configuration | If set, vLLM will pick up the provided Flash Attention MLA Number of GPUs per worker in Ray, if it is set to be a fraction, it allows ray to schedule multiple actors on a single GPU, so that users can colocate other actors on the same GPUs as vLLM. | 1.0 | 1.0 | None | No SM70 path association |
VLLM_RINGBUFFER_WARNING_INTERVAL |
configuration | Interval in seconds to log a warning message when the ring buffer is full | 60 | 60 | None | No SM70 path association |
VLLM_ROCM_AITER_MOE_DISPATCH_POLICY |
configuration | MoE sorting dispatch policy for AITER fused MoE kernels. 0 = auto (default): single-pass for small batches, multi-pass for large batches 1 = always single-pass: one kernel launch, no workspace, may be preferred for low-concurrency decode workloads 2 = always multi-pass: can be faster for MoE-heavy models (e.g., +2-5% on Qwen3-Next, +1.5% on DeepSeek-V3 at TP4, see PR #39177 for benchmarks) | 0 | 0 | None | No SM70 path association |
VLLM_ROCM_CUSTOM_PAGED_ATTN |
configuration | Legacy ROCm opt-out for the custom paged attention selector. Default-on preserves latest upstream behavior; setting it false restores the old escape hatch for ROCm numerical or backend issues. | True | True | None | No SM70 path association |
VLLM_ROCM_FP8_MFMA_PAGE_ATTN |
configuration | If set, use the fp8 mfma in rocm paged attention. | False | False | None | No SM70 path association |
VLLM_ROCM_FP8_PADDING |
configuration | Pad the fp8 weights to 256 bytes for ROCm | True | True | None | No SM70 path association |
VLLM_ROCM_MOE_PADDING |
configuration | Pad the weights for the moe kernel | True | True | None | No SM70 path association |
VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16 |
configuration | Custom quick allreduce kernel for MI3* cards Due to the lack of the bfloat16 asm instruction, bfloat16 kernels are slower than fp16, If environment variable is set to 1, the input is converted to fp16 | True | True | None | No SM70 path association |
VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB |
tuning | Custom quick allreduce kernel for MI3* cards. Controls the maximum allowed number of data bytes(MB) for custom quick allreduce communication. Default: 2048 MB. Data exceeding this size will use either custom allreduce or RCCL communication. | None | Computed when unset: maybe_convert_int(None) | None | No SM70 path association |
VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB |
tuning | Custom quick allreduce kernel for MI3* cards. Controls the minimum allowed number of data bytes(MB) required to use custom quick allreduce communication. If unset, use the built-in threshold table. | None | Computed when unset: maybe_convert_int(None) | None | No SM70 path association |
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION |
configuration | Custom quick allreduce kernel for MI3* cards Choice of quantization level: FP, INT8, INT6, INT4 or NONE Recommended for large models to get allreduce | 'NONE' | 'NONE' | None | No SM70 path association |
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB |
tuning | Controls the minimum tensor size (KB, where 1 KB = 1024 bytes) required to use the configured QuickReduce codec. Smaller tensors use FP QuickReduce. This does not affect QuickReduce eligibility. | None | Computed when unset: maybe_convert_int(None) | None | No SM70 path association |
VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT |
configuration | Whether to use the shuffled kv cache layout | False | False | None | No SM70 path association |
VLLM_ROCM_SLEEP_MEM_CHUNK_SIZE |
tuning | flag to control the chunk size (in MB) for sleeping memory allocations under ROCm | 256 | 256 | None | No SM70 path association |
VLLM_ROCM_USE_AITER |
configuration | Disable aiter ops unless specifically enabled. Acts as a parent switch to enable the rest of the other operations. | False | False | None | No SM70 path association |
VLLM_ROCM_USE_AITER_FP4BMM |
configuration | Whether to use aiter triton fp4 bmm kernel By default is enabled. | True | True | None | No SM70 path association |
VLLM_ROCM_USE_AITER_FP4_ASM_GEMM |
configuration | Whether to use aiter fp4 gemm asm. By default is disabled. | False | False | None | No SM70 path association |
VLLM_ROCM_USE_AITER_FP8BMM |
configuration | Whether to use aiter triton fp8 bmm kernel By default is enabled. | True | True | None | No SM70 path association |
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS |
configuration | Whether to use aiter fusion shared experts ops. By default is disabled. | False | False | None | No SM70 path association |
VLLM_ROCM_USE_AITER_LINEAR |
configuration | use aiter linear op if aiter ops are enabled The following list of related ops - scaled_mm (per-tensor / rowwise) - use aiter tuned gemms for unquantized gemms | True | True | None | No SM70 path association |
VLLM_ROCM_USE_AITER_MHA |
configuration | Whether to use aiter mha ops. By default is enabled. | True | True | None | No SM70 path association |
VLLM_ROCM_USE_AITER_MLA |
configuration | Whether to use aiter mla ops. By default is enabled. | True | True | None | No SM70 path association |
VLLM_ROCM_USE_AITER_MOE |
configuration | Whether to use aiter moe ops. By default is enabled. | True | True | None | No SM70 path association |
VLLM_ROCM_USE_AITER_PAGED_ATTN |
configuration | Whether to use aiter paged attention. By default is disabled. | False | False | None | No SM70 path association |
VLLM_ROCM_USE_AITER_RMSNORM |
configuration | use aiter rms norm op if aiter ops are enabled. | True | True | None | No SM70 path association |
VLLM_ROCM_USE_AITER_TRITON_GEMM |
configuration | Whether to use aiter triton kernels for gemm ops. By default is enabled. | True | True | None | No SM70 path association |
VLLM_ROCM_USE_AITER_TRITON_ROPE |
configuration | Whether to use aiter rope. By default is disabled. | False | False | None | No SM70 path association |
VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION |
configuration | Use AITER triton unified attention for V1 attention | False | False | None | No SM70 path association |
VLLM_ROCM_USE_SKINNY_GEMM |
configuration | use rocm skinny gemms | True | True | None | No SM70 path association |
VLLM_RPC_BASE_PATH |
configuration | path used for ipc when the frontend api server is running in multi-processing mode to communicate with the backend engine process. | tempfile.gettempdir() | Computed when unset: tempfile.gettempdir() | None | No SM70 path association |
VLLM_RPC_TIMEOUT |
configuration | Time in ms for the zmq client to wait for a response from the backend server for simple data operations | 10000 | 10000 | None | No SM70 path association |
VLLM_RUST_FRONTEND_PATH |
configuration | Path to the Rust frontend binary. Defaults to "auto" which discovers the binary installed with the vllm package. Only used when VLLM_USE_RUST_FRONTEND=1. | 'auto' | Computed when unset: _resolve_rust_frontend_path() | None | No SM70 path association |
VLLM_SERVER_DEV_MODE |
configuration | If set, vllm will run in development mode, which will enable some additional endpoints for developing and debugging, e.g. /reset_prefix_cache |
False | False | None | No SM70 path association |
VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD |
tuning | Limits when we run shared_experts in a separate stream. We found out that for large batch sizes, the separate stream execution is not beneficial (most likely because of the input clone) TODO(alexm-redhat): Tune to be more dynamic based on GPU type | 256 | 256 | None | No SM70 path association |
VLLM_SKIP_MODEL_NAME_VALIDATION |
configuration | Skip model name validation in OpenAI API requests. When set to 1, any model name will be accepted in the 'model' field of API requests. This is useful for proxy/gateway scenarios where the actual model is served but different names may be used in requests. | False | False | None | No SM70 path association |
VLLM_SKIP_P2P_CHECK |
configuration | We assume drivers can report p2p status correctly. If the program hangs when using custom allreduce, potantially caused by a bug in the driver (535 series), if might be helpful to set VLLM_SKIP_P2P_CHECK=0 so that vLLM can verify if p2p is actually working. See https://github.com/vllm-project/vllm/blob/a9b15c606fea67a072416ea0ea115261a2756058/vllm/distributed/device\_communicators/custom\_all\_reduce\_utils.py#L101-L108 for details. # noqa | False | True | None | No SM70 path association |
VLLM_SKIP_PRECOMPILED_VERSION_SUFFIX |
configuration | If set, skip adding +precompiled suffix to version string | False | False | None | No SM70 path association |
VLLM_SLEEP_WHEN_IDLE |
configuration | Legacy 0.0.3 mp broadcast idle-wait knob. Latest upstream replaced the old SpinTimer/SpinSleepTimer split with SpinCondition, which already spins under load and waits on notification while idle. Kept only for command compatibility and inventory closure. | False | False | None | No SM70 path association |
VLLM_SM70_DEBUG |
debug | Select comma-separated SM70 diagnostics: trace logs, MTP phase timing, decode events, attention routing or eager selector shadow checks (selector). Default empty to avoid logging and observer overhead. Use only to diagnose a route or latency issue; explicit channels override old aliases. | set() | set() | None | SM70 diagnostics |
VLLM_SM70_DFLASH2_TAIL_CUDAGRAPHS |
configuration | Captures supported single-request DFlash2 verifier tails. Default on after paired token/state and speed qualification. Set 0 to trade tail-graph memory and speed for the eager fallback during diagnosis. | True | True | None | DFlash2 verifier |
VLLM_SM70_E4M3_LONG_ATTENTION |
configuration | Controls the built-in E4M3 long-context attention route. Default enabled when the native operator is present; dtype and page-layout checks still apply. Set 0 to diagnose attention with the original full-context route. | None | None; unset admits the built-in native operator or an explicit manifest. | vllm/v1/attention/ops/sm70_e4m3_long.py:long_attention_enabled | Flash-V100 long-context attention |
VLLM_SM70_FLASH_ATTN_V100 |
configuration | Selects Flash-V100 attention automatically on V100. Default on to use the qualified SM70 backend. Set 0 only to diagnose backend selection; prefer the engine attention_backend option when choosing a backend. | True | True | None | FlashAttentionV100Backend |
VLLM_SM70_FUSED_SIGMOID_MIXED_QKV |
configuration | Fuses sigmoid and multiply for supported mixed-QKV layouts without changing FP32 recurrent state. Default on after #703 quality qualification. Set 0 to diagnose the separate epilogue. | True | True; operator dtype, geometry, numerical policy and native availability checks still apply. | None | SM70 runtime/kernel policy, Flash-Next qualified batch |
VLLM_SM70_MOE_ADD_ALLREDUCE |
experimental | Fuses shared and routed MoE outputs with TP all-reduce. Default enabled by the qualified Qwen4Exp policy, otherwise off. Set 0 to compare separate addition and communication when investigating MoE overlap. | False | False; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:_apply_sm70_qwen38_decode_defaults sets '1' when name not in os.environ; automatic defaults apply only when the environment override is absent. | SM70 collectives |
VLLM_SM70_MTP_HC_BATCH |
configuration | Enables qualified M5/M10 HC projection and gather kernels with the existing FP16 split-K boundaries. Default on after #703 qualification. Set 0 to recover HC packed weights when QWEN38_BATCH_FASTPATH is also 0 (330 MiB per Flash-Next TP4 target rank). | True | True; operator dtype, geometry, numerical policy and native availability checks still apply. | None | MTP verifier, Flash-Next qualified batch |
VLLM_SM70_MTP_HC_COOPERATIVE |
configuration | Combines admitted HC projection and gather phases with cooperative barriers. Default on after #703 quality and complete- round qualification. Set 0 to compare the separate-phase implementation when diagnosing HC performance. | True | True; operator dtype, geometry, numerical policy and native availability checks still apply. | None | MTP verifier, Flash-Next qualified batch |
VLLM_SM70_MTP_HC_FULL_UNROLL |
configuration | Selects the qualified fully unrolled cooperative HC schedule. Default on after #703 complete-round qualification. Set 0 to compare the partial-unroll schedule when diagnosing performance. | True | True; operator dtype, geometry, numerical policy and native availability checks still apply. | None | MTP verifier, Flash-Next qualified batch |
VLLM_SM70_MTP_MOE_FP16_EXACT |
configuration | Uses exact native M1/M5 FP16 draft MoE kernels with the original accumulation order. Default on after #703 qualification. Set 0 to compare the original Triton MoE implementation. | True | True; operator dtype, geometry, numerical policy and native availability checks still apply. | None | MTP verifier, Flash-Next qualified batch |
VLLM_SM70_MTP_PLE_CONV |
configuration | Fuses qualified MTP4 PLE rollback, ordered convolution taps, SiLU and state commit. Default on after #703 exact state and model qualification. Set 0 to diagnose PLE state updates with the separate operations. | True | True; operator dtype, geometry, numerical policy and native availability checks still apply. | None | MTP verifier, Flash-Next qualified batch |
VLLM_SM70_MTP_ROUTER_BATCH |
configuration | Enables the exact M5/M10 FP16 router projection with four ordered FP32 splits. Default on after #703 qualification. Set 0 to recover packed router copies (122.5 MiB per Flash-Next TP4 target rank). | True | True; operator dtype, geometry, numerical policy and native availability checks still apply. | None | MTP verifier, Flash-Next qualified batch |
VLLM_SM70_MTP_ROUTER_TOP16 |
configuration | Selects lossless top-16 router keys before the unchanged top-10 normalization. Default on after #703 qualification. Set 0 to diagnose routing with the original full-key selection. | True | True; operator dtype, geometry, numerical policy and native availability checks still apply. | None | MTP verifier, Flash-Next qualified batch |
VLLM_SM70_MTP_SHARED_BATCH |
configuration | Enables exact M5/M10 shared-expert projection and SiLU fusion. Default on after #703 qualification. Set 0 to recover packed shared-expert copies (76.5625 MiB per Flash-Next TP4 target rank). | True | True; operator dtype, geometry, numerical policy and native availability checks still apply. | None | MTP verifier, Flash-Next qualified batch |
VLLM_SM70_MTP_SPLIT_DRAFT_CUDAGRAPHS |
configuration | Separates MTP draft and verifier graph sizes so single-token draft graphs remain available. Default enabled by the qualified Qwen4Exp MTP engine policy, otherwise off. Set 0 only to diagnose graph capture; speculative width is configured through speculative_config. | False | False | None | MTP verifier |
VLLM_SM70_NVFP4_MOE_GROUPED_MTP5 |
configuration | Reuses expert grouping for the qualified NVFP4 MTP5 route without changing W13 split order. Default on after #703 quality and complete-round qualification. Set 0 to compare the direct expert dispatch. | True | True; operator dtype, geometry, numerical policy and native availability checks still apply. | None | TurboMind NVFP4, Flash-Next qualified batch |
VLLM_SM70_QSA_MTP_TOPK |
configuration | Enables the qualified M5/M10 QSA top-k compaction while retaining the original scorer. Default on after #703 qualification. Set 0 to compare full-row top-k when diagnosing selector output. | True | True; operator dtype, geometry, numerical policy and native availability checks still apply. | None | QSA sparse attention/indexer, Flash-Next qualified batch |
VLLM_SM70_QWEN38_BATCH_FASTPATH |
configuration | Enables the qualified small-batch dense, GDN and TP4 HC routes. Default on after #704 quality and throughput qualification. Set 0 to trade speed for KV capacity; Flash-Next TP4 packed copies add up to 1055.625 MiB per rank. | True | True; operator dtype, geometry, numerical policy and native availability checks still apply. | None | SM70 runtime/kernel policy, Flash-Next qualified batch |
VLLM_SM70_QWEN38_FP16_GEMV |
configuration | Selects checkpoint-FP16 SM70 row GEMV for supported Qwen4Exp projections. Default enabled by the engine policy after capability admission; the standalone getter is off. Set 0 to compare the ordinary linear method when investigating a projection regression. | False | False; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:_apply_sm70_qwen38_decode_defaults sets '1' when name not in os.environ; automatic defaults apply only when the environment override is absent. | SM70 FP16 linear |
VLLM_SM70_QWEN38_FUSED_GDN_INPUT_FP16 |
experimental | Fuses supported checkpoint-FP16 GDN input projections and stores. Default enabled by the qualified Qwen4Exp policy, otherwise off. Set 0 to isolate the original input projection chain during diagnosis. | False | False; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:_apply_sm70_qwen38_decode_defaults sets '1' when name not in os.environ; automatic defaults apply only when the environment override is absent. | Gated DeltaNet |
VLLM_SM70_QWEN38_FUSED_HC_FP16 |
experimental | Fuses supported checkpoint-FP16 HyperConnection stages. Default enabled by the Qwen4Exp policy, otherwise off; local geometry and collective guards remain. Set 0 to isolate HC projection and mixing during diagnosis. | False | False; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:_apply_sm70_qwen38_decode_defaults sets '1' when name not in os.environ; automatic defaults apply only when the environment override is absent. | SM70 FP16 linear |
VLLM_SM70_QWEN38_GDN_INPUT_BATCH |
configuration | Automatic packing is qualified for ordinary decode and MTP; other proposers retain explicit opt-in. Enables lossless packed GDN input projections for M2..16. Default on after #703/#704 qualification. Set 0 together with QWEN38_BATCH_FASTPATH=0 to recover the packed-weight memory (725.625 MiB per Flash-Next TP4 target rank). | True | True; operator dtype, geometry, numerical policy and native availability checks still apply. | None | Gated DeltaNet, Flash-Next qualified batch |
VLLM_SM70_QWEN38_HYBRID_PLE |
configuration | Keeps local pinned-host PLE shards for decode while prefill uses CPU/disk offload. Default enabled by the eligible Qwen4Exp engine policy, otherwise off. Set 0 only to compare PLE placement when diagnosing host-memory or transfer behavior. | False | False; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:_apply_sm70_qwen38_hybrid_ple_defaults sets '1' when owner policy; automatic defaults apply only when the environment override is absent. | SM70 runtime/kernel policy |
VLLM_SM70_QWEN4_EXP_ONLINE_QPN8 |
experimental | Requantizes selected FP16 checkpoint weights to channel E4M3 for online QPN8. Default off because this changes weight precision. Set 1 only for an explicitly qualified precision/performance experiment; keep it off for checkpoint-preserving inference. | False | False | None | SM70 runtime/kernel policy |
VLLM_SM70_REQUIRE_PROFILE_ACCELERATION |
debug | Requires the release profile acceleration selfcheck to pass at startup. Default off so other models and diagnostic fallbacks can start. Set 1 when validating the declared release profile; it does not certify that requests hit every kernel. | False | False | None | SM70 runtime/kernel policy |
VLLM_SM70_RMSNORM_GATED_EXACT |
configuration | Uses native FP32 gated RMSNorm for supported SM70 layouts. Default on after #703/#704 quality qualification to preserve arithmetic across compiled graph contexts. Other models remain off until qualified. Set 0 only to diagnose the original normalization route. | False | False at the compatibility getter; resolved per engine to True for qualified Flash-Next ordinary/MTP configurations. Operator dtype, geometry, numerical policy and native availability checks still apply. | Flash-Next ordinary decode or MTP quality qualification | SM70 runtime/kernel policy, Flash-Next qualified batch |
VLLM_SM70_TP4_PUSH_ALLREDUCE |
configuration | Controls the qualified SM70 TP4 push all-reduce. Default on for supported topology, payload and graph layouts; per-engine GLM policy may disable it. Set 0 to isolate collective transport when investigating correctness or latency. | True | True; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:_configure_sm70_glm5_dflash_tp4_push_allreduce sets '0' when env_name not in os.environ; automatic defaults apply only when the environment override is absent. | SM70 collectives |
VLLM_SPARSE_INDEXER_MAX_LOGITS_MB |
tuning | Maximum size (in MB) for logits tensor in sparse MLA indexer prefill chunks. Bounds the [M, N] float32 logits tensor to prevent CUDA OOM. Default: 512 MB | 512 | 512 | None | No SM70 path association |
VLLM_SPEC_DEBUG_CORRUPTION |
debug | Speculative-decoding diagnostic: debug corruption. The consumer locations and unset defaults are listed below. | False | False | None | No SM70 path association |
VLLM_SPEC_DUMP_ALIGNMENT |
debug | 0.0.3 speculative/MTP diagnostics. These are default-off and must not change sampling or model outputs. | False | False | None | No SM70 path association |
VLLM_SPEC_DUMP_ALIGNMENT_DIR |
debug | Speculative-decoding diagnostic: dump alignment dir. The consumer locations and unset defaults are listed below. | None | None | '/tmp' at vllm/v1/sample/rejection_sampler.py:_maybe_dump_alignment '/tmp' at vllm/v1/worker/gpu/spec_decode/dflash2/sparse_rejection.py:_maybe_dump_selector_alignment |
No SM70 path association |
VLLM_SPEC_DUMP_ALIGNMENT_LIMIT |
debug | Speculative-decoding diagnostic: dump alignment limit. The consumer locations and unset defaults are listed below. | 3 | 3 | None | No SM70 path association |
VLLM_SPEC_DUMP_ALIGNMENT_STEPS |
debug | Speculative-decoding diagnostic: dump alignment steps. The consumer locations and unset defaults are listed below. | None | None | None | No SM70 path association |
VLLM_SPEC_DUMP_ALIGNMENT_TAG |
debug | Speculative-decoding diagnostic: dump alignment tag. The consumer locations and unset defaults are listed below. | None | None | '' at vllm/v1/sample/rejection_sampler.py:_maybe_dump_alignment '' at vllm/v1/worker/gpu/spec_decode/dflash2/sparse_rejection.py:_maybe_dump_selector_alignment |
No SM70 path association |
VLLM_SPEC_DUMP_DRAFT_LOGITS |
debug | Speculative-decoding diagnostic: dump draft logits. The consumer locations and unset defaults are listed below. | False | False | None | No SM70 path association |
VLLM_SSM_CONV_STATE_LAYOUT |
configuration | SSM conv state layout used for Mamba models. - SD: (state_len, dim) — dim contiguous (default) - DS: (dim, state_len) — TP-sharded dim on dim1, consistent with SSM temporal state and HND KV cache layout. | None | None | None | No SM70 path association |
VLLM_SYSTEM_START_DATE |
configuration | Pin the conversation start date injected into the Harmony system message. When unset the current date is used, which introduces non-determinism (different tokens -> different model behaviour at temperature=0). Set to an ISO date string, e.g. "2023-09-12", for reproducible inference or testing. | None | None | None | No SM70 path association |
VLLM_TARGET_DEVICE |
configuration | ================== Installation Time Env Vars ================== Target device of vLLM, supporting [cuda (by default), rocm, cpu] | 'cuda' | 'cuda' | None | No SM70 path association |
VLLM_TEST_FORCE_FP8_MARLIN |
configuration | If set, forces FP8 Marlin to be used for FP8 quantization regardless of the hardware support for FP8 compute. | False | False | None | No SM70 path association |
VLLM_TEST_FORCE_LOAD_FORMAT |
configuration | vLLM: test force load format. The consumer locations and unset defaults are listed below. | 'dummy' | 'dummy' | None | No SM70 path association |
VLLM_TOOL_JSON_ERROR_AUTOMATIC_RETRY |
configuration | Enable automatic retry when tool call JSON parsing fails If enabled, returns an error message to the model to retry If disabled (default), raises an exception and fails the request | False | False | None | No SM70 path association |
VLLM_TOOL_PARSE_REGEX_TIMEOUT_SECONDS |
configuration | Regex timeout for use by the vLLM tool parsing plugins. | 1 | 1 | None | No SM70 path association |
VLLM_TPU_BUCKET_PADDING_GAP |
configuration | Gap between padding buckets for the forward pass. So we have 8, we will run forward pass with [16, 24, 32, ...]. | 0 | Computed when unset: int(os.environ['VLLM_TPU_BUCKET_PADDING_GAP']) if 'VLLM_TPU_BUCKET_PADDING_GAP' in os.environ else 0 | None | No SM70 path association |
VLLM_TPU_MOST_MODEL_LEN |
configuration | vLLM: tpu most model len. The consumer locations and unset defaults are listed below. | None | Computed when unset: maybe_convert_int(None) | None | No SM70 path association |
VLLM_TPU_USING_PATHWAYS |
configuration | Whether using Pathways | False | False | None | No SM70 path association |
VLLM_TP_ALLREDUCE_TRACE |
debug | Legacy 0.0.3 diagnostic gate. Logs one TP all-reduce backend decision per group/backend/shape/dtype so route-hit data can distinguish custom AR, pynccl, flashinfer, symmetric-memory, and torch fallback paths. | False | False | None | No SM70 path association |
VLLM_TRACE_FUNCTION |
debug | Trace function calls If set to 1, vllm will trace function calls Useful for debugging | 0 | 0 | None | No SM70 path association |
VLLM_TRITON_ATTN_USE_TD |
configuration | Use tensor descriptors for Q/K/V loads and output stores in the Triton unified-attention kernel. Enables HW 2D block reads on Intel Xe2/Xe3; the non-TD branch is dead-code-eliminated at Triton compile time so other platforms see no overhead. Tri-state override: unset (default) lets the triton_attn backend auto-select per platform (currently auto-enabled on XPU only); 1 forces TD on; 0 forces TD off. Useful for A/B benchmarking the TD path. |
None | None | None | No SM70 path association |
VLLM_TUNED_CONFIG_FOLDER |
configuration | User override folder for tuned Triton-kernel configs. Shared by MoE, Mamba SSU, and LoRA. Filenames are distinct so one folder can hold all. Each component first checks this folder, then the configs shipped with vLLM (if any). If no JSON matches, it uses a hard-coded heuristic. | None | None | None | No SM70 path association |
VLLM_USAGE_SOURCE |
configuration | vLLM: usage source. The consumer locations and unset defaults are listed below. | 'production' | 'production' | None | No SM70 path association |
VLLM_USAGE_STATS_SERVER |
configuration | Usage stats collection | 'https://stats.vllm.ai' | 'https://stats.vllm.ai' | None | No SM70 path association |
VLLM_USE_AOT_COMPILE |
configuration | Feature flag to enable/disable AOT compilation. This will ensure compilation is done in warmup phase and the compilation will be reused in subsequent calls. | False | Computed by use_aot_compile; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:_configure_sm70_glm5_dflash_tp8_pp1_verifier_path sets '0' when name not in os.environ; automatic defaults apply only when the environment override is absent. vllm/config/vllm.py:__post_init__ sets '1' when sm70_flash_0dot3_compile_graph and sm70_compile_disabled_by_user and current_platform.is_cuda() and _any_participating_device_is_capability(self, (7, 0)) and envs.VLLM_SM70_FLASH_ATTN_V100 and 'VLLM_USE_AOT_COMPILE' not in os.environ; automatic defaults apply only when the environment override is absent. |
No SM70 path association |
VLLM_USE_BREAKABLE_CUDAGRAPH |
experimental | Experimental: breakable cudagraph does not rely on torch.compile | False | False; configuration may replace the unset default under the automatic conditions below. | vllm/config/vllm.py:__post_init__ sets '1' when self.model_config is not None and 'VLLM_USE_BREAKABLE_CUDAGRAPH' not in os.environ and any((a in ('DeepseekV4ForCausalLM', 'DeepSeekV4MTPModel') for a in self.model_config.architectures)); automatic defaults apply only when the environment override is absent. vllm/config/vllm.py:__post_init__ sets '1' when envs.VLLM_SM70_USE_BREAKABLE_CUDAGRAPH and current_platform.is_cuda() and _any_participating_device_is_capability(self, (7, 0)) and 'VLLM_USE_BREAKABLE_CUDAGRAPH' not in os.environ; automatic defaults apply only when the environment override is absent. |
No SM70 path association |
VLLM_USE_BYTECODE_HOOK |
configuration | Feature flag to enable/disable bytecode in TorchCompileWithNoGuardsWrapper. | True | True | None | No SM70 path association |
VLLM_USE_DEEP_GEMM |
configuration | Allow use of DeepGemm kernels for fused moe ops. | True | True | None | No SM70 path association |
VLLM_USE_DEEP_GEMM_E8M0 |
configuration | Whether to use E8M0 scaling when DeepGEMM is used on Blackwell GPUs. | True | True | None | No SM70 path association |
VLLM_USE_DEEP_GEMM_TMA_ALIGNED_SCALES |
configuration | Whether to create TMA-aligned scale tensor when DeepGEMM is used. | True | True | None | No SM70 path association |
VLLM_USE_EXPERIMENTAL_PARSER_CONTEXT |
experimental | Experimental: use this to enable MCP tool calling for non harmony models | False | False | None | No SM70 path association |
VLLM_USE_FASTOKENS |
configuration | If true, replace the Rust BPE backend that powers HF fast tokenizers with the fastokens (https://github.com/crusoecloud/fastokens) shim. Applies to any tokenizer mode that loads an HF fast tokenizer (hf, deepseek_v32, deepseek_v4, qwen_vl, …). The fastokens Python package must be installed. |
False | False | None | No SM70 path association |
VLLM_USE_FBGEMM |
deprecated | Flag to enable FBGemm kernels on model execution Deprecated: use --linear-backend fbgemm instead. | False | False | None | No SM70 path association |
VLLM_USE_FLASHINFER_MOE_FP16 |
deprecated | Allow use of FlashInfer BF16 MoE kernels for fused moe ops. Deprecated: use --moe-backend to select a kernel explicitly. | False | False | None | No SM70 path association |
VLLM_USE_FLASHINFER_MOE_FP4 |
deprecated | Allow use of FlashInfer NVFP4 MoE kernels for fused moe ops. Deprecated: use --moe-backend to select a kernel explicitly. | False | False | None | No SM70 path association |
VLLM_USE_FLASHINFER_MOE_FP8 |
deprecated | Allow use of FlashInfer FP8 MoE kernels for fused moe ops. Deprecated: use --moe-backend to select a kernel explicitly. | False | False | None | No SM70 path association |
VLLM_USE_FLASHINFER_MOE_INT4 |
configuration | Allow use of FlashInfer MxInt4 MoE kernels for fused moe ops. | False | False | None | No SM70 path association |
VLLM_USE_FLASHINFER_MOE_MXFP4_BF16 |
deprecated | If set to 1, use the FlashInfer BF16 (activation) x MXFP4 (weight) MoE backend. Deprecated: use --moe-backend to select a kernel explicitly. | False | False | None | No SM70 path association |
VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8 |
deprecated | If set to 1, use the FlashInfer MXFP8 (activation) x MXFP4 (weight) MoE backend. Deprecated: use --moe-backend flashinfer_trtllm combined with --quantization_config.moe.activation mxfp8. | False | False | None | No SM70 path association |
VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8_CUTLASS |
deprecated | If set to 1, use the FlashInfer CUTLASS backend for MXFP8 (activation) x MXFP4 (weight) MoE. Deprecated: use --moe-backend flashinfer_cutlass combined with --quantization_config.moe.activation mxfp8. | False | False | None | No SM70 path association |
VLLM_USE_FLASHINFER_SAMPLER |
configuration | Whether to use the FlashInfer top-k / top-p sampler on CUDA. Enabled by default when the hardware supports it — set to 0 to opt out explicitly, which forces the PyTorch-native (Triton for bs>=8) path. | True | Computed when unset: bool(int(os.environ['VLLM_USE_FLASHINFER_SAMPLER'])) if 'VLLM_USE_FLASHINFER_SAMPLER' in os.environ else True | None | No SM70 path association |
VLLM_USE_FUSED_MOE_GROUPED_TOPK |
configuration | Whether to use fused grouped_topk used for MoE expert selection. | True | True | None | No SM70 path association |
VLLM_USE_LAYERNAME |
configuration | If set to "0", disable LayerName opaque type for layer_name parameters in custom ops. Defaults to enabled on torch >= 2.11. | True | True | None | No SM70 path association |
VLLM_USE_MEGA_AOT_ARTIFACT |
configuration | Enable loading compiled models directly from cached standalone compile artifacts without re-splitting graph modules. This reduces overhead during model loading by using reconstruct_serializable_fn_from_mega_artifact. | False | Computed by use_mega_aot_artifact | None | No SM70 path association |
VLLM_USE_MODELSCOPE |
configuration | If true, will load models from ModelScope instead of Hugging Face Hub. note that the value is true or false, not numbers | False | False | None | No SM70 path association |
VLLM_USE_NCCL_SYMM_MEM |
configuration | Flag to enable NCCL symmetric memory allocation and registration | False | False | None | No SM70 path association |
VLLM_USE_NVFP4_CT_EMULATIONS |
deprecated | Controls whether or not emulations are used for NVFP4 generations on machines < 100 for compressed-tensors models Deprecated: use --linear-backend emulation instead. | False | False | None | No SM70 path association |
VLLM_USE_OINK_OPS |
configuration | Optional: enable external Oink custom ops (e.g., Blackwell RMSNorm). Disabled by default. | False | False | None | No SM70 path association |
VLLM_USE_PRECOMPILED |
configuration | If set, vllm will use precompiled native binaries (*.so and vllm-rs). | False | Computed when unset: ''.strip().lower() in ('1', 'true') or bool(None) | None | No SM70 path association |
VLLM_USE_PRECOMPILED_RUST |
configuration | If set, vllm will use the precompiled Rust frontend binary (vllm-rs). | False | False | None | No SM70 path association |
VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE |
configuration | If the env var is set, Ray Compiled Graph uses the specified channel type to communicate between workers belonging to different pipeline-parallel stages. Available options: - "auto": use the default channel type - "nccl": use NCCL for communication - "shm": use shared memory and gRPC for communication | 'auto' | 'auto' | None | No SM70 path association |
VLLM_USE_RAY_COMPILED_DAG_OVERLAP_COMM |
experimental | If the env var is set, it enables GPU communication overlap (experimental feature) in Ray's Compiled Graph. | False | False | None | No SM70 path association |
VLLM_USE_RAY_V2_EXECUTOR_BACKEND |
configuration | When True and distributed_executor_backend="ray", use RayExecutorV2 (MQ-based) instead of RayDistributedExecutor (compiled-graph backend). | True | True | None | No SM70 path association |
VLLM_USE_RAY_WRAPPED_PP_COMM |
configuration | If the env var is set, it uses a Ray Communicator wrapping vLLM's pipeline parallelism communicator to interact with Ray's Compiled Graph. Otherwise, it uses Ray's NCCL communicator. | True | True | None | No SM70 path association |
VLLM_USE_RUST_FRONTEND |
configuration | If set, use the Rust frontend binary instead of the Python API server process(es). | False | False | None | No SM70 path association |
VLLM_USE_SIMPLE_KV_OFFLOAD |
configuration | Enable simple KV offload. | False | False | None | No SM70 path association |
VLLM_USE_SPINLOOP_EXT |
configuration | If set to 1, use Python spinloop extension to poll in a more efficient way when using the mp backend. | False | False | None | No SM70 path association |
VLLM_USE_STANDALONE_COMPILE |
configuration | Feature flag to enable/disable Inductor standalone compile. In torch <= 2.7 we ignore this flag; in torch >= 2.9 this is enabled by default. | True | True | None | No SM70 path association |
VLLM_USE_TRITON_AWQ |
configuration | If set, vLLM will use Triton implementations of AWQ. | False | False | None | No SM70 path association |
VLLM_USE_V1 |
configuration | Legacy benchmark harness knob from 0.0.3. Latest vLLM routes vllm.engine.llm_engine directly to the V1 engine, so this remains a no-op compatibility value. | True | True | None | No SM70 path association |
VLLM_USE_V2_MODEL_RUNNER |
configuration | Flag to control the v2 model runner. If unset, use config defaults. | None | Computed when unset: maybe_convert_bool(None) | None | No SM70 path association |
VLLM_V1_OUTPUT_PROC_CHUNK_SIZE |
tuning | Controls the maximum number of requests to handle in a single asyncio task when processing per-token outputs in the V1 AsyncLLM interface. It is applicable when handling a high concurrency of streaming requests. Setting this too high can result in a higher variance of inter-message latencies. Setting it too low can negatively impact TTFT and overall throughput. | 128 | 128 | None | No SM70 path association |
VLLM_V1_USE_OUTLINES_CACHE |
configuration | Whether to turn on the outlines cache for V1 This cache is unbounded and on disk, so it's not safe to use in an environment with potentially malicious users. | False | False | None | No SM70 path association |
VLLM_V2_CUDAGRAPH_MEM_MIB |
configuration | Explicit CUDA graph memory reservation for the V2 GPU model runner, which does not profile graph memory yet. The default preserves upstream behavior. Set this from a measured capture result when sizing a tight KV cache pool. | 0.0 | 0.0 | None | No SM70 path association |
VLLM_VIDEO_FETCH_TIMEOUT |
configuration | Timeout for fetching videos when serving multimodal models Default is 30 seconds | 30 | 30 | None | No SM70 path association |
VLLM_VIDEO_LOADER_BACKEND |
configuration | Backend for Video IO — selects the frame-sampling algorithm. - "opencv": uniform sampling. - "opencv_dynamic": duration-aware dynamic sampling. - "identity": returns raw video bytes for model processor to handle. Custom backend implementations can be registered via @VIDEO_LOADER_REGISTRY.register("my_custom_video_loader") and imported at runtime. If a non-existing backend is used, an AssertionError will be thrown. |
'opencv' | 'opencv' | None | No SM70 path association |
VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY |
configuration | Disable using pytorch's pin memory for CPU offloading. | False | False | None | No SM70 path association |
VLLM_WEIGHT_OFFLOADING_DISABLE_UVA |
configuration | Disable using UVA (Unified Virtual Addressing) for CPU offloading. | False | False | None | No SM70 path association |
VLLM_WORKER_MULTIPROC_METHOD |
configuration | Use dedicated multiprocess context for workers. Both spawn and fork work | 'fork' | 'fork' | None | No SM70 path association |
VLLM_XGRAMMAR_CACHE_MB |
configuration | Control the cache sized used by the xgrammar compiler. The default of 512 MB should be enough for roughly 1000 JSON schemas. It can be changed with this variable if needed for some reason. | 512 | 512 | None | No SM70 path association |
VLLM_XLA_CACHE_PATH |
configuration | Path to the XLA persistent cache directory. Only used for XLA devices such as TPUs. | os.path.join(VLLM_CACHE_ROOT, 'xla_cache') | Computed when unset: os.path.expanduser(os.path.join(get_default_cache_root(), 'vllm', 'xla_cache')) | None | No SM70 path association |
VLLM_XLA_CHECK_RECOMPILATION |
configuration | If set, assert on XLA recompilation after each execution step. | False | False | None | No SM70 path association |
VLLM_XLA_USE_SPMD |
configuration | Enable SPMD mode for TPU backend. | False | False | None | No SM70 path association |
VLLM_XPU_ENABLE_XPU_GRAPH |
configuration | Whether enable XPU graph on Intel GPU | False | False | None | No SM70 path association |
VLLM_XPU_USE_SAMPLER_KERNEL |
configuration | whether use xpu specific sample kernel | True | True | None | No SM70 path association |
VLLM_ZENTORCH_WEIGHT_PREPACK |
configuration | (Zen CPU backend) eagerly prepack weights into ZenDNN blocked layout at model load time. Eliminates per-inference layout conversion overhead. | True | True | None | No SM70 path association |
V_SCALE_CONSTANT |
configuration | Divisor for dynamic value scale factor calculation for FP8 KV Cache | 100 | 100 | None | No SM70 path association |
Retained deprecated inputs
Explicit legacy inputs warn once per process and name. Deprecating an alias does not deprecate its implementation. Experiments retain their explicit entry; evidence applies only to the documented workload.
| Variable | Kind | Reason | Evidence | Replacement |
|---|---|---|---|---|
VLLM_FLASHINFER_MOE_BACKEND |
alias | The retained backend selector now accepts explicit kernel configuration; this input remains an initialization compatibility alias | vllm/config/kernel.py | pass --moe-backend flashinfer_{trtllm,cutlass,cutedsl} directly |
VLLM_MXFP4_USE_MARLIN |
alias | The retained backend selector now accepts explicit kernel configuration; this input remains an initialization compatibility alias | vllm/config/kernel.py | use --moe-backend marlin (MoE) or --linear-backend marlin (linear) instead |
VLLM_NVFP4_GEMM_BACKEND |
alias | The retained backend selector now accepts explicit kernel configuration; this input remains an initialization compatibility alias | vllm/config/kernel.py | use --linear-backend instead |
VLLM_USE_FBGEMM |
alias | The retained backend selector now accepts explicit kernel configuration; this input remains an initialization compatibility alias | vllm/config/kernel.py | use --linear-backend fbgemm instead |
VLLM_USE_FLASHINFER_MOE_FP16 |
alias | The retained backend selector now accepts explicit kernel configuration; this input remains an initialization compatibility alias | vllm/config/kernel.py | use --moe-backend to select a kernel explicitly |
VLLM_USE_FLASHINFER_MOE_FP4 |
alias | The retained backend selector now accepts explicit kernel configuration; this input remains an initialization compatibility alias | vllm/config/kernel.py | use --moe-backend to select a kernel explicitly |
VLLM_USE_FLASHINFER_MOE_FP8 |
alias | The retained backend selector now accepts explicit kernel configuration; this input remains an initialization compatibility alias | vllm/config/kernel.py | use --moe-backend to select a kernel explicitly |
VLLM_USE_FLASHINFER_MOE_MXFP4_BF16 |
alias | The retained backend selector now accepts explicit kernel configuration; this input remains an initialization compatibility alias | vllm/config/kernel.py | use --moe-backend to select a kernel explicitly |
VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8 |
alias | The retained backend selector now accepts explicit kernel configuration; this input remains an initialization compatibility alias | vllm/config/kernel.py | use --moe-backend flashinfer_trtllm combined with --quantization_config.moe.activation mxfp8 |
VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8_CUTLASS |
alias | The retained backend selector now accepts explicit kernel configuration; this input remains an initialization compatibility alias | vllm/config/kernel.py | use --moe-backend flashinfer_cutlass combined with --quantization_config.moe.activation mxfp8 |
VLLM_USE_NVFP4_CT_EMULATIONS |
alias | The retained backend selector now accepts explicit kernel configuration; this input remains an initialization compatibility alias | vllm/config/kernel.py | use --linear-backend emulation instead |