NCCL registers the buffers of collectives captured in a CUDA graph and
assumes each one lies in a single physical allocation. With PyTorch
expandable segments a tensor can span several, and a later graph replay
can fault inside the collective (pytorch/pytorch#158029; NVIDIA's CUDA
graph guide gives NCCL_GRAPH_REGISTER=0 as the workaround).
NCCL caches the setting on first use, so env_override, which runs on
`import vllm` before any communicator exists, now defaults it to 0
whenever the allocator configuration PyTorch selects enables expandable
segments. An explicit NCCL_GRAPH_REGISTER is kept.
On 4x V100 PCIe with TP4 and DCP2 (custom all-reduce is disabled there,
so every collective in a decode graph is NCCL), an NCCL AllGather in the
decode graph faulted with an out-of-range address in 7 of 10 server
starts; with the setting, 3 of 3 starts ran through the same step.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Leon <tomsel1010.us2@gmail.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
