developer-roadmap/roadmaps/inference-engineering/content/multi-node-inference@FZx2xsqw_cmYLFS3GMAl-.md

Multi-Node Inference

Multi-node inference spreads a model across more than eight GPUs using InfiniBand for inter-node communication. It is necessary for the largest models where a single node lacks sufficient VRAM for weights and KV cache. InfiniBand bandwidth is an order of magnitude lower than NVLink, so parallelism strategies must minimize cross-node communication. For MoE models, Expert Parallelism is preferred over Tensor Parallelism between nodes.

Visit the following resources to learn more:

• @article@How multi-node inference works for massive LLMs like DeepSeek-R1
• @video@AI Models Are Huge, but Your GPUs Aren’t: Mastering Multi-Node Distributed Inference