github-actions[bot]andnilbuild
a261c81b0e
Co-authored-by: nilbuild <4921183+nilbuild@users.noreply.github.com>
895 B
895 B
Memory
GPU memory determines what fits on a single device and how fast data can move through the compute units. VRAM capacity sets the upper limit on model size and KV cache; VRAM bandwidth determines how quickly the decode phase can load weights per token. Modern datacenter GPUs use high-bandwidth memory (HBM) in generations HBM3, HBM3e, and HBM4, stacked directly on the GPU die to maximize bandwidth.
Visit the following resources to learn more: