GPU Hardware & Computing Stack

LLM inference performance not only depends on the number of model parameters, but also depends on GPU computing power, memory capacity, memory bandwidth, and the underlying computing software stack.

Core concepts:

Hardware
├── CPU
├── GPU
├── RAM / VRAM
├── Unified Memory
├── Memory Bandwidth
└── PCIe / NVLink

GPU Computing Stack
├── CUDA（NVIDIA）
├── ROCm（AMD）
└── Metal（Apple）

1. CPU, GPU and memory

CPU and GPU

RAM and VRAM

• RAM: System memory mainly used by CPU and the operating system.
• VRAM: The video memory used by GPU usually has high memory bandwidth.

Typical independent graphics card architecture:

CPU ─── System RAM
 │
 │ PCIe / NVLink
 │
GPU ─── VRAM

Data transmission between CPU and GPU requires an interconnection link.

GPU local memory bandwidth is usually much higher than PCIe transmission bandwidth. Therefore, performance may degrade significantly if model weights are repeatedly transferred from RAM to VRAM during inference.

2. GPU Memory Bandwidth

Memory Bandwidth represents the amount of data that the GPU memory system can transmit per unit time, usually expressed as GB/s or TB/s.

This is one of the key indicators of LLM Decode performance.

For example:

Model Weights
      ↓
Read from VRAM
      ↓
GPU Compute
      ↓
Generate Token

For Batch, which is smaller and is mainly limited by memory bandwidth, Decode can be approximately understood as:

$$\text{Tokens/s} \lesssim \frac{\text{Effective Memory Bandwidth}} {\text{Bytes Read per Decode Step}}$$

This is just a simplified model for understanding Memory-bound, not an exact formula for actual performance.

FLOPS and Memory Bandwidth

The video memory capacity determines whether the model is easy to install; the video memory bandwidth affects the model running speed. The two cannot be confused.

3. Unified Memory

Apple Silicon adopts Unified Memory Architecture (UMA).

Apple Silicon
├── CPU
├── GPU
├── Neural Engine
└── Unified Memory
       ↑
   Shared Memory Pool

CPU and GPU have access to the same physical memory pool, eliminating the need to copy all data between completely separate RAM and VRAM like traditional discrete graphics cards.

Advantages

• Reduce data copying between CPU/GPU.
• More flexible memory allocation.
• Allows GPU to run the model using a larger proportion of unified memory.

Limit

• CPU, GPU and other system components share the total capacity.
• GPU Actual allocable memory is still limited by the system.
• Large memory capacity does not mean GPU calculation speed is fast.
• The level of support for the inference engine and GPU Kernel also affects performance.

For example, 16 GB Unified Memory of MacBook does not mean that there are 16 GB that are completely independent and all available for the model VRAM.

It is also necessary to distinguish Unified Memory of Apple Silicon from Unified Memory of NVIDIA CUDA: the two names are similar, but the underlying hardware architecture and memory migration mechanism are not necessarily the same.

4. CUDA: NVIDIA GPU computing platform

CUDA is the GPU parallel computing platform and programming ecosystem of NVIDIA.

NVIDIA GPU
     ↑
CUDA Driver / Runtime
     ↑
CUDA Libraries / Kernels
     ↑
PyTorch / vLLM / TensorRT-LLM
     ↑
LLM Application

CUDA ecology includes:

GPU Kernel

Kernel is a calculation program executed on GPU.

For example in Transformer:

• Matrix Multiplication。
• Attention。
• RMSNorm。
• Softmax。
• Quantized Matrix Multiplication。

For the same model and GPU, using different GPU Kernel, the inference performance may be significantly different.

FlashAttention is an important algorithm and Kernel implementation series that optimizes Attention calculation and memory access.

CUDA Stream

CUDA supports asynchronous execution of GPU tasks.

Stream is used to arrange the execution sequence of GPU operations, and to achieve overlapping of partial calculations and data transmission when dependencies and hardware conditions are met.

It is not equal to CPU Thread.

5. ROCm: AMD GPU computing platform

ROCm is the GPU computing software ecosystem of AMD.

AMD GPU
    ↑
ROCm / HIP
    ↑
PyTorch / Supported Inference Engines
    ↑
LLM Application

Among them, HIP provides the GPU programming interface that is similar to the CUDA programming model.

The ROCm ecosystem includes corresponding computing libraries, compilation tool chains, and GPU communication capabilities.

Need to pay attention to:

• Not all AMD GPU receive the same software support.
• Whether PyTorch, vLLM and other tools are available, you need to check the specific GPU model and version compatibility.
• CUDA Kernel generally cannot run directly on AMD GPU without adaptation.

6. Metal: Apple GPU calculation API

Metal is the underlying graph of Apple and GPU computes API.

Apple GPU
    ↑
Metal / Metal Performance Shaders
    ↑
MLX / PyTorch MPS / llama.cpp
    ↑
LLM Application

Related concepts:

PyTorch example:

import torch

device = "mps" if torch.backends.mps.is_available() else "cpu"

x = torch.randn(1024, 1024, device=device)
y = x @ x

where mps is PyTorch and Device is used to access the Apple Metal GPU acceleration capabilities.

7. GPU Computing Stack comparison

Note: PyTorch ROCm builds usually still reuse the torch.cuda API naming, rather than using torch.rocm Device.

8. OOM and model memory usage

OOM (Out of Memory) indicates insufficient memory.

LLM Inference requires more memory than just model weights:

$$\text{Memory Usage} \approx \text{Weights} +\text{KV Cache} +\text{Activations} +\text{Runtime Overhead}$$

Common OOM causes:

• The model weight is too large.
• Context Length is too long.
• Too many concurrent requests.
• KV Cache continues to grow.
• Runtime temporary memory requirements are too high.

Common processing methods:

1. Use a lower precision quantized model.
2. Decrease Context Length.
3. Reduce the maximum number of concurrencies.
4. Use more efficient KV Cache management or quantization.
5. Use GPU with larger video memory capacity.

Some inference engines can offload weights to CPU RAM if the model exceeds available video memory, but this may be significantly slowed down by cross-device data transfer.

Summary

LLM Inference hardware mainly needs to understand three dimensions:

Software layer:

LLM Framework / Inference Engine
               ↓
CUDA / ROCm / Metal Backend
               ↓
GPU Kernels
               ↓
GPU Hardware

For actual deployment:

• NVIDIA GPU: Focus on CUDA, VRAM, Memory Bandwidth.
• AMD GPU: Focus on ROCm compatibility.
• Apple Silicon: Focus on Unified Memory, Metal, MLX and llama.cpp.

core conclusion: Whether LLM can run is mainly constrained by memory capacity. How fast it runs is also affected by GPU's computing power, memory bandwidth, computing core and inference scheduling strategy.

---

Reference documentation

• NVIDIA CUDA Documentation
• AMD ROCm Documentation
• Apple Metal
• PyTorch MPS Backend
• Apple MLX