Inference: Model inference principles and performance

LLM Inference is the process in which the model uses the weights that have been trained to autoregressively generate the output Token based on the input Token.

Core concepts:

Inference
├── Execution
│   ├── Prefill
│   ├── Decode
│   └── KV Cache
├── Scheduling
│   ├── Batching
│   ├── Continuous Batching
│   └── Chunked Prefill
├── Memory
│   ├── Model Weights
│   ├── KV Cache
│   └── PagedAttention
└── Performance
    ├── TTFT
    ├── TPOT / ITL
    ├── Latency
    └── Throughput

1. Prefill: Process input Prompt

Prefill is the initial stage of inference, responsible for processing the input Prompt and building KV Cache.

For example, the input contains 2000 Token:

Prompt（2000 Tokens）
          ↓
       Prefill
          ├── Token Embedding
          ├── Transformer Layers
          ├── Compute Q / K / V
          ├── Build KV Cache
          └── Generate First-token Logits
          ↓
       First Token

Since all Token of Prompt already exist, the model can compute representations of multiple positions in parallel via Causal Mask.

Main features:

• Input Token usually has a larger number.
• There are many large-scale matrix operations.
• GPU has high parallelism.
• Usually prefer Compute-bound.

Compute-bound

Indicates that the execution speed is mainly limited by the computing power of GPU.

For example:

GPU FLOPS is the main bottleneck
Increase computing power
       ↓
possible improvement Prefill speed

Actual performance however also depends on sequence length, Batch Size, Attention implementation and memory access.

Prefill does not necessarily have to be processed completely at once Prompt, it can also be executed in chunks using Chunked Prefill.

2. Decode: generated one by one Token

After Prefill produces the first output Token, the model continues to progressively generate the remaining Token.

Autoregressive generation requires that the next Token depends on the previously generated Token.

Previous Generated Token
           ↓
    Compute New Q/K/V
           ↓
  Read Historical KV Cache
           ↓
       Attention
           ↓
          MLP
           ↓
         Logits
           ↓
        Sampling
           ↓
      Next Token

Each Decode Step usually produces only one new Token for each sequence being generated.

Memory-bandwidth-bound

Decode requires repeatedly reading the model weights and accessing KV Cache which grows with the context.

When Batch is small, the amount of calculation per round is relatively low, and GPU often cannot fully utilize its computing power.

Therefore Decode is generally more susceptible to memory bandwidth limitations.

Decode
   ↓
Read Model Weights
Read KV Cache
   ↓
Compute Next Token

As Batch increases, multiple requests can share the model weight reads in a batch, improving compute utilization.

So Decode is not always Memory-bound, and larger Batch or long contexts may change the performance bottleneck.

Prefill vs. Decode

3. KV Cache: Cache Attention status

KV Cache saves the processed Token in Key and Value in each Transformer Layer.

The effect is to avoid recalculating K/V of all historical Token every time a new Token is generated.

Prefill
   ↓
K/V for Prompt Tokens
   ↓
KV Cache
   ↓
Decode Step 1 → Append K/V
   ↓
Decode Step 2 → Append K/V
   ↓
Decode Step 3 → Append K/V

KV Cache memory estimate

For the common Decoder-only Transformer:

$$M*{\mathrm{KV}} \approx 2 \times L \times H*{kv} \times D \times S \times B \times P$$

Among them:

For example:

32 Layers
8 KV Heads
128 Head Dimension
4096 Cached Tokens
1 Sequence
BF16 = 2 Bytes

The theoretical KV Cache size is approximately 512 MiB.

This is an estimate that does not take into account paging allocation, memory alignment, shared prefixes, and other runtime overhead.

Important conclusions:

• The larger Context Length is, the larger KV Cache is usually.
• The more sequences in Batch, the larger the total KV Cache is usually.
• GQA/MQA can reduce the size of KV Cache.
• KV Cache Quantization can further reduce memory overhead.

4. Context Length

Context Length is the upper limit of the number of context Token that the model is allowed to use during the current inference process.

Usually:

$$\text{Input Tokens}+\text{Generated Tokens}\leq\text{Context Limit}$$

Need to pay attention to:

• The range of theoretical positions of a model architecture is not necessarily equal to the length of contexts that can be reliably supported.
• The inference engine may set max_model_len smaller.
• Long contexts typically increase KV Cache occupancy and Attention overhead.
• In practice, it may also be limited by model accuracy, RoPE configuration, and GPU memory.

Context Length affects the context size that can be used by a single request, while Batch Size affects the number of requests processed simultaneously. The two together determine the pressure of KV Cache.

5. Batching: batch processing

Batching refers to combining multiple requests to perform calculations on GPU.

For example:

Request A ──┐
Request B ──┼── GPU Batch
Request C ──┘

Advantages:

• Improve GPU utilization.
• Amortize model weight read overhead.
• Improve overall server throughput.

Disadvantages:

• May increase queues.
• Consumes more KV Cache memory.
• When Batch is too large, it may reduce the response speed of a single request.

Static Batching

Traditional static Batch usually requires requests within a Batch to be scheduled as a group.

The input and output lengths of different requests may be different, which may easily result in Padding or waste caused by waiting.

Continuous Batching

Continuous Batching (also In-flight Batching) allows requests to dynamically join or leave an executing batch.

Step 1: A B C
Step 2: A B C
Step 3: A   C     ← B Complete
Step 4: A D C     ← D Join
Step 5:   D C     ← A Complete

Core values:

• Requests do not need to come in at the same time.
• Completed requests release resources immediately.
• New requests can be added in subsequent scheduling steps.
• Improve GPU utilization and service throughput.

Continuous Batching is an important scheduling mechanism for modern inference engines such as vLLM and SGLang.

6. Chunked Prefill

Prefill with a long Prompt may occupy a large amount of GPU computing resources, thus affecting the request being Decode.

Chunked Prefill splits the long Prompt into multiple Token Chunk for processing.

For example:

Prompt: 4096 Tokens

Chunk 1: 1024
Chunk 2: 1024
Chunk 3: 1024
Chunk 4: 1024

This allows the scheduler to schedule Prefill and Decode jobs across multiple scheduling steps.

GPU Scheduling
       ↓
Decode Batch
       ↓
Prefill Chunk
       ↓
Decode Batch
       ↓
Prefill Chunk

Main purpose:

• Avoid long Prefill from occupying computing resources for a long time.
• Reduce impact on other requests TPOT.
• Provides more flexible throughput and latency trade-offs.

However, blocking may also increase the time for a single Prompt to complete Prefill, so the scheduling strategy needs to be configured appropriately.

Parameters such as max_num_batched_tokens in vLLM can affect the Token budget of each scheduling step.

7. PagedAttention

KV Cache grows dynamically with the number of Token.

If a contiguous large block of video memory is reserved for each request, memory waste and fragmentation are prone to occur.

PagedAttention divides KV Cache into fixed-size Block and manages them through mapping relationships.

Request A
Logical Blocks
  [0] [1] [2]
   ↓   ↓   ↓
Physical KV Blocks
  [7] [3] [9]

Request B
Logical Blocks
  [0] [1]
   ↓   ↓
Physical KV Blocks
  [2] [5]

Main advantages:

• KV Cache does not require a complete contiguous physical space to be reserved for each request.
• Reduce video memory fragmentation and reservation waste.
• More flexibility in allocating and recycling KV Cache.
• Certain KV Block sharing policies can be supported.

PagedAttention mainly solves the memory management problem of KV Cache, while Continuous Batching mainly solves the request scheduling problem.

The combination of the two can improve the concurrency capability of the inference service.

8. Inference performance indicators

TTFT

The time from when the request is made until the client receives the first output Token.

Request
   ↓
Queue
   ↓
Prefill
   ↓
First Token

TTFT contains not just GPU Prefill time, but may also include network, request queuing, Tokenization, and first output transfer overhead.

TPOT / ITL

TPOT usually represents the average time required for each subsequent Token in the build phase.

Common calculation methods:

$$TPOT=\frac{Latency-TTFT}{N\_{\mathrm{output}}-1}$$

Among them math\(N\_{\mathrm{output}}>1\)math.

For example:

TTFT = 1.0 s
Total Latency = 5.0 s
Output Tokens = 101

Then:

TPOT = 40 ms/token

The corresponding average subsequent generation speed is about 25 tokens/s.

Note: Different Benchmark tools may have different statistical methods for TPOT, ITL, first Token and streaming Chunk. You need to confirm the definition when comparing results.

The end-to-end time from start to completion of the request.

Request Start
      ↓
Queue + Prefill + Decode + Delivery
      ↓
Request Complete

For streaming builds, the total latency of a request can typically be broken down into:

$$Latency \approx TTFT+(N\_{\mathrm{output}}-1)\times TPOT$$

Throughput

The workload processed by the server per unit time.

Common indicators:

Requests/s
Input Tokens/s
Output Tokens/s
Total Tokens/s

where Output Tokens/s is typically used to measure generation throughput.

single user Tokens/s and the server as a whole Tokens/s are not the same indicator.

For example:

• Single user speed: 30 tokens/s.
• The server handles 20 requests simultaneously.
• The overall server generated throughput is probably much higher than 30 tokens/s.

But overall throughput does not simply grow linearly with the number of concurrent requests.

9. Trade-off between Latency and Throughput

Inference services typically pursue both:

• Lower TTFT.
• Lower TPOT.
• Higher Throughput.
• Better GPU utilization.

But these goals may conflict with each other.

Smaller Batch
     ↓
Typically low queuing and scheduling overhead
     ↓
Lower overall throughput risk

Larger Batch
     ↓
higher GPU Utilization
     ↓
Higher overall throughput
     ↓
May increase single request latency

For example, increase concurrency:

• GPU Utilization may increase.
• Total Output Tokens/s may rise.
• Queue Time may also rise.
• TTFT and TPOT may worsen.

So you can't just focus on Tokens/s.

P50 / P95 / P99

Performance benchmarks typically report latency quantiles.

• P50: 50% of requests have latency no greater than this value.
• P95: 95% of requests have latency no greater than this value.
• P99: 99% of requests have latency no greater than this value.

Lower average latency does not mean that there are no significant tail latencies under high load.

10. Inference performance optimization ideas

Prefix Caching

If multiple requests have the same Prompt prefix, the already calculated KV Cache Block can be reused when the conditions are met.

For example:

Request A:
[System Prompt] + [Question A]

Request B:
[System Prompt] + [Question B]

If the system can reuse the common System Prompt prefix, it can reduce repeated Prefill calculations.

However, the benefits of Prefix Caching depend on the prefix repetition rate, cache hits, and memory capacity, and there is no guarantee that all requests will be significantly accelerated.

Summary

LLM inference can be understood with the following structure:

Incoming Requests
        ↓
Request Queue / Scheduler
        ↓
Continuous Batching
        │
        ├── Prefill
        │    └── Build KV Cache
        │
        └── Decode
             ├── Read KV Cache
             ├── Generate Token
             └── Extend KV Cache
        ↓
Streaming Response

Relationships that need to be mastered:

During actual performance testing, the model, input and output Token length, hardware and concurrency conditions should be fixed, otherwise different test results may not be comparable.

---

Reference documentation

• vLLM Documentation
• vLLM Scheduler Configuration
• NVIDIA LLM Inference Benchmarking
• NVIDIA Inference Metrics
• Hugging Face KV Cache