Production Serving & Deployment

Production-level LLM Serving not only starts the inference engine, but also needs to ensure the security, reliability, scalability and observability of the service.

1. Overall architecture

LLM Production Serving
├── API Layer
│   ├── OpenAI-compatible API
│   ├── SSE / Streaming
│   ├── Authentication
│   └── Rate Limiting
│
├── Request Management
│   ├── Queue / Concurrency
│   ├── Timeout / Cancellation
│   ├── Load Balancing
│   └── Backpressure
│
├── Runtime & Resources
│   ├── GPU Scheduling
│   ├── Model Replicas
│   └── Autoscaling
│
└── Operations
    ├── Health Checks
    ├── Metrics / Monitoring
    ├── Logging / Tracing
    └── Deployment / Rollback

Typical architecture:

                 Internet
                    ↓
               API Gateway
         ┌──────────┼──────────┐
         │          │          │
       Auth       TLS       Rate Limit
         └──────────┼──────────┘
                    ↓
              Load Balancer
                    ↓
        ┌───────────┼───────────┐
        ↓           ↓           ↓
      vLLM-1      vLLM-2      vLLM-3
        ↓           ↓           ↓
       GPU         GPU         GPU

Metrics → Prometheus → Grafana
Logs    → Logging System
Health  → Load Balancer / Orchestrator
Scaling ← Queue / Latency / GPU Metrics

2. OpenAI-compatible API

Modern inference engines usually provide a HTTP interface compatible with the OpenAI API style.

For example:

POST /v1/chat/completions
Authorization: Bearer <api-key>
Content-Type: application/json

Request:

{
  "model": "qwen",
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    }
  ],
  "stream": false
}

The response usually contains:

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Hello!"
      }
    }
  ]
}

Common Endpoint:

OpenAI-compatible does not imply consistent support for Tool Calling, Structured Outputs, and Reasoning across all engines.

3. SSE / Streaming

LLM typically uses SSE style HTTP Streaming to incrementally send the generated content to the client.

Client Request
      ↓
Server Prefill
      ↓
First Token
      ↓
Token 1
      ↓
Token 2
      ↓
Token 3
      ↓
Stream Complete

Request:

{
  "model": "qwen",
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    }
  ],
  "stream": true
}

Responses typically use:

Content-Type: text/event-stream

Example:

data: {"choices":[{"delta":{"content":"Hello"}}]}

data: {"choices":[{"delta":{"content":" world"}}]}

data: [DONE]

What is transmitted here is the incremental message Chunk. There is no guarantee that one Chunk exactly corresponds to one Token.

Nginx Streaming

location /v1/ {
    proxy_pass http://vllm:8000;

    proxy_http_version 1.1;
    proxy_buffering off;
    proxy_read_timeout 300s;
}

Key configuration:

• proxy_buffering off: Avoid Nginx buffering generated content.
• proxy_read_timeout: Limit the wait time between two consecutive upstream reads.
• proxy_http_version 1.1: Use HTTP/1.1 proxy upstream.

Streaming requests also need to handle the situation where the client actively disconnects to avoid that the user has canceled the generation and GPU still continues to work.

4. Authentication and Rate Limiting

Inference resources are usually expensive, and unprotected model services cannot be directly exposed to the public network.

Authentication

Common ways:

• API Key。
• JWT。
• OAuth / OIDC。
• Internal service authentication.

For example:

Authorization: Bearer <api-key>

Rate Limiting

Can limit:

For example:

User A:
60 RPM
100,000 TPM
5 Concurrent Requests

For LLM Serving, just restricting RPM may not be enough.

Because the cost of two requests can be very different:

Request A: 100 Input + 50 Output Tokens

Request B: 100K Input + 8K Output Tokens

Therefore it is best to constrain both the Token budget and the amount of concurrency.

security boundary

vLLM's --api-key only protects part of the API path, and not all service endpoints are certified. Production environments cannot rely solely on this parameter and need to protect other management endpoints through reverse proxies and network policies.

5. Queue, Concurrency and Backpressure

An inference service typically has limited GPU compute and KV Cache resources.

Incoming Requests
        ↓
   Admission Control
        ↓
      Queue
        ↓
     Scheduler
        ↓
   Running Requests
        ↓
       GPU

Concurrency

Concurrency represents the number of requests being processed at the same time.

Need to distinguish:

• API layer of In-flight Requests.
• Sequences running in the inference engine.
• Requests being queued.

These quantities are not necessarily the same.

Queue

When the request exceeds the current processing capacity of GPU, some requests need to wait.

Queue can absorb short bursts of traffic, but long queuing times will worsen TTFT.

Backpressure

Backpressure is a mechanism that actively limits the entry of new jobs when the system is overloaded.

For example:

Queue < Limit
     ↓
Accept Request

Queue >= Limit
     ↓
Reject / Throttle
     ↓
HTTP 429 / 503

Common measures:

• Limit the maximum amount of concurrency.
• Set the maximum number of queued requests.
• Limit waiting time in queues.
• Deny new requests that exceed capacity.
• Use a backoff strategy for trusted retryable errors.

The infinite increase of and Queue cannot increase the computing power of GPU, but will only continue to increase the delay.

6. Timeout and Cancellation

LLM requests often have several different timeouts.

Different API Gateway and inference engines may not natively support all types and need to be implemented according to the specific system.

Cancellation

When the user closes the page or clicks Stop Generation:

Browser Cancel
      ↓
API Gateway
      ↓
Inference Server
      ↓
Cancel Request
      ↓
Release KV Cache / Scheduler Resources

The production service should ensure that cancellation signals are delivered to the inference engine.

For Streaming requests, you need to pay special attention to the relationship between network disconnection and server-side task cleanup.

Retry

Not all failures should be retried immediately.

For example, GPU is overloaded and may form Retry Storm if a large number of clients retry at the same time.

Common practices are:

• Exponential Backoff。
• Jitter。
• Retry Limit。
• Retry only on errors that are eligible for retry.

For streaming generation that has been partially output, retrying may also result in duplicate content or additional charges, which needs to be handled with caution.

7. Load Balancing

Multi-instance inference deployment:

                 Load Balancer
                        ↓
           ┌────────────┼────────────┐
           ↓            ↓            ↓
        vLLM-1        vLLM-2        vLLM-3
           ↓            ↓            ↓
          GPU          GPU          GPU

Common strategies:

For LLM, ordinary Round Robin is not necessarily optimal.

Because the request lengths vary greatly, the actual load of two GPU instances with the same number of connections may be completely different.

Cache-aware Routing can use the similar Prompt Prefix between multiple requests to reduce repeated calculations.

However, the load balancing strategy also needs to take into account cache hit rate and load fairness.

8. GPU Scheduling and Autoscaling

GPU Scheduling

Within the inference engine, Scheduler is responsible for scheduling requests to GPU for execution.

For example:

Queue
  ↓
Continuous Batching
  ↓
Prefill / Decode
  ↓
KV Cache Allocation
  ↓
GPU Execution

This is not the same thing as Kubernetes scheduling on platforms such as GPU Pod:

• Inference engine Scheduler: schedules Token and requests.
• Container orchestration Scheduler: Schedule containers to GPU nodes.

Autoscaling

Autoscaling Increase or decrease inference instances based on load.

For example:

Incoming Traffic ↑
       ↓
Queue Length ↑
       ↓
Scale Out
       ↓
More Model Replicas

Commonly used basis for expansion and contraction:

• Queue Length。
• TTFT / TPOT。
• Requests/s。
• Tokens/s。
• GPU Memory / KV Cache Usage。
• GPU Utilization。

It is not recommended to use GPU Utilization as the only metric because GPU busyness does not necessarily reflect queuing and user experience.

Special question for LLM Autoscaling

Starting a new model instance may require:

1. Assign GPU.
2. Pull or load model weights.
3. Initialize the inference engine.
4. Assign KV Cache.
5. Complete warm-up.

So the cost of cold start for LLM instances can be high.

Scaling is not instantaneous and requires a combination of traffic prediction and a reasonable minimum number of replicas.

9. Health Check

Need to distinguish:

For example:

Model Loading
     ↓
Startup Not Ready
     ↓
Model Loaded
     ↓
Readiness Passed
     ↓
Receive Traffic

vLLM provides endpoints such as /health for service health checks.

Note: A successful health check does not mean that the instance must have enough KV Cache capacity to receive new requests.

Therefore, the production environment also needs to combine Queue and resource indicators to determine actual service capabilities.

10. Metrics, Logging and Monitoring

Metrics

Production environments typically use:

vLLM Metrics
      ↓
Prometheus
      ↓
Grafana
      ↓
Dashboards / Alerts

vLLM provides Prometheus-compatible /metrics Endpoint.

Key monitoring points:

Among them, Running Requests, KV Cache usage rate, Prefix Cache Hits, etc. are all vLLM observable indicators.

Logging

Logs are used to record specific requests and system events, such as:

Request ID
Timestamp
Model Version
Status Code
Latency
Input / Output Token Count
Error Type

Production environments should not default to fully logging user Prompt and model responses, especially if they may contain sensitive data.

Tracing

Tracing focuses on the execution path of a single request through different services.

Request ID
    ↓
API Gateway
    ↓
Application
    ↓
Retriever / Tools
    ↓
Inference Server
    ↓
Response

For RAG and Agent applications, Tracing helps determine whether the delay is from retrieval, tool invocation, queuing, or model generation.

11. Deployment, Rolling Update and Rollback

Release process

Model / Container Version
          ↓
Deployment
          ↓
Model Loading
          ↓
Readiness Check
          ↓
Accept Traffic
          ↓
Monitoring

The production environment should be clearly fixed:

• Model Revision / Weight Version。
• Tokenizer / Chat Template。
• Quantization Format。
• Inference Engine Version。
• Serving Configuration。

Merely logging the model name is usually not enough to guarantee a reproducible deployment.

Rolling Update

In the case of multiple instances, you can gradually update:

vLLM-1: Old → New
vLLM-2: Old
vLLM-3: Old

Once the new instance is ready, other old instances are gradually replaced.

Graceful Shutdown / Draining

When stopping the old instance:

Stop Accepting New Requests
            ↓
Allow In-flight Requests to Finish
            ↓
Release Resources
            ↓
Terminate Instance

For long Streaming requests, a reasonable maximum Drain time also needs to be set.

Rollback

If the new model deteriorates in quality or performs abnormally:

New Model Version
       ↓
Monitoring Detects Problem
       ↓
Rollback
       ↓
Previous Stable Version

Rollback not only involves Docker Image, but also needs to ensure that the model weight, Tokenizer and service configuration version are consistent.

12. Deployment plan selection

Kubernetes is not a necessary condition for LLM Serving. For a single GPU or a small number of instances, Docker Compose plus a reverse proxy is usually sufficient to meet initial deployment needs.

Summary

Production-level LLM Serving mainly needs to solve five problems:

Core architecture:

Client
   ↓
API Gateway
   ↓
Authentication / Rate Limit
   ↓
Load Balancer
   ↓
Inference Server
   ├── Queue / Scheduler
   ├── KV Cache Manager
   └── GPU Execution
   ↓
Streaming Response

Observability:
Metrics + Logs + Traces

Operations:
Health Check + Scaling + Deployment

The goal of production-level LLM Serving is not just to enable the model to run, but to continue to provide safe, predictable, and scalable inference services under given GPU resources.

---

Reference documentation

• vLLM Serving
• vLLM Production Metrics
• vLLM Security
• Prometheus
• Grafana
• Kubernetes Autoscaling