LLM Model Structure & Model Files

The structure and files of large language models can be divided into four levels:

Model Architecture
├── Transformer
├── Attention / MLP
├── Position Encoding
└── KV Cache

Model Weights
├── FP32 / FP16 / BF16
└── INT8 / INT4 Quantization

Model Files
├── config.json
├── *.safetensors
├── tokenizer.json
├── tokenizer_config.json
└── generation_config.json

Model Formats
├── Hugging Face Model Directory
└── GGUF

1. Transformer architecture

Modern mainstream generative text LLM mostly uses Decoder-only Transformer.

Input Text
    ↓
Tokenizer → Token IDs
    ↓
Token Embedding
    ↓
Transformer Blocks × N
    ├── RMSNorm / LayerNorm
    ├── Causal Self-Attention
    │   ├── Q / K / V
    │   ├── RoPE
    │   └── KV Cache（inference stage）
    ├── Residual Connection
    ├── MLP / FFN
    └── Residual Connection
    ↓
Final Normalization
    ↓
LM Head
    ↓
Logits
    ↓
Sampling / Decoding Strategy
    ↓
Next Token

Different architectures have differences in normalized position, MLP structure, Attention design, etc.

Attention

Attention allows Token to obtain information about other Token based on correlation.

• Query (Q): What information is currently Token looking for?
• Key (K): What queries can the current Token match?
• Value (V): What does Token currently have to offer?

Core formula:

$$\operatorname{Attention}(Q,K,V) =\operatorname{softmax}\left(QK^T d_k^{-1/2}+M\right)V$$

Among them:

• (d_k): Key vector dimensions.
• (M)：Attention Mask。
• Causal Mask: Block Token and focus on future Token.

MLP / FFN

Attention is mainly responsible for information exchange between Token.

MLP is mainly responsible for non-linear transformation of the features of each Token.

Modern LLM Common MLP structures include gated forms such as SwiGLU.

Residual Connection

Residual connections add module inputs to module outputs, improving the training and information transfer of deep networks.

RoPE: Rotary position encoding

Transformer's Attention itself does not directly know the position of Token.

RoPE (Rotary Position Embedding) encodes position information into Q/K, allowing Attention to perceive relative position relationships.

RoPE is closely related to the following concepts:

• Context Window。
• Long context inference.
• RoPE Scaling。
• The position of Key in KV Cache.

RoPE is not an independent model weight file, but a position encoding mechanism defined in the model architecture. The specific implementation and parameters are determined by the model configuration.

2. KV Cache

KV Cache is used to reuse Key and Value that have been calculated before Token to avoid recalculating the entire context during autoregressive generation.

Prefill

Input Prompt
    ↓
Process Prompt Tokens
    ↓
Calculate Attention
    ↓
Generate K / V for Each Layer
    ↓
Store KV Cache
    ↓
Produce Next-token Logits

Prefill typically processes the entire input Prompt and therefore contains more parallel matrix computations.

Decode

Current Token
    ↓
Calculate New Q / K / V
    ↓
Read Historical KV Cache
    ↓
Calculate Attention
    ↓
Append New K / V
    ↓
Generate Next Token

Decode usually only handles a new Token per round, but requires access to the historical context corresponding to K/V.

Therefore:

• Prefill is usually very computationally intensive.
• Decode is often affected by memory bandwidth, KV Cache access, and batch processing conditions.
• KV Cache takes up more memory as the context length grows.

Q is generally not cached because subsequent generation only requires new Query and historical Key/Value to calculate Attention.

3. MHA、MQA、GQA

The main difference between the three is the quantitative relationship between Query Head and KV Head.

GQA is a common compromise found in modern LLM.

Advantages:

• Reduce KV Cache occupancy.
• Reduce the amount of KV data read.
• Inference efficiency can often be improved at a relatively small cost in quality.

For example, 32 Q Heads and 8 KV Heads. Compared with MHA of the same Head dimension, KV Cache can theoretically be reduced to a quarter of the original size.

4. Model accuracy and weight

Common numerical formats:

Floating point structure:

FP32  1 sign + 8 exponent + 23 fraction
FP16  1 sign + 5 exponent + 10 fraction
BF16  1 sign + 8 exponent + 7 fraction

BF16 has a larger dynamic range but lower mantissa precision than FP16.

Model weight estimation

Theoretically:

$$\text{Weight Memory} \approx \text{Parameter Count}\times \text{Bytes per Parameter}$$

For example, 7B parameter model:

The above uses decimal GB, which is only the theoretical size of the weight.

Actual inference memory also needs to be considered:

Total Memory
≈ Model Weights
+ KV Cache
+ Temporary Activations
+ Runtime Buffers
+ Framework Overhead

Therefore, a 7B INT4 model does not necessarily mean that it will function properly with only 3.5 GB memory.

5. Quantization: quantization

Quantization is to map higher-precision values to lower-precision representations to reduce model memory usage and data transmission costs.

For example:

Weights:       INT4
Activations:   FP16 / BF16
KV Cache:      FP16 / BF16
Accumulation:  FP16 / FP32 Wait

Quantization does not mean that all calculations in the model use INT4.

Actual operation accuracy depends on the inference engine, GPU Kernel, and quantization scheme.

Scale and Zero Point

Common linear quantization can be expressed as:

$$x \approx s(q-z)$$

Among them:

• (x): Raw floating point number.
• (q): quantized integer.
• (s)：Scale。
• (z)：Zero Point。

These auxiliary parameters also need to be stored, so the actual low-bit model size is usually larger than the theoretical number of parameters × bit width.

quantization granularity

Per-group Quantization can reduce the impact of extreme values on other weights, but will increase metadata and implementation complexity.

PTQ and QAT

PTQ（Post-Training Quantization）

Quantize after the model training is completed, and there is no need to re-train the entire model.

QAT（Quantization-Aware Training）

Quantization errors are simulated during training to allow the model to adapt to lower-precision representations; the training update process typically still preserves high-precision parameters and gradients.

Common quantization algorithms

GPTQ, AWQ, and SmoothQuant all belong to the typical PTQ technical route, but the applicable quantization schemes are not exactly the same.

It is also necessary to distinguish:

• Weight-only Quantization: Mainly quantified weight.
• Weight + Activation Quantization: Both weights and activation values ​​are quantized.
• KV Cache Quantization: Reduce the storage precision of KV Cache alone.

6. Hugging Face model directory

Typical model catalog:

model/
├── config.json
├── model.safetensors
├── model.safetensors.index.json
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json
└── generation_config.json

The actual model may not contain all files. Large models often split the weights into multiple .safetensors files.

config.json

Describe the model architecture and structural parameters.

Example (for illustration only, actual fields vary by model):

{
  "architectures": ["LlamaForCausalLM"],
  "hidden_size": 4096,
  "intermediate_size": 14336,
  "num_hidden_layers": 32,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "vocab_size": 128256,
  "max_position_embeddings": 131072,
  "torch_dtype": "bfloat16"
}

Important fields:

max_position_embeddings is not necessarily equivalent to the context length that can be reliably used when the model is actually deployed, and is also affected by the model training and inference engine configuration.

*.safetensors

Store the Tensor weights of the model, for example:

model.embed_tokens.weight
model.layers.0.self_attn.q_proj.weight
model.layers.0.self_attn.k_proj.weight
model.layers.0.self_attn.v_proj.weight
model.layers.0.self_attn.o_proj.weight
model.layers.0.mlp.gate_proj.weight
model.norm.weight
lm_head.weight

safetensors is mainly used to save Tensor safely and efficiently, rather than defining the entire model architecture.

model.safetensors.index.json

Large model weights can be stored in shards:

model-00001-of-00004.safetensors
model-00002-of-00004.safetensors
model-00003-of-00004.safetensors
model-00004-of-00004.safetensors

The Index file records the shards corresponding to each weight Tensor:

{
  "metadata": {
    "total_size": 14000000000
  },
  "weight_map": {
    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors"
  }
}

7. Tokenizer

Tokenizer is responsible for converting text to Token ID and decoding Token ID to text.

Text
  ↓
Tokenizer
  ↓
Token IDs
  ↓
Embedding
  ↓
Transformer

Common word segmentation algorithms:

• BPE。
• Unigram。
• WordPiece。

tokenizer.json can contain:

Vocabulary
Normalizer
Pre-tokenizer
Tokenization Model
Post-processor
Decoder

Tokenizer is strongly related to the model vocabulary and Embedding and cannot be replaced at will.

tokenizer_config.json

Describes the usage configuration of Tokenizer, which may include Chat Template.

{
  "model_max_length": 131072,
  "padding_side": "left",
  "chat_template": "..."
}

Chat Template

Chat Template Converts message roles and content into the input format used when training the model.

For example:

<|im_start|>system
You are a helpful assistant.
<|im_end|>
<|im_start|>user
Hello
<|im_end|>
<|im_start|>assistant

Actual templates vary by model.

Chat Template is very important: even if the weights and Tokenizer are correct, the model output quality may degrade if the input message format does not match.

8. Special Tokens

Common Special Tokens:

For example:

<s>
</s>
<|im_start|>
<|im_end|>
<|eot_id|>

Different models use different Special Token systems, and it cannot be assumed that these markers are universal across models.

9. generation_config.json

Used to configure the default parameters when the model generates text.

For example:

{
  "do_sample": true,
  "temperature": 0.7,
  "top_p": 0.9,
  "eos_token_id": 2
}

Common parameters:

Basic process:

LM Head
    ↓
Logits
    ↓
Logits Processing
    ↓
Sampling / Greedy Decoding
    ↓
Next Token ID

The generated parameters will not change the model weights, but only affect the selection strategy of Token.

10. GGUF and Safetensors

GGUF is the model file format used by the llama.cpp / ggml ecosystem and is often used for local quantized inference.

GGUF quantization type

You may see in the llama.cpp ecosystem:

F16
Q8_0
Q4_0
Q4_K_M
Q5_K_M

These names represent different storage and quantization schemes.

For example, Q4_K_M does not mean that all parameters are stored strictly as 4 bit. The actual average bit width and quantization grouping method depend on the format definition.

Ollama

Ollama's local inference capabilities are largely built on the llama.cpp related technology ecosystem, and common models use GGUF or compatible model resources.

But it needs to be distinguished:

Model Architecture
        ↓
Model Weights
        ↓
Model File Format
        ↓
Inference Engine
        ↓
Model Serving API

For example:

Llama Architecture
        ↓
Quantized Weights
        ↓
GGUF
        ↓
llama.cpp / Ollama
        ↓
HTTP API

GGUF is the file format, llama.cpp is the inference implementation, and Ollama provides model management and service capabilities.

Summary

Understand the model structure and files, mainly master:

The most important relationship:

config.json       → Model structure
*.safetensors     → Model weight
tokenizer.json    → Text and Token ID conversion
tokenizer_config  → Tokenizer behavior and Chat Template
generation_config → Default build parameters
GGUF              → Another common inference model file format

Once you understand these, you can further investigate memory usage, model loading, inference engine, and deployment performance.

---

Reference documentation

• Hugging Face Transformers
• Hugging Face KV Cache
• Hugging Face Quantization
• Safetensors
• GGUF Format
• llama.cpp