A useful way to understand an LLM is to separate it into four pieces:

architecture + weights + tokenizer + runtime configuration.

An inference engine such as vLLM reads those pieces, constructs the neural network in GPU memory, loads the learned weights into it, allocates memory for inference, and then repeatedly runs the model to predict the next token.

1. Typical LLM model directory

When you download a Hugging Face model such as Gemma, Llama, Qwen, or Mistral, you will typically see something like:

my-model/
│
├── config.json
├── generation_config.json
│
├── tokenizer.json
├── tokenizer_config.json
├── special_tokens_map.json
├── chat_template.jinja
│
├── model.safetensors.index.json
│
├── model-00001-of-00004.safetensors
├── model-00002-of-00004.safetensors
├── model-00003-of-00004.safetensors
├── model-00004-of-00004.safetensors
│
├── preprocessor_config.json       # multimodal models
├── processor_config.json          # sometimes
│
├── README.md
└── LICENSE

The most important distinction is:

config.json          = what the neural network looks like
*.safetensors        = what the neural network learned
tokenizer files      = how text becomes numbers
generation_config    = default generation behavior

Hugging Face's loading mechanism uses config.json to determine the model configuration and loads the corresponding pretrained tensor state from the checkpoint files. Hugging Face

2. config.json — the model blueprint

This is essentially the architectural blueprint.

A simplified Llama-style configuration might look like:

{
  "architectures": [
    "LlamaForCausalLM"
  ],
  "hidden_size": 4096,
  "intermediate_size": 14336,
  "num_hidden_layers": 32,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "vocab_size": 128256,
  "max_position_embeddings": 131072,
  "torch_dtype": "bfloat16"
}

Those values tell vLLM what neural network it needs to construct.

For example:

num_hidden_layers = 32

means:

Token Embedding
      │
      ▼
Transformer Layer 0
      │
      ▼
Transformer Layer 1
      │
      ▼
...
      │
      ▼
Transformer Layer 31
      │
      ▼
Final Norm
      │
      ▼
LM Head
      │
      ▼
Vocabulary logits

Inside each transformer layer you generally have something resembling:

              ┌──────────────┐
              │ Input Vector │
              └──────┬───────┘
                     │
                 RMSNorm
                     │
              Self Attention
              ┌──────┼──────┐
              Q      K      V
              │      │      │
              └── Attention ┘
                     │
                Projection
                     │
                 + Residual
                     │
                 RMSNorm
                     │
                    MLP
                     │
                 + Residual
                     │
                     ▼
                 Next Layer

The configuration tells vLLM things like:

• how many layers to create
• dimensions of each matrix
• attention architecture
• number of attention heads
• number of KV heads
• vocabulary size
• positional encoding
• supported context length
• datatype such as BF16 or FP16

But config.json contains essentially no learned intelligence.

It's the blueprint.

3. .safetensors — the actual learned model

This is where most of the model's size comes from.

For example:

model-00001-of-00004.safetensors   4.8 GB
model-00002-of-00004.safetensors   4.8 GB
model-00003-of-00004.safetensors   4.8 GB
model-00004-of-00004.safetensors   3.2 GB

These files contain billions of floating-point numbers arranged into tensors.

A tensor might have a name such as:

model.layers.0.self_attn.q_proj.weight

and a shape:

4096 × 4096

Another might be:

model.layers.0.mlp.up_proj.weight

shape:
14336 × 4096

Conceptually:

model.layers.0.self_attn.q_proj.weight

[
  [ 0.031, -0.017,  0.004, ... ],
  [-0.042,  0.021, -0.018, ... ],
  [ 0.007,  0.012,  0.039, ... ],
  ...
]

There may be billions of these numeric parameters across all tensors.

That is what training created.

So when someone says:

"This is a 12-billion-parameter model."

they're essentially saying:

~12 billion learned numerical values

are stored in those weight files.

4. Why there are multiple .safetensors files

A 30B model might require 60+ GB in BF16.

It's inconvenient to have:

model.safetensors
65 GB

so Hugging Face normally splits it:

model-00001-of-00012.safetensors
model-00002-of-00012.safetensors
...
model-00012-of-00012.safetensors

Then:

model.safetensors.index.json

maps individual tensors to files.

For example:

{
  "weight_map": {
    "model.embed_tokens.weight":
        "model-00001-of-00004.safetensors",

    "model.layers.0.self_attn.q_proj.weight":
        "model-00001-of-00004.safetensors",

    "model.layers.18.mlp.down_proj.weight":
        "model-00003-of-00004.safetensors",

    "lm_head.weight":
        "model-00004-of-00004.safetensors"
  }
}

So vLLM knows exactly which file contains each tensor.

Modern vLLM's load-format=auto prefers Safetensors and falls back to PyTorch checkpoint format when Safetensors isn't available. vLLM

5. Tokenizer files

The neural network doesn't understand:

What is the capital of France?

It understands integers.

The tokenizer converts:

"What is the capital of France?"

into something conceptually like:

[1841, 374, 279, 6864, 315, 9822, 30]

You can think of this as:

"What"     → 1841
" is"      → 374
" the"     → 279
" capital" → 6864
" of"      → 315
" France"  → 9822
"?"        → 30

The actual values depend entirely on the model's tokenizer.

Files like:

tokenizer.json
tokenizer_config.json
special_tokens_map.json

define this conversion.

Special tokens may include things such as:

<BOS>
<EOS>
<PAD>
<assistant>
<user>
<system>

depending on the model.

6. Chat template

Chat models have another important component.

You might send:

{
  "role": "user",
  "content": "What is the capital of France?"
}

But the model may actually expect something more like:

<bos>
<start_of_turn>user
What is the capital of France?
<end_of_turn>
<start_of_turn>model

Gemma, Llama, Qwen, etc. use different formats.

That's what a chat template helps define.

So this request:

POST /v1/chat/completions

goes approximately:

OpenAI API messages
        │
        ▼
Chat template
        │
        ▼
Formatted prompt
        │
        ▼
Tokenizer
        │
        ▼
Token IDs

This is an important reason you should use the tokenizer/chat template associated with the model rather than arbitrarily substituting another one.

7. generation_config.json

This file contains default generation preferences.

For example:

{
  "bos_token_id": 2,
  "eos_token_id": 1,
  "temperature": 1.0,
  "top_p": 0.95,
  "top_k": 64
}

Hugging Face uses generation_config.json, when present, as a source of generation defaults. Hugging Face

You've actually seen this in your vLLM startup logs when it reported that settings such as:

temperature = 1.0
top_k       = 64
top_p       = 0.95

were coming from the model's generation_config.json.

8. Now let's start vLLM

Suppose you run:

vllm serve google/gemma-4-12b-it \
    --tensor-parallel-size 2 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90

The process looks roughly like this:

             Hugging Face Model
                     │
        ┌────────────┴────────────┐
        │                         │
    config.json              tokenizer files
        │                         │
        ▼                         ▼
Determine architecture       Load tokenizer
        │
        ▼
Create model structure
        │
        ▼
Read Safetensors index
        │
        ▼
Read weight shards
        │
        ▼
Place tensors on GPU(s)
        │
        ▼
Initialize attention backend
        │
        ▼
Allocate KV Cache
        │
        ▼
Compile / warm up kernels
        │
        ▼
Open API server
        │
        ▼
Ready for requests

Let's look more closely.

9. Step 1 — locate/download the model

If the model isn't already cached:

google/gemma-4-12b-it
       │
       ▼
Hugging Face Hub
       │
       ▼
~/.cache/huggingface/

Inside your Docker setup you're mounting:

~/.cache/huggingface
        ↓
/root/.cache/huggingface

which lets the model survive container restarts instead of downloading it again.

vLLM's loader supports a configurable download directory, defaulting to the Hugging Face cache location. vLLM

10. Step 2 — construct the empty neural network

Suppose config.json says:

32 transformer layers
4096 hidden dimension
32 attention heads
8 KV heads
14336 MLP dimension

vLLM constructs the corresponding computational structure.

Conceptually it now has:

Model
 ├─ embedding
 ├─ layer 0
 │    ├─ q_proj
 │    ├─ k_proj
 │    ├─ v_proj
 │    ├─ o_proj
 │    ├─ gate_proj
 │    ├─ up_proj
 │    └─ down_proj
 │
 ├─ layer 1
 │    └─ ...
 │
 ...
 ├─ layer 31
 │    └─ ...
 │
 ├─ final norm
 └─ lm_head

At this point, conceptually, it's just the structure.

Now the checkpoint values have to populate it.

11. Step 3 — load the weights

vLLM reads tensors such as:

model.layers.0.self_attn.q_proj.weight

from:

model-00001-of-00004.safetensors

and connects them with:

Layer 0 → Attention → Q Projection

So:

Safetensors
     │
     │ tensor:
     │ model.layers.0.self_attn.q_proj.weight
     ▼
GPU memory
     │
     ▼
Transformer Layer 0
     │
     └── q_proj.weight

This happens for every parameter.

Current vLLM also supports different Safetensors loading strategies, including lazy memory-mapped loading, eager loading, and prefetch behavior—particularly useful when model files live on network storage. vLLM

12. Multi-GPU changes this slightly

Suppose you use:

--tensor-parallel-size 2

Rather than duplicating the entire model:

GPU 0: whole model
GPU 1: whole model

vLLM can partition large matrix operations across the two GPUs.

Conceptually:

              Large weight matrix
              ┌───────────────┐
              │               │
              │     4096      │
              │               │
              └───────────────┘
                       │
                 Tensor Parallel
                    TP = 2
                  /          \
                 /            \
             GPU 0            GPU 1
           Part of W        Part of W

During inference both GPUs participate in matrix operations and communicate intermediate results.

This is why tensor parallelism can allow a model that doesn't fit on one GPU to run across multiple GPUs.

13. Model weights vs KV cache

This distinction is extremely important with vLLM.

GPU memory roughly becomes:

┌─────────────────────────────────────┐
│              GPU VRAM               │
├─────────────────────────────────────┤
│                                     │
│ Model weights                       │
│ ███████████████████████████         │
│                                     │
├─────────────────────────────────────┤
│ CUDA / runtime / activations        │
│ ████                                │
├─────────────────────────────────────┤
│                                     │
│ KV Cache                            │
│ █████████████                       │
│                                     │
└─────────────────────────────────────┘

Weights are basically constant during inference.

The KV cache changes with every request.

That distinction is at the heart of vLLM performance.

vLLM can infer KV-cache capacity from available GPU memory and gpu_memory_utilization, or it can be explicitly controlled with KV-cache memory settings. vLLM

14. What happens when your request arrives?

Suppose you call:

curl https://my-vllm/v1/chat/completions \
  -H "Authorization: Bearer ..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma",
    "messages": [
      {
        "role": "user",
        "content": "The capital of France is"
      }
    ]
  }'

The pipeline becomes:

HTTP request
      │
      ▼
OpenAI-compatible API
      │
      ▼
Chat template
      │
      ▼
Tokenizer
      │
      ▼
Token IDs
      │
      ▼
vLLM Scheduler
      │
      ▼
GPU Model
      │
      ▼
Logits
      │
      ▼
Sampler
      │
      ▼
Next Token
      │
      ▼
Tokenizer decoder
      │
      ▼
"Paris"

15. The first major computation: Prefill

Say the prompt tokenizes to:

[1, 582, 2914, 315, 9822, 374]

All prompt tokens are processed through the transformer.

This phase is called:

prefill.

Token 1 ─┐
Token 2 ─┤
Token 3 ─┤
Token 4 ─┤
Token 5 ─┤──► Transformer ─► KV cache
Token 6 ─┘

For each transformer attention layer the model calculates:

Q = X × Wq
K = X × Wk
V = X × Wv

Then attention resembles:

Attention(Q,K,V)
          =
softmax(QKᵀ / √d) V

The important part is:

K and V

can be cached.

Hence:

Key-Value Cache, or KV Cache.

16. Why KV cache matters

Imagine the prompt is:

The capital of France is

and the model generates:

Paris

For the next token, you don't want to recompute:

The
The capital
The capital of
The capital of France
The capital of France is

all over again.

Instead vLLM keeps the previous attention K/V data:

KV CACHE

The       → K,V
capital   → K,V
of        → K,V
France    → K,V
is        → K,V
Paris     → K,V

Then only the newest token needs to go through much of the incremental decoding computation.

This makes autoregressive inference dramatically more efficient.

17. Decode phase

After prefill:

Prompt
"The capital of France is"
          │
          ▼
      PREFILL
          │
          ▼
       logits
          │
          ▼
       "Paris"

Now decode starts:

Paris
  │
  ▼
model
  │
  ▼
","
  │
  ▼
model
  │
  ▼
"which"
  │
  ▼
model
  │
  ▼
"is"
...

LLMs generate one token at a time.

That is why your vLLM benchmark separates metrics such as:

TTFT = Time To First Token
TPOT = Time Per Output Token
ITL  = Inter-Token Latency

Prefill contributes significantly to TTFT, while repeated decode operations largely determine TPOT/ITL.

18. What exactly comes out of the model?

The model doesn't initially produce the word:

Paris

It produces something called logits.

Imagine vocabulary size = 128,000.

The output might be:

token              logit
-------------------------
Paris               18.7
Lyon                 8.1
France               7.9
London               6.2
Berlin               5.1
...
token #127999       -9.3

After softmax these become probabilities:

Paris      92.4%
Lyon        1.1%
France      0.9%
London      0.3%
...

Then sampling parameters affect selection:

temperature
top_p
top_k

For example:

           Model
             │
             ▼
        128K logits
             │
             ▼
        temperature
             │
             ▼
           top-k
             │
             ▼
           top-p
             │
             ▼
         sampling
             │
             ▼
      token #12345
             │
             ▼
          "Paris"

19. Where vLLM is different from simply running Transformers

This is where vLLM becomes interesting.

A conventional approach looks roughly like:

model.generate(prompt)

vLLM instead behaves more like an LLM operating system.

There might simultaneously be:

Request A
Request B
Request C
Request D
Request E
    │
    ▼
┌──────────────────────┐
│    vLLM Scheduler    │
└──────────┬───────────┘
           │
     dynamic batching
           │
           ▼
┌──────────────────────┐
│       GPU Model      │
└──────────┬───────────┘
           │
     KV Cache manager
           │
           ▼
        responses

Instead of treating each request as an isolated GPU workload, the engine schedules many sequences efficiently.

This is one of the reasons you're seeing much higher total token throughput under concurrent load than you would get by simply executing individual requests sequentially.

20. The whole architecture

You can therefore think of your vLLM deployment like this:

                      DISK / HF CACHE
                ┌────────────────────────┐
                │ config.json            │
                │ tokenizer.json         │
                │ chat template          │
                │ generation_config.json │
                │                        │
                │ model-001.safetensors  │
                │ model-002.safetensors  │
                │ model-003.safetensors  │
                └───────────┬────────────┘
                            │
                         startup
                            │
                            ▼
                 ┌────────────────────┐
                 │    vLLM Engine     │
                 │                    │
                 │ Model Loader       │
                 │ Scheduler          │
                 │ Tokenizer          │
                 │ KV Cache Manager   │
                 │ Attention Backend  │
                 │ Sampling           │
                 └─────────┬──────────┘
                           │
                           ▼
                 ┌────────────────────┐
                 │      GPU VRAM      │
                 │                    │
                 │ Model weights      │
                 │ KV cache           │
                 │ Activations        │
                 │ CUDA graphs        │
                 │ CUDA runtime       │
                 └─────────┬──────────┘
                           │
                         CUDA
                           │
                           ▼
                         GPU

Then the runtime path is:

User
 │
 ▼
HTTP /v1/chat/completions
 │
 ▼
Chat Template
 │
 ▼
Tokenizer
 │
 ▼
Token IDs
 │
 ▼
vLLM Scheduler
 │
 ├──── Request A
 ├──── Request B
 ├──── Request C
 └──── Request D
 │
 ▼
Prefill
 │
 ▼
KV Cache
 │
 ▼
Decode
 │
 ▼
Logits
 │
 ▼
Sampler
 │
 ▼
Next Token
 │
 ▼
Decode ──► Next Token
 │
 ▼
...
 │
 ▼
EOS
 │
 ▼
Detokenize
 │
 ▼
Streaming HTTP response

21. The simplest mental model

If you remember only this, you're in good shape:

MODEL FILES
│
├── config.json
│      "Build this neural network."
│
├── *.safetensors
│      "Put these billions of learned numbers into it."
│
├── tokenizer.*
│      "Convert language ↔ token numbers."
│
└── generation_config.json
       "Here are suggested generation defaults."


                     ↓


vLLM
│
├── loads model onto GPU
├── manages GPU memory
├── manages KV cache
├── schedules concurrent requests
├── runs prefill
├── runs token-by-token decoding
└── exposes an OpenAI-compatible API

And perhaps the most important conceptual point is:

An LLM model file is not a database of sentences or answers.

The .safetensors files are enormous collections of numerical matrices. A prompt is converted into vectors, those vectors pass through repeated matrix operations, and the final layer produces a probability distribution for the next token. Repeating that operation creates the answer.