The key is to think of vLLM GPU memory as a budget shared by several very different kinds of data. Some memory is essentially fixed when the model starts; some grows with request length and concurrency.

For your vLLM workload, the mental model is:

GPU VRAM
│
├── Model weights          ← mostly fixed
├── CUDA runtime / kernels ← mostly fixed
├── CUDA graphs            ← reserved at startup
├── Activations            ← temporary during computation
└── KV cache               ← grows with active token contexts

And most vLLM tuning is really about deciding:

After loading the model, how much VRAM can I leave for KV cache without hurting stability or latency?

vLLM explicitly uses gpu_memory_utilization as part of determining available cache capacity, while kv_cache_memory_bytes can override that automatic sizing. vLLM

1. Start with one Transformer layer

Consider a simplified transformer layer:

                    Input hidden states
                           │
                           ▼
                        RMSNorm
                           │
             ┌─────────────┼─────────────┐
             ▼             ▼             ▼
           Q proj         K proj         V proj
             │             │             │
             └─────────────┼─────────────┘
                           ▼
                       Attention
                           │
                           ▼
                       O projection
                           │
                     + residual
                           │
                           ▼
                        RMSNorm
                           │
             ┌─────────────┼──────────────┐
             ▼             ▼              │
         Gate proj       Up proj          │
             └─────────────┬──────────────┘
                           ▼
                     Activation
                           │
                           ▼
                      Down proj
                           │
                     + residual
                           │
                           ▼
                      Next layer

On the GPU, the pieces map approximately to:

                   GPU MEMORY

Weights
─────────────────────────────────────────────
Q weight
K weight
V weight
O weight
Gate weight
Up weight
Down weight
Normalization weights

Activations
─────────────────────────────────────────────
Current hidden states
Q values
attention intermediate results
MLP intermediates

KV Cache
─────────────────────────────────────────────
K token 1
V token 1
K token 2
V token 2
...
K token N
V token N

The important distinction is that Q is not cached.

Only:

K
V

are retained for previous tokens.

2. Model weights: the large fixed allocation

Suppose you have a model containing:

12 billion parameters

At BF16:

2 bytes / parameter

The rough raw-weight requirement is therefore:

12B × 2 bytes
≈ 24 GB

An 8B model would be roughly:

8B × 2
≈ 16 GB

before accounting for architecture details, tied weights, quantization, metadata, and runtime allocations.

Once loaded:

GPU
│
├─ layer 0 weights
├─ layer 1 weights
├─ layer 2 weights
├─ ...
└─ layer N weights

these weights stay resident.

During normal inference:

Weights = constant

They're repeatedly read by the GPU but not normally changed.

This is fundamentally different from training, where you'd also need gradients and optimizer states.

3. Why inference needs much less memory than training

Training could require something conceptually like:

Weights
+ Gradients
+ Optimizer states
+ Large activation storage

Inference mostly needs:

Weights
+ KV cache
+ temporary activations
+ runtime memory

So a rough comparison is:

Training:

┌───────────────────────────────┐
│ weights                       │
│ gradients                     │
│ optimizer states              │
│ saved activations             │
│ temporary buffers             │
└───────────────────────────────┘


Inference:

┌───────────────────────────────┐
│ weights                       │
│ KV cache                      │
│ small transient activations   │
│ CUDA graphs/runtime           │
└───────────────────────────────┘

For LLM serving, the KV cache often becomes the dominant variable memory component.

4. Activations are temporary working memory

Imagine a hidden dimension of:

4096

When tokens enter a transformer layer, intermediate tensors are created:

X
↓
X × Wq → Q

X × Wk → K

X × Wv → V

Then:

QKᵀ
softmax(...)
attention output
MLP intermediate

These are activations.

Unlike the KV cache, most do not have to remain around after the layer/iteration is finished.

So:

Weights          persistent
KV cache         persistent while request is alive
Activations      temporary

This explains why your vLLM startup profile reports something like a:

peak activation memory

rather than allocating that exact amount permanently to every request.

5. Prefill uses much more compute than a single decode step

Take a prompt containing 2,000 tokens.

During prefill:

2000 tokens
    │
    ▼
┌──────────────────┐
│ Transformer      │
│ Layer 0          │
│ Layer 1          │
│ ...              │
│ Layer N          │
└──────────────────┘

The GPU processes a large amount of token work.

During decode:

1 new token
    │
    ▼
Transformer
    │
    ▼
1 new token

but that new token attends to existing KV cache.

So the workload differs considerably:

PREFILL
──────────────
many input tokens
large matrix operations
high compute utilization

DECODE
──────────────
usually one new token per sequence
reads existing KV cache
smaller matrix operations
often memory-bandwidth sensitive

This is why vLLM scheduling and batching matter so much.

6. The KV cache

This is the most important vLLM concept to understand.

For every token and every attention layer, the model stores its:

Key
Value

representations.

Suppose a request has:

10,000 tokens

then conceptually:

Request A

Layer 0
 K[0...9999]
 V[0...9999]

Layer 1
 K[0...9999]
 V[0...9999]

Layer 2
 K[0...9999]
 V[0...9999]

...

Layer N
 K[0...9999]
 V[0...9999]

So KV usage is roughly proportional to:

active tokens
× number of layers
× number of KV heads
× head dimension
× K + V
× bytes per element

A simplified formula is:

KV bytes/token ≈
2 × layers × KV_heads × head_dim × bytes_per_element

The 2 is for:

K + V

7. Why GQA greatly reduces KV cache

Suppose:

Attention heads = 32
KV heads        = 8

This is grouped-query attention, or GQA.

Instead of storing K/V for all 32 attention heads:

32 KV heads

you only store:

8 KV heads

which makes the KV cache approximately:

4× smaller

for that component.

This is one reason modern architectures heavily use:

GQA
MQA
sliding-window attention

They aren't only architectural choices—they have major serving implications.

vLLM also has specialized KV management for hybrid architectures, including models that combine full and sliding-window attention. vLLM

8. vLLM does not allocate KV cache as one giant contiguous region per request

This is one of vLLM's key innovations.

A naive implementation could look like:

Request A
[                         ]

Request B
[          ]

Request C
[                                      ]

This creates allocation and fragmentation problems.

vLLM instead manages cache using blocks/pages.

Conceptually:

KV cache pool

┌───────┐
│Block 1│ ← Request A
├───────┤
│Block 2│ ← Request A
├───────┤
│Block 3│ ← Request B
├───────┤
│Block 4│ ← Request C
├───────┤
│Block 5│ ← Request A
├───────┤
│Block 6│ ← free
├───────┤
│Block 7│ ← Request B
└───────┘

The cache manager can associate logical request-token positions with physical blocks.

vLLM's current design describes the cache memory pool as being divided into blocks/pages allocated to layers and requests. vLLM

This is the basis of the idea usually associated with PagedAttention.

9. Now map this onto your GPU

From one of your previous startup logs, your GPU was approximately:

Total GPU memory     ~22.03 GiB

Model weights        ~15.18 GiB
Peak activation      ~0.27 GiB
CUDA Graphs          ~0.78 GiB
KV cache             ~4.31 GiB

The exact numbers can vary depending on vLLM version, profiling, allocator behavior, and options, but this is a very useful real-world example.

Visually:

22 GiB GPU

┌─────────────────────────────────────────┐
│                                         │
│ MODEL WEIGHTS                           │
│                                         │
│ ~15.18 GiB                              │
│                                         │
├─────────────────────────────────────────┤
│ CUDA graphs ~0.78 GiB                   │
├─────────────────────────────────────────┤
│ activations/runtime                     │
├─────────────────────────────────────────┤
│                                         │
│ KV CACHE                                │
│ ~4.31 GiB                               │
│                                         │
└─────────────────────────────────────────┘

Notice what dominates:

Weights ≫ everything else

But after the model is loaded, the most valuable remaining resource is:

KV cache

because that controls how many active tokens you can serve.

10. Your real KV-cache numbers are very instructive

Your previous startup output reported approximately:

KV cache memory:       4.31 GiB
KV cache token capacity: 208,625 tokens

That implies roughly:

4.31 GiB / 208,625

≈ 21.7 KiB per cached token

So, very approximately:

1K tokens     ≈ 21 MiB
8K tokens     ≈ 173 MiB
32K tokens    ≈ 693 MiB

A single fully occupied 32,768-token context is therefore around:

~0.68 GiB KV cache

for that specific model/runtime configuration.

This is a very useful operational number.

11. That's where your reported 6.37x concurrency came from

vLLM reported roughly:

KV cache tokens = 208,625
max model len   = 32,768

Therefore:

208625
───────
32768

≈ 6.37

Hence your startup message:

maximum concurrency for
32,768 tokens/request ≈ 6.37x

This does not mean vLLM can only handle six simultaneous HTTP requests.

It means:

If every active request actually consumed the entire 32K context window, the KV cache could hold roughly 6.37 such contexts.

That's an important distinction.

12. Real concurrency can be much higher

Suppose your average active context is only:

4,000 tokens

Using your cache capacity:

208,625 / 4,000
≈ 52

You could theoretically hold around:

~52 such contexts

from a KV-capacity perspective.

At 2K:

208625 / 2000
≈ 104

At 8K:

208625 / 8000
≈ 26

So a useful approximation is:

These are capacity estimates, not promises of good latency.

13. --gpu-memory-utilization

Now this setting becomes easier to understand.

For example:

--gpu-memory-utilization 0.90

means vLLM is allowed to target roughly:

90% of GPU memory

for its model executor.

Current vLLM documentation describes the setting as the fraction of GPU memory usable by that vLLM instance; automatic KV sizing uses that budget after accounting for other required memory. vLLM

Conceptually:

GPU = 24 GB

--gpu-memory-utilization 0.90

Target vLLM budget
≈ 21.6 GB

Then:

21.6 GB
- model
- activations
- CUDA graphs
- other runtime allocations
────────────────────────────
= KV cache

It's therefore useful to think:

gpu-memory-utilization
        ↓
available vLLM budget
        ↓
subtract fixed/runtime allocations
        ↓
remaining KV capacity

14. Increasing --gpu-memory-utilization

For example:

0.85 → 0.90 → 0.95

usually gives more room to the cache:

                  More GPU utilization

0.85        0.90          0.95
 │           │             │
 ▼           ▼             ▼

small KV    larger KV     largest KV
cache       cache         cache

Potential benefit:

↑ active cached tokens
↑ concurrency capacity
↑ throughput potential

Potential cost:

↓ safety margin
↑ chance of OOM

So:

0.70

is conservative but wastes memory.

0.99

may be aggressive enough that runtime fluctuations push you over the edge.

15. gpu-memory-utilization does NOT mean GPU compute utilization

This is commonly misunderstood.

This:

--gpu-memory-utilization 0.90

does not mean:

GPU compute runs at 90%

It means approximately:

vLLM memory budget = 90% of VRAM

Your actual:

SM utilization
Tensor Core utilization
GPU utilization %
memory bandwidth

are separate performance metrics.

16. You can directly control KV cache now

Current vLLM also exposes:

--kv-cache-memory-bytes

When explicitly supplied, it provides more direct control of the KV-cache allocation and overrides automatic cache sizing based on gpu_memory_utilization. vLLM

Conceptually:

--kv-cache-memory-bytes 4G

means:

Give KV cache roughly this explicit budget

rather than saying:

Use 90% total GPU memory and figure it out.

For repeatable benchmark experiments, this can be useful.

17. What --max-model-len really controls

Suppose you use:

--max-model-len 32768

vLLM defines this as the maximum combined model context:

prompt + output

and if it isn't set, it normally derives the supported context length from the model configuration. vLLM

For example:

Prompt         28,000
Generated       4,000
─────────────────────
Total          32,000

✓ fits

but:

Prompt         30,000
Generated       5,000
─────────────────────
Total          35,000

✗ exceeds 32,768

18. max-model-len does not automatically consume all that KV memory

This is another important distinction.

Setting:

--max-model-len 32768

doesn't necessarily mean every request immediately gets a 32K KV buffer.

With paged cache management, memory is consumed according to the blocks actually needed.

So:

Request A: 2K
Request B: 3K
Request C: 1K

does not become:

32K + 32K + 32K

The logical maximum is 32K per request, while actual allocation tracks active token use.

That's a major reason vLLM can achieve high concurrency.

19. Why lowering max-model-len can still help

Current vLLM explicitly recommends reducing max_model_len and maximum batch size when memory is tight. vLLM

Suppose your application never needs more than:

8K

Then using:

--max-model-len 8192

instead of:

32768

has several benefits:

prevents runaway long requests
simplifies capacity planning
reduces worst-case memory pressure
can improve scheduling predictability

It also means your worst-case KV calculation changes dramatically.

Using your approximate cache capacity:

208,625 / 8192
≈ 25.5

instead of:

208,625 / 32768
≈ 6.37

So your worst-case capacity goes conceptually from:

6 × 32K requests

to:

25 × 8K requests

using the same KV pool.

20. Concurrency and memory

Now suppose 10 requests arrive simultaneously.

Each has:

input = 1,500 tokens

After prefill:

10 × 1,500
= 15,000 KV-cache tokens

Then each generates 200 tokens:

10 × 200
= 2,000 additional tokens

So approximately:

17,000 cached tokens

are needed.

Compare that with your roughly:

208,625 token capacity

That's only about:

8%

of the token cache capacity.

So your previous 10-request benchmark wasn't close to exhausting the KV cache purely from token count.

This helps explain why increasing concurrency can still significantly increase throughput.

21. Why throughput rises with concurrency

One request might look like:

GPU

token
 │
 ▼
matrix multiply
 │
 ▼
GPU partially utilized

With many requests:

Request A ─┐
Request B ─┤
Request C ─┤
Request D ─┤
Request E ─┤
            ▼
         BATCH
            │
            ▼
           GPU
            │
            ▼
     larger matrix ops

GPUs love larger parallel workloads.

So:

Concurrency ↑
      ↓
Batching ↑
      ↓
GPU utilization ↑
      ↓
aggregate tok/s ↑

That is why one-user latency and server throughput are very different optimization targets.

22. But concurrency eventually hurts latency

The curve generally looks conceptually like:

Throughput

 ^
 |                  ────────── saturation
 |              ───
 |           ───
 |        ──
 |     ──
 |  ──
 +─────────────────────────────> concurrency

while latency can look like:

Latency

 ^
 |                           /
 |                        /
 |                     /
 |                  /
 |________________/
 +────────────────────────────> concurrency

Initially, concurrency improves hardware utilization.

Eventually:

scheduler queues grow
batches become larger
KV pressure increases
requests wait longer
TTFT rises

The goal isn't:

maximum concurrency

It's:

maximum throughput
while satisfying your latency SLO

23. Why vLLM has --max-num-seqs

Suppose:

--max-num-seqs 32

This limits how many sequences the scheduler can process in a single iteration.

Current vLLM defines it as the maximum number of sequences processed in one iteration. vLLM

Think:

100 waiting requests
       │
       ▼
vLLM scheduler
       │
       ├── 32 active
       └── others queued

Increasing it may:

↑ batch size
↑ throughput

but can also:

↑ activation/workspace requirements
↑ CUDA graph memory
↑ scheduling pressure
↑ per-request latency

24. --max-num-batched-tokens

This controls another dimension:

number of sequences
             vs
number of tokens

Current vLLM defines:

--max-num-batched-tokens

as the maximum number of tokens processed in one scheduler iteration. vLLM

For example:

Request A prefill = 2,000 tokens
Request B prefill = 3,000 tokens
Request C decode  = 1 token
Request D decode  = 1 token

The scheduler needs to decide how much work goes into the next iteration.

A larger token budget tends to favor throughput.

A smaller one can improve interactivity by avoiding huge prefill jobs monopolizing execution.

25. CUDA graphs

Without CUDA graphs, a decode step involves many kernel launches:

CPU
 │
 ├── launch RMSNorm
 ├── launch Q projection
 ├── launch K projection
 ├── launch V projection
 ├── launch attention
 ├── launch O projection
 ├── launch MLP
 ├── ...
 ▼
GPU

Even though GPU operations are fast, repeatedly launching them has CPU/driver overhead.

CUDA graphs let vLLM capture a reusable sequence:

Capture once:

┌─────────────────────────┐
│ RMSNorm                 │
│ QKV                      │
│ Attention                │
│ projection               │
│ MLP                      │
│ ...                      │
└─────────────────────────┘

            ↓

Replay
Replay
Replay
Replay

This reduces launch overhead, particularly for decode. Current vLLM documentation explicitly notes CUDA graph capture improves performance by reducing dispatch overhead. vLLM

26. But CUDA graphs consume memory

The trade-off:

CUDA graphs enabled

+ lower kernel launch overhead
+ lower decode latency
+ better throughput

- additional GPU memory
- startup capture/warmup

This is why your startup log showed something like:

CUDAGraph memory ≈ 0.78 GiB

That ~780 MiB is meaningful on a 22 GiB GPU.

It's memory that could otherwise potentially contribute to the KV cache.

vLLM's memory-conservation guide specifically says CUDA graphs consume extra GPU memory and can be reduced or disabled when needed. vLLM

27. You can disable CUDA graphs

For memory troubleshooting:

--enforce-eager

forces eager execution and disables graph capture. vLLM documents this as a memory-saving option. vLLM

Conceptually:

CUDA graphs

ON
────────────────
more memory
lower overhead

OFF / eager
────────────────
less graph memory
higher dispatch overhead

I wouldn't normally disable them for your production serving unless you need the VRAM or are troubleshooting an incompatibility.

28. Tensor parallelism changes the equation substantially

Suppose:

--tensor-parallel-size 2

vLLM shards model parameters across GPUs within each layer. vLLM

Instead of:

GPU 0
16 GB weights

you might conceptually get:

GPU 0                GPU 1

~8 GB weights        ~8 GB weights
──────────────       ──────────────
Layer 0 shard A      Layer 0 shard B
Layer 1 shard A      Layer 1 shard B
...

It's more complicated than exactly 50/50 in real models, but that's the correct mental model.

29. A matrix gets split

Suppose:

Wq = [4096 × 4096]

With:

TP = 2

conceptually:

           Full Wq

┌──────────────────────────────┐
│                              │
└──────────────────────────────┘
              │
          TP = 2
         /      \
        /        \
GPU 0              GPU 1

┌──────────────┐   ┌──────────────┐
│ Wq shard 0   │   │ Wq shard 1   │
└──────────────┘   └──────────────┘

Both GPUs perform their part:

GPU0 matmul
      \
       ── communication ──► result
      /
GPU1 matmul

vLLM provides TP collective operations such as all-reduce, all-gather and reduce-scatter for this distributed execution. vLLM

30. TP gives you more KV-cache headroom

Imagine a 24 GB GPU.

With TP=1:

GPU 0

15 GB weights
~1 GB runtime
~7 GB available elsewhere

Now two GPUs with TP=2:

GPU 0               GPU 1

~7.5 GB weights     ~7.5 GB weights
runtime             runtime
more KV room        more KV room

This is why vLLM recommends TP not only when a model doesn't fit, but also when reducing per-GPU model pressure can create more cache space for throughput. vLLM

31. But TP isn't free performance

Every transformer layer now needs inter-GPU communication.

Conceptually:

GPU 0
 matrix result
     │
     ├──── NVLink / PCIe ────┐
     │                       │
GPU 1                        │
 matrix result               │
     │                       │
     └───────────────────────┘
              │
              ▼
        combined result

Therefore:

TP ↑
   ↓
weight memory/GPU ↓
compute/GPU ↓
but
communication ↑

If the GPUs have:

NVLink / NVSwitch

TP is much more attractive.

If communication goes across a slower PCIe topology, the benefit may be smaller.

32. So don't automatically use TP=2 just because you have two GPUs

If a model fits comfortably on one GPU, you could instead use:

2 independent replicas

rather than:

TP=2

For example:

Option A

TP = 2

GPU0 ──┐
       ├─ one model instance
GPU1 ──┘

Option B

GPU0 → Model instance A

GPU1 → Model instance B

Option B can often deliver better total request throughput when the model fits on one GPU, because:

no TP synchronization between GPUs

while Option A may give:

more memory per model
lower single-request latency in some workloads
support for bigger models/contexts

So the right answer depends on whether your bottleneck is:

memory
latency
or throughput

33. Putting all four controls together

Here's the tuning relationship I recommend memorizing:

                  --gpu-memory-utilization
                            │
                            ▼
                    Total VRAM budget
                            │
          ┌─────────────────┴────────────────┐
          │                                  │
     fixed/runtime                         KV cache
          │                                  │
          │                         ┌────────┴────────┐
          │                         │                 │
          ▼                         ▼                 ▼
model weights                context length      concurrency
CUDA graphs                 --max-model-len      active tokens
activations
          ▲
          │
          │
 --tensor-parallel-size
          │
          ▼
reduces weights per GPU
but increases communication

That is essentially the entire vLLM memory problem.

34. How each parameter affects your benchmark

Here is the practical mapping:

The ~ means little direct effect unless cache pressure or scheduling changes.

35. Now connect this to your benchmark

You previously had approximately:

Successful requests          10
Concurrent requests          10

Request throughput           0.76 req/s
Output throughput          156.92 tok/s
Total throughput           266.53 tok/s

Mean TTFT                  351 ms
Mean TPOT                 19.82 ms
Median TPOT               15.31 ms

Those numbers suggest a fairly healthy serving regime: batching was providing useful aggregate throughput, while TTFT remained well below one second under 10 concurrent requests.

The important next experiment isn't simply:

Can I make 156 tok/s become 170?

It's finding the saturation curve.

36. The benchmark I would run next

Use the same:

input length
output length
model
sampling parameters

and vary only concurrency:

Concurrency
1
2
4
8
16
32
64

Then record:

TTFT p50
TTFT p99

TPOT p50
TPOT p99

output tok/s
total tok/s

requests/s

You'll probably see something like:

Concurrency   Throughput   TTFT
───────────   ──────────   ────
1                low        low
2                ↑          ~
4                ↑          ~
8                ↑          ↑
16               ↑          ↑
32             plateau      ↑↑
64             plateau      ↑↑↑

The sweet spot is just before the latency curve bends sharply upward.

37. For your ~22 GiB GPU, I would start here

Based on the memory profile you've shown, I'd initially run:

vllm serve <model> \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

and avoid changing a dozen knobs simultaneously.

Then benchmark:

C = 1
C = 4
C = 8
C = 16
C = 32

If production doesn't need 32K context, I'd strongly consider:

--max-model-len 16384

or even:

--max-model-len 8192

because controlling worst-case context length is one of the easiest ways to make inference capacity predictable.

Current vLLM specifically identifies context length and maximum sequence count as memory-conservation levers. vLLM

38. A useful production memory target

For production, I generally wouldn't try to make the picture:

GPU

██████████████████████████ 99.9%

I'd rather aim for something conceptually like:

GPU

Weights          █████████████████
KV Cache         █████
CUDA/runtime     ██
Safety margin    ░░

especially for multimodal models, compilation, unusual batches, or changing traffic patterns.

So your:

--gpu-memory-utilization 0.90

is a sensible starting point.

Then validate under sustained load before considering:

0.92
0.94
0.95

Current vLLM defaults can vary by interface/version, so I would set this explicitly rather than depend on the default. vLLM

39. The whole system in one diagram

You can now connect the model files from the previous explanation all the way to runtime:

HUGGING FACE MODEL
│
├── config.json
├── tokenizer
└── safetensors
       │
       ▼
══════════════════════════════════════════════════
                  vLLM STARTUP
══════════════════════════════════════════════════
       │
       ├─ Read architecture
       │
       ├─ Build transformer
       │
       ├─ Load tensors
       │
       ▼
┌──────────────────────── GPU ────────────────────┐
│                                                │
│ MODEL WEIGHTS                                  │
│ ┌────────────────────────────────────────────┐ │
│ │ Layer 0 Q/K/V/O + MLP                     │ │
│ │ Layer 1 Q/K/V/O + MLP                     │ │
│ │ ...                                        │ │
│ │ Layer N                                    │ │
│ └────────────────────────────────────────────┘ │
│                                                │
│ CUDA GRAPHS / COMPILED KERNELS                 │
│ ┌────────────────────────────────────────────┐ │
│ │ captured execution paths                   │ │
│ └────────────────────────────────────────────┘ │
│                                                │
│ ACTIVATION / WORKSPACE                         │
│ ┌────────────────────────────────────────────┐ │
│ │ temporary computation                     │ │
│ └────────────────────────────────────────────┘ │
│                                                │
│ KV CACHE                                       │
│ ┌────┬────┬────┬────┬────┬────┬────┐         │
│ │ A1 │ A2 │ B1 │ C1 │ A3 │ B2 │free│ ...     │
│ └────┴────┴────┴────┴────┴────┴────┘         │
│                                                │
└────────────────────────────────────────────────┘
       ▲
       │
══════════════════════════════════════════════════
                REQUEST EXECUTION
══════════════════════════════════════════════════
       │
User request
       │
       ▼
Tokenizer
       │
       ▼
PREFILL
       │
       ├──────────────► write K/V
       │                     │
       ▼                     ▼
First token             KV CACHE
       │                     ▲
       ▼                     │
DECODE ──────────────────────┘
       │
       ▼
next token
       │
       ▼
DECODE
       │
       ▼
...

And the vLLM scheduler sits above all of that:

        50 incoming requests
                 │
                 ▼
        ┌────────────────┐
        │ vLLM Scheduler │
        └───────┬────────┘
                │
          batch/chunk work
                │
                ▼
              GPU

The scheduler, KV-cache manager and GPU workers are core pieces of the current vLLM architecture. vLLM

The four rules I'd keep in mind

1. Weights determine whether the model fits.

Model size → base VRAM requirement

2. KV cache determines serving capacity.

context length × concurrent active tokens → KV memory

3. Batching determines throughput.

more useful concurrent GPU work → higher tok/s

until saturation.

4. Tensor parallelism trades communication for memory.

TP ↑
→ model memory/GPU ↓
→ potential KV capacity ↑
→ GPU-to-GPU communication ↑

That leads directly to the practical optimization objective for your vLLM deployment:

Fit the weights comfortably, maximize useful KV cache, batch enough requests to keep the GPU busy, and stop increasing concurrency when TTFT/TPOT begin violating your latency target.

For the benchmark numbers you've already collected, concurrency sweep + GPU/KV utilization is the next experiment I'd run before adjusting lower-level kernel or compilation settings.