SOP Running and Configuring vLLM on Linux using Docker

admin

06 Aug 2026 — 3 min read

Document Version: 1.0
Applies To: Ubuntu 22.04/24.04, Debian 12, RHEL/Rocky 9 (or equivalent Linux distributions)
Model: google/gemma-4-12B-it (recommended)

1. Purpose

This document describes the standard procedure for deploying a production-ready vLLM inference server running the Google Gemma 4 12B model using Docker Compose.

The resulting API is OpenAI-compatible and can be used by:

Open WebUI
LiteLLM
LangChain
LlamaIndex
Custom applications

2. System Requirements

Minimum Hardware

Component	Requirement
CPU	8+ cores
Memory	32 GB RAM
GPU	NVIDIA GPU (24GB VRAM recommended)
CUDA	12.x
Disk	100 GB free SSD
OS	Ubuntu 22.04+

Recommended GPUs:

RTX 4090
RTX 6000 Ada
A100 40GB
H100

3. Install NVIDIA Driver

Verify the NVIDIA driver is installed.

nvidia-smi

Example:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 575.xx |
| CUDA Version: 12.9 |
+-----------------------------------------------------------------------------+

If this command fails, install the NVIDIA driver before proceeding.

4. Install Docker

Update packages.

sudo apt update
sudo apt upgrade -y

Install Docker.

curl -fsSL https://get.docker.com | sh

Allow current user to use Docker.

sudo usermod -aG docker $USER

Logout/login.

Verify.

docker version

5. Install NVIDIA Container Toolkit

Ubuntu:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update

sudo apt install -y nvidia-container-toolkit

Configure Docker.

sudo nvidia-ctk runtime configure --runtime=docker

Restart Docker.

sudo systemctl restart docker

Verify GPU access.

docker run --rm --gpus all nvidia/cuda:12.9.0-runtime-ubuntu22.04 nvidia-smi

6. Obtain Hugging Face Access

Create a Hugging Face account.
Accept the Gemma license.

https://huggingface.co/google/gemma-4-12B-it

Create an Access Token.

https://huggingface.co/settings/tokens

Permission required:

Read

7. Create Deployment Directory

mkdir -p ~/vllm
cd ~/vllm

8. Create Environment File

Create .env

nano .env

Contents:

HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Protect it.

chmod 600 .env

9. Create docker-compose.yml

services:
  vllm:
    image: vllm/vllm-openai:gemma4
    container_name: vllm

    ipc: host

    ports:
      - "8000:8000"

    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
      - vllm-cache:/root/.cache/vllm

    env_file:
      - .env

    command:
      - google/gemma-4-12B-it
      - --trust-remote-code
      - --host
      - 0.0.0.0
      - --max-model-len
      - "16384"
      - --gpu-memory-utilization
      - "0.90"
      - --enable-auto-tool-choice
      - --reasoning-parser
      - gemma4
      - --tool-call-parser
      - gemma4
      - --chat-template
      - examples/tool_chat_template_gemma4.jinja

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu

    restart: unless-stopped

volumes:
  vllm-cache:

10. Download Image

docker compose pull

11. Start vLLM

docker compose up -d

Verify.

docker ps

Expected:

vllm   Up

12. Monitor Startup

The first startup downloads approximately 25–30 GB of model files.

docker compose logs -f

Wait until you see:

Application startup complete

13. Verify API

Check available models.

curl http://localhost:8000/v1/models

Example:

{
  "data":[
    {
      "id":"google/gemma-4-12B-it"
    }
  ]
}

14. Test Inference

curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
  "model":"google/gemma-4-12B-it",
  "messages":[
    {
      "role":"user",
      "content":"Explain Kubernetes in one sentence."
    }
  ]
}'

15. Connect Open WebUI

Open:

Settings

↓

Connections

↓

OpenAI API

Configure:

Setting	Value
API URL	http://SERVER_IP:8000/v1
API Key	any non-empty string (e.g., `local-api-key`)

Save.

The model should appear automatically.

16. Automatic Startup

Enable Docker at boot.

sudo systemctl enable docker

Since the Compose file includes:

restart: unless-stopped

the container will automatically restart after system reboots.

17. Updating the Model

Stop the service.

docker compose down

Pull the latest image.

docker compose pull

Start again.

docker compose up -d

18. Troubleshooting

GPU Not Found

docker run --rm --gpus all nvidia/cuda:12.9.0-runtime-ubuntu22.04 nvidia-smi

If this fails:

Verify the NVIDIA driver is installed.
Verify the NVIDIA Container Toolkit is installed.
Restart Docker.

Authentication Error

401 Unauthorized

Verify:

The Hugging Face token is valid.
You have accepted the Gemma model license.
The .env file contains the correct HF_TOKEN.

Out of Memory (OOM)

Reduce the maximum context length:

--max-model-len 8192

Or lower:

--gpu-memory-utilization 0.80

Model Download Interrupted

Remove the partial download:

rm -rf ~/.cache/huggingface

Then restart:

docker compose up -d

19. Useful Commands

docker compose up -d

Start the service.

docker compose down