Version: 1.0
Platform: Amazon Linux 2023
Deployment Method: Docker (Recommended)
Audience: System Administrators, DevOps Engineers, AI/ML Engineers

1. Purpose

This document describes the recommended procedure for deploying vLLM on Amazon Linux 2023 using the official AWS Deep Learning Container (DLC).

Using the AWS DLC is the simplest and most reliable deployment method because AWS maintains the container image with:

• CUDA libraries
• NVIDIA GPU drivers compatibility
• PyTorch
• vLLM
• Optimized runtime libraries

This approach avoids installing CUDA toolkits and compiling C++ dependencies directly on the operating system.

2. Prerequisites

Hardware

• NVIDIA GPU EC2 Instance
  • g5
  • g6
  • p4
  • p5
  • Other CUDA-capable GPU instances

Operating System

• Amazon Linux 2023

Note: Amazon Linux 2 has reached end of support and is not recommended.

3. Architecture

                +-------------------------+
                | Amazon Linux 2023 EC2   |
                +-----------+-------------+
                            |
                    Docker Engine
                            |
                AWS Deep Learning Container
                            |
                       NVIDIA Runtime
                            |
                          CUDA
                            |
                          vLLM
                            |
                 OpenAI Compatible API
                            |
                   Port 8000 (HTTP)

4. Install Docker

Update the system.

sudo dnf update -y

Install Docker.

sudo dnf install docker -y

Enable Docker.

sudo systemctl enable docker
sudo systemctl start docker

Verify Docker.

docker --version

5. Install NVIDIA Container Toolkit

Install the NVIDIA container runtime.

Follow the NVIDIA Container Toolkit installation guide appropriate for Amazon Linux 2023.

1. Set Up the Repo
2. Install the Toolkit
3. Configure the Container Runtime

curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
  sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

sudo dnf install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Restart Docker afterward.

sudo systemctl restart docker

Verify GPU visibility.

docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu22.04 nvidia-smi

Expected output should display the NVIDIA GPU information.

6. Prepare Hugging Face Cache

Create a local cache directory.

mkdir -p ~/.cache/huggingface

If using gated or private models, create a Hugging Face access token and export it:

export HF_TOKEN=<your_token>

7. Pull the AWS vLLM Container

Download the official AWS Deep Learning Container.

docker pull public.ecr.aws/deep-learning-containers/vllm:latest-gpu-py312-cu130-ubuntu22.04-ec2

This image includes:

• CUDA
• PyTorch
• Python 3.12
• vLLM
• Optimized GPU runtime

8. Start the vLLM Server

Run a model using Docker Compose.

# docker-compose.yaml
services:
  vllm:
    image: vllm/vllm-openai:v0.26.0
    container_name: vllm
    ipc: host
    ports:
      - "8000:8000"
    volumes:
      # Hugging Face model cache
      - ~/.cache/huggingface:/root/.cache/huggingface
      # Persist vLLM / torch compile cache
      - ~/.cache/vllm:/root/.cache/vllm
    environment:
      - HF_TOKEN=${HF_TOKEN}
    command:
      - --model
      - google/gemma-4-E4B-it
      - --host
      - 0.0.0.0
      - --port
      - "8000"
      - --max-model-len
      - "32768"
      - --gpu-memory-utilization
      - "0.90"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

    restart: unless-stopped

The server will automatically:

• Download the model (first launch)
• Cache model files
• Start an OpenAI-compatible API server
• Listen on port 8000

9. Access the API

Verify the server is running.

curl http://localhost:8000/v1/models

Example response:

{
  "data": [
    {
      "id": "facebook/opt-125m"
    }
  ]
}

API Documentation & Service Info

OpenAI-Compatible APIs

Anthropic-Compatible APIs

Tokenization APIs

Rendering APIs

Inference & Scoring APIs

Elastic Expert Parallelism

For most application developers using vLLM as an OpenAI-compatible server, the main endpoints are simply:

GET  /v1/models
POST /v1/chat/completions
POST /v1/completions
POST /v1/responses

GET  /health
GET  /metrics
GET  /docs

And with your current server configuration, the base URL would be:

http://<vllm-server>:8000

For example:

http://localhost:8000/v1/chat/completions

10. Example: Run Gemma 4

Replace the model argument with your desired Hugging Face model.

docker run --rm \
    --gpus all \
    -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -e HF_TOKEN=$HF_TOKEN \
    public.ecr.aws/deep-learning-containers/vllm:latest-gpu-py312-cu130-ubuntu22.04-ec2 \
    --model google/gemma-4-12b

Note: Ensure your Hugging Face account has accepted the model license if the model is gated.

11. Verify GPU Usage

Check GPU utilization.

nvidia-smi

You should see:

• Python process
• CUDA memory usage
• GPU utilization

12. Stopping the Server

If running interactively:

CTRL+C

If running detached:

docker ps
docker stop <container-id>

13. Optional: Run in Background

Launch the container in detached mode.

docker run -d \
    --name vllm \
    --restart unless-stopped \
    --gpus all \
    -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -e HF_TOKEN=$HF_TOKEN \
    public.ecr.aws/deep-learning-containers/vllm:latest-gpu-py312-cu130-ubuntu22.04-ec2 \
    --model google/gemma-4-12b

Useful Docker commands:

View logs:

docker logs -f vllm

Stop:

docker stop vllm

Restart:

docker restart vllm

14. Alternative: Native Installation (Advanced)

Native installation is intended only for advanced users who require a non-containerized deployment.

Requirements:

• Amazon Linux 2023
• Python 3.9 or later
• GCC/G++
• CUDA Toolkit
• Matching PyTorch with CUDA support

vLLM can be deployed as a server that implements the OpenAI API protocol. This allows vLLM to be used as a drop-in replacement for applications using OpenAI API. By default, it starts the server at http://localhost:8000. You can specify the address with --host and --port arguments. The server currently hosts one model at a time and implements endpoints such as list models, create chat completion, and create completion endpoints.

Run the following command to start the vLLM server with the Qwen2.5-1.5B-Instruct model:

sudo dnf install python3 gcc gcc-c++ git -y

pip install torch torchvision torchaudio

pip install vllm

During installation, pip compiles the required CUDA and C++ extensions locally. This process is slower and more prone to version mismatches than the Docker-based deployment.

Testing

vllm serve Qwen/Qwen2.5-1.5B-Instruct
curl http://localhost:8000/v1/models

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen/Qwen2.5-1.5B-Instruct",
        "prompt": "San Francisco is a",
        "max_tokens": 7,
        "temperature": 0
    }'

15. Troubleshooting

Gemma 4 is a heterogeneous model—meaning its layers use two different attention head sizes: sliding/local layers use a head_dim of 256, while global layers use 512. In newer versions of the transformers library, accessing config.head_dim directly on such heterogeneous models throws an AmbiguousGlobalPerLayerAttributeError because the attribute now varies by layer. When vLLM tries to map the model config during startup using its older conversion scripts, it calls config.head_dim and crashes.

Fix 1: Downgrade transformers (Quickest Workaround)If you need to get the engine running immediately, downgrade your environment's transformers package to a version released prior to this strict heterogeneity checking (such as v5.10.x or lower, depending on your setup):

pip install --upgrade vllm

Fix 2: Update vLLMThis is a known compatibility issue between vLLM's internal model_arch_config_convertor.py and the updated transformers library. If you are running an older stable version of vLLM, update it to the latest release or build from main where the config mapper has been updated to query config.per_layer_config[i].head_dim instead.

(If using Docker, pull the absolute latest vllm/vllm-openai:latest image).

Fix 3: Patched Local Config (Zero-Installation Hack)If you cannot change your environment packages, you can manually override the Hugging Face cache config to trick vLLM into seeing a uniform head_dim:

1. Locate your local cache folder for google/gemma-4-E4B-it.
2. Open config.json.
3. Locate the text_config block.
4. Add or hardcode "head_dim": 256 directly into the top level of the text_config dictionary.

(Note: While this bypasses the initialization crash, it can sometimes trigger downstream math mismatches during Triton kernel execution depending on your exact vLLM version).

16. Best Practices

• Use Amazon Linux 2023 for new deployments.
• Prefer the AWS Deep Learning Container over native installations.
• Store Hugging Face models in a persistent cache volume to avoid repeated downloads.
• Use --restart unless-stopped or a systemd service to ensure automatic recovery after reboots.
• Restrict network access to the vLLM API using AWS Security Groups or a reverse proxy if the service is not intended to be publicly accessible.
• Monitor GPU utilization and memory usage with nvidia-smi to optimize model placement and performance.

17. References

• AWS Deep Learning Containers (DLC)
• vLLM Documentation
• Hugging Face Model Hub
• NVIDIA Container Toolkit Documentation