Version: 1.0
Platform: Amazon Linux 2023
Deployment Method: Docker (Recommended)
Audience: System Administrators, DevOps Engineers, AI/ML Engineers

1. Purpose

This document describes the recommended procedure for deploying vLLM on Amazon Linux 2023 using the official AWS Deep Learning Container (DLC).

Using the AWS DLC is the simplest and most reliable deployment method because AWS maintains the container image with:

• CUDA libraries
• NVIDIA GPU drivers compatibility
• PyTorch
• vLLM
• Optimized runtime libraries

This approach avoids installing CUDA toolkits and compiling C++ dependencies directly on the operating system.

2. Prerequisites

Hardware

• NVIDIA GPU EC2 Instance
  • g5
  • g6
  • p4
  • p5
  • Other CUDA-capable GPU instances

Operating System

• Amazon Linux 2023

Note: Amazon Linux 2 has reached end of support and is not recommended.

3. Architecture

                +-------------------------+
                | Amazon Linux 2023 EC2   |
                +-----------+-------------+
                            |
                    Docker Engine
                            |
                AWS Deep Learning Container
                            |
                       NVIDIA Runtime
                            |
                          CUDA
                            |
                          vLLM
                            |
                 OpenAI Compatible API
                            |
                   Port 8000 (HTTP)

4. Install Docker

Update the system.

sudo dnf update -y

Install Docker.

sudo dnf install docker -y

Enable Docker.

sudo systemctl enable docker
sudo systemctl start docker

Verify Docker.

docker --version

5. Install NVIDIA Container Toolkit

Install the NVIDIA container runtime.

Follow the NVIDIA Container Toolkit installation guide appropriate for Amazon Linux 2023.

Restart Docker afterward.

sudo systemctl restart docker

Verify GPU visibility.

docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu22.04 nvidia-smi

Expected output should display the NVIDIA GPU information.

6. Prepare Hugging Face Cache

Create a local cache directory.

mkdir -p ~/.cache/huggingface

If using gated or private models, create a Hugging Face access token and export it:

export HF_TOKEN=<your_token>

7. Pull the AWS vLLM Container

Download the official AWS Deep Learning Container.

docker pull public.ecr.aws/deep-learning-containers/vllm:latest-gpu-py312-cu130-ubuntu22.04-ec2

This image includes:

• CUDA
• PyTorch
• Python 3.12
• vLLM
• Optimized GPU runtime

8. Start the vLLM Server

Run a model using Docker.

docker run --rm \
    --gpus all \
    -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -e HF_TOKEN=$HF_TOKEN \
    public.ecr.aws/deep-learning-containers/vllm:latest-gpu-py312-cu130-ubuntu22.04-ec2 \
    --model facebook/opt-125m

The server will automatically:

• Download the model (first launch)
• Cache model files
• Start an OpenAI-compatible API server
• Listen on port 8000

9. Access the API

Verify the server is running.

curl http://localhost:8000/v1/models

Example response:

{
  "data": [
    {
      "id": "facebook/opt-125m"
    }
  ]
}

10. Example: Run Gemma 4

Replace the model argument with your desired Hugging Face model.

docker run --rm \
    --gpus all \
    -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -e HF_TOKEN=$HF_TOKEN \
    public.ecr.aws/deep-learning-containers/vllm:latest-gpu-py312-cu130-ubuntu22.04-ec2 \
    --model google/gemma-4-12b

Note: Ensure your Hugging Face account has accepted the model license if the model is gated.

11. Verify GPU Usage

Check GPU utilization.

nvidia-smi

You should see:

• Python process
• CUDA memory usage
• GPU utilization

12. Stopping the Server

If running interactively:

CTRL+C

If running detached:

docker ps
docker stop <container-id>

13. Optional: Run in Background

Launch the container in detached mode.

docker run -d \
    --name vllm \
    --restart unless-stopped \
    --gpus all \
    -p 8000:8000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -e HF_TOKEN=$HF_TOKEN \
    public.ecr.aws/deep-learning-containers/vllm:latest-gpu-py312-cu130-ubuntu22.04-ec2 \
    --model google/gemma-4-12b

Useful Docker commands:

View logs:

docker logs -f vllm

Stop:

docker stop vllm

Restart:

docker restart vllm

14. Alternative: Native Installation (Advanced)

Native installation is intended only for advanced users who require a non-containerized deployment.

Requirements:

• Amazon Linux 2023
• Python 3.9 or later
• GCC/G++
• CUDA Toolkit
• Matching PyTorch with CUDA support

Example:

sudo dnf install python3 gcc gcc-c++ git -y

pip install torch torchvision torchaudio

pip install vllm

During installation, pip compiles the required CUDA and C++ extensions locally. This process is slower and more prone to version mismatches than the Docker-based deployment.

15. Troubleshooting

16. Best Practices

• Use Amazon Linux 2023 for new deployments.
• Prefer the AWS Deep Learning Container over native installations.
• Store Hugging Face models in a persistent cache volume to avoid repeated downloads.
• Use --restart unless-stopped or a systemd service to ensure automatic recovery after reboots.
• Restrict network access to the vLLM API using AWS Security Groups or a reverse proxy if the service is not intended to be publicly accessible.
• Monitor GPU utilization and memory usage with nvidia-smi to optimize model placement and performance.

17. References

• AWS Deep Learning Containers (DLC)
• vLLM Documentation
• Hugging Face Model Hub
• NVIDIA Container Toolkit Documentation