1. Purpose

This SOP describes a secure and production-ready deployment of vLLM on Amazon Linux 2023, using an NVIDIA GPU-based EC2 G6 or newer instance and an AWS Application Load Balancer (ALB).

The recommended architecture is:

Client / Application
        |
        | HTTPS :443
        v
+-------------------------+
| AWS Application         |
| Load Balancer           |
|                         |
| - TLS / ACM             |
| - Health Check          |
| - Access Logs           |
| - AWS WAF (optional)    |
+------------+------------+
             |
             | HTTP :8000
             | Private VPC
             v
+-------------------------+
| EC2 - Amazon Linux 2023 |
| G6 or newer GPU         |
|                         |
| Docker                  |
|   └── vLLM :8000        |
|                         |
| NVIDIA GPU              |
+-------------------------+

The EC2 instance should preferably reside in a private subnet. Port 8000 must only be accessible from the ALB security group.

2. Prerequisites

AWS Infrastructure

Prepare the following:

Choose the G6 size based on the model's GPU memory requirements.

For example:

Model
  ↓
Model weights
  +
KV cache
  +
CUDA graphs
  +
runtime overhead
  ↓
Required GPU memory

Do not size the instance based only on model weight size.

3. Configure EC2 Security Groups

Create separate security groups for the ALB and vLLM EC2 instances.

ALB Security Group

Inbound:

HTTPS  TCP/443    0.0.0.0/0
HTTP   TCP/80     0.0.0.0/0     # Optional; redirect to HTTPS

vLLM EC2 Security Group

Inbound:

TCP/8000    Source: ALB Security Group

Do not configure:

TCP/8000    0.0.0.0/0

The vLLM API should never be directly exposed to the Internet.

If Systems Manager Session Manager is used, SSH access can normally be eliminated as well.

AWS recommends using the ALB security group as the source for target-instance inbound rules. AWS ALB Security Groups documentation

4. Update Amazon Linux 2023

Connect using AWS Systems Manager Session Manager or SSH and update the instance:

sudo dnf update -y

Confirm the OS:

cat /etc/os-release

You should see:

Amazon Linux 2023

5. Verify NVIDIA GPU

Check that the GPU and NVIDIA driver are available:

nvidia-smi

Expected output should identify the NVIDIA GPU and display:

Driver Version
CUDA Version
GPU Model
GPU Memory
GPU Utilization

For example:

NVIDIA L4

or the GPU appropriate to your selected G6/newer instance.

Do not proceed with vLLM until nvidia-smi works correctly.

6. Install Docker

Install Docker:

sudo dnf install -y docker

Enable and start Docker:

sudo systemctl enable --now docker

Add the current user to the Docker group:

sudo usermod -aG docker $USER

Log out and reconnect.

Verify:

docker --version

Test:

docker run --rm hello-world

7. Configure NVIDIA Container Support

The Docker runtime must be able to expose the NVIDIA GPU to containers.

After installing/configuring NVIDIA Container Toolkit for your selected NVIDIA driver environment, verify Docker GPU access:

docker run --rm --gpus all \
  nvidia/cuda:13.0.0-base-ubuntu24.04 \
  nvidia-smi

The GPU displayed inside the container should match:

nvidia-smi

on the EC2 host.

Do not deploy vLLM until this test succeeds.

8. Create the vLLM Directory

Create a dedicated directory:

mkdir -p ~/vllm
cd ~/vllm

Create the Hugging Face model cache:

mkdir -p ~/.cache/huggingface

Recommended structure:

/home/ec2-user/
├── vllm/
│   ├── docker-compose.yaml
│   └── .env
│
└── .cache/
    └── huggingface/

9. Configure Secrets

Generate a strong vLLM API key:

openssl rand -hex 32

Create:

vi ~/vllm/.env

Example:

HF_TOKEN=hf_xxxxxxxxxxxxxxxxx
VLLM_API_KEY=<generated-secret>

Restrict permissions:

chmod 600 ~/vllm/.env

Do not store API keys directly in docker-compose.yaml.

For a more mature production deployment, store secrets in AWS Secrets Manager or Systems Manager Parameter Store instead of a local .env file.

10. Create Docker Compose Configuration

Create:

vi ~/vllm/docker-compose.yaml

Example production baseline:

services:

  vllm:
    image: vllm/vllm-openai:<PINNED_VERSION>
    container_name: vllm

    ipc: host
    shm_size: "16gb"

    ports:
      - "8000:8000"

    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface

    environment:
      HF_TOKEN: ${HF_TOKEN}
      VLLM_API_KEY: ${VLLM_API_KEY}

    command: >
      google/gemma-4-E4B-it
      --host 0.0.0.0
      --port 8000
      --max-model-len 16384
      --gpu-memory-utilization 0.90
      --max-num-seqs 16
      --enable-prefix-caching

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu

    restart: unless-stopped

For production, pin the vLLM container version rather than using:

vllm/vllm-openai:latest

This prevents an unexpected container update from changing the production environment.

11. vLLM Performance Baseline

Start conservatively with:

--gpu-memory-utilization 0.90
--max-model-len 16384
--max-num-seqs 16
--enable-prefix-caching

GPU Memory

Start with:

--gpu-memory-utilization 0.90

This leaves approximately 10% headroom for CUDA/runtime allocations.

Avoid immediately increasing this to 0.95+.

Context Length

Use:

--max-model-len 16384

unless your application genuinely requires 32K context.

For long-document workloads:

--max-model-len 32768

The larger the context window, the greater the potential KV-cache memory requirement.

Concurrency

Start with:

--max-num-seqs 16

Then benchmark:

8 → 16 → 32 → 64

Measure latency and throughput rather than assuming that a higher value is better.

Prefix Caching

Enable:

--enable-prefix-caching

This is particularly useful when requests repeatedly use the same system prompt, RAG context, agent instructions, or other shared prefixes.

vLLM Automatic Prefix Caching documentation

12. Start vLLM

Start the container:

cd ~/vllm

docker compose up -d

Verify:

docker ps

Monitor startup:

docker logs -f vllm

Wait until the API server is ready.

13. Test vLLM Locally

Test the health endpoint:

curl http://localhost:8000/health

Test authentication:

source ~/vllm/.env

curl http://localhost:8000/v1/models \
  -H "Authorization: Bearer ${VLLM_API_KEY}"

Then test inference:

curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer ${VLLM_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E4B-it",
    "messages": [
      {
        "role": "user",
        "content": "Hello. Tell me about yourself."
      }
    ],
    "max_tokens": 200
  }'

14. Create the ALB Target Group

Create an ALB target group:

Target type:       Instances
Protocol:          HTTP
Port:              8000

Register the vLLM EC2 instance.

Configure the health check:

Protocol:            HTTP
Port:                Traffic port
Path:                /health

Healthy threshold:   2
Unhealthy threshold: 3
Timeout:             5 seconds
Interval:            30 seconds

Success code:        200

AWS ALB periodically sends these health checks to determine whether the vLLM instance should receive traffic. AWS Target Group Health Checks

Verify that the EC2 target becomes:

Healthy

before proceeding.

15. Configure the ALB

Create two listeners.

HTTP

Port:       80
Protocol:   HTTP

Action:
Redirect → HTTPS:443

HTTPS

Port:       443
Protocol:   HTTPS

Certificate:
AWS Certificate Manager

Forward:
vLLM Target Group

The resulting flow becomes:

https://llm.example.com
        |
       443
        |
       ALB
        |
      HTTP
      :8000
        |
       EC2
        |
      Docker
        |
      vLLM

ALB performs TLS termination, so certificates do not need to be installed inside the vLLM container. AWS HTTPS Listeners documentation

16. Test Through ALB

Test:

curl https://llm.example.com/v1/models \
  -H "Authorization: Bearer ${VLLM_API_KEY}"

Then test inference:

curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer ${VLLM_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E4B-it",
    "messages": [
      {
        "role": "user",
        "content": "Explain zero trust security in three sentences."
      }
    ],
    "max_tokens": 200
  }'

17. Protect the Public API

Do not consider the vLLM API key the only security boundary.

For an Internet-facing ALB, consider attaching AWS WAF.

Recommended protections include:

AWS Managed Rules
        +
IP reputation rules
        +
Request rate limiting
        +
Application-specific restrictions

Also restrict which vLLM endpoints are intended for external clients.

Applications generally only need endpoints such as:

/v1/models
/v1/chat/completions
/v1/responses
/v1/completions

Administrative and diagnostic endpoints such as these should not be intentionally exposed to untrusted clients:

/metrics
/docs
/openapi.json
/tokenize
/detokenize
/version
/load

18. Monitoring

Production monitoring should cover three layers.

AWS / ALB

Monitor:

RequestCount
TargetResponseTime
HTTPCode_ELB_5XX_Count
HTTPCode_Target_5XX_Count
HealthyHostCount
UnHealthyHostCount

Enable ALB access logs when appropriate.

EC2 / GPU

Monitor:

CPU
Memory
Disk
Network

GPU utilization
GPU memory
GPU temperature
GPU power

NVIDIA DCGM Exporter can provide GPU telemetry.

vLLM

Keep /metrics private and collect it using Prometheus or another internal monitoring system.

Important metrics include:

Requests running
Requests waiting

TTFT
Time per output token
End-to-end latency

Prompt throughput
Generation throughput

KV cache utilization
Request failures

19. Production Operations

Use these commands for routine administration.

Check status:

docker compose ps

View logs:

docker logs -f vllm

Check GPU:

nvidia-smi

Restart:

docker compose restart

Stop:

docker compose down

Start:

docker compose up -d

Before upgrading vLLM, test the new pinned container version in a non-production environment.

20. Production Security Checklist

Before going live, verify:

• Amazon Linux 2023 is patched.
• G6 or newer GPU instance is appropriately sized for the model.
• EC2 is preferably deployed in a private subnet.
• Port 8000 is accessible only from the ALB security group.
• Port 8000 is not open to 0.0.0.0/0.
• ALB accepts HTTPS 443.
• ACM manages the TLS certificate.
• HTTP redirects to HTTPS.
• vLLM API authentication is enabled.
• API and Hugging Face secrets are not stored in Compose.
• Docker image version is pinned.
• Model revision is pinned where practical.
• AWS WAF is considered for Internet-facing deployments.
• /metrics and administrative endpoints are not intentionally public.
• ALB health checks report the target as healthy.
• ALB access logging is enabled as required.
• GPU and vLLM metrics are monitored.
• EC2 administration uses SSM instead of public SSH where possible.
• Production configuration is load-tested before release.

Recommended Production Architecture

                    Internet / API Clients
                              |
                         HTTPS :443
                              |
                    +---------v---------+
                    | AWS WAF           |
                    | (recommended)     |
                    +---------+---------+
                              |
                    +---------v---------+
                    | AWS ALB           |
                    |                   |
                    | ACM TLS           |
                    | Health Checks     |
                    | Access Logs       |
                    +---------+---------+
                              |
                         HTTP :8000
                              |
                   Security Group Rule
                     ALB SG → EC2 SG
                              |
                 +------------v------------+
                 | EC2                     |
                 | Amazon Linux 2023       |
                 | G6 or newer             |
                 |                         |
                 | +---------------------+ |
                 | | Docker              | |
                 | |                     | |
                 | | vLLM :8000          | |
                 | | API Key Auth        | |
                 | +----------+----------+ |
                 |            |            |
                 |        NVIDIA GPU       |
                 +-------------------------+

                   No public access :8000
                   No public SSH required

This keeps the deployment relatively simple: ALB handles HTTPS and ingress, AWS security groups provide network isolation, Docker provides the runtime, and vLLM focuses on GPU inference.