1. Purpose

This SOP describes how to use LiveBench to evaluate model quality through a production-style vLLM OpenAI-compatible API.

Environment:

LiveBench is primarily used to evaluate model quality, including reasoning, coding, mathematics, data analysis, language, and instruction following. It should be supplemented with vLLM load testing for latency and throughput measurements.

LiveBench GitHub Repository

2. Architecture

+--------------------+
| LiveBench Client   |
| EC2 / Linux / Mac  |
+---------+----------+
          |
          | HTTPS :443
          | Bearer API Key
          v
+--------------------+
| AWS ALB            |
| dev-va-vllm...     |
+---------+----------+
          |
          | HTTP :8000
          v
+--------------------+
| EC2 GPU Instance   |
| Amazon Linux 2023  |
|                    |
| Docker             |
|   └── vLLM         |
|        └── Model   |
+--------------------+

For accurate infrastructure testing, run LiveBench from a different system than the vLLM GPU server.

3. Prerequisites

The following are required:

Python 3
Git
pip
Network access to vLLM
vLLM API key

Verify Python:

python3 --version

Verify Git:

git --version

4. Verify the vLLM API

Set the API key as an environment variable:

export VLLM_API_KEY="your-api-key"

Do not place the actual API key in scripts, Git repositories, shell history, or documentation.

Verify the vLLM endpoint:

curl https://dev-va-vllm.eveon.com/v1/models \
  -H "Authorization: Bearer ${VLLM_API_KEY}"

A successful response should contain the model being served:

{
  "data": [
    {
      "id": "MODEL_ID"
    }
  ]
}

Record the exact model ID.

For example:

export MODEL="MODEL_ID"

LiveBench must use the same model ID recognized by the vLLM server.

5. Test Chat Completion

Before installing or running LiveBench, verify that inference works:

curl https://dev-va-vllm.eveon.com/v1/chat/completions \
  -H "Authorization: Bearer ${VLLM_API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"${MODEL}\",
    \"messages\": [
      {
        \"role\": \"user\",
        \"content\": \"Explain zero trust security in three sentences.\"
      }
    ],
    \"max_tokens\": 200
  }"

Do not continue until both /v1/models and /v1/chat/completions work successfully.

6. Install LiveBench

Clone LiveBench:

git clone https://github.com/LiveBench/LiveBench.git

cd LiveBench

Create a Python virtual environment:

python3 -m venv .venv

Activate it:

source .venv/bin/activate

Upgrade pip:

pip install --upgrade pip

Install LiveBench:

pip install -e .

Verify:

python run_livebench.py --help

7. Configure the Test Environment

Set the endpoint:

export VLLM_ENDPOINT="https://dev-va-vllm.eveon.com/v1"

Set the API key:

export VLLM_API_KEY="your-api-key"

Set the exact model ID obtained earlier:

export MODEL="MODEL_ID"

Verify:

echo $VLLM_ENDPOINT
echo $MODEL

Do not run:

echo $VLLM_API_KEY

on shared systems or when terminal sessions are being recorded.

8. Run a Small LiveBench Test

Do not begin with the full benchmark.

Start with reasoning and one parallel request:

python run_livebench.py \
  --model "${MODEL}" \
  --bench-name live_bench/reasoning \
  --api-base "${VLLM_ENDPOINT}" \
  --api-key-name VLLM_API_KEY \
  --parallel-requests 1 \
  --livebench-release-option 2024-11-25

This validates:

LiveBench
    ↓
HTTPS
    ↓
AWS ALB
    ↓
API Authentication
    ↓
vLLM
    ↓
Model

If successful, proceed to additional tests.

9. Test Reasoning

Run:

python run_livebench.py \
  --model "${MODEL}" \
  --bench-name live_bench/reasoning \
  --api-base "${VLLM_ENDPOINT}" \
  --api-key-name VLLM_API_KEY \
  --parallel-requests 4 \
  --livebench-release-option 2024-11-25

This evaluates the model's reasoning capabilities.

10. Test Coding

Run:

python run_livebench.py \
  --model "${MODEL}" \
  --bench-name live_bench/coding \
  --api-base "${VLLM_ENDPOINT}" \
  --api-key-name VLLM_API_KEY \
  --parallel-requests 4 \
  --livebench-release-option 2024-11-25

11. Test Mathematics

Run:

python run_livebench.py \
  --model "${MODEL}" \
  --bench-name live_bench/math \
  --api-base "${VLLM_ENDPOINT}" \
  --api-key-name VLLM_API_KEY \
  --parallel-requests 4 \
  --livebench-release-option 2024-11-25

12. Run the Full LiveBench Benchmark

Once individual tests are working, run the complete benchmark:

python run_livebench.py \
  --model "${MODEL}" \
  --bench-name live_bench \
  --api-base "${VLLM_ENDPOINT}" \
  --api-key-name VLLM_API_KEY \
  --parallel-requests 4 \
  --livebench-release-option 2024-11-25

This may take significantly longer and generate substantial inference traffic.

For repeatable comparisons, use the same LiveBench release for every model being compared.

13. Tune Parallel Requests

Start conservatively:

1 → 2 → 4 → 8

Recommended baseline:

--parallel-requests 4

For example:

python run_livebench.py \
  --model "${MODEL}" \
  --bench-name live_bench \
  --api-base "${VLLM_ENDPOINT}" \
  --api-key-name VLLM_API_KEY \
  --parallel-requests 8 \
  --livebench-release-option 2024-11-25

Increasing parallel-requests generates more simultaneous requests against vLLM.

Watch for:

API errors
HTTP 429
HTTP 5xx
Request timeouts
vLLM queue growth
GPU memory pressure

Reduce concurrency if errors occur.

14. View LiveBench Results

Display results with:

python show_livebench_result.py \
  --bench-name live_bench \
  --model-list "${MODEL}" \
  --livebench-release-option 2024-11-25

LiveBench also generates detailed category/task results that can be used for model comparison.

A useful summary is:

15. Compare Multiple Models

Run exactly the same LiveBench configuration for each model.

For example:

Model A
   ↓
Same LiveBench release
Same benchmark
Same generation parameters
Same infrastructure

Model B
   ↓
Same LiveBench release
Same benchmark
Same generation parameters
Same infrastructure

Record:

Avoid comparing scores produced by different LiveBench releases or materially different inference settings.

16. Monitor vLLM During Testing

On the vLLM EC2 instance:

watch -n 1 nvidia-smi

Monitor Docker:

docker stats vllm

Monitor logs:

docker logs -f vllm

If Prometheus monitoring is configured, also watch:

Requests running
Requests waiting
KV-cache utilization
Time to first token
Generation throughput
GPU utilization
GPU memory
Request failures

17. LiveBench vs. vLLM Performance Testing

LiveBench should not be your primary tool for determining vLLM serving capacity.

Use two separate benchmark tracks:

                Production Evaluation
                         |
             +-----------+-----------+
             |                       |
             v                       v
         LiveBench              vLLM Benchmark
             |                       |
         MODEL QUALITY          SYSTEM PERFORMANCE
             |                       |
         Reasoning                   TTFT
         Coding                      TPS
         Math                        QPS
         Language                    P95
         Instruction                 P99
         Data Analysis               Concurrency

LiveBench answers:

How capable is this model?

A vLLM serving benchmark answers:

How efficiently can our infrastructure serve this model?

18. Recommended Production Test Matrix

For each candidate model, record both quality and infrastructure performance.

This provides a much better basis for selecting a production model than LiveBench score alone.

19. Recommended Test Process

For each model:

1. Deploy model to vLLM
        ↓
2. Verify /v1/models
        ↓
3. Verify /v1/chat/completions
        ↓
4. Run small LiveBench reasoning test
        ↓
5. Run full LiveBench
        ↓
6. Record model-quality scores
        ↓
7. Run vLLM performance benchmark
        ↓
8. Test concurrency 1 / 4 / 8 / 16
        ↓
9. Record TTFT / TPS / P95
        ↓
10. Record AWS cost
        ↓
11. Compare quality / performance / cost

The final production decision should consider model quality + latency + throughput + stability + AWS cost, rather than choosing the model with the highest benchmark score alone.

LiveBench GitHub Repository

LiveBench Leaderboard