vLLM

WatchThatSite integrates with vLLM, a high-throughput inference engine for running open-source models locally with exceptional performance.

What is vLLM

vLLM is an open-source LLM inference engine optimized for throughput and latency. It runs models locally with advanced optimization techniques like PagedAttention.

Installation

Prerequisites

• CUDA-capable GPU (NVIDIA) or AMD GPU
• Python 3.9+
• CUDA 11.8+ (for GPU acceleration)

Install vLLM

pip install vllm

With GPU Support

# For NVIDIA GPUs
pip install vllm[cuda]

# For AMD GPUs
pip install vllm[rocm]

Running vLLM Server

Basic Start

vllm serve meta-llama/Llama-2-7b-hf --port 8000

With Parameters

vllm serve meta-llama/Llama-2-13b-hf \
  --port 8000 \
  --gpu-memory-utilization 0.9 \
  --max-num-seqs 256

Docker

docker run --gpus all \
  -p 8000:8000 \
  vllm/vllm:latest \
  --model meta-llama/Llama-2-7b-hf

Hugging Face Models

vLLM works with most Hugging Face models:

• Llama 2: Meta's popular model
• Mistral: Efficient 7B model
• Qwen: Alibaba's models
• Zephyr: Fine-tuned models
• Phi: Small, efficient models
• Others: 10000+ compatible models

Find models at Hugging Face Models.

Add to WatchThatSite

1. Go to Settings > AI Providers
2. Click Add Provider Key
3. Select vLLM from the provider dropdown
4. Enter:
   • Base URL: http://localhost:8000 (or your server)
   • Model Name: The model you're running (e.g., meta-llama/Llama-2-7b-hf)
   • API Key (optional): If you configured authentication
5. Optionally select specific models
6. Click Save

Configuration

Basic URL Setup

• Local: http://localhost:8000
• Remote: http://your-server:8000
• Custom port: http://localhost:9000

Model Name

Must match deployed model:

• Full path: meta-llama/Llama-2-7b-hf
• Custom name if configured
• Check with: curl http://localhost:8000/v1/models

Authentication (Optional)

Add authentication header:

vllm serve meta-llama/Llama-2-7b-hf \
  --api-key sk-1234567890

Performance Tuning

GPU Memory

# Use 90% of GPU memory
vllm serve model-name --gpu-memory-utilization 0.9

Parallelism

# Parallel workers
vllm serve model-name --tensor-parallel-size 2

Request Batching

# Max concurrent requests
vllm serve model-name --max-num-seqs 256

System Requirements

Minimum

• Small models (7B): 8GB VRAM
• Medium models (13B): 16GB VRAM
• Large models (70B): 48GB VRAM

Recommended

• Dedicated GPU
• High-bandwidth GPU interconnect
• Sufficient host RAM (2x VRAM)

Advantages Over Ollama

Recommended Models

Latest models with tool calling support:

• Llama 3.1 70B — Large, powerful, excellent quality
• Mistral Large — Fast, capable, balanced
• Qwen 72B — Alibaba's powerful model
• Zephyr 7B — Fast, instruction-tuned

All vLLM-compatible models are available.

All Available Models

Any model supporting tool calling and vLLM compatibility can run. Popular sources:

• Hugging Face Models — 10,000+ models
• vLLM Docs — Supported models list
• Llama Models — Meta's official models

Run any model:

vllm serve meta-llama/Llama-3.1-70b-instruct --port 8000
# Then point WatchThatSite to http://localhost:8000

Troubleshooting

"Connection Refused"

• Verify vLLM server is running
• Check port is correct
• Test: curl http://localhost:8000/v1/models

"Model Not Found"

• Verify model path is exact
• Pull model from Hugging Face first:

  huggingface-cli download meta-llama/Llama-2-7b-hf

• Check model is compatible

Out of Memory

• Use smaller model
• Reduce gpu-memory-utilization
• Reduce max-num-seqs
• Add more GPU memory or GPUs

Slow Performance

• Check GPU utilization: nvidia-smi
• Increase tensor-parallel-size
• Use better GPU
• Optimize prompt/batch size

CUDA Errors

• Verify CUDA version matches vLLM requirements
• Update GPU drivers
• Check GPU is compatible
• Try CPU mode for debugging

Model Download Issues

• Check internet connection
• Verify Hugging Face token if private model
• Set HF token: huggingface-cli login
• Manually download and point to local path

Monitoring

Health Check

curl http://localhost:8000/health

List Models

curl http://localhost:8000/v1/models

Check Utilization

nvidia-smi  # For NVIDIA
rocm-smi    # For AMD

Advanced Setup

Multiple Models

Load different models on different ports:

# Terminal 1
vllm serve model1 --port 8000

# Terminal 2
vllm serve model2 --port 8001

Distributed Inference

Use tensor parallelism across GPUs:

vllm serve model-name \
  --tensor-parallel-size 2 \
  --distributed-executor-backend ray

Production Deployment

For production:

• Use systemd service or Docker
• Enable logging and monitoring
• Set up reverse proxy (nginx)
• Configure authentication
• Enable rate limiting
• Set up auto-restart

When to Use vLLM

Choose vLLM when:

• Throughput matters (many requests)
• You need low latency
• You have GPU available
• You want maximum performance
• Running complex deployments