Ollama

WatchThatSite integrates with Ollama, enabling local LLM inference on your own infrastructure for complete privacy and control.

What is Ollama

Ollama is an open-source project that makes running large language models locally straightforward. You run a server on your machine, and WatchThatSite connects to it.

Installation

MacOS / Linux

1. Download from Ollama
2. Run the installer
3. Ollama starts automatically in the background

Docker

docker run -d -p 11434:11434 ollama/ollama

Verify Installation

curl http://localhost:11434/api/tags

Running Models Locally

Pull a Model

ollama pull llama2
ollama pull mistral
ollama pull neural-chat

Available models on Ollama Library.

List Downloaded Models

ollama list

Add to WatchThatSite

1. Go to Settings > AI Providers
2. Click Add Provider Key
3. Select Ollama from the provider dropdown
4. Enter the Base URL:
   • http://localhost:11434 (local machine)
   • http://your-server:11434 (remote server)
5. Optionally select specific models
6. Click Save

Recommended Models

Popular models with tool calling support:

• Llama 2 70B — Large, capable, good quality
• Mistral 7B — Fast, efficient, balanced
• Neural Chat — Optimized for conversation
• Zephyr 7B — Fast, instruction-tuned

All locally-run models are available. Download what you want.

All Available Models

Any model supporting tool calling can run on Ollama. Popular sources:

• Ollama Library — Curated models
• Hugging Face — Thousands of models
• Model Hub — Chinese models

Pull and run locally:

ollama pull llama2
ollama pull mistral
# Run in WatchThatSite by pointing to http://localhost:11434

System Requirements

Model requirements vary:

• Llama 2 7B: 4-6GB VRAM
• Llama 2 13B: 8-10GB VRAM
• Llama 2 70B: 48GB+ VRAM
• Mistral 7B: 4-6GB VRAM

More VRAM = better performance. Can run on CPU but slower.

Network Setup

Local Machine

• URL: http://localhost:11434
• No additional setup needed

Remote Server

1. Set Ollama to listen on network:

   OLLAMA_HOST=0.0.0.0:11434 ollama serve

2. Firewall: Allow port 11434 from WatchThatSite

3. URL: http://your-server-ip:11434

Docker Container

Container must be accessible from WatchThatSite:

docker run -d -p 11434:11434 \
  -e OLLAMA_HOST=0.0.0.0:11434 \
  ollama/ollama

Usage Tips

Memory Management

If running multiple models:

• Load one at a time
• Ollama unloads unused models
• Keep high-demand models preloaded

Performance Optimization

• Run on GPU if available
• Use quantized models for faster inference
• Increase batch size for higher throughput

Model Selection

• Start with 7B models for speed
• Use 13B for better quality
• 70B for highest quality (needs GPU)

Troubleshooting

"Connection Refused"

• Verify Ollama is running
• Check URL is correct
• For remote: Check firewall allows port 11434
• Test with: curl http://your-url:11434/api/tags

"Model Not Found"

• Pull model first: ollama pull model-name
• Verify model name exactly matches
• List models: ollama list
• Check model isn't unloaded

Out of Memory

• Reduce model size (use 7B instead of 13B)
• Allocate more GPU memory
• Close other applications
• Run on larger machine

Slow Performance

• Check GPU is being used
• Use smaller model
• Reduce context length
• Optimize network (for remote setup)

Remote Connection Issues

• Verify OLLAMA_HOST is set correctly
• Check firewall rules
• Test connectivity: curl http://url:11434/api/tags
• Check logs: Review Ollama server logs

Advantages

• Privacy: Models run locally, data never leaves your machine
• Offline: No internet required (after model download)
• Cost: No API charges, only hardware costs
• Control: Full control over deployment
• Customization: Can fine-tune models locally

Disadvantages

• Hardware: Needs decent GPU or lots of CPU
• Latency: Slower than cloud APIs (unless well-configured)
• Maintenance: You manage the server
• Updates: Manually update models
• Limited models: Fewer models than cloud providers

When to Use Ollama

Choose Ollama when:

• Privacy is critical
• You need offline capability
• You have dedicated hardware
• You want zero API costs
• You need model customization
• Compliance restricts cloud use

Resource Monitoring

Monitor your Ollama server:

# Check loaded models
curl http://localhost:11434/api/ps

# Get model info
curl http://localhost:11434/api/show --data '{"name":"llama2"}'

Advanced Configuration

GPU Memory

Control GPU allocation:

CUDA_VISIBLE_DEVICES=0 OLLAMA_NUM_PARALLEL=4 ollama serve

Model Loading

Pre-load models on startup by creating scripts:

#!/bin/bash
ollama pull llama2
ollama pull mistral

Run at server startup.