Production Deployment Runbook: Hosting Ollama and RAG Index

2026-07-05 | 3 min read

Production Deployment Runbook: Hosting Ollama and RAG Index

Running conversational AI features, semantic search, and mechanical card synergies in a production environment (like takescake.com) requires a reliable hosting configuration.

This runbook covers how to set up Ollama, manage model keep-alive policies, configure hardware acceleration (CUDA/Vulkan/CPU), and automate nightly database updates.

1. Deploying Ollama in Production (Linux / systemd)

On production Linux servers, Ollama should run as a background service managed by systemd. This ensures that Ollama restarts automatically if it crashes or if the host machine reboots.

Step-by-Step systemd Configuration

  1. Create a dedicated system user for Ollama (optional but recommended for security):

    sudo useradd -r -s /bin/false ollama

  2. Create the systemd service file at /etc/systemd/system/ollama.service:

    [Unit]
Description=Ollama Service
After=network.target

[Service]
Type=simple
User=ollama
Group=ollama
ExecStart=/usr/local/bin/ollama serve
Restart=always
RestartSec=5
Environment="OLLAMA_HOST=127.0.0.1:11434"
# Keep loaded models in memory permanently (infinite keep-alive)
Environment="OLLAMA_KEEP_ALIVE=-1"
# Enable multiple request handling
Environment="OLLAMA_NUM_PARALLEL=1"

[Install]
WantedBy=multi-user.target

  3. Enable and start the service:

    sudo systemctl daemon-reload
sudo systemctl enable ollama
sudo systemctl start ollama

2. Hardware Toggles: CUDA, Vulkan, and CPU Only

Depending on your hosting provider, you will run Ollama on either GPU (CUDA/Vulkan) or CPU.

A. CUDA Mode (Recommended for GPU servers)

If your production server has an NVIDIA GPU (e.g. on AWS, GCP, or runpod), Ollama will auto-detect it.

  • Troubleshooting Driver Mismatches: If your driver version is older than the CUDA compiler version, you may hit a PTX compilation error. Ensure your NVIDIA driver is updated, or override the LLM library manually: 
    Environment="OLLAMA_LLM_LIBRARY=cuda_v11"

B. Vulkan Mode (Alternative GPU Mode)

If you are running on a server or workstation that has driver-level CUDA overrides (such as Windows with mismatching CUDA Toolkits), you can force Ollama to use the Vulkan graphics compute pipeline instead:

Environment="OLLAMA_LLM_LIBRARY=vulkan"

C. CPU-Only Mode (Recommended for Standard VPS)

If you deploy to a standard cloud VPS (like DigitalOcean or Linode) without a dedicated GPU:

  • Ollama will automatically fall back to CPU execution.
  • To prevent memory hogging, limit the parallel threads and allocate standard memory pools: 
    Environment="OLLAMA_NUM_PARALLEL=1"
# Disable GPU loading entirely to avoid driver search overhead
Environment="CUDA_VISIBLE_DEVICES="
Environment="OLLAMA_CUDA_VISIBLE_DEVICES="

3. Keeping Models "Always Online"

By default, Ollama unloads a model from memory (RAM or VRAM) after 5 minutes of inactivity to conserve resources. On a public website, this is undesirable because the first user who triggers a synergy check or chat prompt will experience a 20-30 second lag while the model loads from disk.

To make the models always online for sub-millisecond, instant response times:

  • Set OLLAMA_KEEP_ALIVE="-1" in your systemd service environment (or system environment variables).
  • This keeps loaded models (embeddinggemma and gemma4:e2b) cached in memory indefinitely.

4. Reverse Proxying with Caddy

To secure your API calls and enable conversational features on the web, configure Caddy (which serves takescake.com) to reverse proxy request routes to your local node app and Ollama instance if exposing endpoints.

Example Caddyfile configuration block:

takescake.com {
    # Reverse proxy main Next.js App
    reverse_proxy localhost:3000

    # Optional: Securely expose Ollama locally for server-to-server calls
    # reverse_proxy /api/ollama/* localhost:11434 {
    #     header_up Host {upstream_host}
    # }
}

5. Nightly Database Update Loop

MTG sets and market prices change constantly. To keep your database current:

A. Built-in Automation Scheduler (Node Server)

If running the custom server with automation enabled, add these variables to your production .env file:

AUTOMATION_ENABLED=1
AUTO_PRICES=1
AUTO_CARDS=1
AUTO_EMBEDDINGS=1
# Incremental embeds only (faster)
INCREMENTAL=1

This will run:

  • prices:build - hourly (caches TCGplayer prices locally to tcgcsv-prices.json).
  • cards:update - weekly on Sunday (pulls new sets from Scryfall).
  • cards:embeddings - triggers incrementally to generate vectors for any newly released cards.

B. Standard Linux Crontab (External)

If you prefer system cron tasks, add these jobs to your root crontab (crontab -e):

# Build prices daily at 21:10 UTC (10 mins after TCGcsv updates)
10 21 * * * cd /srv/takescake && bun run prices:build >> /var/log/takescake/prices.log 2>&1

# Update cards index weekly on Sunday at 03:00 UTC
0 3 * * 0 cd /srv/takescake && bun run cards:update >> /var/log/takescake/cards-index.log 2>&1

# Run incremental embeddings sync weekly on Sunday at 04:00 UTC
0 4 * * 0 cd /srv/takescake && INCREMENTAL=1 bun run cards:embeddings >> /var/log/takescake/embeddings.log 2>&1

By following this runbook, your production server will remain performant, up to date, and resilient under high traffic!

Related Posts