Most guides to self-hosted AI stop at ollama run llama3. You get the model running, you have a terminal prompt, and the guide says congratulations. Then you try to use it seriously — share it with a teammate, hook it into your editor, run two models at once, or figure out why responses started getting weird — and you discover that Ollama alone is not a stack. It's one layer of a stack, and you've been building on bare concrete.

A real local AI setup looks like the diagram above. Eight components working together, each solving a specific problem that no other component solves. Remove any one and you either lose a capability or you're running something that will fail quietly in ways that are hard to debug.

This guide builds the full stack in three tiers. Tier 1 is the beginner setup: Ollama, Open WebUI, and Traefik. Thirty minutes, three containers, works immediately. Tier 2 adds LiteLLM as a proxy layer, Qdrant as a vector store for RAG, and Langfuse for observability. Tier 3 adds monitoring, multi-user authentication, and the configuration choices that make a shared team deployment trustworthy. Every tier ships with a complete, copy-pasteable Docker Compose file.

Before you start: this guide assumes Docker and Docker Compose are installed. If they're not, the What is Docker primer covers the concepts, and the install section of the AI agent stack guide has the full installation steps for Ubuntu and Debian.

What Does Each Component Actually Do?

The architecture diagram shows the flow. Here's why every component exists and what breaks without it.

What Is Ollama and Why Is It the Foundation?

Ollama wraps llama.cpp — the high-performance C++ inference library — behind a clean REST API and a CLI. It handles GPU memory allocation, model layer distribution across devices, automatic quantization selection, and model storage as content-addressed layers. You pull a model the same way you pull a Docker image. Switch models in one command. The API is OpenAI-compatible, which means anything built for the OpenAI API works with Ollama without modification.

What Ollama does not handle: multi-user auth, spend tracking, request routing across multiple model instances, fallback to cloud providers when the local model is slow or unavailable, and observability. Those gaps are why the rest of the stack exists.

The local LLM complete guide covers Ollama installation, hardware requirements, model selection, and quantization in depth. Read it first if you're new to running local models. This guide treats Ollama as a configured dependency and focuses on everything that sits on top of it.

What Is Open WebUI and Why Not Just Use the Terminal?

Open WebUI is a self-hosted chat interface that connects to Ollama (and any other OpenAI-compatible backend). It gives you persistent conversation history, model switching without restarting anything, file uploads, image generation integration, Retrieval Augmented Generation (RAG) from your own documents, multi-user accounts with role-based access, and a web interface that works from any browser on your network.

The terminal interface disappears when you close it. Open WebUI persists. Every conversation is stored. You can search your chat history. You can share a link to a specific chat with a teammate. You can upload a PDF and ask questions about it. The terminal is for testing. Open WebUI is for working.

What Is LiteLLM and Why Does It Go Between Open WebUI and Ollama?

LiteLLM exposes a unified OpenAI-compatible API in front of 100+ LLM providers, handles load balancing, fallback routing, virtual key management, rate limiting, and spend tracking — all in a single proxy container.

Without LiteLLM, Open WebUI talks directly to Ollama. That works for a single user with one model. It breaks the moment you want to:

  • Route slow or complex requests to a cloud model while fast/simple requests stay local
  • Add a second Ollama instance (second GPU, second machine) and split load between them
  • Set per-user or per-team spending limits across all models
  • Give different users different model access without giving them all direct API access
  • Log every request centrally for debugging and cost analysis
  • Switch from Ollama to vLLM or a different inference backend without changing Open WebUI's config LiteLLM handles all of this. Load balancing is built in: you configure round-robin, least-busy, usage-based, or latency-based routing across multiple API keys or providers. Fallbacks are declarative: if the primary model fails, LiteLLM automatically retries the next provider in the list with no changes to your application code.

What Is a Vector Database and Why Does RAG Need One?

RAG (Retrieval Augmented Generation) is the technique that lets an LLM answer questions about documents it wasn't trained on — your internal docs, your codebase, your notes, your company's knowledge base. The process: ingest documents, chunk them into pieces, generate embedding vectors for each chunk, store those vectors in a vector database, then at query time embed the user's question and search for the most semantically similar chunks to inject as context.

A vector database stores these embedding vectors and makes similarity search fast. Qdrant is the right choice for this stack: purpose-built for vector search, has a REST API, supports payload filtering (search by semantic similarity AND filter by metadata), and handles collections that grow to millions of vectors without restructuring. Open WebUI's built-in RAG uses Qdrant when configured. So does every other serious RAG pipeline.

What Does Traefik Do and Why Not Just Expose Ports?

Traefik is a reverse proxy that sits in front of all your services. It handles TLS termination (HTTPS certificates from Let's Encrypt, automatically renewed), routes incoming requests to the right container, and provides authentication middleware. Without it, you'd either expose Open WebUI and LiteLLM on raw HTTP, manage certificates manually, or set up Nginx with static configuration files for each service.

Traefik reads Docker labels. You add four labels to a service definition and it appears on a subdomain with a valid HTTPS certificate. No Nginx config files. No manual certbot renew cron jobs. No static upstream blocks that break when container IPs change. Adding a new service is four label lines.

What Does Langfuse Do and Why Do You Need Observability?

Langfuse records every LLM interaction as a structured trace: the full prompt, the model and parameters, every tool call and its result, the response, token counts, and latency at each step. Without it, when your agent gives a wrong answer, you have no systematic way to find out why.

LiteLLM integrates with Langfuse natively — set two environment variables and every request flowing through the proxy is automatically traced. You get dashboards showing which models are being used, which queries are failing, where latency is coming from, and what each model run actually cost. This is the difference between running an LLM stack and operating one.

If you're enjoying what you've read so far, we recommend Build Your Own AI Coding Assistant With Access to Your Entire Codebase (2026) as your next read.

How Do You Choose Between the Three Tiers?

Before writing any configuration, match your situation to the right tier.

Tier 1: Beginner — Solo developer or single user. You want local inference with a proper web UI instead of a terminal. You don't need multi-user access, cost tracking, or production-grade routing. You have one machine with one GPU. Setup time: 30 minutes.

Tier 2: Power User — You've outgrown Tier 1 because you want to access the stack from multiple devices, you want RAG on your own documents, you want LiteLLM's routing so you can fall back to a cloud model when the local model is too slow for a task, or you want to understand what your stack is actually doing. Setup time: 90 minutes.

Tier 3: Small Team — Two to fifteen people sharing a stack. You need user accounts, shared document collections, per-user spending limits, monitoring that tells you when something breaks, and the security configuration that makes it safe to put this behind a domain name and share it with people who aren't developers. Setup time: half a day.

Elevated

Want this stack on a server instead of your desk? An Elevated VPS from SudoSecurity gives it an always-on home with real support.

See Elevated VPS plans →

Tier 1: The Beginner Stack

Three containers. Ollama for inference, Open WebUI for the chat interface, Traefik for HTTPS routing. This is the minimum setup that's actually comfortable to use daily.

Prerequisites

  • A server or desktop with Docker installed
  • A domain name with DNS access (for Traefik's Let's Encrypt certificates)
  • Ollama installed on the host with at least one model pulled If you're running on the same machine where Ollama is already installed (not in Docker), Traefik and Open WebUI can connect to it at http://host.docker.internal:11434.

Directory Setup

mkdir -p /opt/ai-stack/traefik/certs touch /opt/ai-stack/traefik/certs/acme.json chmod 600 /opt/ai-stack/traefik/certs/acme.json docker network create ai-proxy 

Tier 1 Docker Compose

# /opt/ai-stack/docker-compose.yml version: "3.9" networks: ai-proxy: external: true services: traefik: image: traefik:v3.1 container_name: ai-traefik restart: unless-stopped networks: - ai-proxy ports: - "80:80" - "443:443" command: - "--providers.docker=true" - "--providers.docker.exposedbydefault=false" - "--providers.docker.network=ai-proxy" - "--entrypoints.web.address=:80" - "--entrypoints.web.http.redirections.entrypoint.to=websecure" - "--entrypoints.web.http.redirections.entrypoint.scheme=https" - "--entrypoints.websecure.address=:443" - "--certificatesresolvers.le.acme.email=you@yourdomain.com" - "--certificatesresolvers.le.acme.storage=/certs/acme.json" - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web" - "--log.level=WARN" volumes: - /var/run/docker.sock:/var/run/docker.sock:ro - /opt/ai-stack/traefik/certs:/certs open-webui: image: ghcr.io/open-webui/open-webui:main container_name: ai-open-webui restart: unless-stopped networks: - ai-proxy extra_hosts: - "host.docker.internal:host-gateway" volumes: - open_webui_data:/app/backend/data environment: OLLAMA_BASE_URL: http://host.docker.internal:11434 WEBUI_SECRET_KEY: "CHANGE_THIS_generate_with_openssl_rand_hex_32" # Disable signup after creating your first account ENABLE_SIGNUP: "true" labels: - "traefik.enable=true" - "traefik.http.routers.open-webui.rule=Host(`ai.yourdomain.com`)" - "traefik.http.routers.open-webui.entrypoints=websecure" - "traefik.http.routers.open-webui.tls.certresolver=le" - "traefik.http.services.open-webui.loadbalancer.server.port=8080" volumes: open_webui_data: 

Start it:

cd /opt/ai-stack docker compose up -d # Watch for certificate issuance docker logs ai-traefik -f # Look for: "Retrieving certificate" and "Certificate obtained successfully" 

Navigate to https://ai.yourdomain.com. Create your account on first load. Set ENABLE_SIGNUP: "false" in the compose file immediately after and restart the container — otherwise anyone who can reach your domain can create an account.

# Disable signups sed -i 's/ENABLE_SIGNUP: "true"/ENABLE_SIGNUP: "false"/' docker-compose.yml docker compose up -d 

We recommend reading How to Run a Local LLM on Your PC in 2026 (Complete Guide) to continue reading our selection of content. The Tier 1 stack you just built needs Ollama running on the host with the right model pulled and the right GPU configuration. That guide covers every hardware tier, model selection, quantization settings, and the Ollama setup that powers everything above it.

https://sudosecurity.org/run-local-llm-on-your-pc-complete-guide-2026/

Tier 2: The Power User Stack

Tier 2 adds LiteLLM as the proxy layer between Open WebUI and your inference backends, Qdrant as the vector store for RAG, and Langfuse for tracing every request. This is where the stack becomes genuinely powerful.

The Environment File

Create /opt/ai-stack/.env. Never commit this to version control.

# Generate secrets: # openssl rand -hex 32 (for most fields) # openssl rand -base64 32 (for Langfuse fields) # ── Open WebUI ────────────────────────────────────────────── WEBUI_SECRET_KEY=generate_with_openssl_rand_hex_32 # ── LiteLLM ───────────────────────────────────────────────── LITELLM_MASTER_KEY=sk-generate_with_openssl_rand_hex_32 LITELLM_DB_PASSWORD=generate_strong_password # Cloud providers (optional -- leave empty to use local only) OPENAI_API_KEY= ANTHROPIC_API_KEY= # ── Qdrant ────────────────────────────────────────────────── QDRANT_API_KEY=generate_with_openssl_rand_hex_32 # ── Langfuse ──────────────────────────────────────────────── LANGFUSE_SECRET_KEY=generate_with_openssl_rand_base64_32 LANGFUSE_NEXTAUTH_SECRET=generate_with_openssl_rand_base64_32 LANGFUSE_SALT=generate_with_openssl_rand_hex_64 LANGFUSE_ENCRYPTION_KEY=generate_with_openssl_rand_hex_64 LANGFUSE_DB_PASSWORD=generate_strong_password CLICKHOUSE_PASSWORD=generate_strong_password 

Tier 2 Docker Compose

# /opt/ai-stack/docker-compose.yml version: "3.9" networks: ai-proxy: external: true ai-internal: driver: bridge volumes: open_webui_data: qdrant_data: litellm_postgres_data: langfuse_postgres_data: clickhouse_data: clickhouse_logs: services: # ── Traefik (same as Tier 1) ───────────────────────────── traefik: image: traefik:v3.1 container_name: ai-traefik restart: unless-stopped networks: - ai-proxy ports: - "80:80" - "443:443" command: - "--providers.docker=true" - "--providers.docker.exposedbydefault=false" - "--providers.docker.network=ai-proxy" - "--entrypoints.web.address=:80" - "--entrypoints.web.http.redirections.entrypoint.to=websecure" - "--entrypoints.websecure.address=:443" - "--certificatesresolvers.le.acme.email=you@yourdomain.com" - "--certificatesresolvers.le.acme.storage=/certs/acme.json" - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web" - "--log.level=WARN" volumes: - /var/run/docker.sock:/var/run/docker.sock:ro - /opt/ai-stack/traefik/certs:/certs # ── PostgreSQL for LiteLLM ──────────────────────────────── litellm-postgres: image: postgres:16-alpine container_name: ai-litellm-postgres restart: unless-stopped networks: - ai-internal environment: POSTGRES_DB: litellm POSTGRES_USER: litellm POSTGRES_PASSWORD: ${LITELLM_DB_PASSWORD} volumes: - litellm_postgres_data:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U litellm -d litellm"] interval: 10s retries: 5 # ── Redis for LiteLLM caching ──────────────────────────── redis: image: redis:7-alpine container_name: ai-redis restart: unless-stopped networks: - ai-internal command: redis-server --maxmemory 512mb --maxmemory-policy allkeys-lru # ── LiteLLM Proxy ───────────────────────────────────────── litellm: image: ghcr.io/berriai/litellm:main-stable container_name: ai-litellm restart: unless-stopped networks: - ai-internal - ai-proxy extra_hosts: - "host.docker.internal:host-gateway" depends_on: litellm-postgres: condition: service_healthy volumes: - /opt/ai-stack/litellm/config.yaml:/app/config.yaml:ro environment: DATABASE_URL: postgresql://litellm:${LITELLM_DB_PASSWORD}@litellm-postgres:5432/litellm LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY} REDIS_HOST: redis REDIS_PORT: 6379 # Cloud provider keys (passed through to backends when needed) OPENAI_API_KEY: ${OPENAI_API_KEY} ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY} # Langfuse tracing LANGFUSE_PUBLIC_KEY: "pk-lf-..." # Fill after Langfuse is running LANGFUSE_SECRET_KEY: ${LANGFUSE_SECRET_KEY} LANGFUSE_HOST: http://langfuse-web:3000 command: ["--config", "/app/config.yaml", "--port", "4000", "--detailed_debug"] labels: - "traefik.enable=true" - "traefik.http.routers.litellm.rule=Host(`llm.yourdomain.com`)" - "traefik.http.routers.litellm.entrypoints=websecure" - "traefik.http.routers.litellm.tls.certresolver=le" - "traefik.http.services.litellm.loadbalancer.server.port=4000" # ── Open WebUI ──────────────────────────────────────────── open-webui: image: ghcr.io/open-webui/open-webui:main container_name: ai-open-webui restart: unless-stopped networks: - ai-internal - ai-proxy depends_on: - litellm - qdrant volumes: - open_webui_data:/app/backend/data environment: # Point at LiteLLM, not Ollama directly OPENAI_API_KEY: ${LITELLM_MASTER_KEY} OPENAI_API_BASE_URL: http://litellm:4000 WEBUI_SECRET_KEY: ${WEBUI_SECRET_KEY} ENABLE_SIGNUP: "false" # RAG configuration RAG_EMBEDDING_ENGINE: ollama RAG_OLLAMA_BASE_URL: http://host.docker.internal:11434 QDRANT_URI: http://qdrant:6333 QDRANT_API_KEY: ${QDRANT_API_KEY} RAG_EMBEDDING_MODEL: nomic-embed-text CHUNK_SIZE: 1000 CHUNK_OVERLAP: 100 extra_hosts: - "host.docker.internal:host-gateway" labels: - "traefik.enable=true" - "traefik.http.routers.open-webui.rule=Host(`ai.yourdomain.com`)" - "traefik.http.routers.open-webui.entrypoints=websecure" - "traefik.http.routers.open-webui.tls.certresolver=le" - "traefik.http.services.open-webui.loadbalancer.server.port=8080" # ── Qdrant ──────────────────────────────────────────────── qdrant: image: qdrant/qdrant:latest container_name: ai-qdrant restart: unless-stopped networks: - ai-internal environment: QDRANT__SERVICE__API_KEY: ${QDRANT_API_KEY} volumes: - qdrant_data:/qdrant/storage # Internal only -- no external exposure # ── Langfuse PostgreSQL ─────────────────────────────────── langfuse-postgres: image: postgres:16-alpine container_name: ai-langfuse-postgres restart: unless-stopped networks: - ai-internal environment: POSTGRES_DB: langfuse POSTGRES_USER: langfuse POSTGRES_PASSWORD: ${LANGFUSE_DB_PASSWORD} volumes: - langfuse_postgres_data:/var/lib/postgresql/data healthcheck: test: ["CMD-SHELL", "pg_isready -U langfuse -d langfuse"] interval: 10s retries: 5 # ── ClickHouse (Langfuse trace storage) ────────────────── clickhouse: image: clickhouse/clickhouse-server:24.8-alpine container_name: ai-clickhouse restart: unless-stopped networks: - ai-internal environment: CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD} volumes: - clickhouse_data:/var/lib/clickhouse - clickhouse_logs:/var/log/clickhouse-server ulimits: nofile: { soft: 262144, hard: 262144 } healthcheck: test: ["CMD", "clickhouse-client", "--password", "${CLICKHOUSE_PASSWORD}", "--query", "SELECT 1"] interval: 10s retries: 5 # ── Langfuse Web ────────────────────────────────────────── langfuse-web: image: langfuse/langfuse:3 container_name: ai-langfuse-web restart: unless-stopped networks: - ai-internal - ai-proxy depends_on: langfuse-postgres: condition: service_healthy clickhouse: condition: service_healthy environment: DATABASE_URL: postgresql://langfuse:${LANGFUSE_DB_PASSWORD}@langfuse-postgres:5432/langfuse CLICKHOUSE_URL: http://default:${CLICKHOUSE_PASSWORD}@clickhouse:8123 NEXTAUTH_URL: https://traces.yourdomain.com NEXTAUTH_SECRET: ${LANGFUSE_NEXTAUTH_SECRET} SALT: ${LANGFUSE_SALT} ENCRYPTION_KEY: ${LANGFUSE_ENCRYPTION_KEY} TELEMETRY_ENABLED: "false" AUTH_DISABLE_SIGNUP: "false" # Disable after first user labels: - "traefik.enable=true" - "traefik.http.routers.langfuse.rule=Host(`traces.yourdomain.com`)" - "traefik.http.routers.langfuse.entrypoints=websecure" - "traefik.http.routers.langfuse.tls.certresolver=le" - "traefik.http.services.langfuse.loadbalancer.server.port=3000" # ── Langfuse Worker ─────────────────────────────────────── langfuse-worker: image: langfuse/langfuse-worker:3 container_name: ai-langfuse-worker restart: unless-stopped networks: - ai-internal depends_on: langfuse-postgres: condition: service_healthy clickhouse: condition: service_healthy environment: DATABASE_URL: postgresql://langfuse:${LANGFUSE_DB_PASSWORD}@langfuse-postgres:5432/langfuse CLICKHOUSE_URL: http://default:${CLICKHOUSE_PASSWORD}@clickhouse:8123 SALT: ${LANGFUSE_SALT} ENCRYPTION_KEY: ${LANGFUSE_ENCRYPTION_KEY} TELEMETRY_ENABLED: "false" 

LiteLLM Configuration File

The config.yaml drives LiteLLM's routing logic. Create /opt/ai-stack/litellm/config.yaml:

# /opt/ai-stack/litellm/config.yaml model_list: # Primary: local Ollama models - model_name: qwen2.5:14b litellm_params: model: ollama/qwen2.5:14b api_base: http://host.docker.internal:11434 stream: true - model_name: qwen2.5-coder:14b litellm_params: model: ollama/qwen2.5-coder:14b api_base: http://host.docker.internal:11434 - model_name: deepseek-r1:14b litellm_params: model: ollama/deepseek-r1:14b api_base: http://host.docker.internal:11434 # Embedding model (for RAG) - model_name: nomic-embed-text litellm_params: model: ollama/nomic-embed-text api_base: http://host.docker.internal:11434 # Cloud fallbacks (only active when API keys are set) - model_name: gpt-4o litellm_params: model: openai/gpt-4o api_key: os.environ/OPENAI_API_KEY - model_name: claude-sonnet litellm_params: model: anthropic/claude-sonnet-4-5 api_key: os.environ/ANTHROPIC_API_KEY # Router group: smart auto-routing - model_name: auto model_info: mode: embeddings litellm_params: model: ollama/qwen2.5:14b api_base: http://host.docker.internal:11434 router_settings: # Route fast requests local, slow/complex requests to cloud routing_strategy: latency-based-routing fallbacks: - {"qwen2.5:14b": ["gpt-4o"]} - {"qwen2.5-coder:14b": ["claude-sonnet"]} litellm_settings: # Semantic caching -- never pay twice for the same prompt cache: true cache_params: type: redis host: redis port: 6379 ttl: 600 # Cache for 10 minutes # Send all traces to Langfuse automatically success_callback: ["langfuse"] failure_callback: ["langfuse"] # Drop params that local models don't support drop_params: true # Retry on failure before fallback num_retries: 2 request_timeout: 120 general_settings: master_key: os.environ/LITELLM_MASTER_KEY database_url: os.environ/DATABASE_URL # Virtual key management (requires PostgreSQL) store_model_in_db: true # UI for the LiteLLM admin panel ui_access_mode: all allow_user_auth: true 

What the LiteLLM Config Actually Does

routing_strategy: latency-based-routing tracks response time per model and routes to the fastest available option. If your local Qwen2.5 14B is crunching a previous request and latency spikes, LiteLLM shifts the next request to a different instance or cloud fallback. Invisible to the user.

**fallbacks** defines what happens when a model fails entirely (not just slow — actually errors). If qwen2.5:14b returns a 503, the request automatically retries against gpt-4o. This is the config that makes your stack resilient to inference failures. The user sees a slightly slower response; they never see an error.

cache: true with Redis stores semantic embeddings of recent queries. If two users ask the same question within ten minutes, the second answer comes from the cache in milliseconds. The model never gets called for the second request. For common questions in a team context, this can cut your inference load by 30-50%.

**drop_params: true** silently removes parameters that local Ollama models don't support (like response_format: {"type": "json_schema"} which is an OpenAI-only feature). Without this, requests that work fine with cloud models fail with local ones because Open WebUI sometimes sends cloud-specific parameters.

Setting Up Langfuse After First Boot

Navigate to https://traces.yourdomain.com and create the first admin account. Then:

# Get your API keys from Langfuse's UI: Settings → API Keys # Copy the public key (pk-lf-...) and secret key (sk-lf-...) # Update your .env echo "LANGFUSE_PUBLIC_KEY=pk-lf-yourkeyhere" >> /opt/ai-stack/.env # Update the LiteLLM service environment in docker-compose.yml # Replace "pk-lf-..." with your actual public key # Restart LiteLLM to pick up the new config docker compose restart litellm 

Within minutes, every request flowing through LiteLLM appears as a trace in Langfuse. Click any trace to see the full prompt, the model response, token counts, and latency breakdown. If a response was wrong, you'll see exactly what the model received and returned.

We recommend reading Build Your Own AI Agent Stack: The Complete Self-Hosted Guide for 2026 to continue reading our selection of content. The Tier 2 stack you have now is the inference and interface layer. That guide builds the automation and orchestration layer on top of it — n8n workflows, Flowise agent builders, and multi-step RAG pipelines that turn your AI stack into something that actually does work for you.

Tier 3: The Small Team Stack

Tier 3 adds three things: infrastructure monitoring with Prometheus and Grafana, production-grade access control with Open WebUI's organization and group permissions, and the security hardening that makes this safe to run for a team. The Docker Compose additions below build on the Tier 2 file.

Adding Prometheus and Grafana

LiteLLM exposes Prometheus metrics at /metrics when configured. Open WebUI doesn't have native metrics, but Node Exporter covers the host.

# Add to docker-compose.yml prometheus: image: prom/prometheus:v2.52.0 container_name: ai-prometheus restart: unless-stopped networks: - ai-internal command: - "--config.file=/etc/prometheus/prometheus.yml" - "--storage.tsdb.retention.time=30d" - "--web.enable-lifecycle" volumes: - /opt/ai-stack/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro - prometheus_data:/prometheus # Internal only -- access via Grafana grafana: image: grafana/grafana:11.0.0 container_name: ai-grafana restart: unless-stopped networks: - ai-internal - ai-proxy depends_on: - prometheus environment: GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD} GF_ANALYTICS_REPORTING_ENABLED: "false" GF_SERVER_ROOT_URL: "https://grafana.yourdomain.com" volumes: - grafana_data:/var/lib/grafana labels: - "traefik.enable=true" - "traefik.http.routers.grafana.rule=Host(`grafana.yourdomain.com`)" - "traefik.http.routers.grafana.entrypoints=websecure" - "traefik.http.routers.grafana.tls.certresolver=le" - "traefik.http.services.grafana.loadbalancer.server.port=3000" 

Create /opt/ai-stack/prometheus/prometheus.yml:

global: scrape_interval: 15s scrape_configs: - job_name: 'litellm' static_configs: - targets: ['litellm:4000'] metrics_path: /metrics - job_name: 'qdrant' static_configs: - targets: ['qdrant:6333'] metrics_path: /metrics - job_name: 'node' static_configs: - targets: ['host.docker.internal:9100'] 

The full Prometheus and Grafana setup — alert rules, dashboard configuration, and PromQL for the queries that actually matter — is in the Prometheus and Grafana complete guide. The setup above wires them in. That guide fills them with content.

Team Access Control in Open WebUI

Open WebUI's permission model works at three levels: global defaults, workspaces, and individual user roles. For a small team:

Global settings (Admin Panel → Settings):

Default user role: User (not Admin) Enable model file uploads: Admin only Enable community sharing: Disabled Chat export: Enabled for all Document upload for RAG: Enabled for all 

User roles:

  • Admin: Full access including model management, user management, and system settings. One or two people.
  • User: Can chat, use RAG, access models permitted by the admin. The default for everyone else.
  • Pending: New signups before approval. Use this if you want to vet users before they get access. Model filtering: In Admin Panel → Models, you can restrict which models are visible to non-admin users. If you have cloud fallback models configured in LiteLLM but don't want users to accidentally run expensive GPT-4o queries, hide those models from the UI for regular users.

Workspaces (Admin Panel → Workspaces): Group users and share document collections within a workspace without exposing those documents to everyone. A team of ten with two separate projects can have separate document namespaces without separate Open WebUI instances.

Securing Remote Access

For team access from outside the local network, you have two options. The simpler one: Tailscale. The more powerful one: Teleport.

Tailscale: Every team member installs Tailscale, joins the same tailnet, and connects to Open WebUI via the internal Tailscale IP. No public-facing domain needed. Works in minutes. Fine for small teams with technical members.

Teleport: Puts every service behind hardware-MFA-required, zero-trust access control. Every access attempt is logged, audited, and tied to a specific user identity. Short-lived certificates instead of passwords. The Teleport self-hosted guide covers the full setup. At SudoSecurity, we proxy Open WebUI through Teleport rather than exposing it directly — the access log alone is worth the setup time.

Still with us? We think you'll like Self-Host OpenClaw: VPS Setup to Production (2026) as your next read.

How Do You Tune Performance Across the Stack?

Ollama: VRAM Allocation and Context Windows

The single biggest performance lever is keeping your model entirely in VRAM. A model that spills from VRAM into system RAM runs at 20-30% of its VRAM-only speed.

# Check what's loaded and where ollama ps # Output: # NAME ID SIZE PROCESSOR UNTIL # qwen2.5:14b ... 9.1 GB 100% GPU 4 min # ^^^^^^^^^^^ # "100% GPU" = entirely in VRAM. Good. # "76%/24% CPU" = spilling. Load a smaller model. 

Context window size directly affects VRAM usage. The default is 4096 tokens. Setting it to 32768 uses roughly 50% more VRAM for the same model. Set context size explicitly rather than relying on the default:

# Set context size for a specific model via Ollama's REST API curl http://localhost:11434/api/generate -d '{ "model": "qwen2.5:14b", "options": {"num_ctx": 8192} }' 

Or configure it in Open WebUI's model settings per-model.

LiteLLM: Connection Pooling and Worker Counts

LiteLLM defaults are conservative. For a team deployment under regular load:

# In your litellm config.yaml litellm_settings: # Increase concurrent request handling max_parallel_requests: 100 # Aggressive timeout to fail fast to fallback request_timeout: 60 # Keep connections to Ollama warm num_retries: 1 # Try once, fall back quickly # In docker-compose.yml, LiteLLM service command: > --config /app/config.yaml --port 4000 --workers 4 # One worker per CPU core available to the container 

Qdrant: Collection Optimization

Default Qdrant settings work for small collections. For a knowledge base with 100,000+ chunks, enable HNSW indexing optimization:

# Update the collection configuration after initial creation curl -X PATCH "http://localhost:6333/collections/your_collection" \ -H "Content-Type: application/json" \ -H "api-key: ${QDRANT_API_KEY}" \ -d '{ "optimizers_config": { "default_segment_number": 4, "indexing_threshold": 20000 }, "hnsw_config": { "m": 16, "ef_construct": 100, "full_scan_threshold": 10000 } }' 

The hnsw_config settings trade index build time for query speed. Higher m and ef_construct values give better recall accuracy at the cost of more memory and slower index builds. For a static knowledge base, build the index overnight and query it fast during the day. For a frequently updated collection, use the defaults.

Open WebUI: RAG Chunk Tuning

The default chunk size (1000 tokens) and overlap (100 tokens) work for most prose documents. For technical documentation with lots of code:

CHUNK_SIZE: 1500 CHUNK_OVERLAP: 200 

For short-form documents (emails, Slack messages, notes):

CHUNK_SIZE: 500 CHUNK_OVERLAP: 50 

The right chunk size is the one where the model's retrieved context contains the complete answer to a typical query without including irrelevant surrounding text. Test by asking questions you know the answer to and checking whether the retrieved chunks include the right information.

What Security Mistakes Does Every Stack Make at First?

Exposing Services That Should Be Internal

The most common mistake: adding Traefik labels to services that have no reason to be publicly accessible. Qdrant, Redis, LiteLLM's internal metrics endpoint, and Langfuse's worker process do not need Traefik routing. If they have a label, they're reachable.

Correct pattern: services get Traefik labels only if humans need direct browser access. Everything else stays on ai-internal with no external routing.

# Audit which services are exposed docker inspect $(docker compose ps -q) | \ jq '.[] | select(.Labels."traefik.enable"=="true") | .Name' # Should only show: open-webui, langfuse-web, grafana # Should NOT show: qdrant, redis, litellm-postgres, clickhouse 

LiteLLM Master Key Exposure

The LITELLM_MASTER_KEY gives full admin access to LiteLLM's API. Never put it in the OPENAI_API_KEY field of your IDE or code. Create virtual keys for different use cases:

# Create a virtual key with budget limit for a specific user curl -X POST "https://llm.yourdomain.com/key/generate" \ -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \ -H "Content-Type: application/json" \ -d '{ "models": ["qwen2.5:14b", "qwen2.5-coder:14b"], "max_budget": 10.0, "budget_duration": "30d", "metadata": {"user": "alice", "team": "engineering"}, "aliases": {"qwen2.5:14b": "local-main"} }' 

This returns a virtual key (sk-...) that Alice can use in her IDE. It can only access the two listed models, has a $10 monthly budget cap, and her usage is tracked separately in the Langfuse dashboard. The master key stays in your .env and never touches a developer's machine.

Open WebUI's Default Cookie Settings

Open WebUI's session cookies default to non-secure settings in some configurations. Verify in the admin panel that WEBUI_SESSION_COOKIE_SECURE=true is set when serving over HTTPS. This prevents session cookie theft over non-HTTPS connections.

Qdrant Without Authentication

Qdrant's default configuration runs with no authentication. The QDRANT__SERVICE__API_KEY environment variable enables API key validation. Without it, anyone who can reach the Qdrant port can read, write, and delete every collection. The ai-internal network isolation provides one layer of protection. The API key provides the second.

How Do You Upgrade Components Without Downtime?

The stack separates concerns cleanly, which means most components can be upgraded independently.

Open WebUI (ships frequently, usually safe to update):

docker compose pull open-webui docker compose up -d --no-deps open-webui # --no-deps prevents restarting unrelated services 

LiteLLM (check release notes before upgrading -- config format changes occasionally):

# Test the new config format in a separate container first docker run --rm \ -v /opt/ai-stack/litellm/config.yaml:/app/config.yaml \ ghcr.io/berriai/litellm:v1.91.0-stable \ --config /app/config.yaml --test_connection # If it passes, upgrade docker compose pull litellm docker compose up -d --no-deps litellm 

Langfuse (requires coordinated update of web + worker):

docker compose pull langfuse-web langfuse-worker docker compose up -d --no-deps langfuse-web langfuse-worker # Both use the same image version -- always update together 

Qdrant (data format changes happen between major versions -- read the changelog):

# Back up first docker exec ai-qdrant tar -czf /tmp/qdrant-backup.tar.gz /qdrant/storage docker cp ai-qdrant:/tmp/qdrant-backup.tar.gz /opt/ai-stack/backups/ # Then upgrade docker compose pull qdrant docker compose up -d --no-deps qdrant 

How Do You Back Up the Stack?

Each component has data that matters:

Component Data Location Backup Priority
Open WebUI open_webui_data Docker volume High (chat history, user accounts)
LiteLLM PostgreSQL litellm_postgres_data volume Medium (virtual keys, spend data)
Qdrant qdrant_data volume High if RAG knowledge base has custom documents
Langfuse PostgreSQL langfuse_postgres_data volume Medium (project config, user accounts)
ClickHouse clickhouse_data volume Low (traces are re-generated; can accept loss)

Here is the backup script we created.

#!/bin/bash # /opt/ai-stack/backup.sh BACKUP_DATE=$(date +%Y%m%d-%H%M) BACKUP_DIR="/opt/backups/ai-stack/${BACKUP_DATE}" mkdir -p "${BACKUP_DIR}" # Dump Open WebUI SQLite database docker exec ai-open-webui sqlite3 /app/backend/data/webui.db \ ".backup /tmp/webui-backup.db" docker cp ai-open-webui:/tmp/webui-backup.db "${BACKUP_DIR}/webui.db" # Dump LiteLLM PostgreSQL docker exec ai-litellm-postgres pg_dump -U litellm litellm | \ gzip > "${BACKUP_DIR}/litellm-postgres.sql.gz" # Dump Langfuse PostgreSQL docker exec ai-langfuse-postgres pg_dump -U langfuse langfuse | \ gzip > "${BACKUP_DIR}/langfuse-postgres.sql.gz" # Snapshot Qdrant collections docker exec ai-qdrant curl -s \ -X POST "http://localhost:6333/collections/knowledge_base/snapshots" \ -H "api-key: ${QDRANT_API_KEY}" # Compress everything tar -czf "/opt/backups/ai-stack-${BACKUP_DATE}.tar.gz" "${BACKUP_DIR}/" rm -rf "${BACKUP_DIR}" # Sync offsite rsync -avz "/opt/backups/ai-stack-${BACKUP_DATE}.tar.gz" \ "backup-server:/backups/ai-stack/" # Clean up local backups older than 14 days find /opt/backups -name "ai-stack-*.tar.gz" -mtime +14 -delete echo "Backup complete: ai-stack-${BACKUP_DATE}.tar.gz" 

The stack described here is the one SudoSecurity runs internally for research and content work. Tier 1 took thirty minutes. Tier 2 took an afternoon. Tier 3 accumulated over several months as the team grew and monitoring gaps became visible.

The key principle: each tier adds components that solve real problems the previous tier has. Add LiteLLM when you want routing and cost visibility, not before. Add Langfuse when you want to understand why responses are wrong. Add monitoring when you want to know before a user tells you that something broke.

For the self-hosting ecosystem beyond the AI stack — the file storage, password manager, email, and media server that round out a complete self-hosted setup — the SudoSecurity self-hosted productivity guide covers that full picture. For every sysadmin and AI article across the site, the SysAdmin section and the AI section are the places to start.