Transform your TrueNAS SCALE NAS into a full AI operations platform running LLMs, Stable Diffusion, vector databases, and orchestration workflows.

The AI Ops Stack: What Runs on Your TrueNAS

Here's what a practical AI operations platform looks like on TrueNAS SCALE:

Core AI Services Layer

  • Ollama: LLM inference server (qwen2.5-coder:14b, deepseek-r1:14b, etc.)
  • Automatic1111 / ComfyUI: Stable Diffusion web UIs for image generation
  • LiteLLM: Proxy/load balancer for multiple LLM providers with fallback
  • Open-WebUI: Chat interface for interacting with LLMs
  • Qdrant: Vector database for RAG and semantic search
  • LangFuse: LLM observability and tracing
  • N8n: Workflow automation and orchestration
  • PostgresQL: Metadata storage for applications
  • Redis: Caching and message brokering

Infrastructure & Support Services

  • TrueNAS SCALE: Base OS with Kubernetes and Docker
  • GPU Passthrough: Direct access to NVIDIA RTX 5060 Ti for acceleration
  • NGINX Proxy Manager: SSL termination, routing, and load balancing
  • Fail2ban + CrowdSec: Intrusion prevention and IP blocking
  • Netdata: Real-time performance monitoring
  • Uptime Kuma: Service monitoring and alerts
  • Volumes: Fast NVMe for apps/models, Storage Pool for datasets
  • Snapshots: ZFS snapshots for quick rollback and experimentation

Data & Model Management

  • Datasets: Curated collections for training/fine-tuning
  • Model Registry: Versioned LLM and diffusion models
  • Embeddings Cache: Pre-computed vectors for fast retrieval
  • Prompt Library: Tested and optimized prompt templates
  • Output Gallery: Curated collections of generated content
  • Experiment Tracking: Parameters, results, and metadata for ML runs

Hardware Requirements and GPU Passthrough

What you need to make AI workloads work well on TrueNAS:

Minimum Viable Hardware:

  • CPU: Modern 6-core+ processor (AMD Ryzen 5 5600X or Intel i5-12400 equivalent)
  • RAM: 32GB ECC recommended (16GB minimum for light usage)
  • Storage: NVMe pool for OS/apps (500GB+), Storage Pool for data (4TB+ RAIDZ1)
  • GPU: NVIDIA card with recent drivers (RTX 3060+/4060+ recommended for VRAM)
  • Network: 1GbE minimum, 2.5GbE/10GbE preferred for large dataset transfers

Recommended Hardware (Your Current Setup):

  • CPU: AMD Ryzen 9 9900X (12 cores/24 threads) - Excellent
  • RAM: 64GB DDR5 - More than sufficient
  • Storage: 2TB WD Black SN850X NVMe (apps) + 3×8TB RAIDZ1 (data) - Ideal
  • GPU: ASUS RTX 5060 Ti 16GB - Perfect for AI workloads
  • Network: Assumed 1GbE+ - Adequate

GPU Passthrough Configuration:

Making your NVIDIA GPU available to containers:

# TrueNAS SCALE GPU passthrough setup # 1. Enable VM/Device passthrough in TrueNAS UI # System → Advanced → Enable "Allow VMs to access PCI devices" # 2. Identify your GPU and audio controller lspci | grep -i nvidia # Example output: # 01:00.0 VGA compatible controller: NVIDIA Corporation AD106 [GeForce RTX 5060 Ti] (rev a1) # 01:00.1 Audio device: NVIDIA Corporation Device 228e (rev a1) # 3. Note the PCI IDs (01:00.0 and 01:00.1 in example) # 4. Create a VM or configure Kubernetes to passthrough these devices # For Kubernetes (TrueNAS SCALE apps): # - Edit the app's advanced settings # - Add "devices" section: # devices: # - hostPath: /dev/nvidia0 # containerPath: /dev/nvidia0 # permissions: ["rwm"] # - hostPath: /dev/nvidiactl # containerPath: /dev/nvidiactl # permissions: ["rwm"] # - hostPath: /dev/nvidia-uvm # containerPath: /dev/nvidia-uvm # permissions: ["rwm"] # - hostPath: /dev/nvidia-uvm-tools # containerPath: /dev/nvidia-uvm-tools # permissions: ["rwm"] # 5. Install NVIDIA container toolkit in your apps # For Ollama or ComfyUI containers: # - Add nvidia runtime # - Set environment variable: NVIDIA_VISIBLE_DEVICES=all # - Or use the built-in GPU support in TrueNAS SCALE apps # 6. Verify GPU access from within container docker run --gpus all --rm nvidia/cuda:12.2-base nvidia-smi

TrueNAS SCALE Specific Notes:

TrueNAS SCALE includes built-in GPU support for apps:

  • When installing apps via the TrueNAS UI, look for "GPU" or "Hardware Acceleration" options
  • Many community apps (like Ollama, ComfyUI) now have GPU-enabled versions
  • Apps can request GPU resources through the TrueNAS SCALE API
  • The system handles driver compatibility and updates

Service-by-Service Deployment Guide

How to deploy each component of your AI ops platform:

1. Ollama (LLM Inference)

Deployment Options

  • TrueNAS App: Official Ollama app from Community Apps
  • Custom Container: docker run with GPU support (more/ollama or similar
  • Kubernetes: Helm chart or manifest

Configuration for TrueNAS 

# Example: docker run command for TrueNAS docker run -d \ --name ollama \ --restart unless-stopped \ -p 11434:11434 \ -v /mnt/nvme-pool/apps/ollama:/root/.ollama \ -v /mnt/Storage_Pool/models:/models \ --gpus all \ -e OLLAMA_MODELS=/models \ -e OLLAMA_HOST=0.0.0.0 \ ollama/ollama # Alternative: TrueNAS SCALE app configuration # In the TrueNAS UI when installing Ollama app: # - Port: 11434 # - Volumes: # Host Path: /mnt/nvme-pool/apps/ollama → Container Path: /root/.ollama # Host Path: /mnt/Storage_Pool/models → Container Path: /models # - Runtime: NVIDIA (enables GPU access) # - Environment Variables: # OLLAMA_MODELS: /models # OLLAMA_HOST: 0.0.0.0

Model Management Strategy

  • Storage Location: Use your Storage Pool for model persistence (/mnt/Storage_Pool/models)
  • Organization: Create folders by type: llm/, embedding/, vision/
  • Versioning: Use subfolders for different quantizations (q4_K_M, q5_K_M, etc.)
  • Symlinks: Point Ollama's model directory to your organized storage
  • Backup: Snapshots of your model dataset for experimentation safety

2. ComfyUI / Automatic1111 (Image Generation)

Why ComfyUI Over Automatic1111?

  • Better resource control: Fine-grained VRAM management
  • Workflow-based: Reproducible, shareable generation pipelines
  • Lower overhead: Less complex than full WebUI
  • API-first: Easy to integrate with n8n and other automation
  • TrueNAS-friendly: Less prone to conflicts with system services

Deployment Configuration 

# Example: docker run for ComfyUI with GPU docker run -d \ --name comfyui \ --restart unless-stopped \ -p 8188:8188 \ -v /mnt/nvme-pool/apps/comfyui:/opt/comfyui \ -v /mnt/Storage_Pool/comfyui-input:/opt/comfyui/input \ -v /mnt/Storage_Pool/comfyui-output:/opt/comfyui/output \ -v /mnt/Storage_Pool/comfyui-models:/opt/comfyui/models \ --gpus all \ -e NVIDIA_VISIBLE_DEVICES=all \ comfyui/comfyui:latest # Alternative: Using TrueNAS SCALE app # In TrueNAS UI app installation: # - Port: 8188 # - Volumes: # /mnt/nvme-pool/apps/comfyui → /opt/comfyui (app data) # /mnt/Storage_Pool/comfyui-input → /opt/comfyui/input # /mnt/Storage_Pool/comfyui-output → /opt/comfyui/output # /mnt/Storage_Pool/comfyui-models → /opt/comfyui/models # - Runtime: NVIDIA # - Environment: NVIDIA_VISIBLE_DEVICES=all

Model Organization for ComfyUI

  • Checkpoints: /mnt/Storage_Pool/comfyui-models/checkpoints/
  • LoRAs: /mnt/Storage_Pool/comfyui-models/loras/
  • ControlNet: /mnt/Storage_Pool/comfyui-models/controlnet/
  • VAEs: /mnt/Storage_Pool/comfyui-models/vae/
  • Embeddings: /mnt/Storage_Pool/comfyui-models/embeddings/
  • Upscalers: /mnt/Storage_Pool/comfyui-models/upscale/

3. LiteLLM (Proxy/Layer for Multiple Providers)

Why Use LiteLLM?

  • Provider fallback: If Ollama fails, try local Llama.cpp or remote API
  • Load balancing: Distribute requests across multiple instances
  • Cost tracking: Monitor usage and expenses for paid APIs
  • Standardized interface: OpenAI-compatible endpoint for all providers
  • Cache layers: Redis caching for repeated requests
  • Security: API key management and rate limiting

Typical Configuration 

# lite llm settings.yaml model_list: - model_name: ollama_qwen litellm_provider: ollama model_name: qwen2.5-coder:14b api_base: http://localhost:11434 num_retries: 3 timeout: 120 max_retries: 3 - model_name: ollama_deepseek litellm_provider: ollama model_name: deepseek-r1:14b api_base: http://localhost:11434 num_retries: 3 timeout: 120 - model_name: local_llama_cpp litellm_provider: llama_cpp model_path: /mnt/Storage_Pool/models/llama-3.1-8b-instruct.q4_K_M.gguf num_retries: 3 timeout: 60 - model_name: openai_backup litellm_provider: openai model: gpt-4o-mini api_key: ${OPENAI_API_KEY} num_retries: 3 timeout: 60 # Routing strategy: try local first, then fall back router: type: LLMSingleAgent model_list: - ollama_qwen - ollama_deepseek - local_llama_cpp - openai_backup # Optional: Add Redis caching cache: type: redis host: localhost port: 6379 # Start LiteLLM litellm --config /path/to/settings.yaml --port 4000

4. Qdrant (Vector Database for RAG)

Why Qdrant?

  • TrueNAS-friendly: Official app available
  • Performance: Optimized for vector search
  • Scalability: Handles millions of vectors efficiently
  • Features: Filtering, payload storage, hybrid search
  • Easy integration: Works with LangChain, LlamaIndex, custom code
  • Backup-friendly: Snapshots and exports work well

Deployment and Usage 

# TrueNAS SCALE app deployment # In TrueNAS UI: # - App: Qdrant (from Community Apps) # - Port: 6333 (HTTP) and 6334 (gRPC) # - Volumes: # Host: /mnt/Storage_Pool/qdrant-storage → Container: /qdrant/storage # Host: /mnt/Storage_Pool/qdrant-snapshots → Container: /qdrant/snapshots # - Resources: Limit RAM usage if needed (e.g., 4GB)  # Example: Using Qdrant with Ollama for RAG import requests from ollama import Client # Initialize clients ollama_client = Client(host='http://localhost:11434') qdrant_url = "http://localhost:6333" def embed_text(text): """Generate embedding using Ollama""" response = ollama_client.embeddings( model="nomic-embed-text", prompt=text ) return response['embedding'] def store_document(doc_id, text, metadata=None): """Store document in Qdrant""" vector = embed_text(text) payload = { "text": text, "metadata": metadata or {} } requests.put( f"{qdrant_url}/collections/my_documents/points", json={ "points": [ { "id": doc_id, "vector": vector, "payload": payload } ] } ) def search_documents(query, limit=5): """Search for similar documents""" query_vector = embed_text(query) response = requests.post( f"{qdrant_url}/collections/my_documents/points/search", json={ "vector": query_vector, "limit": limit, "with_payload": true } ) return response.json()['result'] # Usage example store_document( "doc1", "How to optimize SQL queries in SAP S/4HANA", {"topic": "SAP", "type": "guide", "author": "Ayra ix"} ) results = search_documents("SAP performance tuning tips") for result in results: print(f"Score: {result['score']:.3f}") print(f"Text: {result['payload']['text'][:100]}...") print("---")

5. LangFuse (LLM Observability)

Why Observe Your LLMs?

  • Cost tracking: Monitor token usage and expenses
  • Performance: Measure latency and throughput
  • Quality: Track hallucinations, errors, and user feedback
  • Debugging: Trace complex chains and agent workflows
  • Optimization: Identify bottlenecks and inefficient patterns
  • Compliance: Audit trails for regulated industries

Deployment Configuration 

# docker run for LangFuse docker run -d \ --name langfuse \ --restart unless-stopped \ -p 3100:3000 \ -v /mnt/nvme-pool/apps/langfuse:/langfuse \ -e DATABASE_URL=postgresql://langfuse:langfuse@localhost:5432/langfuse \ -e NEXTAUTH_SECRET=$(openssl rand -hex 32) \ -e NEXTAUTH_URL=http://localhost:3100 \ -e SIGNING_KEY=$(openssl rand -hex 32) \ langfuse/langfuse:latest # Optional: With external Postgres # -v /mnt/Storage_Pool/postgres:/var/lib/postgresql/data # -e POSTGRES_USER=langfuse # -e POSTGRES_PASSWORD=langfuse # -e POSTGRES_DB=langfuse # Initialize in your application from langfuse import Langfuse langfuse = Langfuse( public_key="pk-lf-...", secret_key="sk-lf-...", host="http://localhost:3100" ) # Track LLM calls with langfuse.start_as_current_span(name="llm-call") as span: span.update_trace( input={"question": "How do I optimize this ABAP code?"}, metadata={"user_id": "user123"} ) response = ollama_client.generate( model="qwen2.5-coder:14b", prompt="How do I optimize this ABAP code for performance?" ) span.update_trace( output={"answer": response['response']}, metadata={"model": "qwen2.5-coder:14b", "tokens": response.get('eval_count', 0)} )

6. N8n (Workflow Orchestration)

Why N8n for AI Workflows?

  • Visual workflows: Easy to understand and modify
  • Hundreds of integrations: Connect to LLMs, APIs, databases, files
  • Built-in scheduling: Cron-like triggering of workflows
  • Error handling: Retry policies, error workflows, alerts
  • Custom nodes: Write JavaScript or Python for specialized tasks
  • Webhook triggers: Start workflows from external events
  • TrueNAS-proven: Well-established in the TrueNAS community

Example AI Workflows 

# Workflow 1: Automated Content Generation Pipeline # Trigger: Webhook (receive request via HTTP) # Steps: # 1. Receive request: { "topic": "SAP AI Units", "type": "article" } # 2. Query knowledge base: Search Qdrant for relevant documents # 3. Enhance with LLM: Ask Ollama to summarize findings # 4. Generate outline: Use LLM to create article structure # 5. Write sections: Loop through outline, generate each section # 6. Create images: Use ComfyUI API to generate illustrations # 7. Assemble: Combine text and images into final document # 8. Notify: Send completion via email/n8n notification # 9. Store: Save output to /mnt/Storage_Pool/published-content/ # Workflow 2: Model Maintenance and Updates # Trigger: Schedule (weekly) # Steps: # 1. Check for new models: Query Hugging Face API for updates # 2. Download: Fetch new versions to /mnt/Storage_Pool/models/ # 3. Test: Run inference tests on new models # 4. Compare: Evaluate against benchmark suite # 5. Deploy: Promote to production if passes tests # 6. Notify: Send report on updates performed # 7. Cleanup: Remove old versions based on retention policy # Workflow 3: RAG Pipeline Maintenance # Trigger: Schedule (daily) or webhook (when new docs added) # Steps: # 1. Scan for new documents: Check /mnt/Storage_Pool/documents/ # 2. Extract text: Use appropriate loader (PDF, DOCX, TXT, etc.) # 3. Chunk text: Split into optimal segments for embedding # 4. Generate embeddings: Use nomic-embed-text via Ollama # 5. Store vectors: Add to Qdrant collection # 6. Update metadata: Track when documents were last processed # 7. Notify: Report on indexing progress

Data Management and Storage Strategy

Where to put what for optimal performance and reliability:

NVMe Pool (High Performance)

Use for:

  • Operating system and TrueNAS SCALE
  • Application containers and images
  • Frequently accessed model files (active LLMs)
  • Vector database working sets (Qdrant RAM cache overflow)
  • Temporary processing space
  • Logs and metrics that need fast writes

Example paths:

  • /mnt/nvme-pool/apps/ - Container-app data
  • /mnt/nvme-pool/models/active/ - Currently loaded models
  • /mnt/nvme-pool/logs/ - Application logs
  • /mnt/nvme-pool/cache/ - Redis, temporary files

Storage Pool (Capacity & Reliability)

Use for:

  • Master copies of all models (quantized versions)
  • Training datasets and archives
  • Generated outputs (images, text, etc.)
  • Long-term backups and snapshots
  • Vector database persistent storage
  • Media assets for ComfyUI workflows
  • Document libraries for RAG systems

Example paths:

  • /mnt/Storage_Pool/models/ - All model versions
  • /mnt/Storage_Pool/datasets/ - Training data
  • /mnt/Storage_Pool/generated/ - AI outputs
  • /mnt/Storage_Pool/qdrant/ - Vector DB storage
  • /mnt/Storage_Pool/documents/ - Source documents
  • /mnt/Storage_Pool/comfyui/ - Input/output/media

Backup Strategy:

Protect your AI investments:

  • Models: Snapshots of Storage Pool/model datasets (monthly)
  • Data: Snapshots of document libraries (weekly)
  • Configuration: Export app configs and store in version control
  • Generated content: Snapshots of output directories (as needed)
  • Vector databases: Qdrant snapshots + regular exports
  • Test restoration: Quarterly drills to verify backup integrity

Security and Access Control

Keeping your AI ops platform safe:

Network Security:

  • Network segmentation: Put AI services on isolated VLAN
  • Firewall rules: Only allow necessary ports (HTTP/HTTPS to proxy)
  • Fail2ban: Block brute-force attempts on services
  • CrowdSec: Community-based IP reputation blocking
  • NGINX Proxy Manager: Central TLS termination and access control
  • Internal communications: Allow service-to-service on private network
  • Remote access: Use TrueNAS's built-in Tailscale/ZeroTier or VPN

Application Security:

  • Container isolation: Each app runs in its own isolated environment
  • Least privilege: Containers run as non-root when possible
  • Resource limits: CPU/RAM limits prevent noisy neighbors
  • Read-only root: Where supported, mount container root as read-only
  • Secrets management: Use TrueNAS's built-in secrets or external vault
  • Regular updates: Track and apply security patches
  • Vulnerability scanning: Periodic scans of container images

Data Security:

  • Encryption at rest: ZFS native encryption for sensitive datasets
  • Access controls: POSIX permissions and ACLs on datasets
  • Audit logging: Track who accesses what data
  • Data classification: Label datasets by sensitivity (public/internal/confidential)
  • Model security: Verify provenance of downloaded models
  • Output filtering: Screen generated content for inappropriate material
  • PII handling: Special procedures for datasets containing personal information 
# Example: NGINX Proxy Manager configuration for AI services # In TrueNAS UI → NGINX Proxy Manager → Proxy Hosts # 1. Ollama Access # Domain: ai.internal.yourdomain.com # Scheme: http # Forward Hostname/IP: localhost # Forward Port: 11434 # Access List: Require authentication (optional) # Custom Configuration: # proxy_set_header Host $host; # proxy_set_header X-Real-IP $remote_addr; # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # proxy_set_header X-Forwarded-Proto $scheme; # 2. ComfyUI Access # Domain: comfy.internal.yourdomain.com # Scheme: http # Forward Hostname/IP: localhost # Forward Port: 8188 # Access List: Require authentication (recommended for UI) # SSL: Force SSL redirect (if using external certs) # 3. LangFuse Access # Domain: langfuse.internal.yourdomain.com # Scheme: http # Forward Hostname/IP: localhost # Forward Port: 3100 # Access List: Require authentication (recommended) # 4. Qdrant Access (internal only usually) # Domain: qdrant.internal.yourdomain.com # Scheme: http # Forward Hostname/IP: localhost # Forward Port: 6333 # Access List: Local network only (recommended) # Enable HTTP/2 support for better performance # Enable caching headers where appropriate # Set up automatic SSL renewal via Let's Encrypt (if using public domains)

Monitoring, Maintenance, and Optimization

Keeping your AI ops platform running smoothly:

Essential Monitoring:

  • System level: CPU, memory, disk, network utilization
  • GPU level: VRAM usage, utilization, temperature
  • Application level: Response times, error rates, throughput
  • Business level: Token generation, images created, workflows completed
  • Log aggregation: Centralized logging for debugging
  • Alerting: Notifications for anomalies and failures 
# Example: Simple health check script #!/usr/bin/env python3 import requests import time import json SERVICES = { "ollama": {"url": "http://localhost:11434/api/tags", "healthy": lambda r: r.status_code == 200}, "comfyui": {"url": "http://localhost:8188", "healthy": lambda r: r.status_code == 200}, "langfuse": {"url": "http://localhost:3100/api/auth/check", "healthy": lambda r: r.status_code == 200}, "qdrant": {"url": "http://localhost:6333/collections", "healthy": lambda r: r.status_code == 200}, "litellm": {"url": "http://localhost:4000/health", "healthy": lambda r: r.status_code == 200} } def check_service(name, config): try: response = requests.get(config["url"], timeout=5) is_healthy = config["healthy"](response) return { "service": name, "healthy": is_healthy, "status_code": response.status_code, "response_time": response.elapsed.total_seconds() } except requests.exceptions.RequestException as e: return { "service": name, "healthy": False, "error": str(e), "response_time": None } def main(): results = [] for name, config in SERVICES.items(): result = check_service(name, config) results.append(result) # Log result  print(json.dumps({ "timestamp": time.time(), "check": result })) # In a real system, you'd send this to a monitoring system  # or trigger alerts for unhealthy services  return results if __name__ == "__main__": while True: main() time.sleep(30) # Check every 30 seconds

Regular Maintenance Tasks:

  • Daily: Check logs for errors, verify backups completed
  • Weekly: Update container images, check for model updates
  • Monthly: Review resource usage, optimize configurations, test disaster recovery
  • Quarterly: Perform security audits, test restore procedures, review access controls
  • Annually: Review hardware capacity, plan upgrades, renew certificates

Performance Optimization Tips:

  • Model quantization: Use Q4/K_M or Q5/K_M for best balance
  • Batch processing: Group similar requests to amortize overhead
  • Caching: Cache frequent queries in Redis or application level
  • Connection pooling: Reuse HTTP/database connections
  • Resource limits: Set appropriate CPU/memory limits per container
  • GPU scheduling: Use the techniques from the Ollama GPU scheduling article
  • Storage optimization: Place hot data on NVMe, warm/cool on Storage Pool

Real-World Use Cases on Your TrueNAS AI Platform

Practical applications you can build today:

SAP Development Assistant

Components: Ollama (Code Llama), Qdrant (SAP docs), n8n (workflow)

Capabilities:

  • Explain complex ABAP code snippets
  • Generate unit tests for SAP classes
  • Search SAP documentation using natural language
  • Suggest performance improvements for SQL queries
  • Generate functional specs from user stories
  • Explain error messages and suggest fixes

Content Creation Pipeline

Components: Ollama (writing), ComfyUI (images), n8n (orchestration), Qdrant (research)

Capabilities:

  • Research topics and generate outlines
  • Write draft articles and blog posts
  • Create custom illustrations and diagrams
  • Generate social media content variations
  • Optimize content for SEO and readability
  • Publish to multiple platforms automatically

Data Analysis assistant

Components: Ollama (analysis), Postgres (data), LangFuse (tracking)

Capabilities:

  • Generate SQL queries fr...