How to setup a secure Inference Platform using LiteLLM and vLLM

Introduction

In my last post I showed how to run an air-gapped LLM using Ollama on a single machine. That works great for one person, but as soon as a whole team, or even multiple teams, want to use the models, you need something more robust: concurrent requests, API keys per user, cost tracking and a single, secured entry point.

This article will show how I set up exactly that on Kubernetes, using three building blocks:

• vLLM as the inference server, which actually runs the model on the GPU
• LiteLLM as the gateway in front of it, which handles API keys, routing and spend tracking
• An Ingress controller as the single entry point, which terminates TLS

The request flow looks like this:

Client ──HTTPS──► Ingress ──► LiteLLM (API key check) ──► vLLM (GPU) 

As an example model I'm using Qwen3.8-27B in NVFP4, which fits nicely on a single 40 GB GPU.

Info: I'm running this on a VMware Tanzu (VKS) cluster with NVIDIA vGPUs, but everything shown here works on any Kubernetes flavour with GPU nodes, be it AKS, EKS, GKE or a bare metal cluster.

Why not just use Ollama?

Fair question, since Ollama is what I used last time. The short answer is concurrency. Ollama is great for a single user, but it effectively processes requests one after another. vLLM uses continuous batching and PagedAttention, so it can serve many requests at once on the same GPU. On my hardware, with the very same model, I measured this:

As soon as more than one person uses the platform, vLLM wins by a lot.

Prerequisites: The GPU Operator

Before any pod can request a GPU, your nodes need a driver, the NVIDIA container toolkit and the device plugin. The easiest way to get all of that is the NVIDIA GPU Operator. I won't go into detail here, since NVIDIA documents it really well:

• GPU Operator - Getting Started
• GPU Operator with NVIDIA vGPU, if you are running on vSphere like me

Info: If you are on vGPU, the open drivers won't bind to your GPU. You need the licensed GRID guest driver, and its version must match the vGPU Manager on your ESXi host. In my case I installed the driver directly on the nodes and run the operator with driver.enabled: false.

Once the operator is up, you can check that your nodes advertise their GPUs:

kubectl get nodes -o custom-columns='NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu' 

The operator also creates the nvidia RuntimeClass, which we'll need in a moment.

Setting up vLLM

The namespace

I start with a dedicated namespace for vLLM. Since vLLM can run fully rootless, I enforce the strictest Pod Security Standard on it:

apiVersion: v1 kind: Namespace metadata: name: vllm labels: pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/audit: restricted pod-security.kubernetes.io/warn: restricted 

Storage for the model weights

The model weights live on a PersistentVolumeClaim, so they don't have to be downloaded again on every pod restart:

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: vllm-models-pvc namespace: vllm spec: accessModes: - ReadWriteMany storageClassName: <your-rwx-storage-class> resources: requests: storage: 100Gi 

Info: I'm using ReadWriteMany (NFS) here on purpose. The vLLM namespace has no internet access at all (more on that below), so a separate, internet-capable "loader" namespace mounts the same NFS export and downloads the weights from Hugging Face onto it. This way, the pod that actually processes your data never needs to talk to the outside world. It's the same idea as exporting the volume in my Ollama post, just on Kubernetes.

The vLLM Deployment

Now to the interesting part. vLLM ships an official image with an OpenAI-compatible server, so all I have to do is pass the right flags:

apiVersion: apps/v1 kind: Deployment metadata: name: vllm-qwen38 namespace: vllm spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app: vllm instance: vllm-qwen38 template: metadata: labels: app: vllm instance: vllm-qwen38 spec: runtimeClassName: nvidia tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule enableServiceLinks: false securityContext: runAsNonRoot: true runAsUser: 2000 runAsGroup: 2000 fsGroup: 2000 seccompProfile: type: RuntimeDefault containers: - name: vllm image: vllm/vllm-openai:v0.28.0 args: - "--host" - "0.0.0.0" - "--port" - "8000" - "--model" - "Inferact/Qwen3.8-27B-NVFP4" - "--served-model-name" - "qwen3.8-27b-vllm" - "--gpu-memory-utilization" - "0.85" - "--max-model-len" - "32768" - "--kv-cache-dtype" - "fp8" - "--enable-prefix-caching" - "--enable-auto-tool-choice" - "--tool-call-parser" - "qwen3_coder" - "--reasoning-parser" - "qwen3" env: - name: HF_HOME value: /models/hf - name: VLLM_CACHE_ROOT value: /models/hf/vllm-cache - name: HF_HUB_OFFLINE value: "1" - name: HOME value: /home/vllm resources: limits: nvidia.com/gpu: "1" memory: "48Gi" requests: cpu: "8" memory: "16Gi" ports: - name: http containerPort: 8000 startupProbe: httpGet: path: /health port: 8000 periodSeconds: 15 failureThreshold: 80 readinessProbe: httpGet: path: /health port: 8000 periodSeconds: 15 volumeMounts: - name: models mountPath: /models - name: home mountPath: /home/vllm - name: dshm mountPath: /dev/shm securityContext: allowPrivilegeEscalation: false capabilities: drop: ["ALL"] volumes: - name: models persistentVolumeClaim: claimName: vllm-models-pvc - name: home emptyDir: {} - name: dshm emptyDir: medium: Memory sizeLimit: 8Gi 

There are a few things in here I'd like to point out:

• --served-model-name is the name the model is addressed by. We'll need this exact name again in the LiteLLM config.
• HF_HUB_OFFLINE=1 makes vLLM fail fast if a weight file is missing, instead of trying to reach huggingface.co.
• --kv-cache-dtype fp8 halves the memory needed per token in the KV cache. This is what lets you fit a long context and many parallel requests on one GPU.
• strategy: Recreate prevents a rolling update from starting a second pod, which would just sit in Pending because the only GPU is still taken.
• /dev/shm as a memory-backed volume. The container default of 64 MiB is too small for PyTorch and leads to really obscure crashes.
• The generous startupProbe. A cold start (loading the model plus torch.compile) can easily take 5 minutes. Since VLLM_CACHE_ROOT lives on the PVC, the next restart is way faster.

If you are on vGPU: never set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, even though PyTorch's own out-of-memory message recommends it. vGPU doesn't implement the needed CUDA APIs and vLLM will crash right at startup with operation not supported. Also keep in mind that the vGPU reserves around 3 GB of memory for itself, so --gpu-memory-utilization values above ~0.91 won't work on a 40 GB profile.

The Service

vLLM gets a plain ClusterIP Service. It must not be reachable from outside the cluster, since vLLM itself has no authentication at all:

apiVersion: v1 kind: Service metadata: name: vllm-qwen38 namespace: vllm spec: type: ClusterIP selector: app: vllm instance: vllm-qwen38 ports: - name: http port: 8000 targetPort: 8000 

Locking down the network

Now the part that makes this setup actually secure. I add two NetworkPolicies to the vLLM namespace. The first one only allows LiteLLM to talk to vLLM:

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: vllm-allow-litellm-only namespace: vllm spec: podSelector: matchLabels: app: vllm policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: litellm podSelector: matchLabels: app: litellm ports: - protocol: TCP port: 8000 

The second one blocks all egress to the internet. Only DNS and cluster-internal (RFC1918) traffic is allowed:

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: vllm-egress-internal-only namespace: vllm spec: podSelector: {} policyTypes: - Egress egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system podSelector: matchLabels: k8s-app: kube-dns ports: - protocol: UDP port: 53 - protocol: TCP port: 53 - to: - ipBlock: cidr: 10.0.0.0/8 - ipBlock: cidr: 172.16.0.0/12 - ipBlock: cidr: 192.168.0.0/16 

Info: Images are pulled by the kubelet on the node, so they are not affected by this policy. The pod itself however can't reach anything outside of your network anymore. That is basically the Kubernetes version of the iptables DROP rule from my Ollama post.

After applying everything, it's time to wait for the model to load:

kubectl -n vllm rollout status deployment/vllm-qwen38 --timeout=30m 

Setting up LiteLLM

vLLM gives us a fast, OpenAI-compatible API, but without any users, keys or limits. That's where LiteLLM comes in. It sits in front of one or more inference servers and adds:

• Virtual API keys per user or team, with model access, budgets and rate limits
• Spend and usage tracking
• Load balancing across multiple instances of the same model
• An Admin UI to manage all of the above

Postgres and Secrets

For virtual keys and spend tracking, LiteLLM needs a Postgres database. I run a simple single-replica Postgres in the litellm namespace with a ClusterIP Service named postgres. Nothing special, so I'll skip the manifest here.

The secrets for LiteLLM are created straight from the command line, so they never end up in a YAML file in git:

kubectl create namespace litellm kubectl create secret generic litellm-secrets -n litellm \ --from-literal=LITELLM_MASTER_KEY="sk-$(openssl rand -hex 24)" \ --from-literal=LITELLM_SALT_KEY="$(openssl rand -hex 32)" \ --from-literal=DATABASE_URL="postgresql://<user>:<password>@postgres.litellm.svc.cluster.local:5432/litellm" 

Make sure to keep the LITELLM_SALT_KEY safe and never change it after the first start. It's used to encrypt the credentials stored in the database. If it changes, LiteLLM can't decrypt them anymore.

Hooking up LiteLLM with vLLM

This is where LiteLLM gets connected to vLLM. The config lives in a ConfigMap:

apiVersion: v1 kind: ConfigMap metadata: name: litellm-config namespace: litellm data: config.yaml: | model_list: - model_name: qwen3.8-27b litellm_params: model: openai/qwen3.8-27b-vllm api_base: http://vllm-qwen38.vllm.svc.cluster.local:8000/v1 api_key: "none" model_info: max_input_tokens: 32768 max_output_tokens: 32768 router_settings: routing_strategy: usage-based-routing-v2 general_settings: master_key: os.environ/LITELLM_MASTER_KEY store_model_in_db: true 

What is happening here:

• model_name is the public name your users will call the model by.
• model: openai/qwen3.8-27b-vllm: since vLLM speaks the OpenAI API, I use the openai/ provider. The part after the slash must match vLLM's --served-model-name.
• api_base is the cluster-internal DNS name of the vLLM Service: <service>.<namespace>.svc.cluster.local.
• api_key: "none": vLLM has no auth. The security comes from the NetworkPolicy, which only lets LiteLLM in.

Info: If you later want to scale out, just deploy a second vLLM instance with its own Service and add a second entry under the same model_name. LiteLLM then balances the requests across both. I prefer this over a Deployment with replicas: 2, since Kubernetes Services balance per connection and not per request, which doesn't work well with keep-alive HTTP clients.

The LiteLLM Deployment

I use the litellm-database image, since it ships the Prisma binaries and runs the database migrations automatically:

apiVersion: apps/v1 kind: Deployment metadata: name: litellm namespace: litellm spec: replicas: 1 selector: matchLabels: app: litellm template: metadata: labels: app: litellm spec: containers: - name: litellm image: ghcr.io/berriai/litellm-database:main-stable args: ["--config", "/app/proxy_config.yaml"] envFrom: - secretRef: name: litellm-secrets ports: - name: http containerPort: 4000 readinessProbe: httpGet: path: /health/readiness port: 4000 initialDelaySeconds: 60 periodSeconds: 15 livenessProbe: httpGet: path: /health/liveliness port: 4000 initialDelaySeconds: 120 periodSeconds: 15 volumeMounts: - name: config mountPath: /app/proxy_config.yaml subPath: config.yaml readOnly: true volumes: - name: config configMap: name: litellm-config --- apiVersion: v1 kind: Service metadata: name: litellm-service namespace: litellm spec: type: ClusterIP selector: app: litellm ports: - name: http port: 4000 targetPort: 4000 

Again, I'm using a ClusterIP Service here. In my first iteration, LiteLLM had its own LoadBalancer IP. That turned out to be a plaintext back door that skipped TLS completely, so I removed it. The ingress should be the only way in.

Same as for vLLM, I add an egress policy to the litellm namespace that blocks the internet. LiteLLM doesn't need it: the optional fetch of the model cost map falls back to a bundled copy. I also add an ingress policy so that only the ingress controller can reach LiteLLM:

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: litellm-allow-ingress-only namespace: litellm spec: podSelector: matchLabels: app: litellm policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: ingress-nginx ports: - protocol: TCP port: 4000 

Setting up the Ingress

The last piece is the single entry point. I'm using ingress-nginx behind one LoadBalancer Service with a pinned IP, so my DNS record stays valid even if the Service gets recreated.

Heads up: the ingress-nginx project was retired in March 2026 and no longer gets updates or security fixes. I'm still showing it here because that's what my cluster runs today, but for a new setup I'd recommend a Gateway API implementation. The concepts (TLS termination, streaming-friendly timeouts, a single entry point) are exactly the same.

Installing the controller

These are the Helm values I use:

controller: replicaCount: 2 service: type: LoadBalancer loadBalancerIP: "<your-pinned-ip>" ingressClassResource: name: nginx enabled: true default: true config: proxy-body-size: "0" proxy-read-timeout: "3600" proxy-send-timeout: "3600" 

The config block is important for LLM traffic. Prompts can be large, and the responses are streamed as Server-Sent Events, sometimes for minutes. With the nginx defaults (1 MB body limit, 60 second timeout), long generations would simply get cut off.

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \ --namespace ingress-nginx --create-namespace \ -f ingress-nginx-values.yaml 

Hooking up the Ingress with LiteLLM

Now the Ingress resource, which connects the outside world to LiteLLM:

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: litellm-ingress namespace: litellm annotations: cert-manager.io/cluster-issuer: <your-cluster-issuer> nginx.ingress.kubernetes.io/proxy-body-size: "0" nginx.ingress.kubernetes.io/proxy-buffering: "off" nginx.ingress.kubernetes.io/proxy-read-timeout: "3600" nginx.ingress.kubernetes.io/proxy-send-timeout: "3600" spec: ingressClassName: nginx tls: - hosts: - llm.example.com secretName: llm-example-com-tls rules: - host: llm.example.com http: paths: - path: / pathType: Prefix backend: service: name: litellm-service port: number: 4000 

• cert-manager.io/cluster-issuer tells cert-manager to issue a certificate for the host and store it in the Secret from the tls block. No manual certificate handling needed.
• proxy-buffering: "off" is the one people forget most often. Without it, nginx buffers the streamed response and your users see the whole answer at once, after a long wait, instead of token by token.
• ingress-nginx redirects plain HTTP to HTTPS by default, so nothing goes over the wire unencrypted.

Finally, point a DNS A record for llm.example.com at the pinned IP of the ingress controller.

Testing the whole chain

First, a quick health check through the ingress:

curl https://llm.example.com/health/liveliness 

If LiteLLM answers with "I'm alive!", the Ingress → LiteLLM part works. Next, I create a virtual key for a user, which only has access to our Qwen model:

curl -X POST https://llm.example.com/key/generate \ -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ -H "Content-Type: application/json" \ -d '{"models": ["qwen3.8-27b"], "key_alias": "florian-test", "max_budget": 10}' 

And with the returned key, I can now throw my first prompt at the model:

curl https://llm.example.com/v1/chat/completions \ -H "Authorization: Bearer <virtual-key>" \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3.8-27b", "messages": [{"role": "user", "content": "Explain Kubernetes NetworkPolicies in one sentence."}] }' 

If an answer comes back, the full chain Ingress → LiteLLM → vLLM is working. And since the whole thing speaks the OpenAI API, you can point basically any tool at it, be it a chat frontend, a VS Code extension or your own app using the OpenAI SDK, just by changing the base URL and the API key.

Where to go from here

The setup above already gives you a solid baseline. In my cluster I went a few steps further, and I might cover these in future posts:

• A service mesh (Linkerd) for mTLS on every hop inside the cluster, so even the traffic from the ingress to LiteLLM and from LiteLLM to vLLM is encrypted and mutually authenticated.
• Monitoring with Prometheus and Grafana: GPU utilization via DCGM, time to first token, and usage per team, using LiteLLM's built-in Prometheus metrics.
• Multiple models and instances behind the same LiteLLM gateway, including embedding models for RAG.

Conclusion

With vLLM serving the model, LiteLLM handling keys and routing, and a single TLS-secured ingress in front, I now have my own OpenAI-compatible inference platform. My data never leaves my network: the GPU workload can't reach the internet, vLLM is only reachable by LiteLLM, and LiteLLM is only reachable through HTTPS with a valid key. That's a great basis for a secure coding assistant, an internal chatbot, or any other Gen AI use case that has to work with restricted data.