The combination of DeepSeek R1 going open-weight and LLMs becoming genuinely useful for coding, summarization, and analysis has accelerated interest in self-hosted AI inference. Running models locally or in your own cluster gives you privacy, latency, and cost control that API-dependent approaches can’t match.
This is a practical guide to deploying LLM inference in Kubernetes — not theoretical architecture, but actual working deployments.
Architecture Overview
A production LLM deployment has several components:
User Request → API Gateway → Inference Server → Model (GPU Memory)
↓
Metrics + Monitoring
↓
Model Registry (weights storage)
The inference server is where the model runs. The main options:
| Tool | Best For | Strengths |
|---|---|---|
| Ollama | Development, homelab | Simple, Docker-friendly, broad model support |
| vLLM | Production, high throughput | Continuous batching, PagedAttention, OpenAI API compatible |
| Text Generation Inference (TGI) | Production | Hugging Face native, quantization support |
| llama.cpp server | CPU inference, edge | No GPU required, GGUF format |
GPU Node Setup in Kubernetes
Before deploying anything, you need GPU support in your cluster:
# Install NVIDIA device plugin
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm install nvidia-device-plugin nvdp/nvidia-device-plugin \
--namespace kube-system \
--set failOnInitError=false
Verify GPU detection:
kubectl get nodes -o json | jq '.items[].status.allocatable | select(."nvidia.com/gpu")'
# {"nvidia.com/gpu": "1"}
kubectl describe node gpu-node | grep -A5 "Allocatable:"
# nvidia.com/gpu: 1
Label your GPU nodes for targeting:
kubectl label node gpu-node gpu=true gpu-type=rtx-4090
Deploying Ollama for Development
Ollama is the easiest path for homelab and development:
apiVersion: apps/v1
kind: Deployment
metadata:
name: ollama
namespace: ai
spec:
replicas: 1
selector:
matchLabels:
app: ollama
template:
metadata:
labels:
app: ollama
spec:
nodeSelector:
gpu: "true"
tolerations:
- key: gpu
operator: Exists
effect: NoSchedule
containers:
- name: ollama
image: ollama/ollama:0.6.0
ports:
- containerPort: 11434
name: api
resources:
requests:
memory: "4Gi"
nvidia.com/gpu: "1"
limits:
memory: "32Gi"
nvidia.com/gpu: "1"
volumeMounts:
- name: models
mountPath: /root/.ollama
env:
- name: OLLAMA_HOST
value: "0.0.0.0"
- name: OLLAMA_NUM_PARALLEL
value: "4"
livenessProbe:
httpGet:
path: /api/tags
port: 11434
initialDelaySeconds: 30
periodSeconds: 30
readinessProbe:
httpGet:
path: /api/tags
port: 11434
initialDelaySeconds: 10
periodSeconds: 10
volumes:
- name: models
persistentVolumeClaim:
claimName: ollama-models
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ollama-models
namespace: ai
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: local-path
---
apiVersion: v1
kind: Service
metadata:
name: ollama
namespace: ai
spec:
selector:
app: ollama
ports:
- name: api
port: 11434
targetPort: 11434
Pull a model via an init job:
apiVersion: batch/v1
kind: Job
metadata:
name: pull-deepseek-r1
namespace: ai
spec:
template:
spec:
restartPolicy: OnFailure
containers:
- name: pull
image: curlimages/curl:8.5.0
command:
- sh
- -c
- |
curl -X POST http://ollama:11434/api/pull \
-H "Content-Type: application/json" \
-d '{"name": "deepseek-r1:14b", "stream": false}'
Deploying vLLM for Production
vLLM is the inference server you want when throughput matters. Its PagedAttention algorithm dramatically improves GPU memory utilization:
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-deepseek
namespace: ai
spec:
replicas: 1
selector:
matchLabels:
app: vllm-deepseek
template:
metadata:
labels:
app: vllm-deepseek
spec:
nodeSelector:
gpu: "true"
containers:
- name: vllm
image: vllm/vllm-openai:v0.6.4
args:
- "--model"
- "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B"
- "--dtype"
- "bfloat16"
- "--max-model-len"
- "32768"
- "--gpu-memory-utilization"
- "0.90"
- "--enable-prefix-caching" # Cache common prefixes
- "--tensor-parallel-size"
- "1" # Increase for multi-GPU
ports:
- containerPort: 8000
name: api
resources:
requests:
memory: "16Gi"
nvidia.com/gpu: "1"
limits:
memory: "48Gi"
nvidia.com/gpu: "1"
env:
- name: HUGGING_FACE_HUB_TOKEN
valueFrom:
secretKeyRef:
name: huggingface-token
key: token
volumeMounts:
- name: model-cache
mountPath: /root/.cache/huggingface
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 120 # Model loading takes time
periodSeconds: 30
failureThreshold: 10
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 60
periodSeconds: 10
volumes:
- name: model-cache
persistentVolumeClaim:
claimName: vllm-model-cache
vLLM exposes an OpenAI-compatible API:
# Use vLLM with the OpenAI Python SDK
from openai import OpenAI
client = OpenAI(
base_url="http://vllm-deepseek.ai.svc.cluster.local:8000/v1",
api_key="not-required" # vLLM doesn't need a key by default
)
response = client.chat.completions.create(
model="deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
messages=[
{"role": "user", "content": "Explain quantum entanglement simply."}
],
max_tokens=500
)
print(response.choices[0].message.content)
Autoscaling for LLM Workloads
Standard HPA doesn’t work well for LLMs — requests/second isn’t the right metric. GPU utilization or queue depth is better:
# KEDA (Kubernetes Event Driven Autoscaling) with custom metrics
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: vllm-scaler
namespace: ai
spec:
scaleTargetRef:
name: vllm-deepseek
minReplicaCount: 1
maxReplicaCount: 4 # Limited by GPU nodes
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus.monitoring.svc.cluster.local:9090
metricName: vllm_pending_requests
threshold: "5" # Scale up when >5 requests pending
query: sum(vllm_num_requests_waiting{namespace="ai"})
Model Quantization for Memory Efficiency
Running 14B parameter models in FP16 requires ~28GB of VRAM. With quantization, you can fit larger models on smaller GPUs:
# vLLM with AWQ quantization (4-bit, ~7GB for 14B model)
args:
- "--model"
- "TheBloke/deepseek-r1-14b-AWQ"
- "--quantization"
- "awq"
- "--dtype"
- "auto"
- "--max-model-len"
- "16384"
Quantization trade-offs:
- INT8 (GPTQ/AWQ): ~50% memory reduction, ~2-3% quality loss
- INT4: ~75% memory reduction, ~5-8% quality loss on most tasks
- For coding and reasoning: INT8 is usually acceptable; INT4 degrades more noticeably
Monitoring LLM Inference
vLLM exposes Prometheus metrics:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: vllm-metrics
namespace: ai
spec:
selector:
matchLabels:
app: vllm-deepseek
endpoints:
- port: api
path: /metrics
interval: 15s
Key metrics to track:
# Average time to first token (user-perceived latency)
rate(vllm_time_to_first_token_seconds_sum[5m]) / rate(vllm_time_to_first_token_seconds_count[5m])
# Tokens generated per second
rate(vllm_generation_tokens_total[5m])
# Cache hit rate (higher is better — reduces recomputation)
vllm_cache_config_info{cache_dtype="auto"}
# Pending requests queue depth
vllm_num_requests_waiting
Security Considerations
Running LLM inference has specific security concerns:
# Restrict which pods can call the inference API
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-llm-clients
namespace: ai
spec:
podSelector:
matchLabels:
app: vllm-deepseek
ingress:
- from:
- namespaceSelector:
matchLabels:
llm-access: "true"
ports:
- port: 8000
Input sanitization: LLMs can be manipulated via prompt injection. If your LLM is acting on user input that triggers other systems, sanitize aggressively.
Output validation: Don’t trust LLM outputs for security decisions. Treat generated code as untrusted.
Model provenance: Verify model checksums before deployment. A tampered model is a supply chain attack vector.
Conclusion
Self-hosted LLM inference in Kubernetes is production-viable in 2026. The tooling has matured — vLLM for throughput, Ollama for simplicity, and a growing ecosystem of quantized models that fit on consumer hardware.
The key operational challenges are GPU scheduling, model storage (100GB+ for large models), and scaling strategies that account for model load times. Get the basics right — proper resource limits, PVC retention for model storage, and readiness probes with appropriate delays — and you’ll have a stable, private inference deployment.
Start with Ollama and DeepSeek R1 14B. That combination runs well on a single 24GB GPU and gives you a capable, private AI inference endpoint for your homelab or internal tooling.
