Two years ago, the observability space was fragmented. Teams were choosing between Datadog’s agent, Jaeger for tracing, Prometheus for metrics, and vendor-specific SDKs that locked you into a platform. OpenTelemetry was promising but immature. Today, the landscape has clarified: OpenTelemetry has won the instrumentation layer. Vendors have standardized on it, and the question is no longer “should I use OTel?” but “how do I set it up well?”
What OpenTelemetry Is
OpenTelemetry (OTel) is a CNCF project that provides:
- A specification: Standard data models for traces, metrics, and logs
- SDKs: Libraries for instrumenting applications in Go, Java, Python, JavaScript, .NET, Rust, etc.
- The Collector: A vendor-neutral agent/gateway for receiving, processing, and exporting telemetry
- Auto-instrumentation: Instrumentation that works without code changes (Java agent, eBPF-based)
The value: instrument once, send to any backend. Same application code regardless of whether you use Datadog, Grafana Cloud, Honeycomb, or your own self-hosted stack.
The OTel Collector: Core Architecture
The Collector is the piece most people deploy in Kubernetes. It receives telemetry from your applications and routes it to your backends:
Application (OTel SDK) → OTel Collector → Prometheus / Jaeger / Grafana Loki / etc.
There are two deployment patterns:
Agent (DaemonSet): One Collector pod per node, collecting from all pods on that node. Lower latency, better isolation.
Gateway (Deployment): Central Collector deployment, all apps send to it. Easier to scale, better for routing/processing.
In Kubernetes, the recommended pattern is both: DaemonSet agents for low-overhead local collection, gateway for processing and routing.
Installing the OTel Operator
The OpenTelemetry Operator simplifies deployment in Kubernetes:
# Install cert-manager (prerequisite)
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.0/cert-manager.yaml
# Install OTel Operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
Deploying the Collector
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
name: otel-collector
namespace: monitoring
spec:
mode: DaemonSet # Run on every node
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
config:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
prometheus:
config:
scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
kubeletstats:
collection_interval: 30s
auth_type: serviceAccount
endpoint: "https://${env:K8S_NODE_NAME}:10250"
insecure_skip_verify: true
processors:
batch:
timeout: 10s
send_batch_size: 1000
memory_limiter:
limit_mib: 400
spike_limit_mib: 100
check_interval: 5s
resource:
attributes:
- key: cluster
value: "production"
action: upsert
- key: k8s.node.name
from_attribute: k8s.node.name
action: insert
exporters:
prometheusremotewrite:
endpoint: http://prometheus.monitoring.svc.cluster.local:9090/api/v1/write
tls:
insecure: true
otlp/jaeger:
endpoint: http://jaeger-collector.monitoring.svc.cluster.local:4317
tls:
insecure: true
loki:
endpoint: http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch, resource]
exporters: [otlp/jaeger]
metrics:
receivers: [otlp, prometheus, kubeletstats]
processors: [memory_limiter, batch, resource]
exporters: [prometheusremotewrite]
logs:
receivers: [otlp]
processors: [memory_limiter, batch, resource]
exporters: [loki]
Auto-Instrumentation in Kubernetes
The most powerful feature of the OTel Operator: automatically instrument pods without changing application code:
# Tell the operator to instrument Java, Python, and Node.js apps
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: auto-instrumentation
namespace: production
spec:
exporter:
endpoint: http://otel-collector.monitoring.svc.cluster.local:4317
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: "0.1" # Sample 10% of traces (adjust for your load)
java:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.10.0
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://otel-collector.monitoring.svc.cluster.local:4317
nodejs:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.53.0
python:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.49b0
go:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-go:0.19.0-alpha
Annotate pods to opt in:
# In your Deployment spec.template.metadata.annotations:
annotations:
instrumentation.opentelemetry.io/inject-java: "true"
# Or for other languages:
# instrumentation.opentelemetry.io/inject-nodejs: "true"
# instrumentation.opentelemetry.io/inject-python: "true"
No code changes. The operator injects the instrumentation library as an init container.
Manual Instrumentation in Go
For custom spans and richer telemetry, add OTel instrumentation directly:
package main
import (
"context"
"net/http"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
func setupTracing(ctx context.Context) (*trace.TracerProvider, error) {
exporter, err := otlptracegrpc.New(ctx,
otlptracegrpc.WithEndpoint("otel-collector.monitoring.svc.cluster.local:4317"),
otlptracegrpc.WithInsecure(),
)
if err != nil {
return nil, err
}
tp := trace.NewTracerProvider(
trace.WithBatcher(exporter),
trace.WithResource(resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceName("my-service"),
semconv.ServiceVersion("1.0.0"),
attribute.String("environment", "production"),
)),
)
otel.SetTracerProvider(tp)
return tp, nil
}
var tracer = otel.Tracer("my-service")
func processOrder(ctx context.Context, orderID string) error {
ctx, span := tracer.Start(ctx, "processOrder",
trace.WithAttributes(attribute.String("order.id", orderID)),
)
defer span.End()
// Nested spans are automatically parented
if err := validateOrder(ctx, orderID); err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())
return err
}
span.SetAttributes(attribute.String("order.status", "processed"))
return nil
}
func main() {
ctx := context.Background()
tp, err := setupTracing(ctx)
if err != nil {
panic(err)
}
defer tp.Shutdown(ctx)
// Auto-instrument the HTTP server
mux := http.NewServeMux()
mux.HandleFunc("/order", func(w http.ResponseWriter, r *http.Request) {
// Context from request already has trace context from incoming headers
if err := processOrder(r.Context(), r.URL.Query().Get("id")); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
}
w.WriteHeader(http.StatusOK)
})
// otelhttp wraps the handler and creates spans for each request
http.ListenAndServe(":8080", otelhttp.NewHandler(mux, "my-service"))
}
The Full Stack: Grafana + Prometheus + Jaeger + Loki
With OTel collecting data, you need backends:
# Deploy the complete observability stack via Helm
# Prometheus for metrics
helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--create-namespace \
--values prometheus-values.yaml
# Grafana Loki for logs
helm upgrade --install loki grafana/loki-stack \
--namespace monitoring \
--values loki-values.yaml
# Jaeger for traces
helm upgrade --install jaeger jaegertracing/jaeger \
--namespace monitoring \
--set storage.type=badger \
--set allInOne.enabled=true
In Grafana, you can correlate across all three:
Trace ID in a slow request → Switch to logs view filtering on that trace ID
→ Switch to metrics showing system load at that timestamp
This correlated observability — clicking from a trace to the relevant logs to the metrics that explain the system state — is what OTel enables that vendor-specific tooling used to silo.
Conclusion
OpenTelemetry is the instrumentation standard for the next decade. The vendor support is universal, the Kubernetes integration is first-class, and the auto-instrumentation story means you don’t need to change application code to get traces and metrics.
If you’re running services in Kubernetes and still using separate, siloed observability tools without OTel in the middle, the migration is worth the effort. The payoff is portable instrumentation, correlated traces/metrics/logs, and freedom to change backends without touching application code.
Start with the OTel Operator and auto-instrumentation — you’ll have distributed tracing across your cluster without writing a line of application code.
