Centralizing Telemetry with Linkerd Multi-Cluster
Running multiple Kubernetes clusters is great until you realize your telemetry traffic is taking an unnecessarily complicated path. Each cluster had its own Grafana Alloy instance dutifully collecting metrics, logs, and traces—and each one was routing through an internal Nginx reverse proxy to reach the centralized observability platform (Loki, Mimir, and Tempo) running in my internal cluster.
This worked, but it had that distinct smell of “technically functional” rather than “actually good.” Traffic was staying on the internal network (thanks to a shortcut DNS entry that bypassed Cloudflare), but why route through an Nginx proxy when the clusters could talk directly to each other? Why maintain those external service URLs when all my clusters are part of the same infrastructure?
Linkerd multi-cluster seemed like the obvious answer for establishing direct cluster-to-cluster connections, but the documentation leaves a lot unsaid when you’re dealing with on-premises clusters without fancy load balancers. Here’s how I made it work.
The Problem: Telemetry Taking the Scenic Route
My setup looked like this:
– Internal cluster: Running Loki, Mimir, and Tempo behind an Nginx gateway
– Production cluster: Grafana Alloy sending telemetry to loki.mattgerega.net, mimir.mattgerega.net, etc.
– Nonproduction cluster: Same deal, different tenant ID
Every metric, log line, and trace span was leaving the cluster, hitting the Nginx reverse proxy, and finally making it to the monitoring services—which were running in a cluster on the same physical network. The inefficiency was bothering me more than it probably should have.
This meant:
– An unnecessary hop through the Nginx proxy layer
– Extra TLS handshakes that didn’t add security value between internal services
– DNS resolution for external service names when direct cluster DNS would suffice
– One more component in the path that could cause issues
The Solution: Hub-and-Spoke with Linkerd Multi-Cluster
Linkerd’s multi-cluster feature does exactly what I needed: it mirrors services from one cluster into another, making them accessible as if they were local. The service mesh handles all the mTLS authentication, routing, and connection management behind the scenes. From the application’s perspective, you’re just calling a local Kubernetes service.
For my setup, a hub-and-spoke topology made the most sense. The internal cluster acts as the hub—it runs the Linkerd gateway and hosts the actual observability services (Loki, Mimir, and Tempo). The production and nonproduction clusters are spokes—they link to the internal cluster and get mirror services that proxy requests back through the gateway.
The beauty of this approach is that only the hub needs to run a gateway. The spoke clusters just run the service mirror controller, which watches for exported services in the hub and automatically creates corresponding proxy services locally. No complex mesh federation, no VPN tunnels, just straightforward service-to-service communication over mTLS.
Gateway Mode vs. Flat Network
(Spoiler: Gateway Mode Won)
Linkerd offers two approaches for multi-cluster communication:
Flat Network Mode: Assumes pod networks are directly routable between clusters. Great if you have that. I don’t. My three clusters each have their own pod CIDR ranges with no interconnect.
Gateway Mode: Routes cross-cluster traffic through a gateway pod that handles the network translation. This is what I needed, but it comes with some quirks when you’re running on-premises without a cloud load balancer.
The documentation assumes you’ll use a LoadBalancer service type, which automatically provisions an external IP. On-premises? Not so much. I went with NodePort instead, exposing the gateway on port 30143.
The Configuration: Getting the Helm Values Right
Here’s what the internal cluster’s Linkerd multi-cluster configuration looks like:
linkerd-multicluster:
gateway:
enabled: true
port: 4143
serviceType: NodePort
nodePort: 30143
probe:
port: 4191
nodePort: 30191
# Grant access to service accounts from other clusters
remoteMirrorServiceAccountName: linkerd-service-mirror-remote-access-production,linkerd-service-mirror-remote-access-nonproductionAnd for the production/nonproduction clusters:
linkerd-multicluster:
gateway:
enabled: false # No gateway needed here
remoteMirrorServiceAccountName: linkerd-service-mirror-remote-access-in-cluster-localThe Link: Connecting Clusters Without Auto-Discovery
Creating the cluster link was where things got interesting. The standard command assumes you want auto-discovery:
linkerd multicluster link --cluster-name internal --gateway-addresses internal.example.com:30143But that command tries to do DNS lookups on the combined hostname+port string, which fails spectacularly. The fix was simple once I found it:
linkerd multicluster link \
--cluster-name internal \
--gateway-addresses tfx-internal.gerega.net \
--gateway-port 30143 \
--gateway-probe-port 30191 \
--api-server-address https://cp-internal.gerega.net:6443 \
--context=internal | kubectl apply -f - --context=productionSeparating --gateway-addresses and --gateway-port made all the difference.
I used DNS (tfx-internal.gerega.net) instead of hard-coded IPs for the gateway address. This is an internal DNS entry that round-robins across all agent node IPs in the internal cluster. The key advantage: when I cycle nodes (stand up new ones and destroy old ones), the DNS entry is maintained automatically. No manual updates to cluster links, no stale IP addresses, no coordination headaches—the round-robin DNS just picks up the new node IPs and drops the old ones.
Service Export: Making Services Visible Across Clusters
Linkerd doesn’t automatically mirror every service. You have to explicitly mark which services should be exported using the mirror.linkerd.io/exported: "true" label.
For the Loki gateway (and similarly for Mimir and Tempo):
gateway:
service:
labels:
mirror.linkerd.io/exported: "true"Once the services were exported, they appeared in the production and nonproduction clusters with an `-internal` suffix:
– loki-gateway-internal.monitoring.svc.cluster.local
– mimir-gateway-internal.monitoring.svc.cluster.local
– tempo-gateway-internal.monitoring.svc.cluster.local
Grafana Alloy: Switching to Mirrored Services
The final piece was updating Grafana Alloy’s configuration to use the mirrored services instead of the external URLs. Here’s the before and after for Loki:
Before:
loki.write "default" {
endpoint {
url = "https://loki.mattgerega.net/loki/api/v1/push"
tenant_id = "production"
}
}After:
loki.write "default" {
endpoint {
url = "http://loki-gateway-internal.monitoring.svc.cluster.local/loki/api/v1/push"
tenant_id = "production"
}
}No more TLS, no more public DNS, no more reverse proxy hops. Just a direct connection through the Linkerd gateway.
But wait—there’s one more step.
The Linkerd Injection Gotcha
Grafana Alloy pods need to be part of the Linkerd mesh to communicate with the mirrored services. Without the Linkerd proxy sidecar, the pods can’t authenticate with the gateway’s mTLS requirements.
This turned into a minor debugging adventure because I initially placed the `podAnnotations` at the wrong level in the Helm values. The Grafana Alloy chart is a wrapper around the official chart, which means the structure is:
alloy:
controller: # Not alloy.alloy!
podAnnotations:
linkerd.io/inject: enabled
alloy:
# ... other configOnce that was fixed and the pods restarted, they came up with 3 containers instead of 2:
– `linkerd-proxy` (the magic sauce)
– `alloy` (the telemetry collector)
– `config-reloader` (for hot config reloads)
Checking the gateway logs confirmed traffic was flowing:
INFO inbound:server:gateway{dst=loki-gateway.monitoring.svc.cluster.local:80}: Adding endpoint addr=10.42.5.4:8080
INFO inbound:server:gateway{dst=mimir-gateway.monitoring.svc.cluster.local:80}: Adding endpoint addr=10.42.9.18:8080
INFO inbound:server:gateway{dst=tempo-gateway.monitoring.svc.cluster.local:4317}: Adding endpoint addr=10.42.10.13:4317Known Issues: Probe Health Checks
There’s one quirk worth mentioning: the multi-cluster probe health checks don’t work in NodePort mode. The service mirror controller tries to check the gateway’s health endpoint and reports it as unreachable, even though service mirroring works perfectly.
From what I can tell, this is because the health check endpoint expects to be accessed through the gateway service, but NodePort doesn’t provide the same service mesh integration as a LoadBalancer. The practical impact? None. Services mirror correctly, traffic routes successfully, mTLS works. The probe check just complains in the logs.
What I Learned
1. Gateway mode is essential for non-routable pod networks. If your clusters don’t have a CNI that supports cross-cluster routing, gateway mode is the way to go.
2. NodePort works fine for on-premises gateways. You don’t need a LoadBalancer if you’re willing to manage DNS.
3. DNS beats hard-coded IPs. Using `tfx-internal.gerega.net` means I can recreate nodes without updating cluster links.
4. Service injection is non-negotiable. Pods must be part of the Linkerd mesh to access mirrored services. No injection, no mTLS, no connection.
5. Helm values hierarchies are tricky. Always check the chart templates when podAnnotations aren’t applying. Wrapper charts add extra nesting.
The Result
Telemetry now flows directly from production and nonproduction clusters to the internal observability stack through Linkerd’s multi-cluster gateway—all authenticated via mTLS, bypassing the Nginx reverse proxy entirely.
I didn’t reduce the number of monitoring stacks (each cluster still runs Grafana Alloy for collection), but I simplified the routing by using direct cluster-to-cluster connections instead of going through the Nginx proxy layer. No more proxy hops. No more external service DNS. Just three Kubernetes clusters talking to each other the way they should have been all along.
The full configuration is in the ops-argo and ops-internal-cluster repositories, managed via ArgoCD ApplicationSets. Because if there’s one thing I’ve learned, it’s that GitOps beats manual kubectl every single time.