Multicluster mTLS: The Trust Model Walked Through

Multicluster mTLS: The Trust Model Walked Through

When operating multiple clusters in a service mesh, you focus on issues around traffic flow and mutual trust. With a single cluster, one trust chain is straightforward to reason about. But each cluster you add brings its own trust chain, and the interactions between them get hard to track quickly. To keep those trust chains healthy, you need to understand how they work and how they fail, ideally before they fail in production. A misconfiguration in a certificate or trust chain can quickly lead to a complete outage and a long, sleepless night.

In a healthy multicluster setup, a pod in one cluster communicates securely with a pod in another cluster over mTLS. While it works, you never think about it. But the moment it breaks, you have to understand it fast. 

This blog post walks you through the trust model so that, in case your multicluster communication breaks, you know exactly how to fix it. To do that, we built the trust model on a local two-cluster lab. We cover the certificate chain behind a workload identity, follow the cross-cluster handshake in both modes, then break the mesh to see how each failure shows up.

The environment

• 2 local kind clusters, east and west, each running Linkerd edge-26.6.3.
• cert-manager v1.20.2 and trust-manager v0.23.0.
• MetalLB v0.15.2 provides the LoadBalancer IP for the gateway in hierarchical mode.

To explain the multicluster trust model, we use a setup with two clusters, east and west, that share the same trust anchor while each keeps its own identity issuer. To test the trust chain and communication across the clusters, east runs a workload called client that calls an nginx backend running in west.

The certificate chain and workload identity

By default, every meshed workload in Linkerd is issued a TLS certificate that is used for automatic mTLS. There are three types of certificates forming the chain:

• The trust anchor is a self-signed, typically long-lived root certificate that is not automatically rotated. Each proxy is configured to trust its public certificate, which Linkerd passes to it as an environment variable. Thus the proxy reads its trust roots once, at startup.
• The identity issuer is an intermediate certificate signed by the trust anchor. It is stored as a certificate and private key in the linkerd-identity-issuer secret in the linkerd namespace, and is used to sign the workload certificates.
• The workload certificate is the leaf certificate, valid for 24 hours. The Linkerd identity controller automatically issues it. Each proxy gets a workload certificate and requests a fresh one before it expires. The private key always stays in the proxy's memory and never touches persistent storage.

In a multicluster setup, the leaf name is important. A workload identity is represented by a ServiceAccount, a namespace, and a trust domain, encoded as a DNS name in the certificate. The leaf for our client workload in the east cluster looks as follows:

Subject: CN=default.mc-demo.serviceaccount.identity.linkerd.cluster.local         DNS:default.mc-demo.serviceaccount.identity.linkerd.cluster.local

It's a plain DNS name (not a spiffe:// URI) that reads as ServiceAccount default, namespace mc-demo, trust domain linkerd.cluster.local. Linkerd uses SPIFFE-style workload identity and encodes it as a DNS name by default. mTLS authenticates both ends of a connection. So when pods communicate across clusters, each side has to validate the other's leaf certificate. For that to work, both clusters have to trust the same root that those leaf certificates chain up to. As the central piece of the multicluster trust model, this is achieved with one shared trust anchor trusted by all clusters. Simple as that.

One root, many issuers

The Linkerd documentation mentions the requirement for "a shared trust anchor to exist between the installations in all clusters that communicate with each other."

Identity issuers may stay separate, each cluster running its own. Both just need to chain to the same trust anchor, as shown below.

kind-east   root=48163E2D...B072   issuer=120A1767...0798kind-west   root=48163E2D...B072   issuer=4E555FDC...EE40

Keeping a unique issuer for each cluster is the safest design, and it comes down to blast radius. A single shared issuer signs for the entire mesh, so leaking that one key exposes every cluster. Per-cluster issuers keep a compromise with the cluster that owns the key and leave the others untouched. One leaked issuer becomes a one-cluster incident instead of a mesh-wide one.

The mechanics of a shared root rely on X.509 basic path validation (RFC 5280), where a leaf from a peer validates only if it chains to a root already present in the trust store. In our case, when the west proxy receives a certificate signed by the east issuer, it cannot recognize the issuer directly. Luckily for us, it doesn't need to, as the east issuer certificate is signed by the shared root, which is already present in the west trust store. The chain closes successfully. If the roots were different, the chain would break, and we'll cover what to do in that situation later.

The cross-cluster handshake

To understand the magic of the cross-cluster handshake, we need to cover the whole chain. Which side presents a certificate? Whose identity survives? The behavior depends on the mode in which we run Linkerd multicluster.

In hierarchical mode (also called gateway mode), clusters communicate through a linkerd-gateway service on a LoadBalancer IP. In flat (pod-to-pod) mode, pods reach each other directly over the network, with no gateway in between. The third, federated mode, joins a service across clusters into one and preserves the identity the same way flat mode does. We set it aside and focus on the two that treat identity differently.

Hierarchical (gateway) mode

By default, in hierarchical mode, the proxies in east and west clusters don't talk to each other directly. This is because the request from the east proxy is sent to the linkerd-gateway in west, which is reachable at a LoadBalancer IP. The gateway then forwards the request to the backend pod. For east to route to west at all, it has to know that the west cluster exists and where the west gateway lives. That is exactly what the Link resource is for. You generate it by running the linkerd multicluster link-gen command against west and applying its output in east. 

linkerd --context=west multicluster link-gen --cluster-name west | kubectl --context=east apply -f -Output:secret/cluster-credentials-west createdlink.multicluster.linkerd.io/west created

The Link records the west gateway address and identity, so the east proxy knows where to reach the gateway and which identity to require.

# kubectl --context=east -n linkerd-multicluster get link west -o yaml (output trimmed)spec:  gatewayAddress: 10.89.0.240  gatewayIdentity: linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local  gatewayPort: "4143"  probeSpec:    path: /ready    period: 3s    port: "4191"

As communication flows through the gateway, it results in two mTLS hops: one from the client proxy to the gateway, and another from the gateway to the backend. Each one of them is secured, as can be seen in the output of linkerd viz edges on west, which marks the edge as SECURED.

SRC              DST      SRC_NS                DST_NS   SECUREDlinkerd-gateway  backend  linkerd-multicluster  mc-demo  √

The important part to mention is that, due to the gateway acting as a middleman, the backend sees the gateway as the caller identity, not the client.

client_id="linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local"

Every cross-cluster request looks like it came from the linkerd-gateway. As the Linkerd docs put it, the "client identity is lost when going through the gateway". This matters for authorization, because a backend that needs to authorize by the calling workload cannot do so when every request arrives with the gateway's identity. Often that is fine, but when you need per-caller authorization across clusters, use flat mode, which preserves the caller's identity end to end.

Flat (pod-to-pod) mode

Flat mode removes the gateway from the equation. The connection between proxies is established directly. As a result, our backend sees the real caller identity, which is the client workload in east. For that to work, the pod networks need to be routable across clusters, which isn't usually the default. This is typical in managed Kubernetes offerings such as EKS, where you set up cross-cluster pod routing yourself. Once the pod networks are mutually reachable, install the Linkerd multicluster extension with --gateway=false, and export the backend service to east with the label mirror.linkerd.io/exported=remote-discovery. As a result, the east proxy obtains the real IP of the west backend pod and connects to it directly. Compared to hierarchical mode, nothing sits in the middle of the traffic, so the caller identity is preserved. As a bonus, it's one mTLS hop less, which improves latency and throughput.

client_id="default.mc-demo.serviceaccount.identity.linkerd.cluster.local"

As is often the case with things that seem to "just work", there is an important catch. Here, the tradeoff is the network. With hierarchical mode, only one routable endpoint per cluster is needed (the gateway LoadBalancer). With flat mode, direct pod-to-pod routing between clusters is required. Depending on your network topology and security restrictions, that may not be an option.

Day 2

If everything were easy all the time, we wouldn't need SREs, DevOps engineers, and posts covering trust-model internals in the first place, yet here we are. Day 2 operations can be tricky, failures happen, and their scale may be terrifying. To help, below we describe the most common ones, with tips on how to recognize and fix them.

Mismatched trust anchors

A mismatch in trust anchors is the most common cause of cross-cluster mTLS failure. This happens when the two clusters don't share the same root certificate, either due to a misconfiguration or a mistake during rotation. In such a case, in-cluster mTLS keeps working for both clusters just fine, and linkerd check passes without any sign of an error, but every cross-cluster call fails with 503. Luckily, the linkerd multicluster gateways command reports the peer gateway as ALIVE=False, and the root cause is also present in the source proxy logs as UnknownIssuer, so it's pretty easy to spot.

service{name=probe-gateway-west}:endpoint{addr=10.89.0.240:4191}: linkerd_reconnect:  Failed to connect error=invalid peer certificate: UnknownIssuer

One important detail is which certificate is actually being rejected. Remember that in hierarchical mode the caller identity terminates at the gateway, so the certificate being rejected here is the gateway's, not the workload leaf you would probably suspect first.

Issuer expiry

If an issuer expires, the cluster can't issue or sign certificates for the workload anymore. When this occurs, the linkerd check command fails on the issuer cert is within its validity period check, while the proxy logs show certificate expired errors. If a pod restarts or is rescheduled, its proxy cannot get a new leaf certificate. Without one, it cannot rejoin the mesh, and traffic to it fails. If we set up separate issuers as per best practice, the damage is contained to the cluster with the expired issuer.

Wrong gateway identity in the Link

The Link resource is a Kubernetes CRD that records the remote gateway address and identity. The source cluster checks the health of the remote gateway over mTLS, so the identity in the Link must match the one presented by the gateway. In the event of a mismatch (a wrong identity recorded in the Link), the source proxy can't authenticate the gateway, the probe fails, and the service-mirror marks the gateway as unhealthy.

level=warning msg="gateway returned unexpected status 503" probe-key=westlevel=warning msg="Failure threshold (3) reached - Marking as unhealthy" probe-key=west

Although anchors and issuers may be perfectly fine, linkerd multicluster gateways still report the gateway as ALIVE=False and cross-cluster traffic fails. This time there is no certificate error. The gateway certificate is trusted and valid, and linkerd check passes in both clusters. All the usual suspects seem fine. It's just the Link that expects an identity the gateway doesn't present. To fix that, regenerate the Link with linkerd multicluster link-gen and apply it to the source cluster, so the recorded identity is correct.

One root, the rest per cluster

The trust model comes down to a simple rule: share a single root, keep an issuer per cluster, and you may live happily ever after. Failure modes bite, so build confidence by preparing before the bad days come.

If you already run a multicluster setup, it is worth checking a handful of things before you ever need them. Make sure that every cluster really chains to the same anchor, that each cluster keeps its own issuer, and that you can recognize the failure signals we covered before you meet them in production. I strongly believe that documentation should be considered a love letter to your future self, so I went ahead and created this one for you. Use it to boost your confidence, understand the devil in the details, and mostly see that there really is no such thing as magic behind it. Read it, understand it, train it, break it, fix it, and repeat, so the next time you need it in production you are well prepared.

Sources

• Linkerd automatic mTLS (the certificate hierarchy and the 24-hour workload certificate)
• Linkerd multicluster communication (the shared trust anchor requirement)
• Linkerd pod-to-pod multicluster communication (identity preserved in flat mode, lost through the gateway)
• Linkerd multicluster reference (hierarchical, flat, and federated modes)
• RFC 5280, section 6 (X.509 certification path validation)
• SPIFFE overview (the workload identity format)

FAQ

Why does Linkerd multicluster require a shared trust anchor?

A cross-cluster connection only comes up if each side can validate the certificate the other presents, and a certificate validates only if it chains to a root already in your trust store. If two clusters have different roots, neither can close the chain to the other's workloads. Issuers can differ freely, as long as they both chain to the shared anchor.

Should each cluster have its own identity issuer?

Yes, and it’s the safer default. Each cluster keeps its own identity issuer, and both chain to the shared anchor, so no cluster holds another cluster's signing key. If an issuer is compromised or expires, the damage is contained to that one cluster instead of the whole mesh.

Does the calling workload keep its identity across clusters?

It depends on the mode. In hierarchical (gateway) mode, it doesn’t. The request goes through the remote linkerd-gateway, and the backend sees client_id=linkerd-gateway, not the caller. In flat (pod-to-pod) mode, it does. The proxies connect directly, so the backend sees the real calling workload and can apply an authorization policy on its ServiceAccount.

How do you rotate a shared trust anchor across multiple clusters?

Add the new root to the trust bundle in every cluster and rolling-restart all meshed pods, so every proxy trusts both the old and new root. Then repoint the identity issuer in each cluster to the new root, one cluster at a time. Once you've switched every issuer, drop the old root everywhere and restart once more. The key constraint is that every cluster has to trust both roots before any cluster starts issuing under the new one.

Why does cross-cluster mTLS fail with “invalid peer certificate: UnknownIssuer”?

The two clusters don’t share the same trust anchor. The certificates in one cluster chain to a root the other doesn’t have. In-cluster mTLS stays healthy and linkerd check passes on both sides, so only cross-cluster calls fail, which makes it easy to misread. The fix is to put the shared root in both trust bundles and restart the proxies, so they pick it up. If the two roots share a subject name but differ, as in a half-finished anchor rotation, the symptom is BadSignature instead, with the same cause.