Testing Multi-Cluster Disaster Recovery on Linkerd Without Touching Production

Oct 2026

TL;DR

You can test multi-cluster failover alongside production traffic by sending synthetic requests to a dedicated test service and limiting faults to its workloads. Linkerd federated services provide cross-cluster routing, and Chaos Mesh or LitmusChaos injects the failure.
Linkerd’s automatic-failover extension is deprecated. For Linkerd 2.17 and later, use federated services for new cross-cluster failover drills.
A useful drill makes the test service unavailable in the primary cluster, checks whether requests reach the secondary, then restores the primary. Measure recovery time and failed requests in both directions.

Why untested disaster recovery plans are a liability

A failover configuration proves little until you make its primary cluster unavailable and watch where requests go. Kubernetes disaster recovery testing can expose a missing service export, a routing mistake, or a secondary workload that can’t serve requests.

Isolate the drill before injecting a fault. Put synthetic clients and their federated test service in dedicated namespaces across the linked clusters. If you use a test hostname or request header, verify that it sends requests only to test workloads. A header alone won’t protect production if the chaos experiment still selects pods that serve real users.

Start with the simplest failure model. Scale the test deployment in the primary cluster to 0, confirm that its synthetic requests reach the secondary cluster, then scale it back and check where traffic goes. Chaos Mesh or LitmusChaos can later replace the scale-down with a scoped fault that better matches the failure you want to rehearse.

Set pass criteria before the drill. Record the time between fault injection and stable service on the secondary, then compare it with your recovery time objective (RTO), the maximum interruption you’ve agreed to accept. Count failed synthetic requests during the same interval to calculate the error rate. Repeat both measurements when you restore the primary.

The deprecated failover extension, and why federated services replaced it

Linkerd’s automatic-failover extension is deprecated and will be removed in a future release. The older drill pattern depended on 2 installed extensions. You wrote a TrafficSplit with primary and fallback weights, and a dedicated controller changed those weights when the primary service failed.

For Linkerd 2.17 and later, federated services provide the recommended cross-cluster failover pattern. A federated service gives clients one service address across linked clusters. Federated services need pod-to-pod (flat) network connectivity between the linked clusters, a shared trust anchor, and meshed clients; the gateway-only topology from the multi-cluster guide isn't enough. When the primary cluster’s endpoints disappear, Linkerd stops sending requests to those endpoints and uses the surviving cluster’s endpoints instead. Linkerd’s load balancing then distributes requests among healthy endpoints without the SMI extension, a failover controller, or a TrafficSplit written for the drill.

I’d check that your federated service works across both clusters before testing a failure. The existing multi-cluster guide covers that setup. The drill here starts with a linked service and tests what happens when one cluster’s endpoints become unavailable.

Scoping a failover drill so it never touches production traffic

Isolate the drill before you inject a fault. Create a test namespace in each cluster, and give its workload a dedicated federated service name. Keep production services out of the experiment’s selectors. A fault aimed at a shared service or cluster-wide network link can still disrupt real requests, even when the client sending test traffic lives in its own namespace.

Send synthetic requests through a test hostname, or have your gateway route requests bearing a dedicated header to the test service. Verify the routing rule first. A header on its own doesn’t isolate traffic, and a namespace doesn’t prevent a test client from calling a production service.

Scope a Chaos Mesh network experiment to pods in the test namespace using namespace and label selectors. With LitmusChaos, select the same test workload when choosing a pod or network fault. Check both the pods that originate the fault and its targets before applying it. For a production-adjacent drill, avoid node drains or cluster-wide partitions on infrastructure that production workloads share.

Istio can match a request header when injecting a route-level fault. In this Linkerd drill, the hostname or header directs synthetic requests to a separate service, while the chaos tool’s selectors limit which pods lose connectivity. Those are separate controls, and you need both for an isolated test.

Running the drill: inject, verify, measure, revert

Run the drill against the federated service dr-echo-federated (backed by dr-echo deployment) in dr-test. The commands below assume east is primary, west is secondary, and a dr-client pod in east sends synthetic requests. Each dr-echo response should identify its cluster as east or west. Keep these test pods separate from production workloads.

Establish the baseline. Start requests in one shell and confirm that east serves them. Keep that shell open for the rest of the drill.

kubectl --context=east -n dr-test exec deploy/dr-client -- sh -c '
  while :; do
    response=$(curl -s --max-time 2 -w " %{http_code}" \
      http://dr-echo-federated.dr-test.svc.cluster.local | tr -d '\n') [ -n "$response" ] || response="error 000"
    echo "$response"
    sleep 1
  done
' | awk '{ print systime(), $0; fflush() }' > /tmp/dr-drill.log &
sleep 5
linkerd viz stat svc/dr-echo-federated -n dr-test --context=east
tail /tmp/dr-drill.log

Inject a scoped fault. Apply this Chaos Mesh NetworkChaos resource in east. It partitions the test client from the primary test pods. The selectors leave production namespaces alone.

cat > /tmp/dr-partition.yaml <<'EOF'
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: dr-partition
  namespace: dr-test
spec:
  action: partition
  mode: all
  selector:
    namespaces:
      - dr-test
    labelSelectors:
      app: dr-echo
  direction: both
  target:
    mode: all
    selector:
      namespaces:
        - dr-test
      labelSelectors:
        app: dr-client
EOF

date +%s > /tmp/dr-fault-epoch
kubectl --context=east apply -f /tmp/dr-partition.yaml

Verify the shift. Watch for west 200 in the request log, then check traffic at the secondary deployment. A healthy federated service alone doesn't prove that requests reached west.

tail -f /tmp/dr-drill.log
# In another shell, after west appears
linkerd viz stat deploy/dr-echo -n dr-test --context=west

Measure recovery and errors. Stop following the log with Ctrl-C, then run the commands below. The first west 200 gives a sampled recovery time measured from fault submission. Count every non-200 response during that interval and compare both results with the targets set before the drill.

awk -v start="$(cat /tmp/dr-fault-epoch)" \
  '$2 == "west" && $3 == 200 && $1 >= start {
    print "Recovery seconds:", $1 - start
    exit
  }' /tmp/dr-drill.log

awk -v start="$(cat /tmp/dr-fault-epoch)" \
  '$1 >= start && $3 != 200 { errors++ }
   END { print "Failed requests:", errors+0 }' /tmp/dr-drill.log

Remove the fault, then verify reversion. Record the removal time and watch for healthy east responses again. Measure that interval as failback time. If both clusters remain eligible for the federated service, east may resume serving only a share of requests.

Reading the results: RTO targets and what a failed drill looks like

Judge the drill against an RTO target you set before injecting the fault. Start the clock when the primary becomes unreachable, and stop it when the secondary serves requests consistently. Record failed requests during that interval too. A shift that takes longer than your target is a slow but successful failover.

If requests never reach the secondary, inspect the verification step before changing the RTO target. Check whether the fault reached the test workload, whether the federated service exposes a healthy secondary backend, and whether the synthetic client actually uses that service. A drill that never shifts traffic points to a fault-scope or routing problem, not a slow recovery.

Measure failback separately. Start a second clock when you remove the fault, then check when the primary becomes healthy and receives traffic again. Don’t assume restored reachability forces every request back to the primary. Judge reversion against the routing behavior you intended, and record errors during that interval as well.

How Linkerd's approach compares to Istio and Cilium ClusterMesh

Linkerd, Istio, and Cilium ClusterMesh test different layers of a failure. Istio can inject an HTTP delay or abort for a matched request, which lets you test application behavior without taking down a backend. Its locality failover and multicluster routing handle the separate job of redirecting traffic when backends fail. Cilium ClusterMesh makes backends in connected clusters available through a global service and shifts connections among healthy backends at the network layer.

Project	Failover trigger	Test-traffic isolation	Granularity	Setup for this drill	Best for
Linkerd federated services	Traffic moves to healthy backends across clusters.	Use a dedicated test namespace and service. An HTTPRoute can match test requests when you need header-based routing.	Service-level cross-cluster traffic, with application-aware routing available.	Federate the test service, then scope the fault and synthetic client to it.	Automated cross-cluster failover with few extra components.
Istio	Locality failover redirects traffic after backend failure. A VirtualService can separately inject delays or aborts.	VirtualService rules can match a test header for injected faults. Keep the cross-cluster outage drill scoped to test workloads.	Per-request HTTP faults, plus cross-cluster routing.	Configure the multicluster topology and locality routing. Add VirtualService fault rules if the drill needs them.	Fine-grained synthetic fault injection alongside failover testing.
Cilium ClusterMesh	A global service can send connections to healthy backends in another cluster.	Use a separate test service or namespace and a synthetic client. ClusterMesh doesn’t isolate requests by HTTP header.	Network-level service and backend selection.	Connect the clusters, expose the test service globally, and observe where its connections land.	Network-level L3/L4 high availability when you already run Cilium.

For a cluster-outage drill, keep the injected fault comparable across products. Istio’s header-matched HTTP abort tests a different failure than a partition that makes a cluster unreachable.

Next steps for running this drill on your own clusters

Use Linkerd’s multi-cluster and federated-services documentation to check the cluster links and service exports this drill assumes. Then run the drill against a dedicated test service and keep the recorded RTO and error rate with your incident exercises.

If you want a cross-cluster topology view during the drill, ask Buoyant about Buoyant Enterprise for Linkerd and Buoyant Cloud. You’ll still need to decide what counts as a passing recovery.

FAQ

Is the Linkerd automatic-failover extension still usable, or does it break on upgrade?

The extension still works on releases that include it, but Linkerd has deprecated it and plans to remove it. Check your target release before upgrading, and migrate drills to federated services rather than depending on the extension.

How often should you run a multi-cluster DR drill?

Start with a quarterly production-adjacent drill and repeat it after changes to cluster links, service exports, or routing. Pick a schedule your team can maintain, and compare each run against the same RTO and error-rate targets.

Do you need both Chaos Mesh and LitmusChaos?

Chaos Mesh alone can inject the scoped network partition used in this drill. LitmusChaos is an alternative if you already use it for the failure you want to test.

What RTO should you expect compared with Istio locality failover?

Neither federated services nor Istio locality failover gives you a universal RTO. Fault detection, connection behavior, and your routing configuration affect recovery, so measure both approaches under the same failure and traffic pattern.

Can you use this approach for a single-cluster, multi-zone failure?

Yes, you can isolate test workloads and inject a zone-scoped fault, then measure recovery and failback. For that drill, verify your zone-routing behavior rather than treating cross-cluster federated-service failover as the mechanism under test.