The Service Mesh Operator Pattern: What It Actually Does

Why your mesh upgrade should be a one-line diff, and what the controller behind it is doing

If you run Kubernetes in production, you already run operators. They come from OperatorHub, from Helm charts, from the platform team, and they promise to manage something for you. Have you ever looked at what they actually do? If you’ve ever upgraded a service mesh by hand, you know what that management looks like when a person does it. You have a runbook in a wiki, a sequence of kubectl and helm commands, and one engineer who knows the order.

So what is the operator pattern, really? Start from the control loop and the pieces make sense. First, understand how controllers and CRDs (Custom Resource Definitions) relate inside a cluster, why Gateway API is the same declarative idea applied to traffic, and what a mesh lifecycle operator changes for whoever is on call. Every claim about reconciliation behavior below is backed by output from a live cluster. A few examples come from project documentation instead of the cluster, and those are marked as such.

Cluster details:

• K3s v1.34.6, single node
• Linkerd edge-26.8.3 
• Gateway API v1.2.1
• Envoy Gateway v1.2.8

Before Operators

Here’s what a mesh upgrade looks like without an operator. This is a real sequence, run on the live cluster to move Linkerd from edge-26.6.3 to edge-26.8.3. It follows the official Linkerd upgrade guide, and most internal runbooks I’ve seen look the same. Same steps, in the same order, with the same gaps between them where things go wrong. If you’re following along, this manual upgrade is the baseline. We will compare the operator version with it later.

First, the CLI, because the CLI renders the control plane manifests and an old CLI would render the old version. Then the CRDs, because the new control plane expects the new schemas to already exist. Nothing enforces that order except you:

$ linkerd upgrade --crds | kubectl apply -f -

customresourcedefinition.apiextensions.k8s.io/authorizationpolicies.policy.linkerd.io configured
# ... (10 CRDs configured)

$ linkerd upgrade | kubectl apply -f -

deployment.apps/linkerd-identity configured
deployment.apps/linkerd-destination configured
# ... (webhooks, RBAC, services configured)

Three Linkerd control plane deployments roll and come back healthy. It looks like the upgrade is done, but you’re still at step 3 of 6:

$ linkerd version --proxy

Client version: edge-26.8.3
Server version: edge-26.8.3
Proxy versions:
        edge-26.6.3 (29 pods)
        edge-26.8.3 (3 pods)

The control plane is on the new version now. Every workload proxy in the cluster is still on edge-26.6.3, and linkerd check --proxy will list all 29 pods by name until you do something about it. So you restart the data plane yourself, namespace by namespace. The command you reach for is kubectl rollout restart deploy. That command misses the DaemonSet and the StatefulSet. The demo namespace on this cluster runs 25 Deployments, 1 StatefulSet, and 1 DaemonSet, so the restart is 3 commands, times every meshed namespace, and the engineer has to remember all of them.

If they forget one, nothing fails loudly. The DaemonSet keeps running on the old proxy, the control plane keeps serving it, and the two versions drift apart quietly. The first sign shows up weeks later. A proxy behaves differently, someone opens a support ticket nobody can reproduce, and your team spends one hour on incident response just to realize that the cluster was never fully upgraded. Every dashboard says it was. And if the engineer who ran the upgrade has since left the team, nobody knows which step was skipped, because the only record of the procedure was the person.

One routine upgrade takes 6 steps: 2 CLI versions, 3 workload kinds across 2 namespaces, and 1 ordering rule that lives entirely in the engineer’s head. The problem is that this is fragile. Every step is an opportunity to be interrupted, and the cluster has no idea what end state you’re aiming for. “Declare the end state and let the cluster converge” was not an option, and that’s how mesh upgrades become the task everyone agrees is important and nobody wants to do.

The operator pattern, in depth

Three words get used interchangeably in this space, but shouldn’t. A control loop is the algorithm: observe, diff, act, and repeat. A controller is a process that runs a control loop for one kind of resource: the Deployment controller, the ReplicaSet controller, or the reconciler inside an operator. The control plane is the set of components that host those controllers and the API they talk to: the API server, etcd, the scheduler, and kube-controller-manager, which bundles the built-in controllers into one binary. Linkerd has its own control plane: the identity, destination, and proxy-injector components that its proxies talk to. An operator is a controller plus the CRDs it watches, packaged together and carrying domain knowledge that the built-in controllers don’t have. When I use these four words, this is what I'm referring to.

The control loop: observe, diff, act

Kubernetes already solved this problem once, for its own resources. Every built-in controller runs the same loop: observe the actual state, diff it against the declared state (compare what exists to what’s declared), act to close the gap, repeat forever. You can watch the machinery directly. A Deployment is a declaration of pods, and the Deployment controller writes a ReplicaSet to make it so. The ReplicaSet is the intermediate object that holds the actual pod count and pod template. The Deployment exists so that rollouts can swap one ReplicaSet for another:

$ kubectl get rs -n linkerd \
  -l linkerd.io/control-plane-component=destination \
  -o custom-columns=\
"REPLICASET:.metadata.name,\
DESIRED:.spec.replicas,\
CURRENT:.status.replicas,\
OWNER:.metadata.ownerReferences[0].kind"

REPLICASET                       DESIRED   CURRENT   OWNER
linkerd-destination-5fb4cddcbb   1         1         Deployment

Look at the OWNER column. The ReplicaSet exists because the Deployment controller wrote it, reconciling someone’s spec.replicas: 1 into a running pod. Other kinds have other owners (a StatefulSet owns its pods directly, a Job owns its pods), but the mechanism is the same. The loop is durable against drift, and the quickest proof is to break something on purpose. Delete the pod:

$ kubectl delete pod linkerd-destination-5fb4cddcbb-5rmbx -n linkerd --wait=false

$ sleep 4; kubectl get pods -n linkerd -l linkerd.io/control-plane-component=destination

NAME                                   READY   STATUS        RESTARTS   AGE
linkerd-destination-5fb4cddcbb-5rmbx   4/4     Terminating   3          42d
linkerd-destination-5fb4cddcbb-v9gs9   0/4     Init:0/2      0          4s

Four seconds after the delete, a replacement already exists. No one had to restart it manually because the controller saw one pod declared and zero running, so it closed the gap. That loop running forever is the whole idea. The Kubernetes docs describe controllers as control loops that watch the state of your cluster and make or request changes where needed. An operator is the same machinery pointed at knowledge Kubernetes doesn’t have. The same docs define operators as “software extensions to Kubernetes that make use of custom resources to manage applications and their components,” following “Kubernetes principles, notably the control loop.” The Deployment controller knows how to keep 3 replicas alive. An operator knows the things the Deployment controller doesn’t.,For example, that a database upgrade requires a backup first, or that a mesh upgrade must move the control plane before the proxies. That domain knowledge is what an operator is for.

CRDs: teaching the API server a new noun

The other half of the pattern is the API surface. A CRD registers a new kind with the API server, and the moment it’s applied, the API server stores, validates, and serves objects of that kind like any built-in. You get kubectl get, kubectl explain, role-based access control (RBAC), and watches for free. A CR (custom resource) is just another object. Kubernetes handles it the same way as everything else.

This cluster runs a canonical example, the Prometheus Operator, which registers 10 CRDs including prometheuses.monitoring.coreos.com. Here’s the custom resource and what the controller did with it, live:

$ kubectl get prometheus -n monitoring

NAME                                   VERSION              DESIRED   READY   RECONCILED   AVAILABLE   AGE
kps-kube-prometheus-stack-prometheus   v3.13.1-distroless   1         1       True         True        42d

$ kubectl get statefulset -n monitoring

NAME                                              READY   AGE
prometheus-kps-kube-prometheus-stack-prometheus   1/1     42d

The Prometheus object is a declaration: version, replicas, and storage. The operator reconciled it into a StatefulSet, generated config, and wrote RECONCILED=True back into status. That status column is the controller reporting that the diff is closed, and it’ll keep it closed across restarts, node loss, and any other drift that shows up.

So this is how the 3 pieces relate. The CRD is the noun, the controller is the verb, and the control loop is what keeps the sentence true over time. For the Prometheus operator the verbs are things like create the StatefulSet, render the config, reload on rule changes, roll the pods on a version bump. Register the noun, declare an object, and a controller with domain knowledge converges reality onto it, forever.

Once you know the pattern, you see it in many places. The CNCF Operator White Paper notes that “the Prometheus Operator was one of the first ever Operators written, along with etcd, that proved the use case for this problem space”: the etcd operator encoded failover and rolling upgrades for a consensus store, exactly the kind of work you don’t want to do from memory. cert-manager encodes certificate issuance and renewal, watching Certificate resources and renewing them ahead of expiry so nobody has to remember to. In each case the CRD captures what the team wants to say (“a certificate for this domain”), and the controller captures what an expert would do about it.

The tooling that makes writing one tractable

You generally shouldn’t write the reconcile loop’s plumbing (watches, caches, work queues, retries, and leader election) yourself. This problem is already solved, and it is harder than it looks. Every hand-rolled version gets the same things wrong. A missed event after a watch reconnects, a cache that serves stale objects, two replicas reconciling the same resource at once because leader election was never set up. None of that is your domain knowledge, and 4 projects mean you don’t have to touch it:

• Kubebuilder is the standard Go path: it scaffolds the project, generates the CRD manifests and clients from your Go types, and hands you one Reconcile() function to fill with domain knowledge.
• Operator SDK (part of the Operator Framework) builds on the same runtime and adds packaging and lifecycle tooling, plus Helm-based and Ansible-based operators when the reconcile logic is “render this chart with these values.”
• kopf is the Python framework: decorators on event handlers, quickest path for a Python team, in exchange for Go’s type safety and the ergonomics of the Go client machinery.
• Metacontroller removes code generation entirely: it runs the loop in-cluster and calls webhooks you write in any language, which makes it the cheapest way to test whether your domain knowledge fits a control loop at all.

All 4 give you the same result. The loop is solved, so the code you write is the operational knowledge and nothing else. I will come back to this at the end of this post, because it has a downside too.

‍

Gateway API: the same pattern, applied to traffic

A traffic API might look off-topic in an operator post. It belongs here because of its structure: Gateway API is a set of CRDs (GatewayClass, Gateway, HTTPRoute) with controllers implementing them, which is everything from the previous section, with one difference that matters. The nouns are standardized upstream, and so is what the verb has to do. Conformance tests require every implementation to act on an HTTPRoute the same way. In practice that means you write the same HTTPRoute whether Envoy Gateway, Istio, or Linkerd is underneath, and what implementations compete on is everything around the verb: performance, cost, reliability, how fast the controller converges, and which extensions it adds on top. 

A role split is also built into the CRDs, and it’s the part most teams under-use. GatewayClass belongs to the infrastructure provider (“this is the implementation on offer”). Gateway belongs to the cluster operator (“run a listener here”). HTTPRoute belongs to the application developer (“route my traffic like this”). Three personas, three resources, and RBAC boundaries that finally match the org chart. The under-use usually looks like this. A platform team installs Gateway API, then keeps writing every HTTPRoute itself because that’s how Ingress worked, so application teams still file tickets to change a path prefix. The split only pays off when the RBAC follows it. That means cluster-admin on Gateway and GatewayClass, namespace-scoped write on HTTPRoute for the teams that own the services, and a ReferenceGrant wherever a route needs to cross a namespace boundary. Once that is in place, the tickets stop, because the object each persona owns is the one they actually need to change. 

You can watch an implementation register itself, and it’s the operator pattern in miniature. On a cluster with the CRDs but no gateway controller, you can declare a GatewayClass and nothing happens. It sits there with an empty ACCEPTED column because nothing is watching for it. Install a controller (here, Envoy Gateway), declare a GatewayClass naming it, and 5 seconds later the controller has claimed it and let you know it did so by updating status:

$ kubectl get gatewayclass

NAME    CONTROLLER                                      ACCEPTED   AGE
envoy   gateway.envoyproxy.io/gatewayclass-controller   True       5s

$ kubectl get gatewayclass envoy -o jsonpath='{.status.conditions}'

Accepted=True reason=Accepted msg="Valid GatewayClass"

Observe, diff, act, status. It is the same loop, only with different resources.

For a mesh, there’s exactly one more thing to know: in a mesh, an HTTPRoute attaches to a Service, and there’s no Gateway involved. Mesh support has been generally available (GA) in Gateway API’s standard channel since v1.1, and it dissolved into the spec rather than becoming a separate mesh dialect. That was a deliberate decision after a lot of debate in the project. What you do with HTTP routing is independent of which traffic you’re routing, so a second dialect would have duplicated every verb for no gain. The only visible difference is the parentRef, and the cleanest way to read it is that the parentRef defines the kind of traffic you’re routing. Point it at a Gateway and the route governs north-south traffic. Point it at a Service and it governs east-west traffic inside the mesh. Live, from a route on this cluster’s frontend Service (the same routes that feed the per-route metrics in my SLO playbook):

$ kubectl get httproute frontend-products -n otel-demo -o jsonpath='{.spec.parentRefs}'

[{"group":"","kind":"Service","name":"frontend","port":8080}]

kind: Service instead of kind: Gateway, and the mesh applies the route to east-west traffic heading for that Service. That is all there is to it.

One field, one attachment rule, no mesh-specific verbs. That’s why every major mesh implements Gateway API, and why Linkerd treats it as the routing API rather than maintaining a parallel one. What the standardization actually gets you isn't necessarily what the internet buzz might imply, but it's still useful. Your routing config survives a change of implementation, your role boundaries survive a reorg, and the tooling around the nouns is shared. That holds for the standard-channel core, though, and stops there. Implementation-specific policies and annotations still exist (retries, timeouts, and rate limits are the usual suspects), and every one you use is a line that won’t carry over when you switch implementations. Switching still means rewriting those parts by hand. The standard just keeps the rewrite small. 

The mesh lifecycle operator

Now close the loop on the upgrade from the first section. Buoyant Enterprise for Linkerd (BEL) ships lifecycle automation as 2 operators in 1 Helm chart: linkerd-control-plane-operator, which reconciles a cluster-scoped ControlPlane resource, and linkerd-data-plane-operator, which reconciles namespaced DataPlane resources. You’ll need a BEL license, which you get by signing up at the Buoyant portal. Then the install is:

$ helm repo add linkerd-buoyant https://helm.buoyant.cloud

$ helm install linkerd-buoyant linkerd-buoyant/linkerd-buoyant \
    --namespace linkerd-buoyant --create-namespace \
    --set license=$BUOYANT_LICENSE \
    --set controlPlaneOperator.enabled=true \
    --set dataPlaneOperator.enabled=true

That registers the 2 CRDs and starts both operator deployments in linkerd-buoyant. From here on, the mesh’s lifecycle has an API.

The ControlPlane CRD declares what the core control plane should be, and its one load-bearing field is the version:

apiVersion: linkerd.buoyant.io/v1alpha1
kind: ControlPlane
metadata:
  name: linkerd-control-plane
spec:
  components:
    linkerd:
      version: enterprise-2.20.1
      controlPlaneConfig:
        license: <your license>
        # identity, HA, and any control-plane values you'd otherwise pass to Helm

Everything the operator does hangs off that version string. The rest is configuration it carries along.

The DataPlane CRD declares which meshed workloads the data-plane operator keeps on the current proxy. It’s namespaced, and its schema requires a workloadSelector. An empty selector means every meshed workload in the namespace:

apiVersion: linkerd.buoyant.io/v1alpha1
kind: DataPlane
metadata:
  name: linkerd-data-plane
  namespace: otel-demo
spec:
  workloadSelector:
    matchLabels: {}

Apply one per meshed namespace and the whole data plane is under management.

The upgrade flow is now a 1-field change. Edit the resource (kubectl edit controlplane/linkerd-control-plane) and set spec.components.linkerd.version to the new release. Per the BEL docs, the control-plane operator converges first, and kubectl get controlplane reports the diff while it works. STATUS sits at Pending with DESIRED showing the new version and CURRENT the old, then flips to UpToDate. Only after the control plane completes does the data-plane operator restart workloads onto the new proxy, and kubectl get dataplane -A reports that rollout the same way. Per the BEL docs, mid-upgrade it looks like this:

$ kubectl get dataplane -A
NAMESPACE   NAME                 DESIRED   CURRENT   STATUS
otel-demo   linkerd-data-plane   27        11        Pending

Map that onto the manual runbook from the top of this post and the match is exact: steps 1 through 3 (CLI, CRDs, control plane, in that order) belong to the control-plane operator, and steps 4 through 6 (restart every workload kind in every meshed namespace, verify convergence) belong to the data-plane operator, DaemonSets included. The ordering rule moved out of the engineer’s head and into a controller, so it does not disappear when the engineer goes on vacation or leaves the team.

Rollback is the same operation in the other direction. That is the point of declared state. There’s no separate rollback procedure to memorize. You set the version back to the previous release, and the operators converge onto the declaration again. When the declaration was the mistake, you fix the declaration, and the audit trail of the incident is a 2-line diff in Git.

Be honest about what the operator will not do for you. It won’t check whether the target version is a good decision for your infrastructure. It applies your declaration faithfully, even if it is the wrong one. It won’t read the release notes, and a breaking change (a CRD schema migration, a trust-anchor rotation) is still yours to plan before you touch the field. The safety it adds is enforced ordering and visible convergence, and that’s a lot. The judgment call stays with a human.

What changes for the on-call team

The runbook shrinks to a manifest diff. That sentence sounds like marketing until you hold the two artifacts side by side. On one side a wiki page with 6 steps, 3 workload kinds, and a warning about DaemonSets. On the other side a 1-line change to a version field in a Git repo. The sequence is enforced by the controller, so the order no longer depends on what someone remembers, and the answer to “where are we in the upgrade” is a status column instead of a terminal history.

This is also what holds up at fleet scale. Imagine Learning runs Linkerd across several Amazon EKS clusters and hundreds of microservices serving over 18 million students, and manages the mesh with the lifecycle operator. In their words, it “makes it easy to manage our Linkerd deployments with a single custom resource.” They report a 20% reduction in operational overhead, and they drive canary releases with Argo Rollouts through Linkerd’s Gateway API support, which is the same declarative surface from 2 sections ago doing traffic instead of lifecycle.

There is also a cost. An operator is another controller to debug. When it’s stuck reconciling, you have 2 problems, the upgrade and the thing supervising the upgrade. This is where the pattern’s plumbing helps more than a black box would. A stuck ControlPlane reports lastUpdateAttempt, lastUpdateAttemptResult, and lastUpdateAttemptMessage in its status, which is where you start debugging, and the operator’s own logs are ordinary pod logs. But the skill you’re exercising has changed. You’ve traded command-sequence toil for controller-debugging, and a team that has never debugged a reconcile loop should know it’s signing up to learn.

In my experience the trade lands clearly on the operator’s side for anything with a strict order. A controller’s failure modes are visible in one status block, while a manual sequence’s failure modes live in whichever step got skipped, and nothing tells you which one that was.

Where the pattern earns its place

Reach for an operator where the lifecycle is ordered and the domain knowledge is real. That means anything whose runbook contains “first this, then that, and never on a Friday.” That’s databases, certificate authorities, message brokers, and yes, service meshes. Gateway API is the same idea standardized for traffic, which is why it’s the routing surface every major mesh converged on. Linkerd supports Gateway API for routing. BEL ships the lifecycle operator for the mesh’s own runbook. Between the two of them, the mesh’s nouns are declared, and controllers hold the knowledge.

Writing your own is real work. The frameworks made the loop cheap on purpose. Kubebuilder scaffolds a working operator in an afternoon, and then the real work begins, which is deciding which of your engineers’ knowledge is worth encoding into Reconcile(). Look for the runbooks that get exercised monthly, the procedures with a strict order, the steps that page people when they’re skipped. Those are the places where an operator makes sense. If none of that applies, a Helm chart and a good wiki page are enough, and you should say so in the design review.

The manual upgrade at the top of this post took 6 steps, 2 CLIs, and 1 ordering rule that only the engineer knew. The declared version takes 1 field. That difference is what the operator pattern gives you, and now you know exactly what you’re installing when you install it.

Sources

• CNCF, Kubernetes operators: what are they? Some examples
• CNCF, Understanding Kubernetes Gateway API‍
• CNCF TAG App Delivery, Operator White Paper‍
• Gateway API project docs‍
• Kubernetes blog, Gateway API v1.1: service mesh support to Standard
• Kubernetes docs, Controllers‍
• Kubernetes docs, Operator pattern‍
• Linkerd docs, Gateway API support‍
• Buoyant, BEL lifecycle operator docs‍
• Buoyant, Imagine Learning case study

FAQ

What is a Kubernetes operator?

A Kubernetes operator is a controller plus the CRDs it watches, packaged to carry domain knowledge the built-in controllers don’t have, like knowing a mesh upgrade must move the control plane before the proxies. 

What’s the difference between a Kubernetes controller and an operator?

A controller runs a control loop for one resource kind, like the Deployment controller. An operator is a controller plus CRDs carrying domain knowledge, like knowing a database upgrade needs a backup first. 

How does the observe, diff, act control loop work in Kubernetes?

Every Kubernetes controller runs the same loop: observe the actual state, diff it against the declared state, act to close the gap, then repeat. Delete a pod on purpose and the loop replaces it in seconds. 

How does Gateway API handle mesh traffic versus ingress traffic?

In Gateway API, an HTTPRoute’s parentRef sets the traffic type. Point it at a Gateway and you’re routing north-south ingress traffic. Point it at a Service and you're routing east-west mesh traffic instead. 

How does Buoyant Enterprise for Linkerd automate service mesh upgrades?

Buoyant Enterprise for Linkerd ships 2 operators, one for the control plane and one for the data plane. Set 1 version field and both converge the whole mesh, replacing a 6-step manual upgrade with 1 change.