Service meshes have been around for a while, but when do you adopt one, and how do you do it?

The concept of microservices wasn’t debuted by the creator of Docker  under bright lights on a keynote stage. Microservices, or the idea of them, have always been around in software engineering. Some articles date back to the early 1970s, quoting these lightweight, performant and specialized units of software. However, microservices were not practical for a long time due to the complexities to implement them. It’s easy to look at the Unix philosophy of “do one thing and do it well” and think of that itself as microservices - at least in ideology. 

It’s hard to talk about microservices without mentioning containers nowadays – and that’s not a new idea either. Linux containers (LXC) date back to 1999, built on the same technology of `cgroups` and OS namespaces that Docker marketed later. A fun blue whale and a man named Solomon made technology that has lived on our Arch machines for decades into a technological revolution. 

Now, looking upon 2015 and 2016, there was a type of “boom” that occurred. Docker debuted, inspiring lightweight, ephemeral environments. This concept of ephemerality led into a problem: how do we manage a bunch of ephemeral, unruly things? Engineering managers have worked on exactly this for decades, of course (they call it “herding cats”), and fortunately for engineers, Google released Kubernetes to shepherd containers. But then, of course, a new issue arose.

By adopting microservices, we adopted containers, and by adopting containers, we now need to adopt something to allow these services to communicate and connect without hardcoding in IP addresses. How do we handle connectivity and routing, if all the addresses of these environments are constantly changing?

This was already an issue in mega-enterprises such as Google, Twitter, and IBM. At Twitter they made proxies to enable applications built from disparate teams to have a shared way to communicate to each other. As long as all the teams agreed on the common API of the proxies, they could build and deploy independently, using whatever language and runtime they liked, and trust that things would work. These types of proxies inspired what is now called service mesh. 

A mesh walks into a bar in Rome

A “service mesh” wasn’t a real thing until a few open source engineers trying to capitalize on their passion and had to figure out how to market their tool. The creators of Linkerd had issues trying to describe what it was their software did. 

Imagine you’re at Statizone di Roma Termini - Rome’s central train station - with a heavy backpack and a paper ticket in hand. Around you, hundreds of other people navigate a thicket of coffee shops, ticket gates, information stations, and - most importantly - the trains themselves. You go to the information desk to ask what platform your train is on, forming a line out into the Italian sun with a mass of other tourists who speak a plethora of different languages. The information desk attendant only speaks Italian. People in line start to miss their trains, and many cannot even communicate with the attendant to find their platform. Those who are able to get past the attendant are not always routed to the proper platform. Some are attempting to board incorrect trains, there is no tracking about who is on what train. 

But in this scenario, you’re not just a tourist on holiday, you’re also a request within a Kubernetes cluster. That paper ticket in your hand is a packet, it’s telling you your destination, some metadata, and the type of transit you should be taking. 

So, how do we solve this?

CNI sets the stage

In Kubernetes clusters, with applications built of many different languages, a service mesh allows for a standard form of communication. This enables routing for polyglot distributed applications, with minimal overhead of operations teams. Service mesh communication is monitored, encrypted, and traced. Instead of tens or hundreds of tourists hitting your ingress, a service mesh acts as an automated gate with multiple check points to ensure people get to their destination.

Service mesh being a decade old, it is easy to think they are ubiquitous as containers themselves. But, they are not as widely adopted. It is important to understand the state of a Kubernetes cluster and Kubernetes networking without a service mesh. When Kubernetes released, and for some time after, the operator (SRE, DevOps Engineer, choose your player) of the cluster was in charge of creating connectivity between containers on different hosts. This was a painful amount of YAML and bash strung together in a Makefile. Service mesh aimed to help ease this pain on operators, handling pods to pod communication and navigating L4 & L7 routing within clusters.

Kubernetes and Operators evolved, eventually debuting the Kubernetes Network Model which set a standard for Kubernetes communication. The network model states that all pods must communicate with all other pods, whether on the same or different nodes - and these pods can communicate to each other without proxies or address translation. This allows for applications to be deployed onto clusters without the need of a service mesh, or much human intervention, enabling inherent connectivity. The “NetworkPolicy” resource allows controlling traffic flow both between two pods and between pods and the outside world – but only at OSI Layer 4, which is to say the level of IP addresses and port numbers. At OSI Layer 7 - HTTP and gRPC - you use Gateway API for the same kinds of functionality.

To implement the Kubernetes Network Model within a cluster, an operator must install a Container Network Interface (CNI) plugin, which runs as a pod on your node, handling all internal and external traffic. CNI provides a standard in the Kubernetes networking space, enabling cloud providers to fit Kubernetes clusters into their existing network models by creating custom CNI plugins and leaving the rest of Kubernetes alone.

Of course, the CNI plugins have gained functionality over time, raising questions about whether service meshes are even still needed today.

It depends (as always!) on your needs. Most service meshes provide granular traffic control at OSI Layer 7 (HTTP and gRPC). While a CNI plugin’s focus is on traffic at the IP address and port level, a service mesh can handle a variety of different protocols including gRPC. 

Service meshes can enable more secure communication for sensitive information through the use of encryption. In private sectors this is especially important for maintaining compliance, and this is where service meshes shine. Most modern meshes provide end to end traffic encryption, audit logging and tracing. This functionality gives Kubernetes operators the insight they need into the daily activities of their cluster, enabling quicker debugging during those late night pages.

Simply put, basic Kubernetes networking using only a CNI plugin is a great way to getting started. But even in unregulated environments, to create confidence in deployments and provide reliable services a mesh is necessary.

Navigating adoption discussions with empathy

To adopt another piece of software in your cluster can be a challenge. Navigating a plethora of opinions from different teams and trying to explain the impact to senior leadership is a real challenge, and it’s easy to fall into the trap that a mesh is unnecessary due to the social burden of adoption. That is why it is important to know how to talk to your boss about service mesh. 

There are always questions when Kubernetes operators, platform engineers, or software engineers advocate for additional software. It is important to understand what senior management and leadership care about to be able to advocate effectively and nullify their concerns. Here are the top three questions, and how to approach answering them:

What is the current problem of today?

Describing the toil teams are experiencing, how late night pages are having people fall asleep midday, and… can prompt understanding from leadership teams. To gain an engineering manager on the side of advocating for a job to be easier, they must understand the pain points.

What are the long term costs of waiting, and what are the costs of adoption?

No piece of software seamlessly integrates without some sort of issue. Management wants to know the current cost of this problem, the amount they need to spend to fix it – and if it doesn’t work out, they want to know the cost to get rid of it.

Why Now?

Management may feel that everything has been working fine without a service mesh. But fine doesn’t mean forever, and if management wants growth and 10x revenue, that may come with 10x the engineering burden. “Why now?” can be answered by tying back to leadership goals; for example, if a Q4 goal is to retain 20% of the platform’s subscribers, it is easy to tie the line between retention and reliability, and between platform reliability and the observability and reliability features a mesh provides (for example, gRPC load balancing or OTel integration).

Personally, I find the Hashicorp RFC template the primary way I approach these discussions. It gets me to write down my ideas in a way where I can articulate them more aptly, and it works as a take home worksheet for my engineering manager - now they can use it to help senior leadership understand why we need to adopt a service mesh. Whatever you do, when having the service mesh discussion, show up with metrics and empathy. 

FAQ

What is the Kubernetes Network Model?

The Kubernetes Network Model is the standard that says every pod must be able to reach every other pod, with no proxies or address translation needed.

What are the most popular CNI plugins?

Cloud provider CNI plugins are the most common choice. Each one implements a different networking model, from overlay to BGP, which changes your security posture.

Do I still need a service mesh if I already have a CNI plugin?

Yes, if you need Layer 7 control. A CNI plugin handles traffic at the IP and port level. A service mesh adds HTTP and gRPC routing, encryption, and audit logging on top.

When is the right time to talk to my boss about service mesh?

The best time to talk to your boss about service mesh was yesterday, but today still beats waiting until you're underwater. Start the conversation before an outage forces it.

What should I focus on when pitching a service mesh to my boss?

Focus on the toil your team faces today, the cost of waiting versus adopting, and why now, tied to a leadership goal like retention or reliability.