gRPC Got Us - Or Did It?

The other day I ran the Gateway API conformance tests against Linkerd. This is normally a pretty routine chore, but this time, a gRPC routing test failed. This was extremely startling, since Linkerd has supported gRPC routing for about two years now (full GRPCRoute support shipped in Linkerd 2.16). Even more confusingly, the specific failure was around routing with weights, and I had used exactly that feature a few days before.

This sort of thing can happen for a lot of different reasons. Maybe someone snuck in a change in Linkerd; maybe there's a bug in the test; maybe the test uses a configuration that's different from my usage. So which is it?

The easiest thing in the world is to just blame it on a bug in the test -- and that's almost never the right reflex. Figuring out what was going on in this case was quite the ride through a lot of different pieces of the ecosystem; it's well worth a more detailed look.

Intro to gRPC

gRPC's website defines it as "a modern open source high performance Remote Procedure Call (RPC) framework". RPC is basically the same idea as a function call, just across machines¹: you have a named bit of executable code that takes some inputs and returns some data, and you have a defined way to encode those inputs and outputs that preserves type².

Over the wire, gRPC is layered within HTTP/2, so every gRPC call is an HTTP/2 stream with HEADERS frames carrying metadata, DATA frames carrying the inputs and outputs, and trailers (HEADERS frames after all the DATA frames) carrying the final status of the gRPC call. Valid gRPC calls are always HTTP POST requests with Content-Type: application/grpc.

This layering means that, at the most basic level, you can route a gRPC request by routing its HTTP/2 stream -- even as far as routing based on which specific gRPC call you're trying to make (which is encoded in the HTTP/2 PATH) or routing on custom gRPC metadata (which appears in HTTP/2 headers).

On the other hand, metrics and reliability mechanisms (like canaries or retries) are not the same for gRPC as for HTTP/2, because - crucially - every gRPC call returns an HTTP status of 200 even if the RPC fails. To understand the real result of a gRPC call, you must look into the trailers to get the gRPC status. If you don't do that, you'll just see the HTTP 200 and believe that you're seeing a 100% success rate.

So - and this is the first piece of the test-failure puzzle - to do anything significant with gRPC, you have to know you're dealing with gRPC, which means that you have to start by knowing you're dealing with HTTP/2.

Protocol Detection

This need to know the details of the connection isn't only a gRPC thing, of course. In general, anything more subtle than just passing bytes back and forth with no analysis requires you to know what the protocol is. This includes not just golden metrics and reliability, but even basic features like per-request load balancing -- after all, you can't choose endpoints for individual requests if you don't know where the requests start and end in the data stream.

There are fundamentally only two ways to know what protocol is in play: you can to look at the bytes in transit and figure out what protocol it is (protocol detection), or you can be told up front what protocol it is (protocol declaration). Linkerd takes the attitude that detection is friendlier than declaration, so it leans heavily on protocol detection.

However, protocol detection requires the client to send some data (think for a second about things like session affinity, where there's no way to know which replica should get the traffic before you hear from the client which session they're part of). Without any data, protocol detection will wait 10 seconds, then give up. This most commonly happens for protocols like SMTP where it's the server that speaks first, but it can also happen when things are overloaded: on a really heavily loaded cluster, it's possible for a client to not ever get enough CPU to send its first bytes until after protocol detection has timed out.

When Protocol Detection Goes Bad

Here's the second piece of the test-failure puzzle: if protocol detection doesn't get the client data it needs, it doesn't just cause a 10-second delay. Linkerd will also treat the connection as opaque: we do mTLS, but otherwise we just pass bytes back and forth with no visibility into the connection at all.

"No visibility into the connection" is very broad. No per-request routing, since we can't know what constitutes a request. No metrics, since we can't see whether a request succeeds or fails. No retries, no canaries, no circuit breaking, and - arguably the most serious problem - no authorization policy.

Preventing this situation no matter what kind of shape the cluster is in is why protocol declaration exists.

Protocol Declaration

Protocol declaration works based on the (optional) appProtocol attribute of a Service: setting appProtocol completely disables protocol detection and just uses the protocol you declare. This is the third piece of the puzzle: protocol declaration is an override, and nothing else can overrule it.

appProtocol on Service became standard in Kubernetes 1.20 and supports only certain values: service names from the the IANA Service Name and Transport Protocol Port Number Registry, and custom names of the form domain/protocol, including the three values defined by Kubernetes itself:

• kubernetes.io/h2c: HTTP/2 over cleartext as described in RFC 9113;
• kubernetes.io/ws: WebSocket over cleartext as described in RFC 6455; and
• kubernetes.io/wss: WebSocket over TLS as described in RFC 6455.

Since gRPC is not on this list, there is no standard way to use appProtocol to specifically declare gRPC.

This is normally less problematic than it might seem, because gRPC doesn't change HTTP/2, it is encapsulated in HTTP/2. Knowing that the protocol is HTTP/2 (which is still true for gRPC) is enough for Linkerd to know how to interpret individual requests, including looking into the headers to see if Content-Type: application/grpc shows up. If so, great, this is a gRPC call! and we can interpret the trailers to get the gRPC status.

In other words, setting appProtocol: kubernetes/h2c will work just fine for gRPC: Linkerd will trust that the wire protocol is HTTP/2, which will let it use the Content-Type to know when to interpret trailers to get the gRPC status.

What you can't do is set appProtocol: grpc.

When appProtocol Goes Bad

Linkerd is very strict about accepting only the standard appProtocol values. This is the fourth and final piece of the test-failure puzzle: any invalid value for appProtocol means that Linkerd assumes the connection is opaque. This may seem a bit draconian, but it's the only safe way to go: if we decide to interpret appProtocol: grpc to mean cleartext gRPC, what happens if the ecosystem later decides that it should mean gRPC using TLS?

This isn't just an academic concern, to be clear: KEP 3726 originally attempted to define a kubernetes.io/grpc value, but dropped it because no consensus could be reached on whether it would mean cleartext or TLS. (Linkerd could define a linkerd.io/grpc and sidestep that confusion; to date, this hasn't seemed necessary, but if you think it is, let me know.)

Putting the Puzzle Together

To bring all these threads together: for gRPC to work correctly, service meshes (including Linkerd!) need to know that a connection is using HTTP/2, so that they can figure out to interpret HTTP/2 trailers to understand gRPC status. They can detect HTTP/2, or they can have it declared. For Linkerd, a protocol declaration is an absolute override, and Linkerd treats nonstandard appProtocol values as opaque.

And the test that failed had been changed to set appProtocol: grpc since the last time I'd tried to run it. That was the whole problem: since grpc is a nonstandard value, Linkerd switched to opaque, which shut off all gRPC processing, including the weighted per-request routing that the test required. The failure isn't subtle at all, but understanding why it happened very much is.³

So if you're using protocol declaration - which we're sure many of you are - first, pay attention to the standards when you set up your protocols! A wrong value can produce a very confusing error.

Second, though, is a debugging tip. Linkerd as a project is very big on the idea of "first, do no harm": if you add Linkerd to a working application it should keep working. That means we take the standards quite seriously, and we're very careful about how we handle both protocol detection and protocol declaration. If Linkerd seems to be ignoring a Route that it's marked as Accepted, that's always something of a red flag, and incorrect protocol declarations are a great place to start looking.

And props to all the unsung heroes writing tests! Testing is hard, important, and often thankless: more of us should spend time there than do.

Feedback is always welcome; just ping @flynn on the Linkerd Slack.

Footnotes

1. I know it says "procedure", but I stand by describing it as a function call. Practical RPC implementations date back to the 1980s, but RPC as a concept dates back to the 1960s, so its terminology was influenced by languages like PL/I and ALGOL, which used "procedure" where pretty much all the common languages today (notably the entire C family) use "function". To me, a "procedure" never returns a value and a "function" does, but that's because I started out working in Pascal (seriously) before switching to C. ↩
2. gRPC uses Protocol Buffers for this. ↩
3. I ended up going all the way down to the linkerd diagnostics policy command to make absolutely certain that I knew what was happening before working on getting the test fixed. If you can't be that certain, it's probably too early to blame the test. ↩

‍