Linkerd, like every other tool that uses mTLS, relies on properly-managed TLS certificates to function. In the cloud native world, Venafi’s cert-manager and trust-manager tools have become the de facto standard for automating certificate management: cert-manager handles issuing and rotating certificates independently or in conjunction with tools like Vault, and trust-manager handles packaging multiple certificates into bundles, letting other members of the ecosystem know which certificates they can trust.

‍

A great many Linkerd installations use cert-manager and trust-manager to excellent effect, and you can read all about how to set this up in the Linkerd documentation. Recently, though, cert-manager made a change that has the potential to disrupt Linkerd installations with a particularly subtle misconfiguration.

The change is detailed in cert-manager issue 7601: prior to cert-manager 1.17, cert-manager’s default RotationPolicy was Never, and in cert-manager 1.18, it changed to Always. The reason this is important has to do with the nature of “rotating” a certificate.

Rotation Policy Explained

It’s an axiom of security that the longer a secret is used, the more valuable it becomes. Since a certificate has a secret key at its core, this axiom means that if you’re using certificates, you need to replace the secret every so often, which is called “rotating” the certificate. Under the hood, rotating is really replacing the certificate:

• you generate a new secret, thus a new certificate
• you put the new certificate in service
• you then take the old certificate out of service

However, with a RotationPolicy of Never, cert-manager actually leaves the old secret in play: all it does is update the certificate’s expiration date. With Always, cert-manager actually does the rotation, generating a new secret key as one would expect. Never is quite a bit less secure than actually rotating the certificate, so we’re grateful that cert-manager changed the default to Always!

Where Things Can Go Wrong

Unfortunately, this behavior change does open the door to a serious problem. A typical setup with Linkerd and cert-manager will have Linkerd’s trust anchor - the root of all trust in the system - managed by cert-manager. trust-manager will then copy the trust anchor to the trust bundle, which determines which trust anchors are valid for Linkerd. Configured properly, trust-manager will be told how to manage two anchor certificates, so that when cert-manager rotates the trust anchor, the new trust anchor can be put into the bundle before the old one is removed. This is a critical step for avoiding downtime.

A site that leaves the RotationPolicy at the default and only configures trust-manager with one anchor certificate - which some older Linkerd documentation may have described - will work with cert-manager 1.17, but break with 1.18.

The problem happens because at the moment the trust anchor changes, the running Linkerd proxies haven’t heard about the change yet: they’ll still be using the old trust anchor until they restart – but trust-manager with only a single anchor certificate configured will remove the old trust anchor from the trust bundle instantly. When cert-manager with Never doesn’t actually change the trust anchor’s key, this isn’t a problem: the effect is that trust-manager is just updating the dates on the trust anchor that the proxies are using. When cert-manager with Always does change the key, though, the effect is that trust-manager throws away the trust anchor that the proxies are using, which breaks communications until the proxies are restarted.

Mitigation

The mitigation, thankfully, is fairly simple: don’t upgrade to cert-manager 1.18 until checking to make sure that you’ve configured trust-manager according to the current Linkerd documentation! A really quick check here is to take a look at your Bundle resource: if you have only one entry in the sources array, that’s a red flag and you’ll need to fix that. As always, if you have questions, you can email us at support@buoyant.io, or reach out on the Linkerd Slack.

In the future, Linkerd itself will be taking steps to further simplify certificate management, and to be able to more carefully check the trust chain for you, so watch this space for further updates – and in the meantime, stay calm and mesh on!