‍

A great many Linkerd installations use cert-manager and trust-manager to excellent effect, and you can read all about how to set this up in the Linkerd documentation. Recently, though, cert-manager made a change that has the potential to disrupt Linkerd installations with a particularly subtle misconfiguration.

The change is detailed in cert-manager issue 7601: prior to cert-manager 1.17, cert-manager’s default RotationPolicy was Never, and in cert-manager 1.18, it changed to Always. The reason this is important has to do with the nature of “rotating” a certificate.

Rotating Certificates

It’s an axiom of security that the longer a secret is used, the more valuable it becomes. Since a certificate has a secret key at its core, this axiom means that if you’re using certificates, you need to replace the secret every so often, which is called “rotating” the certificate. Under the hood, rotating is really replacing the certificate:

• you generate a new secret, thus a new certificate
• you put the new certificate in service
• you then take the old certificate out of service

However, wIth a RotationPolicy of Never, cert-manager actually left the old secret in play, just updating the certificate’s expiration date. With Always, cert-manager actually does the rotation, generating a new secret key as one would expect. Never is quite a bit less secure than actually rotating the certificate, so we’re grateful that cert-manager changed the default to Always!

The Outage

However, this behavior change does open the door to a serious problem. A typical setup with Linkerd and cert-manager will have Linkerd’s trust anchor - the root of all trust in the system - managed by cert-manager. trust-manager will then copy the trust anchor to the trust bundle, which determines which trust anchors are valid for Linkerd. Configured properly, trust-manager will be told how to manage two anchor certificates, so that when cert-manager rotates the trust anchor, the new trust anchor can be put into the bundle before the old one is removed. This is a critical step for avoiding downtime.

A site that leaves the RotationPolicy at the default and only configures trust-manager with one anchor certificate - which some older Linkerd documentation may have described - will work with cert-manager 1.17, but break with 1.18. This happens because, when the trust anchor is rotated, trust-manager will blindly overwrite the trust bundle rather than copying in the new anchor before changing the old one – and while that’s OK when cert-manager with Never didn’t actually change the trust anchor’s key, it will always cause downtime when cert-manager with Always does change the key.

Mitigation

The mitigation, thankfully, is fairly simple: don’t upgrade to cert-manager 1.18 until checking to make sure that you’ve configured trust-manager according to the current Linkerd documentation! (If you’re not sure how to check, you can email us at support@buoyant.io, or reach out on the Linkerd Slack.)

In the future, Linkerd itself will be taking steps to further simplify certificate management, and to be able to more carefully check the trust chain for you, so watch this space for further updates – and in the meantime, stay calm and mesh on!