How PayNearMe stopped a gRPC crisis with Buoyant

After deploying in only two weeks, Linkerd helped PayNearMe stop a gRPC crisis and gain the observability they were missing

About PayNearMe

PayNearMe builds technology to transform the payment experience for non-commerce businesses and their customers. A single platform, contract, and integration supports all major payment types and channels, from PayPal, Venmo, Cash App Pay, Apple Pay, and Google Pay to cards, ACH, and cash at over 62,000 U.S. retail locations. PayNearMe operates a highly available, cloud-based infrastructure across multiple regions.

‍

The challenge: gRPC load balancing and a high-impact outage

As PayNearMe continues to move to more modern infrastructure and expands its microservices footprint, the team ran into a fundamental limitation in Kubernetes. The team encountered limitations in how persistent connections were distributed across services, which introduced uneven traffic patterns and resilience challenges.

A component in the traffic management layer introduced unintended routing behavior. When it experienced connectivity issues, it amplified downstream service instability, contributing to a broader outage.The outage triggered the need for a replacement tool with gRPC load balancing capabilities. 

‍

Why Buoyant Enterprise for Linkerd

Members of the team had been pushing for a service mesh even prior to the outage. After the outage, the need became apparent that a service mesh would solve the gRPC issues and help in other areas that the team had explored. When evaluating which service mesh, the team relied heavily on internal feedback from those who had used a service mesh at previous companies. PayNearMe had already validated Linkerd's open-source version in lower environments and confirmed it could solve the gRPC problem. The decision to move to Buoyant Enterprise for Linkerd (BEL) came down to two factors. 

1. Proven simplicity over the competition: The team knew how easy Linkerd is to use and knew they could onboard quickly and efficiently. Employees who had used Isito at previous companies had watched teams spend enormous resources just to maintain it.
2. Enterprise-grade support for critical infrastructure: As a payments company, PayNearMe requires vendor-backed support for any technology critical to its transaction processing pipeline. Additionally, they knew that both Linkerd, the open source project, and the enterprise distribution of Linkerd are built and maintained by the same team, eliminating additional issues down the line.

"The thing I like most about Linkerd is that it doesn't become your cluster, just a part of your cluster. Istio takes over, and you have to use all the Istio things once you install it, and you basically can't remove it. Linkerd is more of the Kubernetes model, which is what really attracted me to it." — Jordan Gregory, Cloud Architect, PayNearMe

PayNearMe also evaluated Cilium and Kuma, but Linkerd remained the clear winner on reliability and simplicity. As Jordan put it: “Linkerd just works. It always has just worked. And continues to just work."

‍

Implementation: Production-ready in two weeks

Once the decision to move forward with BEL was made, speed was a top priority. PayNearMe's SDLC has a standard two-week lead time to reach production. Configuring Linkerd was no different, and that's exactly how long it took to go from initial access to a fully deployed, production-ready Linkerd environment.

The implementation was led by Jordan and his lead engineer, working primarily from Buoyant's documentation. By the time the onboarding call with the Buoyant team took place, PayNearMe was already live in production.

The only post-deployment correction needed was adjusting certificate duration, which was a simple fix. Everything else was done right the first time. For comparison, Jordan recalled a prior company that deployed Istio and had to hire three full-time engineers just to support it. PayNearMe achieved a production-grade service mesh deployment across environments with their existing team in two weeks. Since deploying Buoyant Enterprise for Linkerd, PayNearMe has eliminated the cascading gRPC failures that caused their high-impact outage.

‍

Buoyant Cloud: Observability on a complex network

Network observability had previously been a challenge at PayNearMe. After implementing Buoyant Cloud, they were able to add a layer of service mesh-specific visibility that changed how the team operated day-to-day.

Today, Linkerd metrics are integrated into centralized observability dashboards used across engineering teams. Engineers can see real-time real-time service health and traffic insights that were previously invisible.

"I didn't even realize what we were missing until we had it. And now I can never not have this. The alerting, the single pane of glass to see everything, and it gives us so many entry points to troubleshoot that we didn't have before." — Jordan Gregory, Cloud Architect, PayNearMe

Buoyant Cloud has become the early warning system for every production release. When a new deployment causes an unexpected drop in success rate, the team sees it immediately and can roll back quickly rather than spending hours diagnosing the root cause. What previously took four hours to resolve now takes roughly 30 minutes.

The team has also put Linkerd's circuit breaker capability to use in production. When a service was intermittently failing, they used resilience features to isolate unstable services while maintaining overall system availability

‍

What's Next

PayNearMe is now moving beyond using BEL as a targeted fix and into leveraging it as a strategic platform capability. Immediate priorities include continuing to enhance availability and resilience across regions, deeper circuit breaking and retry policies across their growing microservices portfolio, and continued Buoyant Cloud integration into developer-facing dashboards.

‍

Interested in what Buoyant Enterprise for Linkerd can do for your team? Contact us to learn more.

‍