LinkedIn
Peak demand shouldn't mean failed SLOs
The platform can decide what happens when GPU pools are saturated and send requests to a hosted inference provider.
GPU pools saturate during peak demand
Policy keeps sensitive requests on your GPUs
Hosted inference providers take the overflow and maintain SLOs
Serve peak demand without adding complexity
Keep requests answered at peak load, without changing the endpoint your app calls
Fewer timeouts at peak
Requests overflow to a provider instead of waiting in a pending queue.
One endpoint
Your app keeps calling the same OpenAI-compatible endpoint.
Inference handled by the platform team
The platform team sets the policy so your app aways meets its SLO.
Decide what happens when the GPU pool saturates using inference-native policies you control.
Policy-driven inference
Inference serving with declarative routing and overflow rules, ownend by the platform team.
Overflow that protects your SLOs
Hosted providers take the overflow if the local pool can't meet the latency SLO. Sensitive traffic stays local.
Hosted providers under your control
Requests route to the endpoint that will serve it fastest, under policy conditions managed by the platform.
Kinference is a policy-driven inference control plane for self-hosted GPUs
Overflow to hosted inference by policy at peak load
The problem. At peak load, requests are sent to pods without knowing which one can respond in time. GPUs get saturated and requests risk missing their latency targets. Sending all traffic to a hosted provider isn’t a fix, because you lose control over response times and what data is allowed to leave the cluster.
The solution. Kinference predicts how quickly each pod on the local pool can respond and sends the request to the fastest one. Pods that are too busy to meet the latency target are bypassed. Only when no local pod can respond in time does the request overflow to a hosted provider, and only if your policy allows that data to leave the cluster.
Serve every request you can locally. Control when and what prompts can overflow.
Frequently asked questions
Do app teams need to change their code?
No. Apps keep calling the same OpenAI-compatible endpoint. Whether a request is served on your GPUs or by a provider is decided by policy, not by the app.
When does a request overflow to the cloud?
Only when the policy has excluded every local pod. In this example, that’s any pod averaging more than 2 queued prompts over the past 30 seconds. Your platform team sets the threshold.
Which providers can requests overflow to?
Any off-cluster provider you configure, such as Bedrock or Databricks. The policy tries them in the order you list.
Do requests come back to our GPUs?
Yes. Every request is routed on its own, so as soon as a local pod drains below the threshold, new requests land on your GPUs again.
Can some requests stay off the cloud?
Yes. Policies can match on the tenant or on classifier tags, so requests flagged as containing PII, for example, never leave the cluster, even when local pods are busy.
Is Kinference open source?
Yes. Kinference is open source and runs in your own Kubernetes cluster, so you can read the code, see exactly how routing and policy decisions are made, and run it alongside the tools you already use. It’s built by Buoyant, the creators of Linkerd, and access to the code is available on GitHub. Kinference on GitHub.