LinkedIn
Use inference efficiency to improve GPU utilization
Kinference sends each request to the endpoint that will respond fastest, using live KV cache data on each GPU, queue status, and other real-time signals.
Predicts the fastest pod by reading each pod’s live KV cache and queue
Routes to the GPU with the prompt already cached as policy
Overall efficiency across the pool is improved and tracked against SLOs
Inference infrastructure made more efficient
Cache-aware routing for faster response times
Faster first tokens
Requests land where the prompt is already cached, so responses start sooner.
Performance-focused SLOs
Specify what the application needs for optimal performance. The platform handles the routing.
Predictable latency under load
Responses stay at target levels even as inference shifts and GPU load increases.
More useful work from every GPU
Predicted TTFT for every prompt
Selects the pod that will deliver the fastest first token, whether its cache is warm or cold.
Cache hit and queue metrics
Inference metrics and dashboards for cache hits and queue depth, with performance per client.
Scale on inference signals
Safely add new pods to handle fresh prompts so cached work isn’t recomputed
Kinference is a policy-driven inference control plane for self-hosted GPUs
Faster first tokens from cache and queue awareness
The problem. Routing that can’t see each pod’s cache and queue either recomputes prompts needlessly or overloads busy pods, slowing first tokens.
The solution. Kinference tracks the queue and KV cache on every vLLM pod and predicts the time to first token for each request on each pod. Every request goes to the pod that will serve it fastest. When the pool has spare capacity, the pod holding its context usually serves it. As load rises and those pods get busy enough to risk the SLO, requests are redirected to less busy GPUs that compute the prompt from scratch but still respond sooner. The result is faster first tokens across the whole pool and better GPU efficiency at high utilization.
Define inference policies that keep every GPU busy with more useful work
Frequently asked questions
Do our apps need to change?
No. Apps keep calling the same OpenAI-compatible endpoint, and Kinference routes each request to the right vLLM pod.,
How does Kinference know where a prompt is cached?
It tracks KV cache state on every vLLM pod and sends each request to the replica that already holds its prompt.
Doesn’t capacity planning already solve this?
Capacity planning tells you how many users a GPU can serve at your SLO, and you can benchmark that with or without Kinference. Kinference makes sure you actually get that capacity. Without cache-aware routing, requests spread across pods miss the cache, and each GPU serves fewer users than you planned.
What happens to response times when we add a replica?
A new vLLM pod still needs time to load its model, and Kinference doesn’t change that. What it changes is what happens next: instead of sending the new pod a share of all traffic, it routes each request by predicted time to first token. Prompts with cached context stay on the pods that hold it, and the new pod picks up requests it can serve quickly, so response times don’t drop as you scale up. If local pods are saturated while the new one starts, a burst-to-cloud policy can send the overflow to a hosted provider.
What is inference efficiency?
SLOs are the targets your clients care about, such as time to first token. GPU utilization is about the hardware: how busy your GPUs are. Inference efficiency connects the two: how much useful output your GPUs deliver while meeting those SLOs. Kinference’s cache-aware routing is one way teams control it. More cache hits mean less recomputation, so first tokens arrive faster from the same GPUs.
Is Kinference open source?
Yes. Kinference is open source and runs in your own Kubernetes cluster, so you can read the code, see exactly how routing and policy decisions are made, and run it alongside the tools you already use. It’s built by Buoyant, the creators of Linkerd, and access to the code is available on GitHub. Kinference on GitHub.