LinkedIn
Let the platform handle performance
Kinference sits between the apps and the models, applying the platform team's policies to every request. It decides which model or provider serves each one, based on predicted response time and live GPU load, so apps get the performance they asked for.
App teams self serve their model selection, response times and data needs
Platform teams encode those choices as routing rules in Kubernetes
Each request is routed to the right model and measured against its SLO
SLOs built around how AI apps run in production
Self-serve your application's service levels without writing them into the code
No code changes
Point your client at the OpenAI-compatible endpoint and the platform handles model selection, fallback, and rate limits as policies.
QoS for every request
Each request can carry the QoS your app needs, and the platform decides how to serve it.
Routing and SLO tracking
Each request goes to the pod predicted to deliver the fastest first token, and every response is tracked against your app’s SLO.
Policy-driven SLOs, managed from the prompt to the pod
Inference-native policy language
Use declarative, inference-native language for writing policies by tenant, model, prompt, and GPU state.
Intelligent routing
Kinference picks the endpoint with the lowest predicted time to first token, using each replica's live queue and cache state.
Inference SLO dashbards
Monitor every app's inference SLOs in Grafana, in the context of queue depth and KV cache state.
Kinference is a policy-driven inference control plane for self-hosted GPUs
The fastest endpoint for every request, and an SLO for every app
The problem.Self-hosted models in production now belong to the platform team, but setting SLOs for each app and model is complex, and vLLM's metrics don't tell you which app a request came from.
The platform team declares an SLO for every combination of app, team, model, and service tier. Kinference routes each request to the vLLM replica with the fastest predicted response, using live queue and KV cache state, and measures response start time, time between tokens, and total completion time on every request. Teams report SLO compliance for inference just as they do for any other production service.
Policies that set how every app, model, and tenant is served
Frequently asked questions
What does Kinference measure for each request?
Time to first token (TTFT), inter-token latency (ITL), and end-to-end completion time (E2E), reported at p95 for every app, team, model, and service tier.
Can different teams have different SLOs for the same model?
Yes. SLOs are set per combination of app, team, model, and service tier, so a premium chat agent and a batch job can share a model with different targets.
How does routing help meet an SLO?
Kinference sends each request to the vLLM replica predicted to meet its target, using live queue depth and KV cache state, instead of spreading requests evenly.
Where do we see SLO compliance?
In SLO dashboards for every app, model, and tenant, alongside the dashboards your team already uses for other production services.
Do developers need to change their apps?
No. Kinference sits in front of the vLLM pods you already run, so apps keep calling the same endpoint.
Is Kinference open source?
Yes. Kinference is open source and runs in your own Kubernetes cluster, so you can read the code, see exactly how routing and policy decisions are made, and run it alongside the tools you already use. It’s built by Buoyant, the creators of Linkerd, and access to the code is available on GitHub. Kinference on GitHub.