LinkedIn
If the prompt declares context, the policy can decide the model
An inference control plane reads each prompt’s context and uses it to find the model that will serve it best, based on predicted time to first token.
Prompts classified for complexity means the policy selects the model
Each model gets a predicted time to first token based on cache and queue
The prompt is served by whichever model is predicted to respond first
The platform decides the model best placed to serve a prompt
Send the prompt with its context and let the platform pick the model that serves it
Intelligent context-aware routing
Full model flexibility using an inference-serving platform to route prompts based on its context.
Classification plus cache state
The combination of live cache state and semantic classification lets the platform pick the right model for every prompt.
Per tenant performance metrics
Know which model served each application, how fast and at what cost.
Inference using context to serve every request to the right model and the fastest latency
Declarative policy
Policy-driven inference gives the platform team control over how requests are handled.
Tagged by complexity and any other context
Use semantic classifiers tag each prompt so the policy can route it to the model that will serve it fastest.
.svg){kind=icon}
Kubernetes-native control
Inference policy, written as Kubernetes resources the platform team owns.
Kinference is a policy-driven inference control plane for self-hosted GPUs
Low complexity prompts served by cheaper models
The problem. Without a context-based routing policy, each prompt goes to the model coded in the application, typically the larger more expensive model. With no signal about complexity,simpler prompts compete for the same replicas as complex ones, which lowers GPU efficiency and slows response times.
When an app asks for glm-5.3 and the classifier tags the prompt complexity-low, the policy automatically adds qwen-3.6-27b as a second option. Kinference sends the request to whichever of the 2 has the lowest predicted TTFT, based on each model's KV cache and queue. Prompts that aren't tagged low complexity go to glm-5.3 as requested.
Inference based on knowing what each prompt needs to be served fastest.
Frequently asked questions
Do app teams need to change their code?
No. Apps keep sending OpenAI-compatible requests for glm-5.3 to the endpoint you publish. Which model serves a simple prompt is set in platform policy.
How does semantic classification work with KV cache?
Classification decides which models a prompt can use. KV cache state helps decide which one answers first. Kinference predicts time to first token for each candidate from its cache overlap and queue length. A smaller model usually wins on a short prompt, unless a larger model already holds most of the prompt in cache and can answer sooner.
How do apps state a model preference if it isn’t hard-coded?
Apps still name a model in each request, or a stable model alias the platform maps to a real model. That name is the preference. Platform policy then decides how the request is served, including whether a smaller model can take a simple prompt when it’s predicted to answer sooner.
Can I use models other than glm-5.3 and qwen-3.6-27b?
Yes. Kinference is model agnostic and supports any model vLLM can run, so the names in the policy are whatever you deploy.
Why use semantic classification?
Without it, the platform knows which model a request names but nothing about the prompt itself. Semantic classification tags each prompt, for example by complexity, which gives policy the context to act on. For a prompt tagged low complexity, the policy can widen the candidate pool to include a smaller model, and Kinference sends the prompt there only when that model is predicted to return the first token soonest. Simple prompts stop taking large-model replicas that complex prompts need, so each pool is used for the work it’s sized for.
Is Kinference open source?
Yes. Kinference is open source and runs in your own Kubernetes cluster, so you can read the code, see exactly how routing and policy decisions are made, and run it alongside the tools you already use. It’s built by Buoyant, the creators of Linkerd, and access to the code is available on GitHub. Kinference on GitHub.