Model-Aware Routing with the Kubernetes Gateway API Inference Extension

Tier: Advanced · Workflow area: Platform-for-AI: serving & gateways

The Gateway API Inference Extension teaches Kubernetes ingress about all of this — routing to model-server pools via an Endpoint Picker that reads live serving metrics. This playbook stands it up in front of the vLLM service from pb-43.

When to use

• •
  You run (or will run) multiple vLLM/Triton replicas or model variants in-cluster and round-robin is producing wildly uneven latency.
• •
  You need K8s-native traffic policy for models: priority between workloads, safe rollout of a new model version, LoRA adapter routing.
• •
  You are evaluating kgateway, Envoy AI Gateway, GKE Inference Gateway, or Istio support for inference routing.

Skip this when you have one replica of one model — a plain Service suffices until you scale out — or when your traffic is mostly to external providers (that's pb-45's LiteLLM layer, which can sit behind or beside this).

Prerequisites

• •
  The pb-43 vLLM Deployment (ideally scaled to 2–3 replicas) on a cluster where you can install CRDs.
• •
  A Gateway API implementation that supports the Inference Extension — kgateway, Envoy AI Gateway/Envoy Gateway, or GKE Gateway are the reference choices; check current conformance before picking.
• •
  Comfort with Gateway API basics: GatewayClass, Gateway, HTTPRoute.

The workflow

1. 1.

   Install the extension CRDs and your gateway. Follow your chosen implementation's install (Helm for kgateway/Envoy AI Gateway). Confirm the CRDs: kubectl get crd | grep inference — the v1 API centers on InferencePool (early alpha releases also had an InferenceModel CRD; it did not survive to v1, so treat older blog YAML with suspicion and match your installed bundle's API version).

2. 2.

   Define the InferencePool. An InferencePool selects your model-server pods and names the Endpoint Picker (EPP) that will choose among them:

   Copy

   apiVersion: inference.networking.k8s.io/v1
   kind: InferencePool
   metadata: {name: qwen-pool}
   spec:
     targetPorts:
     - number: 8000
     selector:
       matchLabels:
         app: vllm-qwen
     endpointPickerRef:
       name: qwen-pool-epp
   

   The EPP (deployed alongside, per your implementation's chart) scrapes vLLM metrics — queue depth, KV-cache utilization, loaded LoRA adapters — and picks the endpoint per request instead of blind round-robin. Field names vary slightly across bundle versions; validate with kubectl explain inferencepool.spec.

3. 3.

   Route to the pool, not a Service. The HTTPRoute's backend is the pool:

   Copy

   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata: {name: llm-route}
   spec:
     parentRefs: [{name: ai-gateway}]
     rules:
     - matches: [{path: {type: PathPrefix, value: /v1}}]
       backendRefs:
       - group: inference.networking.k8s.io
         kind: InferencePool
         name: qwen-pool
   

4. 4.

   Prove it's smarter than round-robin. Send a skewed load: a few long-context requests (big prefill) mixed with many short ones. Watch per-pod vllm:num_requests_waiting — with the EPP, queue depth should stay roughly even and no pod should starve behind a long request. Then repeat against a plain Service and compare P95 TTFT. Keep the numbers; that delta is the business case.

5. 5.

   Add policy in layers. Once basic routing works: use your implementation's support for priority/criticality (protect the interactive workload when batch traffic spikes), model-version traffic splitting via weighted rules for canary model rollout (pairs with pb-52's eval gates), and body-based model-name routing if one gateway fronts several pools.

6. 6.

   Wire it into the platform path. North-south: clients → LiteLLM gateway (auth, keys, budgets — pb-45) → inference gateway (scheduling-aware routing) → vLLM pools. Each layer does one job; don't make one layer do both.

Tool variants

• kgateway / Envoy AI Gateway: OSS reference implementations; Envoy AI Gateway also brings provider-egress features if you want one Envoy for both directions.
• GKE Inference Gateway: managed flavor on GKE with the same InferencePool model — least install effort on Google Cloud.
• Istio: supports Inference Extension routing in recent versions if you're already a mesh shop.
• llm-d: couples this routing with disaggregated prefill/decode serving — where large-scale deployments are heading; know the name.

Verification

• •
  kubectl get inferencepool shows the pool programmed; gateway status conditions are Accepted/Programmed.
• •
  Skewed-load test shows measurably better P95 TTFT and more even per-pod queue depth than a plain Service (keep both runs' numbers).
• •
  Killing one pool pod mid-load: EPP stops routing to it before K8s endpoints catch up, or at worst within probe latency — no request storm of 5xx.
• •
  Priority policy test: saturate with low-priority traffic, confirm high-priority P95 holds.

Pitfalls

• Stale YAML from the alpha era. Pre-v1 examples using InferenceModel or old group names (inference.networking.x-k8s.io) won't apply cleanly. Trust kubectl explain over blog posts — including AI-assistant output trained on them.
• EPP without metrics. If the picker can't scrape your model servers, it degrades to trivial picking and you'll conclude the extension "does nothing." Verify EPP logs show live metrics per endpoint.
• Terminating long streams on rollout. LLM responses stream for minutes; set generous drain/termination grace on gateway and vLLM pods or every deploy kills in-flight generations.
• Double-balancing. Putting the pool behind another L7 balancer that re-picks endpoints defeats the EPP. The pool must be the last hop before pods.

Restricted-environment variant

• •
  All components are OSS and run fully in-cluster — this layer adds no new data-egress surface, which is exactly why regulated and services-company deployments should prefer in-cluster routing over SaaS routers for client traffic.
• •
  Air-gapped installs: mirror the gateway, EPP, and CRD images into your registry; charts pin digests.
• •
  Gateway access logs contain prompt paths/sizes and can contain headers — apply the same log-redaction policy as the model servers.

Success criteria

• •
  Traffic reaches model pods through an InferencePool with the EPP demonstrably influencing placement (documented A/B numbers vs plain Service).
• •
  A model-version canary shifted 10% of traffic via route weights without client changes.
• •
  Priority policy protected interactive latency during a synthetic batch flood.
• •
  You can explain to a colleague, in one whiteboard pass, why LLM traffic breaks classic load balancing and what the EPP reads to fix it.