llm-d
Open-source, Kubernetes-native stack for distributed LLM inference that runs vLLM, SGLang and other engines across a cluster with prefix-cache-aware routing, prefill/decode disaggregation, wide expert parallelism and autoscaling.
About llm-d
Single-node inference engines do not by themselves handle cluster-wide routing, cache reuse and scaling for large production LLM deployments.
Who it’s for
- Platform teams serving large open-weight models on Kubernetes
- Teams running inference across GPUs, TPUs or other accelerators who want tested deployment recipes
When to consider it
Consider llm-d when you already run vLLM or SGLang and need to scale it across a Kubernetes cluster using the project's tested Well-Lit Path recipes.
Tradeoffs & limitations
- It requires a Kubernetes cluster with accelerators and is an infrastructure layer, not a hosted API or a single-machine runtime.
- Apache 2.0; a CNCF Sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA.
SOTA overview · Documentation-based assessment · Sources & review method
Updates
No updates shared yet.
Discussion
Newest firstAsk a question or share how you use llm-d.
Keep it helpful. Community rules
Loading discussion…
Sources & review method
Documentation-based assessment · Nov 3, 2026 · Prepared with AI assistance; not a hands-on benchmark.
Checked by SOTA · AI-assisted documentation review. Selection advice is our assessment; verify current requirements for your deployment.
Import history & original evidence
llm-d official website
Open-source, Kubernetes-native stack for distributed LLM inference that runs vLLM, SGLang and other engines across a cluster with prefix-cache-aware routing, prefill/decode disaggregation, wide expert parallelism and autoscaling.
Based on official pages and announcements checked on 2026-11-02. No hands-on test, performance benchmark or popularity ranking is claimed.