vLLM vs SGLang vs LMDeploy for self-hosted LLM serving

Comparison scope and release snapshot

This comparison covers self-hosted inference and serving. All three projects publish Apache-2.0 code and document an OpenAI-compatible server. Treat API compatibility as a starting point: verify the endpoints, streaming responses and request fields your application actually uses.

PyPI metadata checked November 7, 2026 lists vLLM 0.33.0, released November 2; SGLang 0.5.23, released October 30; and LMDeploy 0.18.0, released September 28. The feature comparison uses current official documentation, without testing every feature in those packages. Pin a package version or container digest and use matching documentation before deployment.

Hardware and quantization entries are examples of documented paths, not a promise that every listed feature works on every device. Both vLLM and SGLang publish quantization compatibility tables; LMDeploy publishes separate model and data-type tables for its engines. Start with the intersection of your model architecture, accelerator, precision and required features.

Check the complete serving configuration

Replace MODEL_PATH in the table commands with a supported Hugging Face model ID or local model directory. vLLM’s quickstart uses localhost:8000/v1, SGLang’s quickstart sends requests to localhost:30000/v1/chat/completions, and LMDeploy’s server guide uses port 23333. These are starting commands; use the model and hardware guide for additional arguments.

LMDeploy’s LoRA guide limits adapters to its PyTorch backend, while its structured-output guide supports both PyTorch and TurboMind. Its speculative-decoding guide labels the feature experimental and shows PyTorch examples. Its NVIDIA installation path also needs careful version selection: the README says default PyPI wheels use CUDA 12.8 from v0.13.0, whereas the current SGLang NVIDIA installation guide requires CUDA 13. Record the actual runtime and driver combination used in your evaluation.

How to choose — SOTA’s assessment

SOTA’s assessment is based on the official documentation, with no hands-on tests or benchmark ranking. Shortlist vLLM when you need to evaluate several distributed-inference strategies or a hardware-plugin path: its documented options include tensor, pipeline, data, expert and context parallelism. Confirm the chosen plugin’s model and feature coverage before investing in deployment.

Shortlist SGLang when shared-prefix workloads and explicit prefill/decode separation are central to your design: its docs cover RadixAttention prefix caching and disaggregation with Mooncake or NIXL. Use those as reasons to test your own request mix, without assuming a latency or throughput advantage.

Shortlist LMDeploy when you want quantization tools alongside serving and can make an explicit choice between TurboMind and PyTorch. Its docs cover AWQ/GPTQ inference, SmoothQuant and KV-cache quantization, with backend-specific model support. Prefer whichever engine passes your model, correctness and operational checks on the target hardware.

A reproducible evaluation plan

This is SOTA’s proposed test protocol; we have not run this benchmark. Select one model supported by all three engines, pin the same weight revision, tokenizer and chat template, and begin with the same unquantized precision. Use the same physical host, accelerator count, memory budget, driver, power settings and client placement for each run. Pin engine versions or container digests and save every launch command, dependency version and configuration flag.

Create a fixed JSONL prompt corpus, publish its checksum and use the same prompt order and seed for each engine. Include separate groups with 128, 1,024 and 4,096 input tokens, using the pinned tokenizer, and request at most 128 output tokens with identical sampling and stop settings. Record actual output lengths and early stops. Test concurrency 1, 8 and 32 using the same client harness; after 30 warm-up requests, run each case for five minutes and repeat it three times, rotating engine order.

Measure successful requests per second and output tokens per second separately. With streaming enabled, measure time to first token (TTFT) from client request submission to the first non-empty generated token; report p50 and p95 alongside end-to-end latency, errors and timeouts. Sample total accelerator memory throughout each run and report idle allocation and peak usage, including KV-cache and draft-model allocations. Preserve raw request timings, token counts, memory samples and logs so another team can reproduce the calculation.

Run distinct cold-cache and warm shared-prefix cases, recording cache settings and reset procedures. Establish a baseline with speculative decoding disabled, then evaluate supported optimizations one at a time. Test a common quantized checkpoint separately if all three support it on the selected hardware; include output-quality checks. Validate streaming, structured-output schemas, LoRA routing and multimodal requests as separate acceptance cases when your application needs them. Publish results only for the combinations actually tested.

What changes the decision

Revisit the choice when the model architecture, accelerator generation, quantization method, shared-prefix rate, prompt/output length mix or concurrency target changes. Re-run the same evaluation after an engine upgrade, a backend switch or a move from one node to several. Set acceptance thresholds for TTFT, throughput, memory, correctness and recovery behavior before comparing results.

A required feature combination should decide the shortlist before a speed comparison. For example, LMDeploy’s documented LoRA path requires PyTorch; vLLM’s hardware plugins need their own compatibility checks, and SGLang’s quantization table distinguishes NVIDIA, AMD and Ascend support. Treat missing documentation or a failed acceptance case as a reason to narrow the configuration or evaluate another engine, rather than infer support from a project-wide feature list.

Sources & method

Official documentation was checked on 2026-11-07. Selection advice is our assessment. Interfaces and requirements may change; check the linked source for the version you intend to use.

• vLLM Apache-2.0 licence
• SGLang Apache-2.0 licence
• LMDeploy Apache-2.0 licence
• vLLM PyPI release history, Python requirements
• SGLang PyPI release history and Python requirements
• LMDeploy PyPI release history
• vLLM quickstart, installation and OpenAI-compatible serving
• vLLM hardware platforms and external hardware plugins
• vLLM repository: serving features, parallelism and hardware-plugin examples
• vLLM quantization methods and hardware compatibility
• vLLM supported models and multimodal inputs
• SGLang quickstart and server launch command
• SGLang installation, Python and CUDA requirements
• SGLang repository: hardware platforms and TPU paths
• SGLang RadixAttention and prefix-caching overview
• SGLang quantization methods and platform compatibility
• SGLang speculative-decoding methods and constraints
• SGLang JSON schema, regex and EBNF output constraints
• SGLang multi-LoRA serving
• SGLang data and tensor parallelism
• SGLang prefill/decode disaggregation with Mooncake and NIXL
• SGLang multimodal language models
• LMDeploy repository: engines, prefix caching, DeepSeek disaggregation, wheel requirements
• LMDeploy installation and hardware build targets
• LMDeploy OpenAI-compatible server, CLI and tensor parallelism
• LMDeploy model and precision support by engine and platform
• LMDeploy TurboMind AWQ/GPTQ 4-bit inference
• LMDeploy SmoothQuant INT8 and FP8
• LMDeploy INT4/INT8 KV-cache quantization
• LMDeploy experimental speculative decoding with PyTorch examples
• LMDeploy structured output on PyTorch and TurboMind
• LMDeploy multi-adapter LoRA serving on PyTorch only
• LMDeploy PyTorch multi-node tensor parallelism through Ray
• vLLM 0.33.0 on PyPI

How SOTA reviews projects · Suggest a correction · Download the evaluation checklist