opea/vllm

Easy, fast, and cheap LLM serving for everyone

| https://docs.vllm.ai | https://blog.vllm.ai/ | https://arxiv.org/abs/2309.06180 | ***/X | https://discuss.vllm.ai | https://slack.vllm.ai |

🔥 We have built a vLLM website to help you get started with vLLM. Please visit https://vllm.ai to learn more. For events, please visit https://vllm.ai/events to join us.

About

vLLM is a fast and easy-to-use library for LLM inference and serving.

Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has grown into one of the most active open-source AI projects built and maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors.

vLLM is fast with:

• State-of-the-art serving throughput
• Efficient management of attention key and value memory with https://blog.vllm.ai/2023/06/20/vllm.html
• Continuous batching of incoming requests, chunked prefill, prefix caching
• Fast and flexible model execution with piecewise and full CUDA/HIP graphs
• Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, ***, /, ***, compressed-tensors, ModelOpt, TorchAO, and https://docs.vllm.ai/en/latest/features/quantization/index.html
• Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton
• Optimized GEMM/MoE kernels for various precisions using CUTLASS, TRTLLM-GEN, CuTeDSL
• Speculative decoding including n-gram, suffix, EAGLE, DFlash
• Automatic kernel generation and graph-level transformations using torch.compile
• Disaggregated prefill, decode, and encode

vLLM is flexible and easy to use with:

• Seamless integration with popular Hugging Face models
• High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more
• Tensor, pipeline, data, expert, and context parallelism for distributed inference
• Streaming outputs
• Generation of structured outputs using xgrammar or guidance
• Tool calling and reasoning parsers
• OpenAI-compatible API server, plus *** Messages API and gRPC support
• Efficient multi-*** support for dense and MoE layers
• Support for NVIDIA GPUs, AMD GPUs, and x86/ARM/PowerPC CPUs. Additionally, diverse hardware plugins such as Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, and more.

vLLM seamlessly supports 200+ model architectures on Hugging Face, including:

• Decoder-only LLMs (e.g., Llama, Qwen, ***)
• Mixture-of-Expert LLMs (e.g., ***, DeepSeek-V3, Qwen-MoE, ***)
• Hybrid attention and state-space models (e.g., Mamba, Qwen3.5)
• Multi-modal models (e.g., ***, Qwen-VL, Pixtral)
• Embedding and retrieval models (e.g., E5-***, GTE, ColBERT)
• Reward and classification models (e.g., Qwen-Math)

Find the full list of supported models https://docs.vllm.ai/en/latest/models/supported_models.html.

Getting Started

Install vLLM with uv (recommended) or pip:

bash
uv pip install vllm

Or https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source for development.

Visit our https://docs.vllm.ai/en/latest/ to learn more.

• https://docs.vllm.ai/en/latest/getting_started/installation.html
• https://docs.vllm.ai/en/latest/getting_started/quickstart.html
• https://docs.vllm.ai/en/latest/models/supported_models.html

Contributing

We welcome and value any contributions and collaborations. Please check out https://docs.vllm.ai/en/latest/contributing/index.html for how to get involved.

Citation

If you use vLLM for your research, please cite our https://arxiv.org/abs/2309.06180:

bibtex
@inproceedings{kwon2023efficient,
  title={Efficient Memory Management for Large Language Model Serving with PagedAttention},
  author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},
  booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},
  year={2023}
}

Contact Us

• For technical questions and feature requests, please use GitHub https://github.com/vllm-project/vllm/issues
• For discussing with fellow users, please use the https://discuss.vllm.ai
• For coordinating contributions and development, please use https://slack.vllm.ai
• For security disclosures, please use GitHub's https://github.com/vllm-project/vllm/security/advisories feature
• For collaborations and partnerships, please contact us at ***

Media Kit

• If you wish to use vLLM's logo, please refer to https://github.com/vllm-project/media-kit