vLLM
High-throughput, memory-efficient inference and serving engine for large language models, built for running LLMs in production at scale rather than on a single local machine.
Overview
vLLM is a library for fast LLM inference and serving, originally developed at UC Berkeley's Sky Computing Lab and now maintained by a large open-source community. Its core contribution is PagedAttention, a memory-management technique for the attention mechanism's key-value cache that lets it serve far more concurrent requests on the same GPU hardware than naive serving approaches, combined with continuous batching so incoming requests are processed efficiently as they arrive rather than waiting for a full batch.
It exposes an OpenAI-compatible API server, supports a wide range of open-weight models from Hugging Face, and includes quantization and speculative decoding options for further throughput and latency tuning.
vLLM is a fit for teams deploying LLMs as a production service that needs to handle many concurrent users efficiently - a different tier than Ollama (also in this catalog), which is built for running a model locally for a single user rather than high-throughput multi-request serving.
- Categories
- AI & Machine Learning
- Languages
- Python, CUDA
- License
- Apache-2.0
Related Projects
llama.cpp
MITHigh-performance C/C++ implementation for running LLM inference locally on consumer hardware, including CPUs, with minimal dependencies.
- AI & Machine Learning
SGLang
Apache-2.0Fast serving framework for large language models and vision-language models, with a structured generation language for complex LLM programs.
- AI & Machine Learning
KServe
Apache-2.0Kubernetes-native platform for serving machine learning models at scale, standardizing model deployment across frameworks.
- AI & Machine Learning
CAMEL
Apache-2.0Open-source framework for research and building systems of multiple communicating, role-playing AI agents that collaborate autonomously.
- AI & Machine Learning
Spotted an error? Suggest an edit on GitHub.