vLLM

An inference engine with PagedAttention for efficient serving of LLMs at high throughput.

Related terms

LatencyThroughputKV Cache (Key-Value Cache)
View on glossary index