KV Offloading

Moving key/value cache tensors to CPU or disk tiers to serve longer contexts under memory limits.

Related terms

KV Cache (Key-Value Cache)LatencyThroughput
View on glossary index