EpochLex entry

KV Cache Quantization

kay-vee kash kwon-tuh-ZAY-shun

The reduction of numerical precision used to store a transformer’s key-value attention cache, such as using FP8, to lower cache memory consumption and potentially improve throughput or support longer contexts.