#kv cache
4 articles
Latest
Research
Five Papers Attack KV Cache Bottleneck With Quantization Methods
New approaches use statistical inference, rate-distortion theory, and learned eviction to reduce memory cost of long-context LLM inference.
2 days ago9 min read
Tools
Gemma 4 Finally Works in llama.cpp After Critical Fixes
KV cache and tokenizer bugs squashed. Local inference actually viable now.
about 1 month ago1 min read
Tools
Alibaba MNN Adds TurboQuant Support for Local LLM Inference
The framework now supports aggressive KV-cache compression, making on-device models faster to run.
about 2 months ago1 min read
Research
TurboQuant Compresses LLM KV Cache to 3-4 Bits Without Accuracy Loss
New quantization algorithm enables longer context windows and 3.2× memory savings for local inference.
about 2 months ago7 min read