LowRAM

Low-rank keys and amortised values for composable KV cache compression at test time

Why store what you can predict? LowRAM reconstructs values from keys and keeps only what it can't predict.

Conventional KV cacheevery key and value

LowRAMreconstructed when needed

Figure 1. A conventional cache stores every key and value. LowRAM stores low-rank keys, a value predictor ƒ_θ fitted to the context, and sparse corrections for the values it predicts worst.

**Table 1.** Selected results from the paper. Settings differ between rows.
4×	value-cache compression, keeping 99.3% of the uncompressed LongBench score	Llama-3.1-8B, values only Table 5
26.9×	combined with eviction and quantisation, 94.5% recovered score on long contexts	Mistral-7B, LongBench >10K Table 11
2	adaptation steps with meta-initialisation: faster prefill, with an accuracy tradeoff	Meta-initialised LowRAM Table 2

1Can it keep the details?

Three facts are tucked inside a 64K-token document, and the model is asked for all three. Pick a LowRAM configuration and a batch size to compare the recorded answers, and the memory they take, with the uncompressed model.

LowRAM configuration Batch size

The document

BeginningEnd

The question

Loading…

Uncompressed—

Waiting for the recorded result.

LowRAM—

Waiting for the recorded result.

Physical prefill cache—

Baseline

—

LowRAM

—

Performance comparison for the selected configuration
Measured metric	Baseline	LowRAM

Figure 2. Recorded model outputs. Answers are scored against the document’s reference facts. Cache sizes are mean bytes per batch, including the stored predictor and cache metadata, and exclude model weights.

2Store the pattern, keep the exceptions

Keys and values come from the same hidden states. LowRAM uses that relationship to store less, in four steps. Select a step to see it.

K ≈ A · B

Figure 3. Shared structure, smaller keys. A shared factor captures patterns across a group of layers. Each layer keeps its own smaller reconstruction factor.

3Results

Quality, speed and capacity tell different parts of the story, so they are shown separately. Hover over or focus a point to read its value.

Select a point to inspect its result.

Figure 4. How much performance is retained? Mistral-7B on LongBench contexts longer than 10K tokens (paper Table 11). Recovered score = method score / uncompressed score × 100.

Select a measured point to inspect its settings.

Figure 5. When does compression help throughput? Llama-3.1-8B, 262K tokens, 4× target. Five repeats, two warmups, 128 decode steps. Points marked OOM ran out of memory.

4Try it yourself

Run the readable example, measure the tradeoffs, and inspect the recorded measurements. Commands run from the repository root.

# Activate the environment and install required packages
# Generate an answer with a compressed cache
python -m examples.lowram_demo

# Compare memory, throughput, latency and accuracy
python -m benchmarks.compare \
  --methods baseline fast lowram \
  --meta-weights /path/to/meta_mlps.pt \
  --contexts 131072 --batch-sizes 1 2 4 6 8 12 \
  --ratios 4 --samples 24 --repeats 2 --warmup 1 \
  --decode-tokens 48 \
  --output results/demo/comparison.json

References

Chang et al. xKV: Cross-Layer SVD for KV-Cache Compression. arXiv:2503.18893, 2025.
Sun et al. ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference. ICML 2025.