Multi-Level Caching for Multimodal LLMs

Key-value caching and its prefix-caching extension eliminate redundant attention computation within a single request, and modern serving engines like vLLM build on them to keep decoding fast. But both are fundamentally within-request: once a request finishes, its cached state is reclaimed. In multimodal deployments that's a real gap, because a dominant chunk of inference cost sits before decoding even starts. Every image has to pass through a vision encoder and get projected into the language model's embedding space, and that computation is re-run from scratch for every request, even when two requests reference the same image or near-identical frames from the same scene.

Exact-match caching can catch some of this, but it's brittle: visually identical images can still produce different embeddings from resolution changes, cropping, or plain floating-point noise, and users rarely repeat prompts verbatim. Multi-turn assistants and streaming camera use cases make the redundancy worse, not better, since consecutive frames from a slowly changing scene, or a person asking several questions about the same picture, are common. We wanted a caching system built around similarity rather than exact matches, one that could recognize this kind of redundancy across both the visual and textual halves of a request.

MLC organizes lookup into three tiers that escalate from strict to approximate matching, running cheapest-first so expensive similarity search only kicks in when it's needed. L0.5 (exact input match) normalizes and hashes incoming text, catching verbatim repeats at near-zero cost. If that misses, L1 (text semantics) embeds the prompt with a sentence transformer and checks cosine similarity against cached prompts, catching paraphrases exact matching would miss. If that also misses, L2 (multimodal semantics) falls back to a CLIP-style embedding over both the prompt and the image, so the cache can recognize a repeated scene even when the wording changes, or correctly reject a match when the wording is similar but the image isn't.

Lookup is deliberately decoupled from what happens on a hit. We implemented response reuse, which returns a stored response outright and skips the model's forward pass entirely, backend-agnostic and best suited to high-confidence hits. The alternative is KV chunk reuse, injecting cached key/value tensors into the decoder so it can resume mid-sequence rather than substituting a whole output. We explored wiring this into vLLM, but its execution path doesn't expose a stable interface for supplying externally-cached KV state, and KV state is tightly coupled to vLLM's own sequence metadata and memory-block allocation. Enabling it looks like a non-trivial change to vLLM itself, so our evaluation is built entirely on response reuse.

Before testing the full hierarchy, we wanted to see response caching in its simplest form: a fine-tuned Vision Transformer classifying rice images into 8 classes, where a cached response is just a class label. We embedded each image with a lightweight encoder (ResNet-18, MobileNetV2, or EfficientNet-B0), and on a new query returned the nearest cached label instead of re-running the ViT whenever cosine similarity cleared a threshold τ. We swept τ from 0.875 to 0.95 across all three encoders on 816 rice images, measuring accuracy, hit rate, and end-to-end runtime against a no-cache baseline (88.97% accuracy).

The results exposed a sharp accuracy–hit-rate tradeoff. Lower thresholds pushed hit rates as high as 78% (ResNet-18, τ=0.875) but let in frequent false hits, dropping accuracy to 74.9%. Raising τ recovered accuracy but hit rates collapsed, down to 5–18% at τ=0.95, at which point the overhead of computing embeddings and running similarity search outweighed the compute saved by skipping the ViT forward pass. Only two configurations beat the baseline on latency at all, ResNet-18 at τ=0.875 and τ=0.90 (1.07× and 1.08× speedup), and both came with a meaningful accuracy hit. Because the unit of reuse here is the final label, any approximate match can directly return a wrong answer, which is what motivated moving to intermediate-computation reuse and a more realistic workload in Experiment 2.

Experiment 2 evaluates the full MLC hierarchy on Qwen3-VL and InternVL3.5 against the GQA dataset, 1,024 sampled visual question-answering pairs that need fine-grained scene understanding, closer to how a real multimodal assistant gets used. Exact-text caching (L0.5) was safe but limited, an 7.8% hit rate with accuracy roughly preserved. Semantic (L1) and embedding-based (L2) caching showed the same pattern across both models: lower similarity thresholds drove hit rates as high as 93% and cut latency dramatically, but accuracy fell off just as sharply, since aggressive reuse means more false hits. Picking a threshold here is a real design decision, not just a tuning knob.

Fixing all thresholds at 0.8 as a balanced tradeoff, MLC cut inference latency across every model we tested, with speedups from 1.75× (Qwen3-VL-2B) up to 2.48× (Qwen3-VL-8B). Larger models benefited more: bigger models are more robust to approximate reuse, so they retained more of the accuracy while still capturing most of the latency win. That's a useful result for deployment: MLC's value grows exactly where inference is most expensive, and for applications with many similar requests it can meaningfully cut latency, provided the accuracy tradeoff at the chosen threshold is acceptable for the use case.

The write-up below covers everything in more depth: the full experimental setup, model comparison tables, and future work on KV-chunk caching, additional modalities, and identifying real-world scenarios where the accuracy–latency tradeoff is worth it. If it doesn't load, you can also open it directly.