KV cache and snapshots

A runner keeps the KV cache of the conversation it served last. Spark adds snapshots, so a conversation does not lose its work when another one takes the cache.

What is reused

• The live cache. A request whose prompt starts with the tokens already in the cache, such as the next turn of the same chat, prefills only what is new. The playground and the request log show how many prompt tokens were cached.
• Snapshots. When another conversation pushes a conversation out of the live cache, the runner saves it as a snapshot, a memory-mapped file. When the first conversation returns, the longest matching prefix is restored.
• Anthropic breakpoints. A cache_control block in an Anthropic request writes the shared prefix as a snapshot while it is prefilled. A top-level cache_control marks everything up to the last message. Usage reports cache reads and writes in each API's terms.

OpenAI requests accept prompt_cache_key, and report prompt_tokens_details.cached_tokens.

Snapshots belong to Spark's own transformer (llama, mistral, qwen2, qwen3 and granite). TensorSharp families keep their prefix cache in memory inside the runner and have no snapshots.

Limits and cleanup

Snapshots are limited, least recently used first, by a budget, an entry count and the disk's free space.

Setting	Default	Meaning
SPARK_KV_CACHE_FOLDER	$SPARK_STORAGE/kvcache	One folder per model.
SPARK_KV_CACHE_MB	auto	Per model. Auto is a quarter of the memory, at most half the free disk. -1 turns snapshots off.
SPARK_KV_CACHE_ENTRIES	64	Snapshots kept per model.
SPARK_KV_CACHE_MIN_TOKENS	64	Shorter conversations are not snapshotted.
SPARK_KV_CACHE_MIN_FREE_MB	auto	Free space kept on the snapshots' disk: 5% of the disk, between 1 GB and 10 GB. Below it, the least recently used snapshots of every model are deleted and runners save none.
SPARK_KV_CACHE_MAX_AGE_HOURS	168	A snapshot unused this long is deleted. 0 keeps it while there is room.
SPARK_KV_CACHE_TOTAL_MB	0	The most all models' snapshots together may take. 0 is no limit beyond the disk margin.
SPARK_KV_CACHE_SWEEP_SECONDS	300	How often the server checks the disk, the age, the total and the folders of deleted models.

A runner trims its own folder when it saves. The server sweeps for the rest: folders of deleted models, age, the disk margin and the total. Deleting a model deletes its folder. Snapshots survive a runner restart. If you change SPARK_KV_CACHE_FOLDER, Spark does not clean up the old folder.

To clear one model's snapshots, use Clear KV snapshots on its page.

Not supported

Concurrent requests to one model queue behind each other. There is no continuous batching, no context shifting when a conversation outgrows its context, and no snapshots in F16.