Runners and placement

Spark never runs a model inside the server process. Each loaded model lives in its own spark-runner child process, so a model that crashes does not take the server down.

Loading and unloading

• On demand. The first request for a model starts a runner for it and loads the model. Requests that arrive meanwhile wait for the load.
• Idle unload. A model unused for SPARK_IDLE_UNLOAD_MINUTES (30) is unloaded and its process stops. 0 never unloads. A model with Unloaded when idle turned off in its settings stays loaded.
• By hand. Load and Unload on the model page, or on Runners.
• Limit. SPARK_MAX_LOADED_MODELS caps the number of loaded models. 0 lets memory decide. At the cap, the least recently used idle model is unloaded to make room.
• One request at a time. A runner answers one request at a time. Requests to the same model queue in arrival order.

A runner is tied to one model and one device. It never loads another model.

Placement

Before it starts a runner, Spark estimates what the model needs from its GGUF header: the weights, the KV cache for the context, the work buffers and the projector. It then puts the model where it fits.

• Idle runners are stopped to make room, least recently used first. A model set to stay loaded goes last.
• Host memory keeps SPARK_MEMORY_RESERVE_MB (1024) free. Each GPU keeps SPARK_VRAM_RESERVE_MB free, by default the larger of 512 MB and a sixteenth of its memory.
• A model that fits nowhere is refused with the arithmetic, and nothing is started to fail.
• The host's live free memory is checked as well as the estimates.

The model page shows the estimate under Needs, for example 606 MB for 4096 tokens of context (weights 138 MB, KV cache 180 MB).

Devices

Setting	Effect
SPARK_DEVICE	Where a model runs unless its own setting says: auto, cpu, cuda or cuda:N. auto means a CUDA device when there is one.
SPARK_CUDA_DEVICES	The CUDA devices runners may use: all, or a list such as 0,1.
SPARK_CPU_RUNNERS	Whether models may run on the CPU: auto (only when there is no CUDA device), true or false.
SPARK_THREADS	CPU threads per runner. 0 uses all.

Spark probes the CUDA devices at start: name, total and free memory, compute capability. A model set to cuda on a machine without a GPU is refused with 503 and the reason. A model set to auto falls back to the CPU.

On a machine with several GPUs, each model is placed on one of them. One model cannot be split across GPUs.

Mixtures of experts larger than the GPU

A mixture of experts model that does not fit a GPU keeps the routed experts of its first layers in host memory, and the GPU caches the most recently used experts. This is SPARK_HOST_EXPERTS (on by default) and SPARK_EXPERT_CACHE_PERCENT (10). Spark picks the fewest layers that fit, from the GGUF header.

Crash recovery

The server watches every runner by its process exit and by a health check every 5 seconds.

• A runner that crashes, is killed or stays silent for SPARK_RUNNER_HANG_SECONDS (120) is killed if needed, restarted and its model reloaded. Restarts back off from 1 second to a minute.
• Requests that were waiting for the model get it once it is back.
• Requests that were running fail with an error event.
• A request that had not started yet on a runner that died runs again.

To test recovery, kill a runner from the Runners page. That is the same as a crash. The next request works.

Runners page

Runners lists each runner with its state, model, device, memory, requests served and tokens. Each row opens the runner's log, unloads the model or kills the process. The devices panel shows the CUDA devices found, with their memory.

Logs

Runners write level: category: message lines, and the server logs each one again at its level. SPARK_LOG_LEVEL is passed to the runners. Prompt text is not logged.