Spark

Request log and zero retention

Spark logs every request to the inference endpoints. The log is the first place to look when a client misbehaves.

What is logged

For each request: the time, the API and endpoint, the model, whether it streamed, the status and finish reason, the prompt, cached and completion tokens, the total time, the time to first token, the prefill and decode speed, the device, the client address, the user agent and the API key. With SPARK_LOG_REQUEST_BODIES=true (the default) it also keeps the request body, the prompt the chat template rendered, the answer, the reasoning and the tool calls.

Requests lists them with filters, search and a live view. Open one to see:

  • the request and the rendered prompt;
  • the answer, the reasoning and the tool calls;
  • a timeline of each step: fetching images, waiting for or loading the model, the vision encoder, cache restore and save, tokenizing, prefill, decode and tool calls, drawn as a waterfall.

The request log with OpenAI, Anthropic and embeddings requests, their tokens, timings and clientsThe request log with OpenAI, Anthropic and embeddings requests, their tokens, timings and clients

One request: who sent it, the timing, a timeline of each step and the answerOne request: who sent it, the timing, a timeline of each step and the answer

SPARK_REQUEST_LOG_RETENTION_DAYS (30) deletes older entries. 0 keeps them for ever. The console also clears the log. The x-request-id header of a response is the id of its request, so you can find a request from the client side.

The client address behind proxies

Behind a reverse proxy or Cloudflare, the log names the real client from CF-Connecting-IP, True-Client-IP, X-Real-IP, Forwarded or X-Forwarded-For (read from the right). It believes these headers only from the proxies named by SPARK_TRUSTED_PROXIES: addresses, CIDR ranges, loopback, private, cloudflare or none. The default is loopback,private,cloudflare.

Zero retention

A key created with Zero retention on has its requests logged without their content.

  • Stored: when, who, which model, token counts and timings.
  • Never stored: the prompt, the request body, the rendered prompt, the answer, the reasoning, the tool calls and the tool names.
  • Held in memory: the content of the most recent requests, in a first in first out buffer of SPARK_ZERO_RETENTION_MEMORY_MB (16). The console still shows a recent request whole. Newer requests push older ones out, and a restart forgets them. 0 keeps none.
  • A central passes the setting to the node that answers, which does the same.
  • It is set when the key is created and cannot be changed.
What zero retention does not cover

It covers the request log only. The KV cache works as for any other key, and its snapshots on disk (SPARK_KV_CACHE_FOLDER) hold the prompt's token ids, which decode back to its text. To avoid that, turn snapshots off with SPARK_KV_CACHE_MB=-1. A node running an older Spark ignores the flag and logs the content.

SPARK_LOG_REQUEST_BODIES=false turns body logging off for every key.

Server logs

The server and the runners log through one logger to the console at SPARK_LOG_LEVEL. The logs contain counts and timings, never prompt text. Under systemd, read them with journalctl -u spark -f.

The playground

Playground chats with a model through the same endpoint. Every exchange is in the request log. It shows the cached tokens, the speed, and links to the answer's timeline and request. It takes images for vision models and streams reasoning as one dimmed line.