Spark

Fleet: central and nodes

A central Spark can route requests to models that live on other Spark servers, called nodes. A node is another machine, possibly with other GPUs. Clients talk to the central only. The central lists the nodes' models in GET /v1/models and serves them, streamed, through both APIs.

flowchart LR C[Client] -->|"/v1/..."| S[Central] S -->|"signed HTTP"| N1[Node A: model X] S -->|"signed HTTP"| N2[Node B: model Y] N1 -.->|"join, check-in every 15 s"| S N2 -.-> S

Roles

SPARK_ROLE What it is
standalone (default) One server.
central Serves its own models, and routes to the nodes' models. Has the Nodes page and join tokens.
node Runs models. Joins a central with a one-time token and keeps its membership.

Join a node

1

Start the central

SPARK_ROLE=central SPARK_ADMIN_PASSWORD='choose-a-password' spark
2

Create a join token

On the central, open Nodes and create a join token with a name. It is valid for 24 hours by default, up to 30 days, and can be used once. The page shows the token once, with the command to run.

3

Start the node with it

SPARK_ROLE=node SPARK_CENTRAL_ADDRESS=http://central.example.com:8080 SPARK_JOIN_TOKEN=… \
SPARK_NODE_NAME=gpu-1 SPARK_NODE_ADDRESS=http://10.0.0.12:8080 SPARK_ADMIN_PASSWORD='choose-a-password' spark
4

Check

The node appears on the central's Nodes page with its state, hardware and models. Its runners show on the central's Runners page.

SPARK_NODE_ADDRESS is the address the central calls the node on. SPARK_NODE_NAME defaults to the machine name.

How it works

  • After the join, every call between the two is signed, with a timestamp and a nonce against replay. Unsigned calls to a node and an unknown join token are refused.
  • The node checks in every 15 seconds with its models, runners, downloads and hardware.
  • Node secrets are sealed with fleet.key in SPARK_STORAGE.
  • A request for a model that only a node has goes to that node. When several nodes hold the same model, the central prefers one where it is loaded and then the one with the fewest busy runners. There is no other load balancing yet.
  • Both servers log the request. The central's log shows the node's phases and the number of images.
  • A zero retention key's setting travels with the request. The node logs the same way.

Download onto a node

On the central's Discover page, Download to picks a node. The Downloads page shows which node each download runs on. Cancel, clear and resume work from the central as well.

Manage nodes

The Nodes page can disable a node, which stops routing to it, and remove it. Delete a join token that is no longer needed.

Limits

  • The central must be able to call each node. Nodes behind NAT are not supported yet.
  • Embeddings for a model that only a node has are not answered by the central (it returns 501).
  • A node that runs an older Spark ignores the zero retention flag.
  • Use HTTPS between the central and the nodes where you can: put a reverse proxy in front of each, as in Run it on a server. Signatures authenticate every call, and TLS hides the traffic.