Fleet: central and nodes

One central Orchestrator (ORC_ROLE=central) can manage nodes (ORC_ROLE=node) on other machines. Each node runs its workspaces exactly as a standalone Orchestrator does, and keeps doing so if the central goes away. The central is the console and, by default, the front door.

Roles

`ORC_ROLE`	What it is
`standalone` (default)	One Orchestrator on one Docker host.
`central`	Everything standalone does, plus the node registry, join tokens, a relay to nodes and the fleet pages. Its own workspaces are the built-in node `local`.
`node`	Runs workspaces. Once enrolled, it takes orders from its central and refuses to create workspaces on its own. `ORC_ADMIN_PASSWORD` is optional, and when it is not set nobody signs in to the node's own console.

The central calls nodes for everything after registration: management, state polling and visitor traffic. A node calls the central for two things only: to join, and to check in every 60 seconds. The central is the only machine that must reach the nodes, so nodes can be firewalled to accept only the central.

Start a central and a node

# central
ORC_ROLE=central ORC_ADMIN_PASSWORD='choose-a-password' ORC_PUBLIC_ADDRESS=https://orchestrator.example.com \
  curiosity-orchestrator

# node, on another machine
ORC_ROLE=node ORC_NODE_NAME=node-a ORC_NODE_ADDRESS=https://10.0.0.12:8443 curiosity-orchestrator

The node prints its id, its certificate key and an enrollment code. To try a fleet on one machine, give the central and each node their own ORC_PORT, ORC_STORAGE, ORC_CONTAINER_PREFIX and ORC_DOCKER_NETWORK, and give each node an ORC_NODE_ADDRESS that the central can call.

Register a node

There are three ways, all on the central's Nodes page.

Use this for a node you just installed by hand.

Start the node

It prints its enrollment code, or uses ORC_NODE_ENROLLMENT_CODE if you set one.

Add node

Type its address (10.0.0.12, node-3:8443 or a URL) and choose Look up. A bare host is tried as https://<host>/, then https://<host>:8443/, then http://<host>:8080/.

Give the code

The code never crosses the wire. The central proves it knows it, bound to the certificate key it saw, so anything that terminates TLS between the two, such as a reverse proxy or load balancer, presents another key and is refused.

Use this for machines that come up by themselves, for example from cloud-init.

New join token shows a token once, with the command to run:

ORC_ROLE=node \
ORC_CENTRAL_ADDRESS=https://orchestrator.example.com \
ORC_CENTRAL_PIN=sha256:… \
ORC_JOIN_TOKEN=orcj_… \
curiosity-orchestrator

A token has an expiry (24 hours by default), a number of uses (1 by default), and whether nodes that use it start approved or wait as pending. The central calls the node back at the address it gives before it accepts it, so a join cannot register an address the joiner does not control. ORC_CENTRAL_PIN pins the central's certificate key when that certificate does not validate normally.

A pending node waits on the Nodes page with Approve and Reject. Until it is approved, the central sends it no work.

After registration, every call between the two is signed with a secret of that node's own (ORC1-HMAC-SHA256). The central rotates it every ORC_FLEET_ROTATE_DAYS (90) days, or on demand.

Use the fleet

Workspaces lists every workspace of the fleet with its node. New workspace picks a node, or the active node with the most free memory that matches the labels you ask for and has room under its ORC_MAX_RUNNING_INSTANCES. Set labels on a node with ORC_NODE_LABELS=zone=eu-1,gpu.
A workspace's page and every action on it work the same wherever it lives. The central relays them to the node.
Settings → Environment for every workspace and Accounts apply to every node. The central pushes them, and a managed node's console refuses to change them.
Names and addresses are unique across the fleet. The central owns the address namespace. A node cannot claim another workspace's address.
Update all and Check for updates run on the central and on every active or draining node, in one operation.
Drain a node to stop placing new workspaces on it. Existing ones keep running.
Disable a node to send it nothing. The node is left alone.

Visitors

Visitors open /<slug>/ or <slug>.<host> on the central, which forwards to the node. The starting page, cookies and subdomains behave as on a single machine. A node needs no public name and no public certificate. The central's Let's Encrypt names for <slug>.<host> come from the placements.

ORC_NODE_ALLOW_DIRECT=false on a node refuses workspace traffic that did not come through the central. With a firewall that admits only the central, the nodes are private.
With ORC_FLEET_PROXY=false on the central, each workspace is linked to its node's own ORC_PUBLIC_ADDRESS, and every node handles its own DNS, certificates and starting page.

Moving a workspace

Move to another node copies a workspace: it stops the workspace, creates it on the target, streams the backup into a restore there, starts it and waits until it is ready, re-points the address, then deletes it at the source while keeping the source's data volume. Moving works node to node, node to central and central to node. Always online, or what is left of a keep warm, goes with it. While it moves, visitors get 503 and the starting page says it is being moved. A failed copy removes the half-made one and leaves the workspace where it was.

When a node does not answer

The central polls every active node every ORC_FLEET_POLL_SECONDS (15). Three failed polls in a row mark it unreachable. A failed visitor request or console call polls it at once, so a node that goes down is usually marked within seconds. While a node is out, its workspaces show as Unavailable, dimmed and counted apart on the landing page. A navigation to one opens a starting page that says its server does not answer and opens the workspace once the node is back. Other requests get 503 at once. Nothing waits on a timeout.

A node keeps idle shutdown, keep warm, capacity and low-memory handling by itself. A central outage stops management and central-proxied traffic, nothing else.

Remove a node

Deleting a node asks it to forget the central first. It is refused while workspaces are placed on it, unless you detach them: they keep running on the node and leave the central's catalogue.

If a node was revoked or can no longer be reached, it logs that its secret is no longer accepted, marks itself orphaned and keeps running its workspaces. To reset it, stop the Orchestrator on that machine and run:

curiosity-orchestrator --leave-fleet

The node then forgets its central and prints a new enrollment code.

Settings

Variable	Role	Default	Meaning
`ORC_ROLE`	all	`standalone`	`standalone`, `central` or `node`.
`ORC_FLEET_SECRET`	central, node	none	Shared secret for automated joins, at least 32 characters.
`ORC_FLEET_AUTO_APPROVE`	central	`false`	Nodes joining with the fleet secret start active.
`ORC_FLEET_PROXY`	central	`true`	Visitors reach nodes through the central. `false` links each node directly.
`ORC_FLEET_POLL_SECONDS`	central	`15`	How often each node's state is read.
`ORC_FLEET_ROTATE_DAYS`	central	`90`	Rotate each node's secret this often. `0` is never.
`ORC_FLEET_HOST`	central	none	Answer `/api/fleet/*` only on this host name.
`ORC_CENTRAL_ADDRESS`	node	none	The central to join.
`ORC_CENTRAL_PIN`	node	none	`sha256:…` key pin of the central, when its certificate does not validate.
`ORC_JOIN_TOKEN`	node	none	A join token. Used once.
`ORC_NODE_NAME`	node	host name	The name the central shows.
`ORC_NODE_ADDRESS`	node	guessed	The address the central calls the node on.
`ORC_NODE_LABELS`	node	none	Labels for placement, for example `zone=eu-1,gpu`.
`ORC_NODE_ENROLLMENT_CODE`	node	random	Choose the enrollment code instead of reading it from the log.
`ORC_NODE_ALLOW_DIRECT`	node	`true`	`false` serves workspaces only to traffic the central forwarded.

Setting ORC_JOIN_TOKEN or ORC_FLEET_SECRET without ORC_CENTRAL_ADDRESS fails at start with exit code 0xBAD (173 in a shell).

Limits

One central. It holds the fleet's catalogue and is not highly available. A central restarted while a node is down knows only the slugs of that node's workspaces, not their details, until the node answers. They are missing from the landing page until then.
Nodes behind NAT are not supported. The central must reach every node.
Sign in as administrator for a workspace on a node that is linked directly stores the token on the central's origin, where that workspace does not run.

The design, the signing scheme and the security reasoning are in the project's design document, docs/fleet.md in the source repository.