Troubleshooting

The Orchestrator does not start

Symptom	Cause and fix
Exit code 173 (`0xBAD`) and "ORC_ADMIN_PASSWORD is required"	Set `ORC_ADMIN_PASSWORD`, at least 8 characters. A node may omit it.
`IO error: While lock file: …/db/LOCK`	Another Orchestrator uses the same `ORC_STORAGE`. Give each its own folder.
The port is in use, or permission is denied on 80 or 443	Choose another `ORC_PORT`. For ports below 1024 grant `CAP_NET_BIND_SERVICE` or run through systemd with `AmbientCapabilities`.

The console cannot reach Docker

The overview says it cannot reach the Docker daemon. Check that the daemon runs, and that the user the Orchestrator runs as is in the docker group. In a container, mount the socket: -v /var/run/docker.sock:/var/run/docker.sock. For rootless Docker mount $XDG_RUNTIME_DIR/docker.sock, or set ORC_DOCKER_ENDPOINT.

A workspace does not start

Read the workspace's Logs tab and the Activity page.
ORC_START_TIMEOUT_MINUTES (15) ends the wait for a start that does not become ready. Restart it from the console.
A start that is refused with the message that the server is full means ORC_MAX_RUNNING_INSTANCES is reached and every running workspace is in use. See Capacity.
A workspace whose container was removed outside the Orchestrator shows as Missing. The data volume is not touched by that.

A workspace opened in a script or monitor returns 503

Only a browser navigation wakes a sleeping workspace. Other requests get 503 on purpose. Start it through the API: POST /api/admin/instances/<slug>/start. See Sleep and wake.

A workspace keeps shutting down

Three things stop a workspace: ORC_IDLE_TIMEOUT_MINUTES, room for another workspace (ORC_MAX_RUNNING_INSTANCES) and low memory (ORC_MIN_FREE_MEMORY_PERCENT). The Activity page lists each stop. Use Keep warm or Always online to hold it. A forgotten browser tab does not count as activity.

Subdomains do not work

Add the wildcard DNS record *.<your domain>.
Use a certificate that covers the wildcard. Let's Encrypt gives each workspace its own, so a new workspace waits up to about 10 minutes for it.
localhost, IP addresses and single-label hosts never match a workspace. Test with <slug>.localhost.
Set ORC_PUBLIC_ADDRESS when the Orchestrator itself lives on a subdomain.
Check that ORC_SUBDOMAIN_ROUTING is not false.

Let's Encrypt does not issue

Settings lists each name's state. A waiting name has not been issued yet.

DNS must point at the server, and ports 443 and 80 must be reachable from the internet.
Failures are retried every 15 minutes. The log says why.
Let's Encrypt allows 5 failed validations per name per hour and 50 certificates per registered domain per week. Try ORC_LETSENCRYPT_STAGING=true first.
A private CA that requires POST-as-GET for account reads fails the first order. See A private CA.

A backup request is refused

409 means the workspace is running. Stop it first. 400 on restore means the upload is not a whole backup made by the Orchestrator: it must be a complete tar with every entry under DATA/.

With landing sign-in, the workspace shows its own form when it has not been given the current version of the account yet, or when a user of that name already existed there. See Accounts and console users. The hand-off also needs a Studio image whose login/check answers the token as text.

Two Orchestrators on one Docker

Container and volume names come from ORC_CONTAINER_PREFIX. Two Orchestrators on one Docker daemon need different prefixes and different ORC_DOCKER_NETWORK values.

A node is unavailable

The central marks a node unreachable after three failed polls. Check that the central can call the node's ORC_NODE_ADDRESS. A TLS-terminating proxy between them breaks enrollment by code: join with a token instead. After a node is revoked, run curiosity-orchestrator --leave-fleet on it.

Referenced by

Introduction