Read-only replicas

A read-only replica is a second Curiosity Studio process that keeps a full copy of a primary workspace, follows the primary's writes as they happen, and serves read traffic: search, node and graph reads, and code endpoints marked read-only. Replicas spread read load off the primary, and they keep search and browsing available while the primary restarts.

This page explains how replication works, how the browser decides which server a request goes to, and what a deployment has to provide for both to work. The pages after it apply this to concrete setups:

• Deploy with DNS routing: one DNS name per node, each node serving HTTPS itself.
• Deploy on AWS: VPC, security groups, one Application Load Balancer, EC2.
• Operate replicas: health checks, lag, upgrades, failover, troubleshooting.

Architecture

A deployment has exactly one primary, the only process that writes, and any number of replicas. The primary does not need to be configured with a list of replicas: each replica connects to the primary, registers itself, and tells the primary the public URL it can be reached at. The primary then advertises that URL to browsers.

Every replica keeps its own complete copy of the data on its own disk. Nothing is shared between the nodes: no shared filesystem, no shared block device, no database server.

What runs where

A replica has everything it needs to serve reads, because all of it lives in the replicated RocksDB database: nodes, edges, schemas, users and access groups, configuration, and the data of every index (text, vector and numeric). A replica does not rebuild indexes; it reads the index data the primary wrote. Files whose content is stored in the graph are replicated with it. When the workspace keeps file blobs in S3 (external blob storage), the bucket is shared, and each replica needs the same MSK_BLOB_STORAGE_AWS_* settings and access to it.

How a replica joins and stays current

1. Register. Each time the replica process starts it picks a new replica id and registers with the primary over gRPC, on port 42999 of the host named in MSK_PRIMARY_ADDRESS. The call is authenticated with a secret derived from MSK_JWT_KEY, so both sides need the same key.
2. Copy the database. The primary takes a RocksDB checkpoint (hard links in <storage>/rep, so no data is copied on the primary) and streams the file list. The replica downloads what it lacks. SST files are immutable, so an .sst already on the replica's disk with the same name and size is kept; everything else is always downloaded. A replica restarted on its existing storage therefore downloads only what changed, and a new replica seeded from a snapshot of the primary's storage downloads only what changed since the snapshot.
3. Open and sync the front-end. The replica opens its copy read-only and mirrors the primary's front-end folder, so it serves the same application, including a custom front-end. It re-syncs whenever the primary's front-end changes.
4. Tail the write-ahead log. The replica asks for every change after its last sequence number. The primary streams write batches as it commits them, and the replica applies each chunk with a single write. The caches a read is served from are refreshed 5 ms after a chunk lands, and search readers reopen 250 ms after a replicated index commit.
5. Go online. The replica sends its public and internal addresses. The primary adds the public address to the MSKREPLICAURLS value it injects into index.html, and tells connected browsers over the WebSocket.
6. Report. The replica reports the last sequence number it applied every 250 ms while changes arrive, and a heartbeat with its replication counters every second. The primary uses these reports to measure lag and to decide which WAL files it can delete.

If the primary restarts, the next heartbeat tells the replica it is unknown. The replica registers again and resumes the WAL stream from its last sequence number, without copying the database again. It keeps serving reads, from its last state, while the primary is down.

If a replica stops reporting for 10 minutes, the primary drops it: it stops keeping WAL for it and removes its URL from the list sent to browsers. A replica that shuts down cleanly is dropped the same way, 10 minutes later, because replicas do not deregister.

Consistency

Replication is asynchronous. A write returns once the primary has committed it, before any replica has it. On a local primary and replica pair, a write became readable on the replica after a median of 4.3 ms (maximum 10 ms, 20 writes); network latency, sustained write load and an under-sized replica all add to that.

Two things follow:

• No read-your-writes guarantee. A user who saves something and immediately reads it back through a replica can briefly see the previous version. In a custom front-end, leave a read that must see a write the user just made unmarked, so it goes to the primary.
• Search results lag slightly more than node reads, because the search readers reopen 250 ms after an index commit replicates.

How the browser routes requests

The routing is done by the front-end in the browser, not by the server or a load balancer. The front-end keeps a list of servers and marks one of them as the primary:

A custom front-end can set AppSettings.ReplicaURLs in its settings callback instead. That list replaces MSKREPLICAURLS; the URLs must include the /api suffix.

The front-end checks each server with GET <server>/api/graph/ready on its first request, and again when a request receives 503 Service Unavailable (at most once every 5 seconds). A server counts as available when that returns true.

Each request is then routed like this:

• Only marked requests use replicas. The built-in front-end marks its read calls with ReadOnlyCompatible(): search, node and folder reads, graph queries, file text and blobs, and exports. Everything else goes to the primary.
• The primary is part of the pool. A replica is picked at random from all available servers, including the primary, and the choice is kept for 5 to 30 minutes (random per browser) while the server stays available. Load spreads across browsers, not across the requests of one browser.
• A 303 sends the request back to the primary. A replica answers 303 See Other to a request that would write, and to a code endpoint that is not marked read-only. The front-end then repeats the request without the read-only flag.
• When the primary is not ready, reads fail over. When the app starts and the primary is unreachable or still loading, the front-end checks the replicas. If one is available the app opens in read-only mode, with a Read-only mode banner, and reloads when the primary is ready again. While the app is already open, a 503 from the primary (which is what it answers while loading or shutting down) triggers a re-check of the servers, after which reads go to replicas. Writes fail, and keep failing after the primary is back until the page is reloaded.
• Network errors do not trigger failover. A refused connection, a 502 or a 504 fails the request without re-checking the servers. Open browsers therefore switch to replicas while the primary is starting or shutting down, but not when its process has stopped or its host is unreachable; they recover on the next page load.
• Opening a replica's URL directly loads the same application, with a READ-ONLY REPLICA banner. In that tab every request goes to replicas, and writes cannot reach the primary. Use it for diagnostics, or as the fallback address during a primary outage; do not send users there otherwise.

Manage → Developer tools shows the server list as the current browser sees it, with each server's availability.

Code endpoints

A code endpoint runs on a replica only when both sides allow it:

• The endpoint has Read Only enabled in the endpoint editor. It then compiles against the read-only scope, which has no write methods.
• The caller marks the call: API.Endpoints.CallAsync<T>(name, body, readOnlyCompatible: true) in the front-end SDK, or .ReadOnlyCompatible() on a raw REQ.

An endpoint that is not marked Read Only answers 303 on a replica, and the front-end repeats the call on the primary. A read-only endpoint running on a replica can call RunEndpointOnPrimaryAsync to run another endpoint on the primary; the replica forwards that call over HTTP to MSK_PRIMARY_ADDRESS, authenticated with a token the primary issued at registration. See Global scope.

AI tools that wrap an endpoint follow the endpoint's flag.

Clients other than the browser

The replica routing lives only in the workspace front-end. Data connectors, curiosity-cli, the C# and Python libraries and anything calling the REST API with a token talk to the URL they are given, and most of them write. Point them at the primary.

Assumptions and requirements

The routing and replication above only work when the deployment provides the following. Each item names what breaks without it.

Configuration

Replica

Settings stored in the workspace (Manage pages) replicate with the data, so a replica needs no copy of them. The license is part of that stored configuration unless it is passed with MSK_LICENSE, in which case pass it to the replicas too.

Primary

The primary needs no replica-specific setting. It accepts any replica that presents the secret derived from its key.

Ports

Where to go next

• Deploy with DNS routing
• Deploy on AWS
• Operate replicas
• Scaling: when to add replicas and when to scale the primary instead.
• Backup and restore: a replica is not a backup. A deletion on the primary reaches every replica within milliseconds.