External blob storage

By default a workspace keeps the content of every file it ingests inside its own graph database, on the disk behind MSK_GRAPH_STORAGE. You can keep that content in a cloud bucket instead:

  • AWS S3
  • Azure Blob Storage
  • Google Cloud Storage
  • Cloudflare R2

Only file content moves to the bucket. Nodes, edges, indexes, text extracted from files and configuration stay in the graph database, so the disk is still needed and still needs backups. Those backups do not include the bucket: protect it with the provider's own tools, such as object versioning or replication to a second region.

Choose a provider

Each provider has one variable that turns it on. Set exactly one of them, plus that provider's settings below. If more than one is set, the workspace refuses to start.

Provider Turned on by
AWS S3 MSK_BLOB_STORAGE_AWS_S3_BUCKET
Azure Blob Storage MSK_BLOB_STORAGE_AZURE_CONTAINER
Google Cloud Storage MSK_BLOB_STORAGE_GCS_BUCKET
Cloudflare R2 MSK_BLOB_STORAGE_R2_BUCKET

Create the bucket or container before starting the workspace: the workspace does not create it. At every start it writes, reads back and deletes a small test object, and it does not start if any of those steps fails or the content it reads back is different. So a wrong name, missing credentials or missing permissions show up at startup, not on the first upload.

AWS S3

Environment variable Required Description
MSK_BLOB_STORAGE_AWS_S3_BUCKET yes Bucket name.
MSK_BLOB_STORAGE_AWS_REGION yes Region of the bucket, for example eu-central-1.
MSK_BLOB_STORAGE_AWS_ACCESS_KEY_ID no Access key. Used together with the secret key.
MSK_BLOB_STORAGE_AWS_SECRET_ACCESS_KEY no Secret key.
MSK_BLOB_STORAGE_AWS_PROFILE_NAME no Named profile from the AWS credentials file, used when no access key is set.

With neither keys nor a profile, the workspace uses the AWS SDK's default credential chain: an EC2 instance profile, an ECS task role, EKS pod identity or IRSA, or the standard AWS_* variables.

The credentials need these permissions on the bucket's objects:

{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
  "Resource": "arn:aws:s3:::curiosity-prod-blobs/*"
}

Azure Blob Storage

Environment variable Required Description
MSK_BLOB_STORAGE_AZURE_CONTAINER yes Container name.
MSK_BLOB_STORAGE_AZURE_CONNECTION_STRING one of these two Storage account connection string. Use this for sovereign clouds or a custom endpoint.
MSK_BLOB_STORAGE_AZURE_ACCOUNT_NAME one of these two Storage account name. The workspace connects to https://<account>.blob.core.windows.net.
MSK_BLOB_STORAGE_AZURE_ACCOUNT_KEY no Account key, used with the account name.

With an account name and no key, the workspace signs in with DefaultAzureCredential: a managed identity on a VM, Workload Identity on AKS, or the standard AZURE_* variables for a service principal. That identity needs the Storage Blob Data Contributor role on the container or the storage account.

docker run -d \
  -e MSK_BLOB_STORAGE_AZURE_CONTAINER=curiosity-blobs \
  -e MSK_BLOB_STORAGE_AZURE_ACCOUNT_NAME=curiosityprod \
  ...
  curiosityai/curiosity:<version>

Google Cloud Storage

Environment variable Required Description
MSK_BLOB_STORAGE_GCS_BUCKET yes Bucket name.
MSK_BLOB_STORAGE_GCS_CREDENTIALS_FILE no Path to a service-account key file (JSON) inside the container.

Without a key file, the workspace uses Application Default Credentials: Workload Identity on GKE, the VM's service account on Compute Engine, or GOOGLE_APPLICATION_CREDENTIALS. The service account needs the Storage Object User role (roles/storage.objectUser) on the bucket.

gcloud storage buckets add-iam-policy-binding gs://curiosity-prod-blobs \
  --member=serviceAccount:curiosity@my-project.iam.gserviceaccount.com \
  --role=roles/storage.objectUser

Cloudflare R2

Environment variable Required Description
MSK_BLOB_STORAGE_R2_BUCKET yes Bucket name.
MSK_BLOB_STORAGE_R2_ACCOUNT_ID yes Cloudflare account ID.
MSK_BLOB_STORAGE_R2_ACCESS_KEY_ID yes Access key ID of an R2 API token.
MSK_BLOB_STORAGE_R2_SECRET_ACCESS_KEY yes Secret access key of the same token.
MSK_BLOB_STORAGE_R2_JURISDICTION no eu or fedramp, for a bucket created in that jurisdiction. Leave unset otherwise.

Create the token in the Cloudflare dashboard under R2 → Manage API tokens, with Object Read & Write permission, scoped to the bucket. The workspace talks to R2 through its S3-compatible API at https://<account-id>.r2.cloudflarestorage.com (or <account-id>.<jurisdiction>.r2.cloudflarestorage.com).

A workspace stays with its bucket

The first time a workspace starts with external blob storage, it records where its files are:

Provider Recorded as
AWS S3 <region>-<bucket>
Azure Blob Storage azure:<account>/<container>
Google Cloud Storage gcs:<bucket>
Cloudflare R2 r2:<host>/<bucket>

On every later start the configured location has to match. If it doesn't, the workspace refuses to start rather than run against a bucket that doesn't have its files. It also refuses to start when it has a recorded location but no provider is configured.

The workspace does not move files between buckets. To move to a new bucket or provider:

  1. Stop the workspace.
  2. Copy every object to the new bucket, keeping the object names. All four providers use the same names, so a copy between providers works as is, for example with rclone copy.
  3. Change the variables to the new location, and set the provider's _FORCE variable to true for the first start.
  4. Remove the _FORCE variable after the workspace has started.
Provider Variable
AWS S3 MSK_BLOB_STORAGE_AWS_S3_BUCKET_FORCE
Azure Blob Storage MSK_BLOB_STORAGE_AZURE_CONTAINER_FORCE
Google Cloud Storage MSK_BLOB_STORAGE_GCS_BUCKET_FORCE
Cloudflare R2 MSK_BLOB_STORAGE_R2_BUCKET_FORCE

Setting _FORCE without copying the objects first starts the workspace against a bucket that doesn't have its files: every file's preview, download and re-extraction then fails.

Replicas

Read-only replicas share the primary's bucket. Give every replica the same MSK_BLOB_STORAGE_* values and the same access to the bucket as the primary: each replica runs the startup test too.

See also