External blob storage
By default a workspace keeps the content of every file it ingests inside its own graph database, on the disk behind MSK_GRAPH_STORAGE. You can keep that content in a cloud bucket instead:
- AWS S3
- Azure Blob Storage
- Google Cloud Storage
- Cloudflare R2
Only file content moves to the bucket. Nodes, edges, indexes, text extracted from files and configuration stay in the graph database, so the disk is still needed and still needs backups. Those backups do not include the bucket: protect it with the provider's own tools, such as object versioning or replication to a second region.
Choose a provider
Each provider has one variable that turns it on. Set exactly one of them, plus that provider's settings below. If more than one is set, the workspace refuses to start.
| Provider | Turned on by |
|---|---|
| AWS S3 | MSK_BLOB_STORAGE_AWS_S3_BUCKET |
| Azure Blob Storage | MSK_BLOB_STORAGE_AZURE_CONTAINER |
| Google Cloud Storage | MSK_BLOB_STORAGE_GCS_BUCKET |
| Cloudflare R2 | MSK_BLOB_STORAGE_R2_BUCKET |
Create the bucket or container before starting the workspace: the workspace does not create it. At every start it writes, reads back and deletes a small test object, and it does not start if any of those steps fails or the content it reads back is different. So a wrong name, missing credentials or missing permissions show up at startup, not on the first upload.
AWS S3
| Environment variable | Required | Description |
|---|---|---|
MSK_BLOB_STORAGE_AWS_S3_BUCKET |
yes | Bucket name. |
MSK_BLOB_STORAGE_AWS_REGION |
yes | Region of the bucket, for example eu-central-1. |
MSK_BLOB_STORAGE_AWS_ACCESS_KEY_ID |
no | Access key. Used together with the secret key. |
MSK_BLOB_STORAGE_AWS_SECRET_ACCESS_KEY |
no | Secret key. |
MSK_BLOB_STORAGE_AWS_PROFILE_NAME |
no | Named profile from the AWS credentials file, used when no access key is set. |
With neither keys nor a profile, the workspace uses the AWS SDK's default credential chain: an EC2 instance profile, an ECS task role, EKS pod identity or IRSA, or the standard AWS_* variables.
The credentials need these permissions on the bucket's objects:
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
"Resource": "arn:aws:s3:::curiosity-prod-blobs/*"
}
Azure Blob Storage
| Environment variable | Required | Description |
|---|---|---|
MSK_BLOB_STORAGE_AZURE_CONTAINER |
yes | Container name. |
MSK_BLOB_STORAGE_AZURE_CONNECTION_STRING |
one of these two | Storage account connection string. Use this for sovereign clouds or a custom endpoint. |
MSK_BLOB_STORAGE_AZURE_ACCOUNT_NAME |
one of these two | Storage account name. The workspace connects to https://<account>.blob.core.windows.net. |
MSK_BLOB_STORAGE_AZURE_ACCOUNT_KEY |
no | Account key, used with the account name. |
With an account name and no key, the workspace signs in with DefaultAzureCredential: a managed identity on a VM, Workload Identity on AKS, or the standard AZURE_* variables for a service principal. That identity needs the Storage Blob Data Contributor role on the container or the storage account.
docker run -d \
-e MSK_BLOB_STORAGE_AZURE_CONTAINER=curiosity-blobs \
-e MSK_BLOB_STORAGE_AZURE_ACCOUNT_NAME=curiosityprod \
...
curiosityai/curiosity:<version>
Google Cloud Storage
| Environment variable | Required | Description |
|---|---|---|
MSK_BLOB_STORAGE_GCS_BUCKET |
yes | Bucket name. |
MSK_BLOB_STORAGE_GCS_CREDENTIALS_FILE |
no | Path to a service-account key file (JSON) inside the container. |
Without a key file, the workspace uses Application Default Credentials: Workload Identity on GKE, the VM's service account on Compute Engine, or GOOGLE_APPLICATION_CREDENTIALS. The service account needs the Storage Object User role (roles/storage.objectUser) on the bucket.
gcloud storage buckets add-iam-policy-binding gs://curiosity-prod-blobs \
--member=serviceAccount:curiosity@my-project.iam.gserviceaccount.com \
--role=roles/storage.objectUser
Cloudflare R2
| Environment variable | Required | Description |
|---|---|---|
MSK_BLOB_STORAGE_R2_BUCKET |
yes | Bucket name. |
MSK_BLOB_STORAGE_R2_ACCOUNT_ID |
yes | Cloudflare account ID. |
MSK_BLOB_STORAGE_R2_ACCESS_KEY_ID |
yes | Access key ID of an R2 API token. |
MSK_BLOB_STORAGE_R2_SECRET_ACCESS_KEY |
yes | Secret access key of the same token. |
MSK_BLOB_STORAGE_R2_JURISDICTION |
no | eu or fedramp, for a bucket created in that jurisdiction. Leave unset otherwise. |
Create the token in the Cloudflare dashboard under R2 → Manage API tokens, with Object Read & Write permission, scoped to the bucket. The workspace talks to R2 through its S3-compatible API at https://<account-id>.r2.cloudflarestorage.com (or <account-id>.<jurisdiction>.r2.cloudflarestorage.com).
A workspace stays with its bucket
The first time a workspace starts with external blob storage, it records where its files are:
| Provider | Recorded as |
|---|---|
| AWS S3 | <region>-<bucket> |
| Azure Blob Storage | azure:<account>/<container> |
| Google Cloud Storage | gcs:<bucket> |
| Cloudflare R2 | r2:<host>/<bucket> |
On every later start the configured location has to match. If it doesn't, the workspace refuses to start rather than run against a bucket that doesn't have its files. It also refuses to start when it has a recorded location but no provider is configured.
The workspace does not move files between buckets. To move to a new bucket or provider:
- Stop the workspace.
- Copy every object to the new bucket, keeping the object names. All four providers use the same names, so a copy between providers works as is, for example with
rclone copy. - Change the variables to the new location, and set the provider's
_FORCEvariable totruefor the first start. - Remove the
_FORCEvariable after the workspace has started.
| Provider | Variable |
|---|---|
| AWS S3 | MSK_BLOB_STORAGE_AWS_S3_BUCKET_FORCE |
| Azure Blob Storage | MSK_BLOB_STORAGE_AZURE_CONTAINER_FORCE |
| Google Cloud Storage | MSK_BLOB_STORAGE_GCS_BUCKET_FORCE |
| Cloudflare R2 | MSK_BLOB_STORAGE_R2_BUCKET_FORCE |
Setting _FORCE without copying the objects first starts the workspace against a bucket that doesn't have its files: every file's preview, download and re-extraction then fails.
Replicas
Read-only replicas share the primary's bucket. Give every replica the same MSK_BLOB_STORAGE_* values and the same access to the bucket as the primary: each replica runs the startup test too.