Workspace-hosted Connectors

A connector is normally a stand-alone program that writes into the workspace over HTTP with the Curiosity.Library SDK. The same ingestion code can also run inside the workspace process, as a code integration: no separate host to deploy, no API token to manage, and the run is scheduled and logged like any other workspace task.

The ingestion code is the same either way. A workspace-hosted connector's Graph is a Curiosity.Library.ConnectorGraph — the in-process equivalent of what a stand-alone connector connects to — so a connector can be prototyped here and later moved out to run next to its source system, or the other way round, without its ingestion code changing.

When to host it in the workspace

Creating one

A code integration is a scheduled task of the Data Connector type. Create it from Manage → Build (#/manage/build) → Code Integrations → the + button, or import a set exported from another workspace with Import integrations on the same tree node.

It carries a schedule like any scheduled task, and each run is recorded under Manage → Operate → Integrations Logs (#/manage/operate/api-integrations) next to the external connectors', with its progress and its log output.

The execution scope

The code runs in a DataConnectorExecutionScope. Its members are top-level identifiers, as in the other code scopes:

Write through Graph, not through Workspace, so every write goes through the connector's batching.

var apiKey = await GetSecretAsync(Secrets.CrmApiKey);

using var http = new HttpClient();
http.DefaultRequestHeaders.Add("Authorization", "Bearer " + apiKey);

await Graph.CreateEdgeSchemaAsync(typeof(E));

var page = 0;

while (true)
{
    CancellationToken.ThrowIfCancellationRequested();

    var batch = await http.GetFromJsonAsync<CustomerPage>($"https://crm.example.com/customers?page={page}", CancellationToken);

    if (batch.Items.Count == 0) break;

    foreach (var c in batch.Items)
    {
        var customer = Key(N.Customer.Type, c.Id);

        Graph.AddOrUpdate(customer, new { Name = c.Name, Tier = c.Tier });
        Graph.Link(customer, Key(N.Account.Type, c.AccountId), E.OfAccount, E.HasCustomer);
    }

    ProgressReporter.Message($"Ingested page {page}");
    page++;
}

await Graph.LogAsync(LogLevel.Information, $"Ingested {page} pages of customers");

Committing

Writes are queued and committed in batches in the background — Graph.SetAutoCommitCost(everyNodes) sets the batch size (5,000 nodes by default).

Whatever is still pending when the code returns is committed before the run is reported as finished, so a closing await Graph.CommitPendingAsync() is not needed. Call it only where later work has to see the earlier writes — a query only sees committed data, so commit before reading back something you just wrote.

await Graph.LogAsync(LogLevel.Information, "…") writes to this run's log, which is what an operator reads under Integrations Logs. Prefer it over Logger for anything they should see.

Writing data

The write surface is the connector API, documented in full in the connector SDK:

• Graph.AddOrUpdate(node, content) replaces the node's content; Graph.Update(node, content) writes the fields in content and leaves the rest; Graph.TryAdd(node, content) writes only when the node is new. content is an anonymous object or an op => op.Set(field, value) builder.
• Graph.Link(a, b, E.Forward, E.Backward) / Graph.Unlink(...) for edges. A link to a node that does not exist yet is fine — Key(type, key) derives the same UID the node will get, so the edge activates once both ends exist.
• Access control: Graph.RestrictAccessToUser / RestrictAccessToTeam / AddOwners / ClearPermissionsExcept, or write a node and its owners atomically with Graph.AddOrUpdateWithOwnership(node, content, owners).
• Files and folders: Graph.CreateFolderAsync, Graph.UploadFileAsync, Graph.UploadFileToFolderAsync, Graph.TryGetFileNodeAsync (to skip an unchanged file), Graph.DeleteFileAsync, Graph.DeleteFolderAsync.
• Users and teams: Graph.CreateUserAsync, Graph.CreateTeamAsync, Graph.AddUserToTeam.
• Schema: Graph.CreateEdgeSchemaAsync(typeof(E)) registers the edge types the connector uses.

Related

• Connectors — the stand-alone model, and when to prefer it.
• Connector SDK — the full Curiosity.Library write surface.
• Data → Tools — generate a starting connector class from a CSV or JSON file.
• Scheduled tasks — the scheduling the run inherits.