Troubleshooting Coding and Queries

This guide covers common issues and performance tips when working with custom code and queries in Curiosity Studio.

Node locks and timeouts

Every write to the graph goes through a node lock: TryGetLockedAsync / GetOrAddLockedAsync take an exclusive lock on the node, and CommitAsync or AbandonChanges releases it. Only one caller can hold the lock on a given node at a time. Two time limits apply to those locks, and each has its own exception:

Both exceptions live in the Mosaik.GraphDB namespace, and both carry the node's UID and type, plus the caller that holds the lock, in Exception.Data (also printed by ToString()).

NodeDeadlockException

What it means

A request for a lock (TryGetLockedAsync, GetOrAddLockedAsync, or one of their multi-node overloads) did not succeed within the graph's maximum lock wait. The graph retries with an increasing back-off for the whole wait period, then gives up and throws. The wait is 10 seconds by default, and the workspace never uses less than 10 seconds.

The name says "deadlock", but the graph cannot tell a real deadlock from a slow lock holder. All it knows is that the node stayed locked for longer than the wait limit. The exception data says which one you are looking at:

• UID: the node and its type. With the multi-node overloads, UIDs lists every node that was requested.
• TryingToLock: the method, file and line of the code that was waiting.
• LockedBy: the method, file and line of the code that first took the lock that is still held.

The workspace log records the same information as an error, in the form Node <uid> of type <type> trying to lock at <caller> might be deadlocked, was locked by <caller>, throwing.

Common causes

1. Inconsistent locking order (a real deadlock):
   • Thread A locks Node 1, then tries to lock Node 2.
   • Thread B locks Node 2, then tries to lock Node 1.
   • Neither can proceed, and one of them receives NodeDeadlockException after the wait limit.
2. Locking the same node twice in one flow: code that holds a lock on a node and then calls a helper that locks the same node again waits on itself. The scoped Graph object handed to endpoints, AI tools and scheduled tasks detects this and throws RequestedAlreadyLockedNodeException straight away; code that goes through Graph.Internals does not, and waits for the full wait limit instead.
3. Long-held locks: the lock holder is doing slow work (a network call, an LLM call, a large computation, a long loop) between taking the lock and committing. Nothing is stuck, but other writers to the same node time out.
4. A forgotten commit: code took a lock and returned, or threw, without calling CommitAsync or AbandonChanges. Every other writer to that node times out until the lock is released (see Uncommitted Nodes).
5. A hot node: many concurrent writers update the same node, for example a shared counter, a "last updated" field on a parent node, or an edge to a popular node that every ingestion item adds. Each writer is quick, but the queue for the lock grows longer than the wait limit under load.

What to do

• Read LockedBy first. It names the code that held the lock: that is the code to fix, not the code that received the exception.
• If LockedBy and TryingToLock point at the same flow, the code is locking a node it already holds. Pass the LockedNode you already have into the helper instead of locking again.
• If two flows lock the same pair of nodes, lock both in one call with TryGetLockedAsync(uid1, uid2), which takes both locks together, or always lock them in the same order (see below).
• If the holder is slow, move the slow work out of the locked section: prepare everything first, lock, apply, commit.
• If many writers contend for one node, reduce how often it is written: aggregate changes and write once per batch, or move the shared value to a separate node.
• In batch jobs (connectors, scheduled tasks, migrations), the failing item can be retried after a short delay. Retrying only helps for a busy node; it does not fix a real deadlock or a forgotten commit, so log the exception data even when you retry.

NodeLockExpiredException

What it means

Your code held a lock for so long that the graph force-released it. When that happens, the graph throws away the pending changes on the node and restores it from storage, and marks your LockedNode as released. The next time your code uses that LockedNode to CommitAsync, release or abandon it, the call throws NodeLockExpiredException instead of writing anything. None of the changes made on that LockedNode were saved.

The exception data contains maxWait (the limit that was exceeded), UID (the node and its type) and LockedBy (the code that took the lock).

The graph force-releases a lock in two situations:

• The lock was held too long: the graph's background maintenance, which runs about every 15 seconds, releases every lock held for longer than ten times the maximum lock wait (about 100 seconds with the default settings). The workspace log records Found possibly deadlocked node <uid> of type <type> that was locked for <time> (removing any changes from node cache & restoring the node from storage).
• The workspace is shutting down: saving the graph to storage on shutdown releases every lock that is still held.

Common causes

1. Slow work while holding a lock: an external API call, an LLM request, file extraction, or a long loop between GetOrAddLockedAsync and CommitAsync.
2. Holding locks across a whole batch: locking every node of a large batch first and committing them all at the end, so the earliest locks are held for the length of the whole batch.
3. A lock that is never released, and code that comes back to the same LockedNode much later, for example a LockedNode kept in a field or a cache.

What to do

• Treat the write as failed. The changes are gone, so a retry has to lock the node again and redo the changes; calling CommitAsync again on the same LockedNode throws again.
• Keep the time between locking and committing short: do the slow work before you lock, then lock, change, and commit straight away.
• For large batches, commit in chunks rather than holding hundreds of locks until the end.
• Never keep a LockedNode beyond the method that locked it. Read nodes with queries, and lock only at the point of writing.
• A NodeLockExpiredException usually has NodeDeadlockException errors close by in the log: other writers that timed out waiting for the same node while it was held. Fixing the long-held lock fixes both.

Avoiding lock problems

• Consistent Ordering: If you need to lock multiple nodes, lock them in one call, or always sort them by their UID before acquiring locks.

  // CORRECT: Consistent order prevents deadlocks
  var uidsToLock = new[] { uidA, uidB }.OrderBy(u => u).ToList();
  // Lock in this order
  

  // WRONG: Locking in arbitrary order
  await Graph.GetOrAddLockedAsync(uidA);
  // ... some work ...
  await Graph.GetOrAddLockedAsync(uidB); // Risk of deadlock if another thread locked B then A
  

• Lock Late, Commit Early: Acquire locks immediately before you need to modify data, and call CommitAsync as soon as the modification is done.

• Avoid Expensive Operations Inside Locks: Do not perform network requests or heavy calculations while holding a lock.

  // WRONG
  var node = await Graph.GetOrAddLockedAsync(uid);
  var data = await CallSlowExternalApi(); // Slow! Blocks other threads needing 'node'
  node.UpdateProperty("data", data);
  await Graph.CommitAsync(node);
  
  // CORRECT
  var data = await CallSlowExternalApi(); // Do slow work first
  var node = await Graph.GetOrAddLockedAsync(uid); // Lock briefly
  node.UpdateProperty("data", data);
  await Graph.CommitAsync(node);
  

Uncommitted Nodes

One of the most common errors is forgetting to commit a locked node.

var node = await Graph.GetOrAddLockedAsync(uid);
node.UpdateProperty("status", "processing");

// ... exception thrown here ...

// Commit is never reached! The node stays locked until the lock is force-released.
await Graph.CommitAsync(node);

Fix: Use try/finally or ensure code paths are safe. If an endpoint, AI tool or scheduled task finishes while a node is still locked, the scoped Graph releases the lock, discards the changes, and throws DeadlockHadToBePreventedException. Until then, every other writer to that node waits, and may receive NodeDeadlockException. Code that bypasses the scoped Graph gets no such clean-up: its lock is held until the graph force-releases it (see NodeLockExpiredException).

Performance Optimization

Curiosity's GraphDB is an in-memory graph database, but understanding its performance characteristics is crucial for writing efficient code.

Edge Traversal is Fast

Relationships are stored as direct memory pointers. Operations like Out(), IsRelatedTo(), and SortByConnectivity() are extremely fast because they traverse these pointers without deserializing data.

Property Access can be Slower

Reading a property value (e.g., node.GetString("name")) involves deserializing the node's content blob. While fast, doing this for millions of nodes in a loop is significantly slower than edge traversal.

Implication:

• Avoid Where(n => n.GetString("status") == "Active") if possible. This forces the engine to load and deserialize every node to check the property.
• Instead, use WhereString("status", "Active"). This uses internal indexes to filter UIDs before loading the content.

Indexing

• Exact Match: Use StartWhereString or WhereString to leverage hash indexes for exact value lookups.
• Full Text: Use StartSearch for fuzzy matching or relevance-based search. This uses the inverted index.

Large Result Sets

If a query returns thousands of nodes:

• Don't use ToList() immediately if you don't need all data in memory.
• Do use AsEnumerableAsync() to stream results.
• Do use Take() and Skip() for pagination.

Common Exceptions