# The Document Class

The Document class is the primary data structure in Catalyst. It represents the text being processed and stores all the linguistic annotations generated by the NLP pipeline.

You can create a new document by providing the raw text and its language.

using Catalyst;
using Mosaik.Core;

var doc = new Document("The quick brown fox jumps over the lazy dog", Language.English);

• Value: The original raw text of the document.
• Language: The language of the document.
• UID: A unique identifier for the document.
• Metadata: A dictionary for storing arbitrary key-value pairs associated with the document.
• Labels: A list of string labels (e.g., for document classification).
• Spans: An enumerable of all Span objects (sentences) in the document.
• SpansCount: The total number of spans in the document.
• TokensCount: The total number of tokens across all spans.
• EntitiesCount: The total number of recognized entities in the document.
• IsParsed: A boolean indicating if the document has been tokenized.

Returns the tokenized text of the document. You can optionally merge recognized entities into single tokens.

string tokenized = doc.TokenizedValue(mergeEntities: true);

Flattens all tokens from all spans into a single list of IToken objects.

List<IToken> allTokens = doc.ToTokenList();

Manually adds a span to the document by specifying the start and end character indices.

var span = doc.AddSpan(0, 10);

Allows you to generate a new string where recognized entities are replaced based on a custom function.

string anonymized = doc.ToStringWithReplacements(entity =>
{
    if (entity.EntityType.Type == "Person") return "[REDACTED]";
    return null; // Keep original
});

Removes all tokens and spans from the document, but keeps the raw text and metadata.

doc.Clear();

Catalyst provides several ways to save and load documents.

You can easily convert a document to and from a JSON string.

// Serialize to JSON
string json = doc.ToJson();

// Deserialize from JSON
Document doc2 = Document.FromJson(json);

For high-performance scenarios, Catalyst supports binary serialization using MessagePack. This is often used internally when storing models or large corpora.

The ImmutableDocument class provides an immutable, memory-efficient representation of a document. It is useful for scenarios where you want to ensure the document data is not changed after processing.

// Convert to ImmutableDocument
ImmutableDocument immutableDoc = doc.ToImmutable();

// Convert back to mutable Document
Document mutableDoc = immutableDoc.ToMutable();