Pattern Spotter

Pattern Spotter models capture entities based on complex text patterns, similar to advanced regular expressions.

Defining Patterns

Patterns use a proprietary syntax to match sequences of tokens, token types, and shapes. The pattern editor carries a cheatsheet covering the whole syntax — the document structure, the token rules, the shape characters, the character flags and the modifiers — and its completion and hover give the same information as you type.

Inside one pattern, each non-empty line is a separate alternative and the matcher keeps the longest match; a blank line ends the pattern and starts a new one.

Shapes

A shape: rule matches a token by its character shape: every character of the token is reduced to one shape character, and the result must equal your shape character for character. Separate alternatives with commas.

shape:XX999      matches AB123
shape:XXXX       matches NASA
shape:Xx         matches Go
shape:$999       matches €100 and $100

Your shape is reduced the same way the token is, so a literal example is also a valid shape — shape:$999 works because $ reduces to #. The corollary is that a literal # counts as punctuation and reduces to ., so write a real symbol rather than # for such a position.

Use _ when any token will do at that position.

Testing and Experiments

Before deploying, use the Experiment and Capture interface to test your patterns against your actual data. Running an experiment allows you to see what would be captured without modifying the graph.

Deployment

If the patterns are useful, create a new node type (e.g., PotentialIdentifiers) to store the results and add the pattern spotter to your pipeline.