Module gf::pivot::tokenizer

🟢 Tokenizers.

Structs

Tokenizer	🟢 Tokenizer in a sealed module.
TokenizerId	⚪ Tokenizer identifier.
TokenizerInstance	🟢 Instance of a tokenizer.
TokenizerSealingData	⚪ Data collected for a tokenizer during sealing.
UnsealedTokenizer	⚪ Tokenizer in an unsealed module.
UnsealedTokenizerRule	⚪ Rule in an unsealed tokenizer.

Enums

TokenizerResult	🟢 Result of a tokenizer instance invocation.
UnsealedTokenCtor	⚪ Token constructor in an unsealed tokenizer rule.
UnsealedTokenizerPat	⚪ Pattern in an unsealed tokenizer rule.

Functions

call_tokenizer

🟢 Call a tokenizer.