[][src]Module gf::pivot::tokenizer

🟢 Tokenizers.

Structs

Tokenizer

🟢 Tokenizer in a sealed module.

TokenizerId

⚪ Tokenizer identifier.

TokenizerInstance

🟢 Instance of a tokenizer.

TokenizerSealingData

⚪ Data collected for a tokenizer during sealing.

UnsealedTokenizer

⚪ Tokenizer in an unsealed module.

UnsealedTokenizerRule

⚪ Rule in an unsealed tokenizer.

Enums

TokenizerResult

🟢 Result of a tokenizer instance invocation.

UnsealedTokenCtor

⚪ Token constructor in an unsealed tokenizer rule.

UnsealedTokenizerPat

⚪ Pattern in an unsealed tokenizer rule.

Functions

call_tokenizer

🟢 Call a tokenizer.