Enum gf::pivot::tokenizer::TokenizerResult

pub enum TokenizerResult {
    Accept(VarSize),
    Reject,
    TentativeReject(VarSize),
}

🟢 Result of a tokenizer instance invocation.

Variants

Accept(VarSize)

A prefix of the input of given length was accepted.

Reject

Input was rejected.

TentativeReject(VarSize)

Input was rejected, but at least one pattern failed to match because of a short input. The associated value contains the minimum number of bits to append to the input before retrying.

Trait Implementations

impl Clone for TokenizerResult

fn clone(&self) -> TokenizerResult

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for TokenizerResult

impl Debug for TokenizerResult

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenizerResult

impl Hash for TokenizerResult

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H) where
    H: Hasher, 

Feeds a slice of this type into the given Hasher.

impl PartialEq<TokenizerResult> for TokenizerResult

fn eq(&self, other: &TokenizerResult) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &TokenizerResult) -> bool

This method tests for !=.

impl StructuralEq for TokenizerResult

impl StructuralPartialEq for TokenizerResult