TokenizerGenerator

Undocumented in source.

struct TokenizerGenerator (

alias tokens

alias reservedTokens

) {

alias TokenIDRep = ubyte;

alias TokenIDRep = ushort;

alias TokenIDRep = uint;

struct TokenType2;

template tk(string symbol);

static string generateCases(string[] tokens, size_t index, bool* mayFallThrough);

static Tuple!(size_t, size_t, TokenType2) match(R r);

}

Members

Aliases

TokenIDRep alias TokenIDRep = ubyte: Representation for token types.
TokenIDRep alias TokenIDRep = ushort: Undocumented in source.
TokenIDRep alias TokenIDRep = uint: Undocumented in source.

Static functions

generateCases string generateCases(string[] tokens, size_t index, bool* mayFallThrough): Undocumented in source. Be warned that the author may not have intended to support it.
match Tuple!(size_t, size_t, TokenType2) match(R r): Undocumented in source. Be warned that the author may not have intended to support it.

Structs

Token struct Token: Defines one token together with file and line information. The precedingComment_ is set if there was one comment before the token.
TokenType2 struct TokenType2: Undocumented in source.

Templates

tk template tk(string symbol): Undocumented in source.

Meta

Source

See Implementation

tooling Tokenizer aliases

functions

static functions

static variables

specialTokens

structs

TokenizerGenerator