tooling.Tokenizer

Undocumented in source.

Members

Aliases

CppLexer alias CppLexer = TokenizerGenerator!([NonAlphaTokens, Keywords], specialTokens): Undocumented in source.
Keywords alias Keywords = TypeTuple!("and", "and_eq", "asm", "auto", "bitand", "bitor", "bool", "break", "case", "catch", "char", "class", "compl", "const", "const_cast", "constexpr", "continue", "default", "delete", "do", "double", "dynamic_cast", "else", "enum", "explicit", "extern", "false", "float", "for", "friend", "goto", "if", "inline", "int", "long", "mutable", "namespace", "new", "noexcept", "not", "not_eq", "operator", "or", "or_eq", "private", "protected", "public", "register", "reinterpret_cast", "return", "short", "signed", "sizeof", "static", "static_cast", "struct", "switch", "template", "this", "throw", "true", "try", "typedef", "typeid", "typename", "union", "unsigned", "using", "virtual", "void", "volatile", "wchar_t", "while", "xor", "xor_eq"): Undocumented in source.
NonAlphaTokens alias NonAlphaTokens = TypeTuple!("~", "(", ")", "[", "]", "{", "}", ";", ",", "?", "<", "<=", "<<", "<<=", ">", ">=", ">>", ">>=", "%", "%=", "=", "==", "!", "!=", "^", "^=", "*", "*=", ":", "::", "+", "++", "+=", "&", "&&", "&=", "|", "||", "|=", "-", "--", "-=", "->", "->*", "/", "/=", "//", "/*", "\\", ".", ".*", "...", "'", "\"", "#", "##", "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "@", "$", "R\"("): Undocumented in source.
Token alias Token = CppLexer.Token: Undocumented in source.
TokenType alias TokenType = CppLexer.TokenType2: Undocumented in source.
tk alias tk = CppLexer.tk: Undocumented in source.

Functions

nextToken CppLexer.Token nextToken(string pc, size_t line): Helper function, gets next token and updates pc and line.
tokenize CppLexer.Token[] tokenize(string input, string initialFilename): This is the quintessential function. Given a string containing C++ code and a filename, fills output with the tokens in the file. Warning - don't use temporaries for input and filename because the resulting tokens contain StringPiece objects pointing into them.
tokenize void tokenize(string input, string initialFilename, CppLexer.Token[] t): Undocumented in source. Be warned that the author may not have intended to support it.

Static functions

munchCharLiteral string munchCharLiteral(string pc, size_t line): Assuming pc is positioned at the start of a character literal, munches it from pc and returns it. A reference to line is passed in order to track multiline character literals (yeah, that can actually happen) correctly.
munchChars string munchChars(string pc, size_t howMany): Eats howMany characters out of pc, avances pc appropriately, and returns the eaten portion.
munchComment string munchComment(string pc, size_t line): Assuming pc is positioned at the start of a C-style comment, munches it from pc and returns it.
munchIdentifier string munchIdentifier(string pc): Assuming pc is positioned at the start of an identifier, munches it from pc and returns it.
munchNumber string munchNumber(string pc): Assuming pc is positioned at the start of a number (be it decimal or floating-point), munches it off pc and returns it. Note that the number is assumed to be correct so a number of checks are not necessary.
munchPreprocessorDirective string munchPreprocessorDirective(string pc, size_t line): Assuming pc is positioned at the start of a specified preprocessor directive, munches it from pc and returns it.
munchRawString string munchRawString(string pc, size_t line): Assuming pc is positioned at the start og a raw string, munches it from pc and returns it.
munchSingleLineComment string munchSingleLineComment(string pc, size_t line): Assuming pc is positioned at the start of a single-line comment, munches it from pc and returns it.
munchString string munchString(string pc, size_t line): Assuming pc is positioned at the start of a string literal, munches it from pc and returns it. A reference to line is passed in order to track multiline strings correctly.

Static variables

specialTokens string[] specialTokens;: Undocumented in source.

Structs

TokenizerGenerator struct TokenizerGenerator(alias tokens, alias reservedTokens): Undocumented in source.

tooling.Tokenizer

Members

Aliases

Functions

Static functions

Static variables

Structs

Meta

Source