TokenStream
class TokenStream
| internal |
The tokens of one statement, read in runs and never past its terminator.
A token is its source offset and upper-case source text; its type is derived from the text when first needed unless the lexer recorded it. Tokens that were read are kept, so a parser can look behind the current one or rewind. Source skipped by skipGroup() or skipRest() is never read, so it cannot be replayed.
Constants
| private OPERATORS |
Keyword operators that take an operand after them, in the expressions that clause keywords can end. |
Methods
Consume the current token.
Get the type of the current token or one after it.
Get the source offset of the current token or one after it.
Get the upper-case keyword of the current token or one after it, or null when it is not a word.
Get the upper-case source text of the current token or one after it, or an empty string for the terminator.
Determine whether the next tokens are the given sequence of keywords, without consuming them.
Determine whether the current token, or one before or after it, is the given punctuation.
Consume a sequence of keywords when all of them are next.
Consume the current token and get its decoded value when it is a word or quoted identifier.
Consume the given punctuation when it is next.
Consume required punctuation.
Consume a group from its opening parenthesis, the current token, through the matching parenthesis.
Consume an expression that ends before a top-level comma, an unmatched closing bracket or one of the given keywords.
Get a mark of the current position to return to.
Return to a marked position within the statement.
Determine whether the statement has no more tokens.
Get the decoded value of the current token or one after it.
Get the source text from an offset through the most recently consumed token.
Skip the unread remainder of the statement without reading its tokens.
Create a parse error located at the current token or one after it.
Details
at line 58
__construct(Lexer $lexer, int $start)
Begin reading a statement at its first token.
at line 68
void
next()
Consume the current token.
at line 80
TokenType
type(int $ahead = 0)
Get the type of the current token or one after it.
at line 92
int
offset(int $ahead = 0)
Get the source offset of the current token or one after it.
at line 102
string|null
keyword(int $ahead = 0)
Get the upper-case keyword of the current token or one after it, or null when it is not a word.
at line 114
string
text(int $ahead = 0)
Get the upper-case source text of the current token or one after it, or an empty string for the terminator.
at line 124
bool
is(string $keyword, string ...$following)
Determine whether the next tokens are the given sequence of keywords, without consuming them.
at line 134
bool
isSymbol(string $symbol, int $ahead = 0)
Determine whether the current token, or one before or after it, is the given punctuation.
at line 144
bool
accept(string $keyword, string ...$following)
Consume a sequence of keywords when all of them are next.
at line 161
string|null
acceptName()
Consume the current token and get its decoded value when it is a word or quoted identifier.
at line 181
bool
acceptSymbol(string $symbol)
Consume the given punctuation when it is next.
at line 195
void
expectSymbol(string $symbol, string $description)
Consume required punctuation.
at line 207
void
skipGroup()
Consume a group from its opening parenthesis, the current token, through the matching parenthesis.
at line 249
bool
skipExpression(array $keywords, bool $afterOperand = false)
Consume an expression that ends before a top-level comma, an unmatched closing bracket or one of the given keywords.
Brackets and CASE ... END nest. A keyword ends the expression only after a complete operand, so DEFAULT NULL keeps its NULL and 1 + NULL keeps both operands, and a word before a qualification dot is a name. Returns whether anything was consumed.
at line 348
int
mark()
Get a mark of the current position to return to.
at line 358
void
rewind(int $mark)
Return to a marked position within the statement.
A group consumed by skipGroup() keeps only its brackets as tokens, so a mark cannot be reused across one.
at line 366
bool
atEnd()
Determine whether the statement has no more tokens.
at line 376
string
value(int $ahead = 0)
Get the decoded value of the current token or one after it.
at line 388
string
spanFrom(int $start)
Get the source text from an offset through the most recently consumed token.
at line 398
void
skipRest()
Skip the unread remainder of the statement without reading its tokens.
at line 415
ParseException
error(string $message, int $ahead = 0)
Create a parse error located at the current token or one after it.