Lexer
abstract class Lexer
| internal |
Split SQL into statements and tokens located in the original source.
Reading and skipping share one scanner. Tokens are only created as a parser reads them, so the unread remainder of a statement, and the body of a group whose source text is all a parser needs, cost a scan.
Constants
| protected HIGH_BYTES |
Bytes of multibyte characters, which unquoted identifiers may contain. |
| protected LETTERS |
|
| protected DIGITS |
|
| protected WHITESPACE |
|
| protected COMMENT_STARTS |
Bytes that can begin a comment, so skipComment() is only called where one may start. |
| private WINDOW_LENGTH |
The most source that runs are matched against at once. |
| private WINDOW_CUTS |
The bytes a window can be cut before, as no unquoted token contains them. |
Properties
| protected int | $length | ||
| protected int | $position | ||
| protected int | $statementStart | ||
| protected int | $tokenStart | The start of the token or terminator most recently scanned. |
|
| protected bool | $ended | ||
| protected int | $windowStart |
Methods
Create a lexer over the complete SQL input.
Begin the next non-empty statement, or return null at the end of the input.
Append the next tokens of the current statement, through its terminator at most.
Get the type of a token appended without one, which follows from its upper-case text.
Match a run of tokens at the position, or return null where the scanner must read the next token.
Skip the rest of the current statement and return the offsets of its terminator.
Skip to the parenthesis that closes the given number of open groups and return its start.
Decode a token's value.
Get the one-based line containing a source offset.
Create a parse error located at a source offset.
Scan one token or the statement terminator.
Mark the current statement as ended at its terminator.
Scan the token starting at the current position, or a terminator as End.
Scan to the end of the current statement.
Get the length of a statement terminator at the current position, or zero.
Consume a client command between statements.
Reset state tracked within one statement.
Handle input that belongs to a statement after its terminator.
Skip whitespace and comments.
Skip a comment at the current position.
Skip to the start of the next line.
Scan a quoted run whose opening quote is at the position, escaped by doubling and optionally backslashes.
Normalize an unquoted identifier or keyword.
Decode a quoted identifier.
Decode a string literal whose quotes are escaped by doubling.
Details
at line 67
__construct(string $sql)
Create a lexer over the complete SQL input.
at line 77
TokenStream|null
nextStatement()
Begin the next non-empty statement, or return null at the end of the input.
at line 118
void
read(array $types, array $starts, array $ends, array $texts)
Append the next tokens of the current statement, through its terminator at most.
Each token is appended with its start offset and upper-case source text; a token scanned on its own also records its type. A terminator has an empty text and its end recorded, as its span can include COPY data.
at line 138
abstract TokenType
type(string $text)
Get the type of a token appended without one, which follows from its upper-case text.
Tokens whose type depends on the lexer's state, such as string literals, always have it recorded.
at line 149
protected array|null
matchRun(string $pattern)
Match a run of tokens at the position, or return null where the scanner must read the next token.
Dialects read runs of ordinary tokens with one native match each, so only tokens that need boundary handling go through the scanner. A statement's first token is matched alone, as parsers skip most statements after it.
at line 194
array
skipStatement()
Skip the rest of the current statement and return the offsets of its terminator.
at line 208
int|null
skipGroup(int $depth)
Skip to the parenthesis that closes the given number of open groups and return its start.
Returns null when the statement ends first.
at line 226
string
value(TokenType $type, int $start, int $end)
Decode a token's value.
at line 239
int
line(int $offset)
Get the one-based line containing a source offset.
at line 247
ParseException
error(string $message, int $offset)
Create a parse error located at a source offset.
at line 255
protected TokenType
scan()
Scan one token or the statement terminator.
at line 272
protected void
finishStatement()
Mark the current statement as ended at its terminator.
at line 281
abstract protected TokenType
scanToken()
Scan the token starting at the current position, or a terminator as End.
at line 286
protected void
skip()
Scan to the end of the current statement.
at line 294
protected int
terminator()
Get the length of a statement terminator at the current position, or zero.
at line 302
protected bool
skipCommand()
Consume a client command between statements.
at line 310
protected void
startStatement()
Reset state tracked within one statement.
at line 317
protected void
endStatement()
Handle input that belongs to a statement after its terminator.
at line 324
protected void
skipWhitespace()
Skip whitespace and comments.
at line 334
protected bool
skipComment()
Skip a comment at the current position.
at line 363
protected void
skipLine()
Skip to the start of the next line.
at line 373
protected void
scanQuoted(string $quote, bool $backslashes, string $description)
Scan a quoted run whose opening quote is at the position, escaped by doubling and optionally backslashes.
at line 402
protected string
word(int $start, int $end)
Normalize an unquoted identifier or keyword.
at line 410
protected string
identifier(int $start, int $end)
Decode a quoted identifier.
at line 421
protected string
string(TokenType $type, int $start, int $end)
Decode a string literal whose quotes are escaped by doubling.