1. class
  2. Hypervel
  3. \
  4. SchemaParser
  5. \
  6. Parsing
  7. \
  8. PostgresLexer

PostgresLexer

class PostgresLexer extends Lexer

internal  
 

Lex plain-text pg_dump output with psql's statement boundaries.

psql ends a statement at a semicolon outside parentheses and outside the BEGIN ... END body of a SQL-standard routine, treats backslash commands as its own, and reads COPY FROM STDIN data through the \. line.

Constants

protected HIGH_BYTES

Bytes of multibyte characters, which unquoted identifiers may contain.
protected LETTERS

protected DIGITS

protected WHITESPACE

protected COMMENT_STARTS

private WINDOW_LENGTH

The most source that runs are matched against at once.
private WINDOW_CUTS

The bytes a window can be cut before, as no unquoted token contains them.
private IDENTIFIER_BYTES

private DOLLAR_TAG_BYTES

private SKIP_STOPS

The bytes where skipping must stop, as they can begin a quoted token, a comment or a boundary.
private SPACE

Whitespace and line comments before a token.

Repetitions in the native patterns are bounded so one match stays within PCRE's limits; longer input stops the run and is read by the scanner.
private WORD

A word that is not a string prefix and does not run into a string.
private NUMBER

private QUOTED_IDENTIFIER

A quoted identifier whose closing quote does not begin a doubled one.
private OPERATOR

An operator, which cannot contain the start of a comment.
private PUNCTUATION

Single-byte punctuation other than parentheses.
private STRING_END

The closing quote of a string literal, not the first of a doubled one, followed by more on its line so it cannot continue on a later one.
private STANDARD_STRING

A plain string literal where backslashes are ordinary characters.
private ESCAPED_STRING

A plain string literal where backslashes are escapes.
private STANDARD_RUN

A run of tokens and standard strings, read in one native call through a semicolon or up to a token that needs boundary handling.
private ESCAPED_RUN

A run of tokens and escaped strings.
private SPACE_END

Whitespace and line comments, matched to report where they end.
private TOKEN_END

A number, operator or punctuation, matched to report where it ends.
private LEADING_WORDS

The letters psql records for the leading words that identify routines and COPY FROM STDIN.
private ROUTINE_STARTS

The leading words of CREATE [OR REPLACE] FUNCTION and PROCEDURE, whose bodies can contain semicolons.
private ROUTINE_PREFIXES

The leading words that can still become a routine start.

Properties

protected int $length from  Lexer
protected int $position from  Lexer
protected int $statementStart from  Lexer
protected int $tokenStart

The start of the token or terminator most recently scanned.

 from  Lexer
protected bool $ended from  Lexer
protected int $windowStart from  Lexer
bool $standardConformingStrings

Whether ordinary string literals treat backslashes literally.

Methods

__construct(string $sql)

Create a lexer, skipping a UTF-8 byte order mark as psql does.

TokenStream|null
nextStatement()

Begin the next non-empty statement, or return null at the end of the input.

from  Lexer
void
read(array $types, array $starts, array $ends, array $texts)

Append a run of regular tokens, or one token that needs boundary handling.

TokenType
type(string $text)

Get the type of a token read in a run, which follows from its upper-case text.

array|null
matchRun(string $pattern)

Match a run of tokens at the position, or return null where the scanner must read the next token.

from  Lexer
array
skipStatement()

Skip the rest of the current statement and return the offsets of its terminator.

from  Lexer
int|null
skipGroup(int $depth)

Skip to the parenthesis that closes the given number of open groups and return its start.

string
value(TokenType $type, int $start, int $end)

Decode a token's value.

from  Lexer
int
line(int $offset)

Get the one-based line containing a source offset.

from  Lexer
ParseException
error(string $message, int $offset)

Create a parse error located at a source offset.

from  Lexer
TokenType
scan()

Scan one token or the statement terminator.

from  Lexer
void
finishStatement()

Mark the current statement as ended at its terminator.

from  Lexer
TokenType
scanToken()

Scan the token starting at the current position, or a terminator as End.

void
skip()

Skip to the terminator.

int
terminator()

Get the length of a statement terminator at the current position, or zero.

from  Lexer
bool
skipCommand()

Consume a client command between statements.

from  Lexer
void
startStatement()

Reset the boundary state tracked within one statement.

void
endStatement()

Skip the data of a COPY FROM STDIN statement through its \. line.

void
skipWhitespace()

Skip whitespace and comments.

bool
skipComment()

Skip comments, including nested block comments and psql backslash commands.

void
skipLine()

Skip to the start of the next line.

from  Lexer
void
scanQuoted(string $quote, bool $backslashes, string $description)

Scan a quoted run whose opening quote is at the position, escaped by doubling and optionally backslashes.

from  Lexer
string
word(int $start, int $end)

Fold an unquoted identifier or keyword to lower case.

string
identifier(int $start, int $end)

Decode a quoted identifier, including Unicode escapes.

string
string(TokenType $type, int $start, int $end)

Decode any PostgreSQL string literal form.

Details

at line 107
__construct(string $sql)

Create a lexer, skipping a UTF-8 byte order mark as psql does.

Parameters

string $sql

in Lexer at line 77
TokenStream|null nextStatement()

Begin the next non-empty statement, or return null at the end of the input.

Return Value

TokenStream|null

at line 117
void read(array $types, array $starts, array $ends, array $texts)

Append a run of regular tokens, or one token that needs boundary handling.

Parameters

array $types
array $starts
array $ends
array $texts

Return Value

void

at line 164
TokenType type(string $text)

Get the type of a token read in a run, which follows from its upper-case text.

Parameters

string $text

Return Value

TokenType

in Lexer at line 149
protected array|null matchRun(string $pattern)

Match a run of tokens at the position, or return null where the scanner must read the next token.

Dialects read runs of ordinary tokens with one native match each, so only tokens that need boundary handling go through the scanner. A statement's first token is matched alone, as parsers skip most statements after it.

Parameters

string $pattern

Return Value

array|null

the upper-case texts and their offsets from $windowStart

in Lexer at line 194
array skipStatement()

Skip the rest of the current statement and return the offsets of its terminator.

Return Value

array

at line 184
int|null skipGroup(int $depth)

Skip to the parenthesis that closes the given number of open groups and return its start.

Returns null when the statement ends first.

Parameters

int $depth

Return Value

int|null

in Lexer at line 226
string value(TokenType $type, int $start, int $end)

Decode a token's value.

Parameters

TokenType $type
int $start
int $end

Return Value

string

in Lexer at line 239
int line(int $offset)

Get the one-based line containing a source offset.

Parameters

int $offset

Return Value

int

in Lexer at line 247
ParseException error(string $message, int $offset)

Create a parse error located at a source offset.

Parameters

string $message
int $offset

Return Value

ParseException

in Lexer at line 255
protected TokenType scan()

Scan one token or the statement terminator.

Return Value

TokenType

in Lexer at line 272
protected void finishStatement()

Mark the current statement as ended at its terminator.

Return Value

void

at line 192
protected TokenType scanToken()

Scan the token starting at the current position, or a terminator as End.

Return Value

TokenType

at line 237
protected void skip()

Skip to the terminator.

Return Value

void

in Lexer at line 294
protected int terminator()

Get the length of a statement terminator at the current position, or zero.

Return Value

int

in Lexer at line 302
protected bool skipCommand()

Consume a client command between statements.

Return Value

bool

at line 308
protected void startStatement()

Reset the boundary state tracked within one statement.

Return Value

void

at line 320
protected void endStatement()

Skip the data of a COPY FROM STDIN statement through its \. line.

Return Value

void

at line 359
protected void skipWhitespace()

Skip whitespace and comments.

Return Value

void

at line 377
protected bool skipComment()

Skip comments, including nested block comments and psql backslash commands.

Return Value

bool

in Lexer at line 363
protected void skipLine()

Skip to the start of the next line.

Return Value

void

in Lexer at line 373
protected void scanQuoted(string $quote, bool $backslashes, string $description)

Scan a quoted run whose opening quote is at the position, escaped by doubling and optionally backslashes.

Parameters

string $quote
bool $backslashes
string $description

Return Value

void

at line 431
protected string word(int $start, int $end)

Fold an unquoted identifier or keyword to lower case.

Parameters

int $start
int $end

Return Value

string

at line 439
protected string identifier(int $start, int $end)

Decode a quoted identifier, including Unicode escapes.

Parameters

int $start
int $end

Return Value

string

at line 453
protected string string(TokenType $type, int $start, int $end)

Decode any PostgreSQL string literal form.

Parameters

TokenType $type
int $start
int $end

Return Value

string