The full Python source text to tokenize. Line endings
are normalized to \n first (CPython's universal-newline handling —
\r\n and lone \r both become \n, including inside string
literals), so indentation tracking isn't thrown off by \r and
lineno/col_offset land where CPython puts them regardless of the
source file's original line-ending style.
Tokenizes the source text passed to the constructor.
Resets all internal lexer state first, so the same Lexer
instance can be safely re-tokenized by calling this method again.
Emits any trailing DEDENT tokens needed to close open indentation
levels, followed by a final EOF token.
The complete list of tokens, terminated by an EOF token.
Tokenizes Python source code into a flat array of Tokens.
Handles Python's significant-whitespace grammar (emitting synthetic
INDENT/DEDENT/NEWLINEtokens), string literals (plain, prefixed, triple-quoted, and f-strings), numeric literals (decimal, hex, octal, binary, float, scientific notation, complex), comments, line continuations (`` followed by a newline), and all Python operators and delimiters.