Mirrors CPython's tokenize module categories: literals, keywords,
operators/delimiters, comparison and assignment operators, and the
structural tokens (NEWLINE/INDENT/DEDENT/EOF) used to represent
Python's significant whitespace. F-string specific values
(FSTRING_START/FSTRING_MIDDLE/FSTRING_END) are reserved for future
use; this lexer currently emits whole f-strings as a single STRING
token (see Lexer.scanInterpolatedStringLiteral).
The kind of a lexical token produced by Lexer.
Mirrors CPython's
tokenizemodule categories: literals, keywords, operators/delimiters, comparison and assignment operators, and the structural tokens (NEWLINE/INDENT/DEDENT/EOF) used to represent Python's significant whitespace. F-string specific values (FSTRING_START/FSTRING_MIDDLE/FSTRING_END) are reserved for future use; this lexer currently emits whole f-strings as a singleSTRINGtoken (see Lexer.scanInterpolatedStringLiteral).