IndentationError in new repl

[ronaldoussoren]

Bug report

Bug description:

Python version: Python 3.14.0a7+ (heads/main-dirty:8d0e07eb899.

To reproduce type the following class definition in the REPL:

class C:
    def
    m

Then set the cursor to the space before m and press backspace to correct the incorrect newline.

Behaviour: the REPL exits with an IndentationError.

Typing backspace works correctly when the cursor is on m.

The exit/crash does not happen in 3.14a7.

CPython versions tested on:

3.14, CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-133541: Handle SyntaxError raised by the tokenizer on user input #133606
• [3.14] gh-133541: Handle SyntaxError raised by the tokenizer on user input (GH-133606) #133709

[terryjreedy]

Win 10: with a space between the cursor and 'm', so 3 spaces to left of cursor
IndentationError: unindent does not match any outer indentation level

[ronaldoussoren]

Full traceback:

Python 3.14.0a7+ (heads/main-dirty:8d0e07eb899, May  6 2025, 18:45:02) [Clang 17.0.0 (clang-1700.0.13.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class C:
...     def
...    Traceback (most recent call last):
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
                     "__main__", mod_spec)
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/runpy.py", line 88, in _run_code
    exec(code, run_globals)
    ~~~~^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/_pyrepl/__main__.py", line 10, in <module>
    __pyrepl_interactive_console()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/_pyrepl/main.py", line 59, in interactive_console
    run_multiline_interactive_console(console)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/_pyrepl/simple_interact.py", line 138, in run_multiline_interactive_console
    statement = multiline_input(more_lines, ps1, ps2)
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/_pyrepl/readline.py", line 393, in multiline_input
    return reader.readline()
           ~~~~~~~~~~~~~~~^^
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/_pyrepl/reader.py", line 751, in readline
    self.handle1()
    ~~~~~~~~~~~~^^
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/_pyrepl/reader.py", line 734, in handle1
    self.do_cmd(cmd)
    ~~~~~~~~~~~^^^^^
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/_pyrepl/reader.py", line 664, in do_cmd
    self.refresh()
    ~~~~~~~~~~~~^^
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/_pyrepl/reader.py", line 641, in refresh
    self.screen = self.calc_screen()
                  ~~~~~~~~~~~~~~~~^^
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/_pyrepl/completing_reader.py", line 261, in calc_screen
    screen = super().calc_screen()
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/_pyrepl/reader.py", line 315, in calc_screen
    colors = list(gen_colors(self.get_unicode()))
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/_pyrepl/utils.py", line 97, in gen_colors
    for color in gen_colors_from_token_stream(gen, line_lengths):
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/_pyrepl/utils.py", line 155, in gen_colors_from_token_stream
    for prev_token, token, next_token in token_window:
                                         ^^^^^^^^^^^^
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/_pyrepl/utils.py", line 348, in prev_next_window
    for x in iterator:
             ^^^^^^^^
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/tokenize.py", line 586, in _generate_tokens_from_c_tokenizer
    raise e from None
  File "/Library/Frameworks/PythonDev.framework/Versions/3.14/lib/python3.14/tokenize.py", line 582, in _generate_tokens_from_c_tokenizer
    for info in it:
                ^^
  File "<string>", line 3
    method(self): pass
                      ^
IndentationError: unindent does not match any outer indentation level

[StanFromIreland]

I can reproduce but with a varied traceback.

Python 3.14.0a7+ (heads/main:3dfed230928, May  6 2025, 17:44:25) [GCC 15.1.1 20250425 (Red Hat 15.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class C:
...     def
...    Traceback (most recent call last):
  File "/usr/local/lib/python3.14/_pyrepl/readline.py", line 393, in multiline_input
    return reader.readline()
           ~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.14/_pyrepl/reader.py", line 748, in readline
    self.handle1()
    ~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.14/_pyrepl/reader.py", line 731, in handle1
    self.do_cmd(cmd)
    ~~~~~~~~~~~^^^^^
  File "/usr/local/lib/python3.14/_pyrepl/reader.py", line 661, in do_cmd
    self.refresh()
    ~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.14/_pyrepl/reader.py", line 638, in refresh
    self.screen = self.calc_screen()
                  ~~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.14/_pyrepl/completing_reader.py", line 261, in calc_screen
    screen = super().calc_screen()
  File "/usr/local/lib/python3.14/_pyrepl/reader.py", line 315, in calc_screen
    colors = list(gen_colors(self.get_unicode()))
  File "/usr/local/lib/python3.14/_pyrepl/utils.py", line 102, in gen_colors
    for color in gen_colors_from_token_stream(gen, line_lengths):
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.14/_pyrepl/utils.py", line 160, in gen_colors_from_token_stream
    for prev_token, token, next_token in token_window:
                                         ^^^^^^^^^^^^
  File "/usr/local/lib/python3.14/_pyrepl/utils.py", line 357, in prev_next_window
    for x in iterator:
             ^^^^^^^^
  File "/usr/local/lib/python3.14/tokenize.py", line 586, in _generate_tokens_from_c_tokenizer
    raise e from None
  File "/usr/local/lib/python3.14/tokenize.py", line 582, in _generate_tokens_from_c_tokenizer
    for info in it:
                ^^
  File "<string>", line 3
    m
     ^
IndentationError: unindent does not match any outer indentation level

[danielhollas]

cc @ambv based on the stacktrace it looks like it's coming from the new syntax highlighting code.

[ambv]

Oof. OK, yes, indeed. Sorry about that omission.