1.7.8 crashes with a tokenize ValueError on a multi-line f-string inside parentheses

[alexander-turner]

docformatter 1.7.8 aborts with a ValueError out of tokenize.untokenize when a file contains a multi-line f-string inside a parenthesized expression. 1.7.7 formats the same file cleanly.

Reproducer

def build(x):
    return (
        f"""a
{x}""",
    )

$ docformatter --in-place t.py
  File ".../docformatter/format.py", line 876, in _do_format_code
    _code = tokenize.untokenize(self.new_tokens)
  File "/usr/lib/python3.11/tokenize.py", line 216, in untokenize
    self.add_whitespace(start)
  File "/usr/lib/python3.11/tokenize.py", line 177, in add_whitespace
    raise ValueError("start ({},{}) precedes previous end ({},{})"
ValueError: start (2,8) precedes previous end (3,4)
$ echo $?
1

No config file and no flags are needed. Two things narrow it:

• Remove the f prefix and the run exits 3 (would reformat) with no error, so the f-string prefix is the trigger.
• Assign the same f-string to a name instead of putting it in a tuple and it does not crash, so it has to sit inside parentheses.

Affected versions

docformatter	Python 3.11	Python 3.12	Python 3.13
1.7.7	ok	ok	ok
1.7.8	ValueError	ok	ok
master	ValueError	ok	ok

Root cause

Bisected to d9ace1f ("fix: resolve several empty line regressions", #330). Its parent 7798699 is clean.

That commit adds a Python < 3.12 workaround to classify.is_f_string:

elif any(
    [
        token.string.startswith('f"""'),
        prev_token.string.startswith('f"""'),
        token.string.startswith("f'''"),
        prev_token.string.startswith("f'''"),
    ]
):
    return True

The PY312 branch above it tests tokenize.FSTRING_MIDDLE, which only appears inside an f-string. The < 3.12 branch instead matches any token whose text begins with f""", including an ordinary triple-quoted f-string that is not a docstring — and, via prev_token, the token that follows one.

The single consumer is _get_unmatched_start_end_indices (format.py:499), where a true verdict sets _start_row = prev_token.end[0] (format.py:526-535). That rewrites the f-string's start row back to where the previous token ended, so untokenize sees a token starting above one it has already emitted and raises.

Impact

With several files on one command line, the run aborts at the first offending file and exits 1, leaving every file after it unprocessed. On a 1,255-file tree this stopped at file 318 with no indication that the remainder was skipped.

Environment

CPython 3.11.15, Linux x86-64, docformatter==1.7.8 from PyPI.

[adityaanikam]

Your bisection and root cause are exactly right, checked both files on master. The branch of is_f_string that handles Python versions under 3.12 matches on prev_token.string.startswith('f"""'), so it returns true for the token right after a triple quoted f-string too, not just tokens inside one. That feeds straight into _get_unmatched_start_end_indices, which rewinds _start_row for a token that was never a docstring boundary in the first place. I'd like to fix the branch to only match when the current or previous token is genuinely still inside the f-string, not just prefixed similarly.