`UnicodeEncodeError` in pluggy with surrogate escape in parametrization and `--debug`

[The-Compiler]

On Linux, with the current git main of pytest (8.5.0.dev60+g67738403d), pluggy 1.6.0 and Python 3.13.1, this test runs fine:

import pytest

@pytest.mark.parametrize("s", ["\ud800"])
def test_x(s):
    pass

but, somewhat contrary to its name, --debug=debug.log causes a bug instead:

_____________________ ERROR collecting test_surrogate.py _____________________
.venv/lib/python3.13/site-packages/pluggy/_hooks.py:512: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:475: in traced_hookexec
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:472: in <lambda>
    lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:240: in pytest_pycollect_makeitem
    return list(collector._genfunctions(name, obj))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:466: in _genfunctions
    self.ihook.pytest_generate_tests.call_extra(methods, dict(metafunc=metafunc))
.venv/lib/python3.13/site-packages/pluggy/_hooks.py:573: in call_extra
    return self._hookexec(self.name, hookimpls, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:475: in traced_hookexec
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:472: in <lambda>
    lambda: oldcall(hook_name, hook_impls, caller_kwargs, firstresult)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:114: in pytest_generate_tests
    metafunc.parametrize(*marker.args, **marker.kwargs, _param_mark=marker)
src/_pytest/python.py:1274: in parametrize
    ids = self._resolve_parameter_set_ids(
src/_pytest/python.py:1395: in _resolve_parameter_set_ids
    return id_maker.make_unique_parameterset_ids()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:902: in make_unique_parameterset_ids
    resolved_ids = list(self._resolve_ids())
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:945: in _resolve_ids
    yield "-".join(
src/_pytest/python.py:946: in <genexpr>
    self._idval(val, argname, idx)
src/_pytest/python.py:957: in _idval
    idval = self._idval_from_hook(val, argname)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/_pytest/python.py:985: in _idval_from_hook
    id: str | None = self.config.hook.pytest_make_parametrize_id(
.venv/lib/python3.13/site-packages/pluggy/_hooks.py:512: in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:120: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/pluggy/_manager.py:470: in traced_hookexec
    before(hook_name, hook_impls, caller_kwargs)
.venv/lib/python3.13/site-packages/pluggy/_manager.py:495: in before
    hooktrace(hook_name, kwargs)
.venv/lib/python3.13/site-packages/pluggy/_tracing.py:69: in __call__
    self.root._processmessage(self.tags, args)
.venv/lib/python3.13/site-packages/pluggy/_tracing.py:44: in _processmessage
    self._writer(self._format_message(tags, args))
E   UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 140: surrogates not allowed

Reporting here as it's somewhat unclear whether a proper fix for this should be in pytest or in pluggy (possibly the latter though, given that it can probably be reproduced with any hook execution with a string argument that has surrogate escapes?).

[RonnyPfannschmidt]

Seems like we need to configure surrogate escape error handling in pluggy

[bluetech]

The fact that there are strings in Python that cannot be UTF-8 encoded is such a massive gotcha that no one knows about.

I agree with @RonnyPfannschmidt that the fix should be in pluggy, it should use errors='replace' in its tracing function.

[Ghanchu]

Hello, my partner and I are students at the University of Michigan who have been tasked with contributing to an open source project for our class. We agree that the error is in pluggy's tracing function and will create a pull request to fix this bug.

[RonnyPfannschmidt]

It may possibly be sufficient to set the encoder to surrogate escape

[jokeena]

Hello, I'm the other project partner with Ghanchu. Here is the link to the PR