Scoped reload to measure source imported before the coverage engine starts

[jaraco]

Summary

Explore a scoped, opt-in mechanism for pytest-cov to measure code that was imported before the coverage engine started — the classic "plugin imports the project under test" problem — by reloading only the affected source modules once coverage is active.

This is a solution-focused follow-up to #635 (and the cluster #682, #414, #439, #397, #372, #359, #587). In #635 the conclusion was essentially "use coverage run -m pytest, or manually pre-start via the pth/env-var machinery, or scrub sys.modules yourself in the offending plugin," and @ionelmc rightly cautioned that broad sys.modules manipulation is a "big no-no" (it invites blame reports like #620). The goal here is to discuss whether a narrowly-scoped reload could sidestep that objection.

The limitation

When a plugin is imported before pytest-cov's engine starts (entry-point plugins, or anything pytest-enabler-style that imports the project under test to decide whether to enable coverage), every line executed at import time — import, class, def, decorators — is recorded as missed, because it never executes again under coverage. The plugins docs document this and state there's currently "no way to measure your pytest plugin."

Concretely: pytest-perf is itself a pytest plugin, so its own runner.py import-time lines are unmeasurable via pytest --cov; a 100% diff-coverage gate can't be met without either coverage run -m pytest or a per-project reload hack (jaraco/pytest-perf#18, jaraco/pytest-perf#23).

Approaches worth exploring

Both aim to reload only modules that (a) were imported before coverage started and (b) fall under the configured --cov/source — never stdlib, never third-party, never the whole sys.modules.

1. Snapshot-and-diff. Capture the set of sys.modules keys as early as possible (e.g. via the pth/embed machinery that already exists for subprocess start-up). After the coverage engine is active, reload only the modules that appeared since the snapshot and that match the source configuration — i.e. exactly the modules a plugin pulled in early. This bounds the blast radius to plugin-activated source.

2. Source-discovery. From the resolved --cov/source/source_pkgs configuration, discover the target modules and, for any already present in sys.modules when coverage starts, importlib.reload just those. Reloading is scoped strictly to the code under measurement.

Caveats to discuss

• Reloading has real hazards: module-level singletons, registries, dataclass/enum identity, C extensions, and already-bound references to pre-reload objects. Scoping to source-under-measurement reduces but doesn't eliminate this.
• It would need to be opt-in (a config flag), off by default, and clearly documented as "measures import-time lines at the cost of a scoped reload."
• The existing coverage run -m pytest remains the clean answer for those who can wrap the invocation; this is for the plugin-activation workflows (pytest-enabler et al.) where wrapping is undesirable, per coverage starts too late when a plugin imports the project under test #635.

Question

Would a scoped, opt-in reload — bounded to the configured source and to modules imported before the engine started — be an acceptable direction, or are the reload hazards fundamental enough that documenting coverage run -m pytest should remain the only recommendation?