Skip to content

fix(pytest): surface diagnostic context when no tests are collected#2006

Open
YOMXXX wants to merge 1 commit into
rtk-ai:developfrom
YOMXXX:fix/pytest-no-args-collection
Open

fix(pytest): surface diagnostic context when no tests are collected#2006
YOMXXX wants to merge 1 commit into
rtk-ai:developfrom
YOMXXX:fix/pytest-no-args-collection

Conversation

@YOMXXX
Copy link
Copy Markdown
Contributor

@YOMXXX YOMXXX commented May 21, 2026

Summary

`rtk pytest` (no args) collapsed pytest's collection output to a single terse line — `Pytest: No tests collected` — whenever the summary parser couldn't find a `passed/failed/skipped` count. Reporter (#1417) saw this on a real project where `rtk pytest tests/test_cli_dispatch.py -v` worked but `rtk pytest` did not, with no way to tell whether it was a cwd issue, a pytest config issue, or an RTK bug.

Two real situations both hit this:

  1. No tests in cwd — pytest emits `no tests ran in X.XXs` instead of a counted summary line. The user was left guessing why nothing was found.
  2. Collection error — `ImportError` in a conftest, missing fixture import, etc. pytest emits `errors during collection` + an `ERROR` block, then `no tests ran`. The actionable error was being thrown away.

Fix

Add `diagnose_no_tests`: when the existing summary collapses to `No tests collected`, scan the raw pytest output for the lines that explain why — `rootdir:`, `configfile:`, `testpaths:`, `collected N items`, any `ERROR …` line, plus the `no tests ran` / `no tests collected` / `errors during collection` markers — and emit them verbatim under a header so the LLM and the user can act on them.

```
Pytest: no tests collected
═══════════════════════════════════════
rootdir: /Users/x/proj
configfile: pyproject.toml
testpaths: tests, integration
collected 0 items
no tests ran in 0.05s
```

Capped at 15 lines (a tee-style overflow hint would be wrong here — diagnostic lines are bounded in practice).

Falls back to the terse default when no recognisable marker is present (e.g. RTK's parser broke but tests actually ran), so we don't flood the output with arbitrary noise on a parser regression.

Behaviour

Scenario Before After
Empty cwd, `rtk pytest` `Pytest: No tests collected` + rootdir / configfile / testpaths / "no tests ran"
Collection `ImportError` `Pytest: No tests collected` + the actual `ERROR collecting …` line
Quiet-only `no tests ran in 0.05s` (no rootdir present) terse terse (markers fence keeps it from over-firing)
Tests pass unchanged unchanged
Tests fail unchanged unchanged

Test plan

  • `test_filter_no_tests_verbose_surfaces_rootdir_and_configfile` — rootdir / configfile / testpaths / collected count / "no tests ran" all reach the consumer.
  • `test_filter_no_tests_collection_error_surfaced` — ImportError-during-collection scenario shows the `ERROR` line.
  • `test_filter_no_tests_quiet_mode_falls_back_to_terse` — bare `no tests ran in 0.05s` (no rootdir to anchor on) keeps the original short verdict.
  • Pre-existing `test_filter_pytest_no_tests` updated to assert the new enriched output while keeping the same verdict header for grep-ability.
  • `cargo fmt --all` clean.
  • `cargo clippy --all-targets` zero warnings.
  • `cargo test --bin rtk -- --test-threads=8` → 1912 passed (was 1909).

Notes

  • The original reporter's underlying confusion (was the cwd wrong? was it pytest config? was it RTK?) is now answerable straight from the rtk output: rootdir tells you where pytest looked, configfile tells you which config it used, and testpaths tells you which subtree it scanned.
  • Doesn't touch the `-q` defaults or the runner's cwd — those are still pytest's standard behaviour.

Fixes #1417

`rtk pytest` (no args) collapsed pytest's collection output to a single
terse line "Pytest: No tests collected" whenever the summary parser
couldn't find a `passed/failed/skipped` count. That happens in two real
situations both reported in rtk-ai#1417:

1. `pytest` discovers nothing in the cwd and emits `no tests ran in
   X.XXs` instead of a counted summary line — the LLM/user is left
   guessing whether it's a cwd issue, a pytest config issue, or an RTK
   bug.

2. Collection itself fails (`ImportError` in a conftest, typo in a test
   file) and pytest emits `errors during collection` plus an `ERROR`
   block, then `no tests ran`. The actionable error was being thrown
   away.

Add `diagnose_no_tests`: when the existing summary collapses to
"No tests collected", scan the raw pytest output for the lines that
explain *why* — `rootdir:`, `configfile:`, `testpaths:`, `collected N
items`, any `ERROR …` line, the `no tests ran` / `no tests collected` /
`errors during collection` markers — and emit them verbatim under a
header so the LLM and the user can act on them. Caps at 15 lines (a
tee-style overflow hint would be wrong here — diagnostics are bounded
in practice).

Falls back to the terse default when *no* recognisable marker is
present (e.g. RTK's parser failed but tests actually ran), so we don't
flood the output with arbitrary noise on a parser regression.

Tests:
- test_filter_no_tests_quiet_mode_falls_back_to_terse — quiet-mode
  output `no tests ran in 0.05s` carries no rootdir/configfile, so the
  terse verdict is the right answer; this fence prevents over-eager
  diagnostic surfacing from leaking the original raw line.
- test_filter_no_tests_verbose_surfaces_rootdir_and_configfile — locks
  in that rootdir / configfile / testpaths / collected count / the
  `no tests ran` summary are all surfaced from a verbose pytest run.
- test_filter_no_tests_collection_error_surfaced — covers situation
  (2) above: collection-time ImportError must reach the consumer.
- The pre-existing test_filter_pytest_no_tests now asserts the new
  enriched output (collected 0 items + no tests ran both visible)
  while keeping the same verdict header for grep-ability.

cargo fmt / clippy --all-targets / test --bin rtk: 1912 passed,
0 warnings.

Fixes rtk-ai#1417
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pytest collection fails with no-args invocation

1 participant