feat: documentation snippet testing with literalinclude and Pydantic validation#704
Closed
raballew wants to merge 91 commits into
Closed
feat: documentation snippet testing with literalinclude and Pydantic validation#704raballew wants to merge 91 commits into
raballew wants to merge 91 commits into
Conversation
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
raballew
force-pushed
the
497-doc-snippet-testing-literalinclude
branch
3 times, most recently
from
0a6bb6b to
fa1f535
Compare
3 tasks
raballew
force-pushed
the
497-doc-snippet-testing-literalinclude
branch
from
a3314bf to
7d342fa
Compare
Create the infrastructure for extracting inline code snippets from documentation into standalone files with corresponding e2e tests. Adds examples/tests/ with conftest fixtures and a docs-snippet-test Makefile target. Generated-By: Forge/20260526_125902_2796304_6b066394
Extract the inline driver implementation example from introduction/drivers.md into a standalone Python file at examples/introduction/driver_example.py. Replace the inline code block with a literalinclude directive. Add e2e test that imports and runs the driver example using serve() for real validation. Generated-By: Forge/20260526_125902_2796304_6b066394
Extract YAML exporter configuration snippets from introduction/exporters.md and introduction/drivers.md into standalone files. Replace inline code blocks with literalinclude directives. Add tests that validate each YAML file against the ExporterConfigV1Alpha1 Pydantic model for real structural validation. Generated-By: Forge/20260526_125902_2796304_6b066394
…ests Extract all YAML hook configs and the Python hook example from introduction/hooks.md into standalone files under examples/introduction/. Replace inline code blocks with literalinclude directives. Add tests that validate each YAML file against HookConfigV1Alpha1 and the full ExporterConfigV1Alpha1 Pydantic models, and verify the Python example compiles successfully. Generated-By: Forge/20260526_125902_2796304_6b066394
Document the pattern for converting inline code blocks to literalinclude directives with corresponding tests. Covers Python, YAML, and bash snippets, test fixtures, and the docs-snippet-test Makefile target. Generated-By: Forge/20260526_125902_2796304_6b066394
Ruff F401 (unused imports) would fail CI. The three test files imported Path from pathlib but only used the examples_root fixture which already returns a Path object. Generated-By: Forge/20260526_125902_2796304_6b066394
The test only imports the module (does not execute the __main__ block), so rename from test_driver_example_executes_successfully to test_driver_example_imports_successfully. Generated-By: Forge/20260526_125902_2796304_6b066394
…ample Separate `import os` (stdlib) from `from jumpstarter.utils.env import env` (third-party) with a blank line to satisfy ruff I001 import sorting rule. Generated-By: Forge/20260526_125902_2796304_6b066394
Remove YAML comments from driver_exporter_config.yaml, remove redundant file-existence tests superseded by validation tests, add trailing newline to drivers.md, and fix grammar in contributor guidelines. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…r tests Convert inline code blocks in 41 driver READMEs to literalinclude directives pointing at standalone files under each driver's examples/ directory. Generate examples_test.py per driver that validates extracted YAML configs (yaml.safe_load) and Python snippets (compile). Tests run as part of existing make pkg-test-<driver> infrastructure. 104 example tests across 41 drivers. Doctest blocks and invalid YAML are left inline. Install commands, CLI sessions, and console output remain as inline code blocks. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…t YAML syntax Upgrade all 41 driver examples_test.py files to validate YAML configs against ExporterConfigV1Alpha1DriverInstance.model_validate() (for export sections) or ExporterConfigV1Alpha1.model_validate() (for full exporter configs). Non-config YAML (scenarios, method definitions) falls back to yaml.safe_load() validation. Remove one-shot conversion scripts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move jumpstarter.config.exporter imports from module level into test functions using pytest.importorskip so tests skip gracefully when jumpstarter is not installed rather than failing at collection time. Remove incorrectly extracted doctest file from tftp driver and invalid YAML config from ridesx driver. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ples These files contained doctest >>> blocks that were incorrectly extracted during conversion. They are not valid standalone Python and had no corresponding test. The doctest blocks remain inline in the READMEs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move YAML config files from docs/source/reference/package-apis/drivers/ into each driver's examples/ directory. Update literalinclude paths in READMEs. Remove hidden doctest blocks and replace with examples_test.py that validates configs against ExporterConfigV1Alpha1DriverInstance. 44/45 drivers now have example tests. The remaining driver (uds) is a base package with no config of its own. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ocs-test Convert the last 2 doctest blocks (opendal, tftp) to standalone example files with literalinclude. Move opendal.yaml from docs/ to driver examples/. Simplify Makefile: docs-test now runs pytest on examples/ instead of Sphinx doctest (no doctest blocks remain). docs-snippet-test kept as alias for backwards compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The documentation CI ran build, check-warnings, and linkcheck but never ran the snippet validation tests. Add a docs-test job so documentation examples are validated on every PR that touches python/docs/ or python/packages/. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add jumpstarter.testing.examples module with validate_yaml_example and validate_python_example functions. YAML validation dispatches by kind (ExporterConfig, ClientConfig, UserConfig) or section (hooks, export) to the appropriate Pydantic model. Python validation checks syntax and imports. Wrap all 67 config fragments in full Kubernetes-style YAML with apiVersion, kind, and metadata so the kind discriminator drives model selection automatically. Replace 44 hand-written examples_test.py files and 3 central docs test files with a single parametrized pattern that calls the shared module. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… tests
- README template now uses {literalinclude} for config instead of inline YAML
- Add examples/config.yaml.tmpl with full Kubernetes-style ExporterConfig
- Add examples_test.py.tmpl using shared jumpstarter.testing.examples
- Update creating-new-drivers rules with new directory structure and
post-creation steps mentioning config.yaml and examples_test.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rename 49 example files from auto-generated heading slugs to concise, descriptive names. Update literalinclude paths in 17 READMEs. Examples: config_exporterconfig_example_1.yaml -> config_avh.yaml, config_example_configuration_for_shelly_smart_p.yaml -> config_shelly.yaml, usage_flash_with_compressed_images.py -> usage_compressed_flash.py Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each driver's examples_test.py now checks:
- test_no_unused_examples: fails if config*.yaml or usage*.py files in
examples/ are not referenced in README.md
- test_no_inline_code_blocks: fails if README.md contains bare
```yaml or ```python fences that should use {literalinclude}
Convert 9 remaining inline fragments to {code-block} directives. Fix
stale literalinclude paths in someip and ssh-mitm READMEs. Update driver
creation template with the same checks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move discover_example_files, find_unused_examples, and find_inline_code_blocks into jumpstarter.testing.checks (no model dependencies). Remove duplicate implementations from 44 examples_test.py files. Each test file is now 30 lines importing from the shared modules. Changing the skip directive list, naming convention, or check logic now requires editing one file instead of 44. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
validate_example silently did nothing for kinds other than yaml or python (e.g. bash, shell). Add an else branch that raises ValueError to surface unsupported kinds explicitly rather than silently skipping validation. Generated-By: Forge/20260529_102529_1255558_c89788a1
A non-dict hooks value (e.g. a list) was silently skipped without validation. Now raises TypeError with a descriptive message. Extracted section validation into _validate_section helper to stay under C901 complexity limit. Generated-By: Forge/20260529_160937_1659706_722b0319
Convert inline bash/shell code blocks in driver READMEs to either literalinclude directives (for standalone examples) or code-block directives (for output/placeholder/interactive snippets). Drivers converted: adb, androidemulator, composite, doip, dut-network, energenie, esp32, flashers. Generated-By: Forge/20260529_164302_1718250_e7b18c65
Convert inline bash/shell code blocks in driver READMEs for http-power, mitmproxy, noyito-relay, pyserial, and snmp. Generated-By: Forge/20260529_164302_1718250_e7b18c65
Convert inline bash/shell code blocks in driver READMEs for someip, ssh, ssh-mitm, ssh-mount, and stlink-msd. Generated-By: Forge/20260529_164302_1718250_e7b18c65
Convert inline bash/shell code blocks in driver READMEs for tmt, uds-can, uds-doip, vnc, xcp, and yepkit. Generated-By: Forge/20260529_164302_1718250_e7b18c65
…-block Convert inline python/yaml/bash/shell blocks in getting-started docs and CI/CD guides. Create external example files for standalone Python test examples; use code-block directives for yaml config fragments, CI workflow snippets, and contextual code blocks. Generated-By: Forge/20260529_164302_1718250_e7b18c65
…sions All 23 driver READMEs and 31 docs pages have been converted from inline code blocks to either literalinclude directives (for standalone examples) or code-block directives (for output/placeholder/interactive snippets). Both xfail sets are now empty. Generated-By: Forge/20260529_164302_1718250_e7b18c65
Align 13 test files with the project convention of *_test.py suffix naming. test_utils.py is kept as-is since it contains test utilities, not tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Align with project convention of underscores in Python filenames. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The docs-test target now runs pytest instead of Sphinx doctest, making the sphinx.ext.doctest extension and doctest_test_doctest_blocks config dead code. Generated-By: Forge/20260605_210308_337267_d5a99a2b Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
discover_example_files only globbed for *.yaml and *.py, leaving 16 .bash example files invisible to find_unused_examples. Add *.bash glob and handle the bash kind in validate_example. Generated-By: Forge/20260605_210308_337267_d5a99a2b Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Prevents pytest from collecting Python files in examples/ directories as test modules when running pkg-test-<name> for these driver packages. Generated-By: Forge/20260605_210308_337267_d5a99a2b Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tion Use type[pydantic.BaseModel] instead of bare type for return and parameter annotations, enabling type checkers to verify .model_validate() calls. Generated-By: Forge/20260605_210308_337267_d5a99a2b Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The fixture raises on import failure, making the is-not-None assertion unreachable as a failure path. The behavioral test test_driver_example_serve_creates_client provides the actual verification. Generated-By: Forge/20260605_210308_337267_d5a99a2b Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The template that generates new driver packages was missing norecursedirs = ["examples"], causing newly created drivers to have pytest attempt collection inside their examples directory. Generated-By: Forge/20260605_210308_337267_d5a99a2b
…_example Covers the compile() path that raises SyntaxError when a Python example file has invalid syntax, completing the negative-path test coverage for FR-004. Generated-By: Forge/20260605_210308_337267_d5a99a2b
Add symmetric TypeError guard for non-dict export sections, matching the existing hooks handling. Previously a list or string export section was silently accepted. Generated-By: Forge/20260605_210308_337267_d5a99a2b
The function previously only read the file content without performing any validation. Now it runs bash -n for syntax checking as documented in the contributing guidelines. Generated-By: Forge/20260605_210308_337267_d5a99a2b
…d suite The centralized test_example_files.py already discovers and validates all docs examples including introduction/ via _example_file_params(). This per-directory test duplicated that coverage. Generated-By: Forge/20260605_210308_337267_d5a99a2b
Replace box drawing characters (U+2500) and rightwards arrow (U+2192) with ASCII dashes and -> in section separators and comments. Generated-By: Forge/20260605_210308_337267_d5a99a2b
Remove comments that restate method names (e.g., "# Turn device power on" before power_client.on()) and class/method docstrings that restate test names. Condense module docstrings to run commands only. Retain comments that clarify non-obvious consequences. Generated-By: Forge/20260605_210308_337267_d5a99a2b
importlib.util.spec_from_file_location returns ModuleSpec | None, and spec.loader is Loader | None. Add assertions before use to satisfy type checkers and fail fast with clear errors. Generated-By: Forge/20260605_210308_337267_d5a99a2b
raballew
force-pushed
the
497-doc-snippet-testing-literalinclude
branch
from
7d342fa to
a4e2eb7
Compare
…ackage checks.py and examples.py were added under jumpstarter.testing, shipping test-only code (Pydantic validation, bash -n checks, inline code block detection) to end users. Move them to docs/source/examples/tests/ where they belong alongside the test suite that uses them. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace typo-triggering "ExporerConfig" with "UnknownKind" in test - Add blank line between stdlib and third-party imports in scripting_env.py - Fix mitmproxy literalinclude reference: test_device.py -> device_test.py - Convert OBD driver inline code blocks to literalinclude with example files - Add norecursedirs to OBD pyproject.toml Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The network driver has no examples/ directory, so adding [tool.pytest.ini_options] with norecursedirs was unnecessary and caused the fabric_test to fail by changing pytest's config resolution and breaking Fabric's stdin handling under capture. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Member
Author
|
this got a bit out of hand. closing in favor of a lighter approach. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
examples/directories, referenced via{literalinclude}directivesExporterConfigV1Alpha1,HookConfigV1Alpha1, etc.) and Python examples viacompile()and import checksjumpstarter.testing.checksandjumpstarter.testing.examples, replacing 44 per-driver test files with a single auto-discovering test suitedocs-testjob to the documentation workflowStrictHostKeyChecking=accept-new, escape shell metacharacters in PS1, add--insecureflagexamples/config.yamland{literalinclude}usageTest plan
make docs-testvalidates all example filesmake docsbuilds without literalinclude errorsmake pkg-test-jumpstartercovers shared test modulescreate_driver.shincludes correct examples structureGenerated with Claude Code