feat: add doctest tool and Bats harness for Markdown code blocks

[trevor-vaughan]

Go tool (cmd/doctest/) uses goldmark to parse fenced code blocks
annotated with {test="..."} from Markdown files. Bats test harness
runs extracted snippets. Motivated by 11 open issues where Getting
Started page commands were broken or untested.

• Add cmd/doctest/ with extract and coverage subcommands
• Add Bats test harness via npm (bats-core, bats-support, bats-assert)
• Add Makefile targets: test-docs-extract, test-docs, test-docs-coverage
• Wire doc tests into CI (informational, non-blocking until blocks are annotated)
• Add spec and plan in specs/015-testable-documentation/
• Update CONTRIBUTING.md and README.md; create AGENTS.md

Details:

Go tool (cmd/doctest/):

• goldmark AST parser extracts annotated fenced code blocks to disk
• coverage subcommand exits non-zero when untested executable blocks exist
• Table-driven unit tests cover parsing, extraction, slug derivation, and coverage

Bats harness (tests/docs/):

• Installed as npm devDependencies (project already uses npm for Hugo)
• Helper scripts wrap snippet execution; skeleton test for getting-started guide

Build and CI:

• DOCTEST_DIR variable decouples extract output path from test input path
• Go tests run before doc tests so coverage failures don't block unit tests
• Doc test step uses || true until remaining blocks are annotated
• .gitignore: .test-output/, doctest binary, cmd/sync-content/sync-content

Assisted-by: Claude Opus 4.6
Signed-off-by: Trevor Vaughan tvaughan@redhat.com

[github-actions]

✅ CRAP Load Analysis: PASS (no baseline)

No baseline file found at .gaze/baseline.json. Showing current scores without regression detection.

How to Enable Regression Detection

Generate and commit a baseline file to track CRAP score changes over time:

# 1. Install gaze
go install github.com/unbound-force/gaze/cmd/gaze@latest

# 2. Run tests and generate baseline
go test -coverprofile=coverage.out ./...
mkdir -p .gaze
gaze crap --format=json --coverprofile=coverage.out ./... > .gaze/baseline.json

# 3. Commit the baseline
git add .gaze/baseline.json
git commit -m "chore: add CRAP baseline for regression detection"

For more information:

• Gaze README
• CI Integration Guide

Summary

Metric	Value
Functions analysed	12
Avg complexity	8.3
Avg line coverage	74.8%
Avg CRAP score	19
CRAPload (>= 15)	3
Avg contract coverage	0%
Avg GazeCRAP score	0
GazeCRAPload (>= 15)	0

View full analysis logs

[em-redhat]

Code Review: PR #84 — Doctest Tool and Bats Harness

Overview

This PR adds solid infrastructure for testing documentation code blocks — the Go extractor is well-designed (goldmark is the right choice, matching Hugo's own parser), the test suite is comprehensive for the parsing logic, and the git-tracking integration is smart. The motivation (11 open issues for broken Getting Started commands) is clear and the approach is sound.

However, there are several issues that need attention before merging, ranging from CI pipeline concerns to security considerations and test coverage gaps.

---

CI Status

The "Standardized CI / Run linters" check is failing on this PR (passes on main):

• golangci-lint: 2 gosec findings (G122, G306) — addressed in inline comments below
• markdownlint: 709 errors in specs/015-testable-documentation/plan.md and spec.md (tabs, missing code fence languages, table formatting)

The markdownlint errors in spec files should be cleaned up — 709 errors is a lot of noise for a linter check. At minimum, the hard tabs and missing code fence languages in plan.md need fixing if the project's markdownlint config applies to specs/.

---

Summary of Findings

Severity	Count	Category
HIGH	4	CI visibility, file permissions, stale output dir, test coverage
MEDIUM	5	Deploy workflow, /tmp path, plan divergence, output cleanup, run() testability
LOW	3	Bats placeholder, test assertion gaps, frontmatter edge case
NIT	2	Comment suggestions

HIGH Issues

1. || true makes doc test failures completely invisible in CI — Both ci.yml and deploy-gh-pages.yml use make test-docs || true. Combined with the - prefix in the Makefile, there is no path through which a doctest failure surfaces to a developer. Use continue-on-error: true on the GitHub Actions step instead — failures show as a yellow warning in the UI without blocking the pipeline. (See inline comments.)

2. gosec G306: os.WriteFile with 0o644 — Extracted snippets and manifests should use 0o600 per least-privilege. On shared CI runners, world-readable build artifacts are unnecessary. (See inline comment on extract.go.)

3. No output directory cleanup before extraction — runExtract writes to outputDir but never cleans it. If a code block is renamed or removed, stale snippets persist and Bats tests run against outdated content. Add rm -rf $(DOCTEST_DIR) before extraction in the Makefile, or os.RemoveAll(outputDir) in the Go code.

4. run() in main.go is untested — The main() → run() int pattern exists specifically for testability, but there are zero tests for run(). It has 6+ exit paths (no args, missing flags, unknown subcommand, extract error, coverage error, success). Additionally, run() uses os.Args directly and flag.ExitOnError, making it difficult to test without global state mutation. Consider accepting []string args and using flag.ContinueOnError.

MEDIUM Issues

5. Doc tests in the deploy workflow — deploy-gh-pages.yml runs make test-docs || true between the Hugo build and actions/upload-pages-artifact. Tests should gate PRs (in ci.yml), not deployments. When Phase 2 removes || true, a flaky doc test would block production deploys.

6. /tmp/doctest-snippets as default output path — Predictable path in world-writable /tmp. On shared runners or multi-user systems, concurrent runs could collide. Consider .test-output/doctest-snippets (already gitignored) or mktemp -d.

7. Plan-to-implementation divergence — The 1590-line plan.md contains full source code copies that are already stale: pageSlug behavior differs (plan would cause slug collisions), runCoverage return behavior differs (plan returns nil always, code returns error), gitTrackedFiles is absent from the plan entirely, nonTestableLangs is defined in the plan but not implemented. Consider either updating the plan or removing full code listings (they're a maintenance burden).

8. runCoverage happy path not tested — Tests cover the "untested blocks found" error path and the opt-out path, but not the case where all executable blocks are annotated and runCoverage returns nil.

9. - prefix in Makefile undocumented — The make test target says "Run all tests" but silently ignores doc test and coverage failures via -. Add inline comments explaining the Phase 1/Phase 2 intent so future contributors understand why errors are swallowed.

LOW Issues

10. Bats test file is a pure placeholder — getting-started.bats has zero @test blocks. make test-docs runs 0 tests = always passes. Consider adding at least one real test to validate the end-to-end pipeline.

11. lineNumber never asserted in tests — TestExtractBlocksBasic checks lang, testName, and content but never asserts b.line. Line number accuracy matters for developer experience (error messages, coverage reports).

12. parseFrontmatter edge case — Uses bytes.HasPrefix(source, []byte("---")) without requiring a trailing newline. ---foo at the start of a file would match. Consider requiring ---\n as the opening delimiter.

---

What's Good

• goldmark is the right dependency — Same Markdown parser Hugo uses internally, avoiding parser divergence bugs
• gitTrackedFiles is a smart design choice — Prevents false positives from generated/synced content
• Test isolation is thorough — t.TempDir(), GIT_CONFIG_GLOBAL=/dev/null, isolated git repos
• Code quality is clean — Good error messages, clear function boundaries, proper run() int pattern
• Spec was updated — spec.md correctly reflects implementation decisions (npm vs git submodules, exit codes)

[em-redhat]

[HIGH] || true completely silences doc test failures — CI shows green regardless of outcome. There is no mechanism to surface failures (no annotation, no artifact, no separate status check).

Recommendation: Replace with a proper GitHub Actions continue-on-error:

      - name: Run documentation tests (informational)
        run: make test-docs
        continue-on-error: true

This keeps the step non-blocking but shows a yellow warning icon when tests fail, making failures visible without blocking the pipeline. The current || true pattern means a regression in the doctest tool or any tested snippet will be completely hidden.

[em-redhat]

[MEDIUM] Consider removing this step from the deploy workflow entirely. Tests should gate PRs (via ci.yml), not deployments. This step adds compile + test latency to every deploy, and when Phase 2 removes || true, a flaky doc test would block production deployments.

If deployment-time validation is needed, it should be a separate job that does not block the deploy job.

[em-redhat]

[HIGH] gosec G306: 0o644 makes extracted snippets world-readable. These are temporary build artifacts — use 0o600 per least-privilege principle. The Bats runner (same user) can still read them. The manifest os.WriteFile below (around line 380) has the same issue.

Suggested change

	if err := os.WriteFile(outPath, b.content, 0o644); err != nil {
	if err := os.WriteFile(outPath, b.content, 0o600); err != nil {

[em-redhat]

[MEDIUM] gosec G122: filepath.WalkDir follows symlinks by default. Consider skipping symlinks in the callback to address the TOCTOU race:

if d.Type()&fs.ModeSymlink != 0 {
    return nil
}

This is low-risk for this use case (git-tracked files are unlikely to be symlinks), but it resolves the gosec finding and is trivial to add.

[em-redhat]

[MEDIUM] The - prefix silently ignores errors from test-docs and test-docs-coverage. While intentional for Phase 1, the target description says "Run all tests" — misleading when failures are silently swallowed.

Suggestions:

1. Add inline comments explaining the Phase 1 non-blocking intent and when this changes
2. Add @rm -rf $(DOCTEST_DIR) at the start of test-docs-extract to prevent stale snippets from previous runs persisting
3. Consider echoing a warning when doc tests fail so developers notice

[em-redhat]

[MEDIUM] run() reads os.Args directly and uses flag.ExitOnError, making it impossible to unit test without mutating global state. The main() → run() int pattern exists specifically for testability, but there are zero tests for run().

Consider accepting args as a parameter and using flag.ContinueOnError:

func run(args []string) int {
    if len(args) < 1 { ... }
    subcmd := args[0]
    fs := flag.NewFlagSet("extract", flag.ContinueOnError)
    ...
}

This enables straightforward table-driven tests for all 6+ exit paths.