ci: check links in skill content - #312
Open
Peetee06 wants to merge 5 commits into
Open
Conversation
A dead link in context/ ships into agent context windows and degrades every wizard and MCP user until someone notices by hand. Two recent fixes (0741482, fda834d) were exactly that: link rot found by accident, after shipping. Split into two workflows so a PR is only blocked by what it changes: - link-check-offline: relative paths and #anchors. No network, so it cannot flake; baseline is green today, so any red is a real regression. Blocking. - link-check-external: the 59 external URLs, nightly and non-blocking. Reports via the same PostHog webhook e2e.yml/integration.yml use, on exit code 2 only (1 and 3 are lychee's own errors, not link rot). Only reports links that are GONE. 403/429/timeouts are accepted, each with a reason in-file: they say the server failed to answer, not that the resource is missing. 5xx stays reportable — retries absorb blips, and a durable 5xx means the link is effectively dead for an agent. example-apps/ is excluded on purpose: human-read, not agent-read. Verified against lychee 0.24.2 (the version the action bundles): offline exits 0, external exits 2 with three genuine findings, fixed separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2iPGUrMb11u5o6RfxnTkq
Peetee06
commented
Aug 2, 2026
- Trim comments that restate the diff or belong in the PR description. - Drop the fix-instruction steps duplicating lychee's own output; keep a one-line pointer, since logs are read from the failing step upwards. - Keep the false-positive guidance only in .lycheeignore. - Fail the nightly job after the webhook fires, so GitHub shows it red too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2iPGUrMb11u5o6RfxnTkq
Red-teaming found the checks didn't cover what they claimed: - The offline check verified 0 links in context/ — skill content has no relative links, so the blocking job only ever checked the 4 root docs. - example-apps/ was excluded as "human-read", but example-processor.js inlines README markdown raw (unfenced) into references/EXAMPLE.md inside the shipped skill ZIPs. Its links are agent-read. Both go away by globbing every .md in the repo instead of naming paths, which also means a new directory is covered without editing these files. Scoped to *.md rather than '.' because bare '.' parses HTML too, where root-relative asset paths can't resolve. This found one more real 404, in an example README. Dropped --include-fragments from the nightly: posthog.com renders some headings client-side, so a #fragment that works in a browser is absent from the HTML lychee sees. Anchors are still checked offline against local files. Also: pin lycheeVersion (the action SHA doesn't pin the binary it fetches), fail the job on any nonzero exit rather than only 2 (it would otherwise go green having checked nothing), curl -f so a dead webhook is visible, add timeout-minutes, and exclude localhost dev-server URLs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2iPGUrMb11u5o6RfxnTkq
Peetee06
commented
Aug 2, 2026
State the glob rationale in both workflows rather than pointing at the other file, which would drift silently if either is renamed. A failing check is assumed real until proven flaky, so saying so adds nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2iPGUrMb11u5o6RfxnTkq
Backtesting against the two commits that motivated this check found 0741482's markdown fix was a 308 to live content, not a 404 — so this check would not have caught it. That is the intended behaviour: posthog.com redirects moved docs rather than 404ing, and flagging ~40 redirects a night, most of them auth gates, costs more than it catches. A retired redirect becomes a 404 and the next nightly catches it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P2iPGUrMb11u5o6RfxnTkq
Peetee06
marked this pull request as ready for review
August 2, 2026 13:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds link checking for every markdown file in the repo.
Two recent fixes (0741482, fda834d) were dead links found by hand, after they had already shipped into agent context windows.
Two workflows:
link-check-external.yml– runs nightly, potentially flaky external links shouldn't block PRslink-check-offline.yml– runs on PRs and main and checks relative paths and #anchorsThe failure webhook is the one
e2e.yml/integration.ymluse.Known gap:
docs_urlsincontext/**/config.yamlare fetched into shipped skills but aren't markdown, so lychee doesn't see them.fetchDocOncethrows on 404 andbuild.ymlruns per-PR, so hard 404s are caught there.🤖 Generated with Claude Code