DOC-6951 Add a scanner for pages that moved without an alias - #3767
Open
andy-stark-redis wants to merge 7 commits into
Open
DOC-6951 Add a scanner for pages that moved without an alias#3767andy-stark-redis wants to merge 7 commits into
andy-stark-redis wants to merge 7 commits into
Conversation
Renaming a content file changes its URL, and the old URL dies unless the page declares an alias for it. That is author-declared and measurably unreliable -- 611 URL-changing moves in this repo's history, only 302 aliased. It fails inconsistently even inside one commit: 1552778 moved LangCache and Agent Memory under context-engine together, LangCache got an alias, Agent Memory did not, and its old URL 404s today. So the check has to be mechanical. The scanner walks git rename records, works out which renames actually changed a published URL, and reports the ones with no alias. One implementation serves three modes -- a branch range for a PR check, a full-history sweep, and a writing fix -- deliberately, because the recurring failure on DOC-6939 has been the same rule implemented twice and then drifting apart. The whole difficulty is false positives. A naive first version reported 961 missing aliases against a true 259, and every one of the extras looked plausible in a list, which is the dangerous kind of wrong. Two thirds of that noise was Hugo bundles: index.md and _index.md both publish at the containing directory's URL, so renaming commands/lpushx/index.md to commands/lpushx.md changes no URL at all, and 519 of this repo's renames are that shape. Writing the scanner a second time is what found the rest, after a 25-URL live sample had already blessed the first version. Five things it had wrong. 49 files spell the empty key "aliases: null", which the first fixer promoted into a list containing a literal null. 104 files give aliases a bare scalar instead of a list -- a shape I had not seen at all, and my earlier "192 empty keys" figure was really 88 empty plus those 104. One file uses a folded multi-line scalar, which is valid YAML that quietly folds two intended aliases into one string, so that author's second alias has never worked; the fixer refuses that file rather than guess. content/embeds/ is excluded from the site by a build.render never cascade rather than by any naming convention, so the underscore heuristic missed 119 files. And 28 of the gaps are collisions where another page already claims the old URL, where Hugo picks a winner arbitrarily and warns -- adding those automatically would have made 28 redirects ambiguous instead of fixing them. The frontmatter edit is line-based on purpose. Round-tripping through PyYAML reorders keys alphabetically and renormalizes quoting, which would rewrite every file it touched into an unreviewable diff. Verified by running the fix over the whole corpus: 257 aliases across 203 files, every changed file still valid YAML with no empty or null entries, no new duplicate alias claims anywhere in content, and a rescan closing to the single file the fixer correctly declines. Then reverted -- the backfill is its own item and wants its own PR. Learned: writing the measurement a second time found five bugs that a live-sampled first version had already passed, and the bundle trap alone was two thirds of the false positives Constraint: --fix must skip any move whose old URL is already claimed by another page or wanted by another moved page, because Hugo resolves a duplicate alias by picking one arbitrarily and only warning Constraint: frontmatter edits here stay line-based -- a yaml.safe_load/yaml.dump round-trip reorders keys and renormalizes quoting, rewriting every touched file Constraint: whether a path is published is read off the tree via a build.render never cascade, not inferred from an underscore-prefixed directory name Gaps: no full Hugo build has been run against the fixed corpus yet, so freedom from build-time alias warnings is argued from duplicate-claim analysis rather than observed Ticket: DOC-6951 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at feceb6d |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Both Bugbot findings were real, and one of them led somewhere it had not pointed. The first is the one that mattered. With --fix, main returned 0 whether or not apply_fixes had managed to place every alias, so the weekly sweep planned as item C3 would have opened a pull request captioned as a complete fix while leaving files untouched. There is already one such file, the folded multi-line scalar the fixer declines. apply_fixes now returns what it skipped, main lists those files, and --fail exits 1. The second was assigning history[new] rather than merging, which silently drops an earlier rename chain if a second file later moves onto the same path. Measured across the whole history that happens exactly once, and in a degenerate form where the dropped record carries the same old_path, so nothing was actually being lost. Merging anyway, because the loss would be invisible and this script exists to stop precisely that class of thing. Retaining that record is what exposed the real problem underneath. It pushed the totals to 612 moves and 29 collisions, and chasing the extra record showed that 14 old_url-to-new_path pairs arrive twice -- 13 of them through chains, so predating the merge fix entirely. A page moved away and back, or a recurring rename like the monthly Cloud changelog, produces two records for one redirect. A redirect is identified by where it comes from and where it goes, so those are now deduplicated, keeping the earliest. That moves the published figures: 598 URL-changing moves rather than 611, 290 already aliased rather than 302, 258 safely fixable rather than 259. Collisions are unchanged at 28, being 24 where another page already claims the URL and 4 where two moved pages both want it. Every figure in the module docstring was re-measured rather than adjusted by hand, since they had already drifted once. Learned: a review bot found a latent silent-drop and an automation trap in code that unit tests and a full-corpus dry run had both passed, and fixing the smaller one surfaced a larger duplicate-redirect bug that predated it Constraint: a redirect is identified by the pair of old URL and target path, so moves are deduplicated on that pair -- chains and recurring renames otherwise report one redirect twice Constraint: --fix must not exit 0 when it skipped a file, or an automated sweep reports a complete fix it did not make Ticket: DOC-6951 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at eea0761 |
Corrects a false positive I had already written up as a content bug in three places. One file declares its aliases as a YAML scalar folded across two lines. PyYAML reads that as the single string "/a/ /b/", so the scanner reported the page as missing an alias and declined to fix it, and I concluded the author's second alias had never worked. Hugo disagrees, and Hugo is the authority. It casts a scalar aliases value with cast.ToStringSlice, which runs strings.Fields, so a bare string is split on whitespace into several aliases. Reproduced against Hugo 0.143.1 in a throwaway site: the folded frontmatter yields two entries from .Aliases and Hugo writes both alias stubs. Both of that page's aliases resolve on the live site today, so there was never anything wrong with it. The lesson is one I had already written down and did not apply here. An authoritative oracle beats a reimplementation of a spec, and a YAML library is a reimplementation as far as Hugo frontmatter semantics are concerned. What made it convincing was that PyYAML's reading is correct YAML -- the folding is real, and it is Hugo that layers a whitespace split on top. Only list items are exempt, since cast.ToStringSlice splits a string but converts each element of a slice individually. Exactly one file in the corpus is affected, and it now reads as already aliased, which takes the count from 290 aliased and 258 fixable to 291 and 257, and leaves --fix with no skipped files at all. Learned: an authoritative oracle beats a library even when the library is right on its own terms -- PyYAML folds a multi-line scalar correctly, and Hugo then splits the result on whitespace, so only Hugo settles what an alias means Constraint: a scalar aliases value is whitespace-separated because Hugo casts it with cast.ToStringSlice; list items are not split Ticket: DOC-6951 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at f72e0a5 |
An eighth trap, and the first one found by building the site rather than by reading code. Applying the backfill and running a full Hugo build, then checking every stub the build actually emitted, turned up two aliases Hugo had declined to write. Both belonged to content/integrate/write-behind/_index.md, which carries draft: true. Production builds run plain hugo with no --buildDrafts, so a draft page emits nothing at all -- and that includes its aliases. So the scanner was reporting a gap it could not close: --fix wrote two aliases into a draft, reported success, and Hugo ignored them. The same blindness runs the other way, because published_urls treated drafts as occupying their URL, which would have suppressed a real redirect had one pointed there. Nothing was affected today, but only by luck: 31 files are drafts and 13 of them hold an eligible URL. Drafts are now excluded from published_urls, a move whose target is a draft is reported in its own category rather than counted as fixable, and the draft set is found by grep before parsing so this does not read all 5,867 content files. Worth recording how this was caught, because it is the argument for the build step existing at all. The gap was invisible to unit tests, to a full-corpus dry run, to duplicate-claim analysis, and to Bugbot. Absence of build warnings would not have caught it either -- the build was clean at --logLevel warn, and Hugo says nothing when it skips a draft's aliases. Only comparing the aliases we wrote against the stubs Hugo emitted showed it. Learned: the only check that found this compared what we wrote against what Hugo actually emitted; a clean build log is not evidence, because Hugo skips a draft's aliases silently Constraint: a draft page publishes nothing including its aliases, so drafts are excluded from published_urls and a move whose target is a draft is never auto-fixed Ticket: DOC-6951 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at 782d3b5 |
Diffing two full builds -- one with the backfill applied, one without -- showed
256 new alias stubs, no real page overwritten, and one page lost. The lost path
was operate/kubernetes/release-notes/7-4-6-2, with a trailing comma, and it
returns 200 on the live site today while the comma-free spelling returns 404.
The cause was promoting a scalar aliases value to an inline list. One file holds
aliases: /operate/kubernetes/release-notes/7-4-6-2,
because an author wrote a list without brackets, and Hugo publishes an alias whose
path ends in a comma. Written inline as [/...7-4-6-2, /new/] that comma becomes
the list separator, so the alias silently changes to the comma-free path: a
working URL turned into a 404 and a 404 into a working URL. I had also been
stripping the trailing comma deliberately, on the theory that it was a typo. It
probably is, but a mechanical backfill is not the place to decide that, and
nothing in the output would have shown the decision being made.
Scalars are now promoted to a block list instead, which has no separator to
confuse with content, so each existing token survives byte for byte. Tokens are
split on whitespace to match Hugo's cast.ToStringSlice. The repo's dominant style
is a block list anyway, so the diffs read better as a side effect. No inline list
is now written in the whole corpus.
The general point is that this was invisible in every artifact except a
build-to-build page-set diff. The build log was clean, the alias count was
correct at 1185, and 270 of 270 expected stubs were present -- the loss only
appeared as an index.html count of 6826 against a baseline 6571, one short of the
256 added.
Learned: comparing two full builds' page sets caught a silently changed live URL that a clean build log, a correct alias count and a stub-by-stub check all missed
Constraint: existing alias values are preserved byte for byte -- promote a scalar to a block list, never inline, because a value containing a comma changes meaning inside brackets
Rejected: stripping a stray trailing comma from an author's alias | it is probably a typo but the comma'd URL is live, so a mechanical backfill must not silently retire it
Ticket: DOC-6951
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at 6c78b09 |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6c78b09. Configure here.
Three findings from a second Bugbot pass, plus the one that matters, which came from a review comment on the backfill rather than from the code. The important one is conceptual. git records lineage, and lineage is not equivalence. When a page is split up -- X.md becoming X/child.md -- the old URL turns into a section URL while the file itself becomes one page inside it. The scanner faithfully followed the file and proposed redirecting the old landing page to that child. On the Java client docs that meant a bookmark for the generic develop/connect/clients/java page landing on a Jedis connection guide, four renames later, rather than on the Jedis hub that exists today. Reviewers spotted two instances; there are six, including the same mistake in the redis-py docs. Splits to X/_index.md are unaffected, because the URL does not change. Only a split to a named child demotes the old URL, and those chains are now reported for a person to decide rather than fixed. That drops the backfill from 256 to 253 and moves three cases out of the collision bucket, where they had been landing for the wrong reason. Also fixed, both raised by Bugbot and both real but unrealized here. Renames within a single commit were applied in git's listing order, so a commit holding both A->B and B->C would lose the A move if git listed them the other way round; ordering is now derived from the dependencies, with a cycle guard, and extracted into order_renames so it can be tested directly. And the folded-scalar guard treated any indented line as a continuation, including a whitespace-only one, so an ordinary scalar followed by a padded blank line was skipped; it now requires actual content. One Bugbot finding was a false positive and is deliberately unchanged: it read the inline-list branch as collapsing a two-item list into one string. The intermediate does hold both items in one string, but the join puts them back correctly -- [/a/one/, /a/two/] plus /a/three/ yields all three, verified by parsing the result back out. Learned: the deepest bug in this scanner was not a coding error but a category error -- following the file instead of the meaning -- and it took a human reviewer looking at content to see it, after four rounds of measurement had passed Constraint: a chain crossing a page-into-section split is never auto-fixed, because the old URL was a landing page and its lineage ends at one child Ticket: DOC-6951 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at 9a39894 |
Contributor
Author
|
Thanks @dwdougherty ! |
A finding and a failure shared exit code 1: --fail returned it when gaps remained, and so did an internal git failure. The workflow in the companion PR reads that code to decide whether the fixer declined a file, so a git failure would have produced "some aliases could not be added automatically" -- pointing whoever read it at the content rather than at the broken scan. Named codes now: 0 nothing to report, 1 findings under --fail, 2 the scan itself failed. The workflow already treats anything above 1 as a hard error, so it needs no change beyond a comment. Learned: a caller that cannot distinguish "I found a problem" from "I broke" will report the wrong one, and the cost lands on whoever reads the message rather than on the code Constraint: exit 1 means findings and exit 2 means the scan failed; do not collapse them, because CI decides what to say from this Ticket: DOC-6951 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
🧠 Redis MemoryFound 5 related items from repository history:
Memory updated at b72cfc1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a scanner that finds content pages which moved without gaining an alias for their old URL — item A of DOC-6951. Tooling and tests only: no content changes, and nothing published yet.
Why
Renaming a content file changes its URL, and the old URL 404s unless the page declares an
aliases:entry. That's author-declared and measurably unreliable — across this repo's history, 291 of 598 URL-changing moves carry a matching alias. It also fails inconsistently within a single commit:155277839moved LangCache and Agent Memory undercontext-enginetogether, LangCache got an alias, Agent Memory didn't, anddevelop/ai/agent-memoryis a live 404 today.What's here
build/check_missing_aliases.pybuild/test_check_missing_aliases.pyMakefilemake check_aliases,make check_aliases_fixOne implementation, three modes, so the rule can't drift between them:
Also
--json,--github(Actions annotations),--fail(exit non-zero). Warn-only by default, matchingcheck_page_sizes.Current output:
The point of it is avoiding false positives
A naive first version reported 961 missing aliases against a true 257 — nearly three quarters noise, and every extra looked plausible in a list. Seven traps are handled explicitly, each documented in the module docstring with the measurement behind it:
index.mdand_index.mdboth publish at the containing directory's URL, socommands/lpushx/index.md→commands/lpushx.mdchanges no URL. 527 renames are this shape: the largest single source of noise.content/embeds/has abuild.render: nevercascade (240 renames). Read off the tree, not hardcoded, so a future one is caught free.url:frontmatter — set on exactly the 3,107 versioned files and nowhere else, so path→URL derivation is exact for the rest and invalid for those.aliases: null, 44 single-line inline, 39 bare, 19 multi-line inline, and 1 folded multi-line scalar. A scalar is whitespace-separated, because Hugo casts it withcast.ToStringSlice.Verification
--fixwas run over the whole corpus and then reverted:nullentriescontent/(29 before, 29 after — all pre-existing)Review round (
eea076141)Both Bugbot findings were valid and are fixed:
--fixalways exited 0, even when it couldn't place some aliases — which would have let the planned weekly sweep open a PR captioned as a complete fix.apply_fixesnow reports what it skipped and--failexits 1.Retaining that record then exposed something Bugbot hadn't flagged: 14
(old_url, new_path)pairs arrive twice, 13 of them via chains and so predating the merge fix. A page moved away and back, or a recurring rename like the monthly Cloud changelog, yields two records for one redirect. Now deduplicated.Correction (
f72e0a55b)I had reported one file as a content bug — a folded multi-line
aliasesscalar whose second alias "had never worked". That was wrong, and it was my scanner's bug, not the content's. Hugo casts a scalaraliasesvalue withcast.ToStringSlice, which runsstrings.Fields, so a bare string is split on whitespace into several aliases. PyYAML instead folds it to the single string"/a/ /b/"— correct YAML, but not what Hugo does. Reproduced against Hugo 0.143.1 in a throwaway site, and both of that page's aliases resolve live today. The scanner now matches Hugo, that page reads as already aliased, and--fixhas no skipped files at all.Together these moved the figures from 611/302/259 to 598/291/257.
One content issue found on the way (not fixed here)
27 pre-existing duplicate aliases involving non-versioned pages, unrelated to this change — two or more pages claiming the same alias, which Hugo resolves arbitrarily.
(An earlier revision of this description also listed
query-performance-factor.mdas broken. It isn't — see the correction above.)Known gap
No full Hugo build has been run against a fixed corpus, so freedom from build-time alias warnings is argued from duplicate-claim analysis rather than observed. That belongs with item B1, the historical backfill, where content actually changes.
Not in this PR
The 257-alias backfill (B1), the PR check and weekly sweep (C), and the published redirect map plus JSON tombstones (D). Deliberately split so the backfill's content diff is reviewable on its own.
🤖 Generated with Claude Code
Note
Low Risk
Build-time git analysis and optional frontmatter edits; no runtime or published-site behavior until CI or
check_aliases_fixis adopted.Overview
Adds warn-only tooling to catch Hugo content renames that change a published URL without an
aliases:redirect, so old links 404.build/check_missing_aliases.pywalks git rename history (defaultorigin/main..HEADor--all), derives old/new URLs with Hugo-aware rules (bundles,build.render: never, versionedurl:trees, draft pages, section splits, occupied paths, alias collisions), and reports gaps. Optional--fixinserts missing aliases via line-oriented frontmatter edits (no YAML round-trip), plus--json,--github, and--failfor CI.Makefileexposesmake check_aliasesandmake check_aliases_fix, mirroring the existing passivecheck_page_sizespattern.build/test_check_missing_aliases.pycovers URL derivation, move classification, and everyaliasesfrontmatter shape the fixer handles.No doc content is changed in this PR—scanner and tests only.
Reviewed by Cursor Bugbot for commit b72cfc1. Bugbot is set up for automated code reviews on this repo. Configure here.