[docs-agent] Exclude bscscan.com and tenderly.co from lychee#1428
[docs-agent] Exclude bscscan.com and tenderly.co from lychee#1428alchemy-bot wants to merge 2 commits into
Conversation
Weekly link check on 2026-07-06 flagged 3 broken links, all third-party bot-block / rate-limit failures: - bscscan.com/token/0x8AC76a51cc950d9822D68b83fE1Ad97B32Cd580d (403) - referenced from content/wallets/pages/gas-manager-admin-api/gas-manager-faqs.mdx - tenderly.co/ (429) - referenced from two MDX pages under content/wallets/pages/ Both are legitimate destinations, just crawler-blocked. Add exclude patterns matching the existing Etherscan-family and rate-limited-third-party conventions in the same file. Refs DOCS-133
🔗 Preview Mode
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: effb852e1c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| ".*celoscan\\.io.*", # Bot protection blocks crawlers with 403 (Etherscan-family explorer) | ||
| ".*bitsofco\\.de.*", # Intermittent connection failures from crawlers (page loads fine in browser) | ||
| ".*bscscan\\.com.*", # Bot protection blocks crawlers with 403 (Etherscan-family explorer) | ||
| ".*tenderly\\.co.*", # Rate-limits crawlers with 429 (page loads fine in browser) |
There was a problem hiding this comment.
In the scheduled link check, .github/workflows/link-check-scheduled.yml runs lychee over ./content with this config, and lychee excludes regex-matching links. This unanchored pattern matches every Tenderly subdomain, so current links like https://docs.tenderly.co/debugger and https://dashboard.tenderly.co/register are now skipped even though only https://tenderly.co/ was failing with 429. If any of those docs/dashboard URLs typo or return 404, the weekly check will stay green; please restrict the regex to the root host/path or add includes for docs/dashboard.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch, applied in 1439da1. Tightened to .*//tenderly\.co/?$ so only the bare root URL (which is what returned 429) is excluded. docs.tenderly.co/debugger, docs.tenderly.co/simulator-ui, and dashboard.tenderly.co/* will keep getting checked by the weekly lychee run.
Verified with the Rust regex::is_match-equivalent search behavior against the actual URLs in content/wallets/pages/transactions/debug-transactions/debug-with-tenderly.mdx and content/wallets/pages/bundler-api/entrypoint-revert-codes/entrypoint-v06-revert-codes.mdx — only https://tenderly.co/ (and https://tenderly.co) match.
bscscan.com intentionally stays broad because it's Etherscan-family bot protection (uniform across the whole domain, matching the celoscan.io precedent), and only the token page under the root is linked in the docs.
Codex P2 review flagged that the unanchored '.*tenderly\.co.*' pattern also excludes legitimate docs.tenderly.co/dashboard.tenderly.co links used elsewhere in the docs (e.g. debug-with-tenderly.mdx links to https://docs.tenderly.co/debugger and https://docs.tenderly.co/simulator-ui). Tightening to '.*//tenderly\.co/?$' matches only the bare root URL that was returning 429 while keeping the subdomain URLs discoverable to the weekly link check. Refs DOCS-133
1 participant
Summary
The 2026-07-06 weekly link check flagged 3 broken links, all third-party bot-block / rate-limit failures (matching the classification the bot itself recommends handling via the
excludelist):content/wallets/pages/gas-manager-admin-api/gas-manager-faqs.mdxhttps://bscscan.com/token/0x8AC76a51cc950d9822D68b83fE1Ad97B32Cd580dcontent/wallets/pages/bundler-api/entrypoint-revert-codes/entrypoint-v06-revert-codes.mdxhttps://tenderly.co/content/wallets/pages/transactions/debug-transactions/debug-with-tenderly.mdxhttps://tenderly.co/All three URLs point to valid, current destinations (BSC's block explorer page for the USDT token contract; Tenderly's homepage). The failures are purely crawler-side. Following the established pattern in
lychee.toml(existing precedent:celoscan.io,docs.arbitrum.io,developer.offchainlabs.com,medium.*, etc.), added two new exclude patterns:No MDX content changes needed. Next week's link check should report clean for these domains.
Linear
DOCS-133 — https://linear.app/alchemyapi/issue/DOCS-133/weekly-link-check-exclude-bscscan-and-tenderly-from-lychee
Slack
Originating thread: https://alchemyplatform.slack.com/archives/C0852FMMUMR/p1783360729767349
Requested by
Weekly Link Check bot (no human originator; no
Requested-by:trailer per no-self-approval workflow's degraded-attribution path).