APPROVED: Add bot protection: rescue MissingTemplate, robots.txt#1339
Open
APPROVED: Add bot protection: rescue MissingTemplate, robots.txt#1339
Conversation
…page Bots/crawlers requesting text/plain or other non-HTML formats caused a 500 error because no text template exists. Wrapping in respond_to returns 406 Not Acceptable for unsupported formats instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ttack - Rescue ActionView::MissingTemplate globally in ApplicationController to return 406 instead of 500 when bots request non-HTML formats - Revert per-controller respond_to in HomeController (global rescue covers it) - Add allowlist-based robots.txt that only permits crawling public pages - Add rack-attack gem with login throttle and global rate limit Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jmilljr24
approved these changes
Mar 4, 2026
Collaborator
jmilljr24
left a comment
jmilljr24
left a comment
There was a problem hiding this comment.
Can we wait to merge this to staging at a separate time from the bug fixes? I don't think it will break any but it's an app wise change.
Also was rail built in rate limit considered? Not sure exactly what we need but if something included with rails works it's worth avoiding a dependency.
Rails 8.1 provides native rate_limit in controllers, so no need for an external gem. Added global rate limit (300 req / 5 min per IP) in ApplicationController. Devise already handles login-specific protection via account locking after 10 failed attempts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
A blanket rate limit risks blocking legitimate users (e.g. search fields firing per-keystroke). Devise account locking already handles login brute-force. Targeted rate limits can be added later where needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Collaborator
Author
|
@jmilljr24 yes ofc re waiting! we just might get some more of those index template errors in honeybadger. rack-attack is a very common gem and i've used it before. TIL that rails 8.1 built in some rate-limiting options! still, i'll remove the rate limiting stuff from this pr to keep it focused on the error we saw and updating robots.txt. |
maebeale
changed the title
maebeale
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the goal of this PR and why is this important?
Accept: text/plain) cause 500 errors (ActionView::MissingTemplate) across all public-facing controllersrobots.txt, leaving it open to unnecessary crawler traffic on internal pagesHow did you approach the change?
rescue_from ActionView::MissingTemplateinApplicationControllerreturns 406 Not Acceptable instead of 500 — covers all controllers at oncerobots.txt— blocks everything by default (Disallow: /), explicitly allows only public pages (workshops, resources, stories, etc.). Avoids leaking internal app structureAnything else to add?
rate_limit) but deferred — a blanket limit risks false positives on legitimate use (e.g. search fields firing per-keystroke), and Devise account locking already handles login brute-force🤖 Generated with Claude Code