Move docs-dev to S3 backends as the docs.ruby-lang.org migration canary

[znz]

Summary

Canary for moving docs.ruby-lang.org off the docs-origin server onto S3 +
Fastly, on the docs-dev service (production docs.tf is untouched).

A targeted plan has been run and comes out exactly as intended —
Plan: 6 to add, 1 to change, 0 to destroy (the five bucket resources and
the dictionary items created, docs_dev updated in place, nothing destroyed).
Please apply with the real datadog_token: the local plan passed
-var datadog_token=dummy, which is why it shows the logging_datadog block
replaced; applying that literally would break the logging endpoint.

• cdn/s3.tf: new public-read docs.r-l.o bucket (us-east-1 like
  ftp.r-l.o — everything is fetched through Fastly, so the cheapest tier wins
  over client proximity). Prefix layout and writers are documented in the
  file; write access is meant to be split per pipeline with IAM prefix
  conditions rather than per-bucket.
• cdn/docs_dev.tf: three backends — the docs bucket (shielded iad-va-us,
  default_host/domain arrangement from cache.tf), the rubyci bucket for
  /capi/en/master/ (step 1 keeps Doxygen where it is; step 2 later moves it
  into the docs bucket), and docs-origin as the fallback for unflagged paths
  so prefixes can be moved one at a time. Routing is request_conditions on a
  header flag set before #FASTLY recv (assigning req.backend in VCL would
  bypass shielding, per the cache.tf comment). Adds the docs_versions
  dictionary (latest=4.0, master=4.1).
• cdn/vcl/docs_dev.vcl: replaces the behavior-neutral boilerplate with
  the actual origin logic:
  • the five nginx redirect rules, unchanged semantics and order
  • 302 (temporary) for the unreleased version: /en/4.1/* and /ja/4.1/*
    go to the master alias until the release; keyed off the dictionary, so
    a release only updates dictionary values
  • query-string stripping (S3 would interpret them as API parameters)
  • trailing-slash redirect for extensionless directory-looking URLs, and
    index.html appended internally for trailing-slash URLs
  • /ja/latest/・/ja/master/ resolved internally via the dictionary (the
    bucket holds no symlink objects; client URLs stay on the alias)
  • Markdown content negotiation on /ja/ pages: Accept: text/markdown on
    an .html URL internally fetches the .md twin, with a restart fallback
    to .html where no twin exists (generated index pages, frozen RD
    versions); Accept is normalized to two values to keep Vary cacheable
  • Surrogate-Key / Cache-Control emitted at the edge with the existing
    purge-key scheme (alias and real version share one object, so its key
    carries both names)
  • S3 AccessDenied (public-read grants GetObject only) mapped to 404 with a
    short negative-cache TTL; x-amz-* headers stripped; 50x synthetic page

Testing (once content is synced to the bucket)

Against https://docs-rlo-dev.global.ssl.fastly.net/ (no DNS/TLS setup needed,
per the existing comment in docs_dev.tf). Minimum sync for testing:
rurema/generated-documents html/ja/4.0 → ja/4.0/, docs repo public/ →
root, one en version → en/<v>/.

• / /ja/ /ja/latest/ /ja/latest/doc/index.html 200
• /ja/latest/class/Array.html == /ja/4.0/class/Array.html
• /ja/latest/class/Array.md → text/markdown; charset=utf-8
• curl -H 'Accept: text/markdown' /ja/latest/class/Array.html → the .md
  twin; same on /ja/latest/class/index.html → falls back to HTML
• /ja/latest → 301 /ja/latest/
• /en/trunk/x → 301, /en/3.4/doc/y → 301, /ja/2.8.0/z → 301 /ja/3.0/z
• /ja/search/query:%E9%85%8D%E5%88%97/ → 301 /ja/search/?q=%E9%85%8D%E5%88%97
• /en/4.1/anything → 302 /en/master/anything, /ja/4.1/x → 302 /ja/master/x
• /capi/en/master/ serves Doxygen, /capi/en/master/sitemap.xml 200
• a missing path is 404 (not 403); Fastly-Debug: 1 shows the Surrogate-Key
  and purging ja/4.0 / ja/latest works

Follow-ups (separate PRs)

• GitHub Actions upload credentials: OIDC roles (id-token: write +
  aws-actions/configure-aws-credentials) with per-pipeline IAM prefix
  conditions, replacing the long-lived secrets currently used for rubyci
• Deploy pipelines: gd → ja/, ruby/actions docs.yml → en/<v>/, docs repo
  public/ → root; one-off sync for frozen en versions and old ja versions
• Adopting the proven shape in docs.tf, then retiring docs-origin

🤖 Generated with Claude Code