Reduce Stemcell Distribution Egress Costs via Cloudflare CDN

[beyhan]

Summary

Stemcell downloads from https://storage.googleapis.com/bosh-aws-light-stemcells/...
(and equivalent GCS buckets for other IaaS) generate significant GCP egress costs (~2K $ per month).
We can drastically reduce these costs by fronting the GCS buckets with Cloudflare CDN under a community-owned domain such as stemcells.cloudfoundry.org.

Related migration: cloudfoundry/community#926

Motivation

• Stemcells are immutable, versioned, large artifacts — ideal CDN cache candidates.
• Current GCS egress is billed at ~$0.12/GB; Cloudflare egress to end users is free.
• A single cached version is fetched from origin once, then served from edge globally.
• Improves download speed and reliability for the community.

Proposed Architecture

User → https://stemcells.cloudfoundry.org/<iaas>/<version>/<file>.tgz
         │
         └── Cloudflare CDN (cached at edge)
                │  (cache miss only)
                └── storage.googleapis.com/bosh-<iaas>-light-stemcells/...

Example URL mapping:

• Before: https://storage.googleapis.com/bosh-aws-light-stemcells/1.425/light-bosh-stemcell-1.425-aws-xen-hvm-ubuntu-noble.tgz
• After: https://stemcells.cloudfoundry.org/aws/1.425/light-bosh-stemcell-1.425-aws-xen-hvm-ubuntu-noble.tgz

Action Items

Phase 1 — Infrastructure setup

• Create Cloudflare zone (or subdomain delegation) for stemcells.cloudfoundry.org
• Verify stemcell file sizes vs Cloudflare cache limits (Free 512MB / Pro 1GB / Business 5GB / Enterprise unlimited with Cache Reserve)
• Decide Cloudflare plan (or apply for Cloudflare's open-source / OSS sponsorship program)
• Ensure GCS buckets (bosh-aws-light-stemcells, bosh-gce-light-stemcells, bosh-azure-light-stemcells, etc.) have allUsers:objectViewer

Phase 2 — Cloudflare configuration

• Add CNAME stemcells.cloudfoundry.org → storage.googleapis.com (proxied)
• Add Origin Rule: override Host header to storage.googleapis.com
• Add URL Rewrite rule mapping /<iaas>/<path> → /bosh-<iaas>-light-stemcells/<path>
• Configure Cache Rule: Edge TTL = 1 month, Browser TTL = 1 day, cache eligible
• Enable Tiered Cache for better hit ratio
• (Optional) Enable Cache Reserve for large objects

Phase 3 — Release automation integration

• Update stemcell publishing pipeline (bosh-io / stemcells-index and equivalents) to:
  • Continue uploading to GCS as origin of truth
  • Issue a curl -I warm-up request after upload to pre-fill Cloudflare cache
  • Publish CDN URL (stemcells.cloudfoundry.org/...) in release metadata / bosh.io index
• Update bosh.io UI/API to point downloads at CDN URL
• Keep GCS URLs functional for backward compatibility (no breaking change)

Phase 4 — Monitoring & rollout

• Add Cloudflare Analytics dashboard (cache hit ratio, bandwidth saved)
• Compare GCP egress bill before/after (1 month baseline)
• Document new URL pattern in BOSH / stemcell docs
• Announce on cloudfoundry-dev mailing list & Slack #bosh

Phase 5 — Optional follow-up

• Evaluate full migration of origin to Cloudflare R2 (zero egress, no large-file cache limits) if cost savings justify operational change

Success Criteria

• >90% cache hit ratio after warm-up
• Measurable reduction in GCP egress costs (target: >70%)
• No regressions in stemcell download reliability
• Zero breakage for existing consumers using old GCS URLs

[beyhan]

We found that stemcell candidates are stored in buckets which are in a different region then the Concourse workers are running, which generates egress costs when the Concourse workers use the stemcell candidates. The first optimization is to relocate the buckets to the same GCP region like the Concourse workers and continue from the optimizations from there.

[aramprice]

@beyhan - it looks like production stemcells (aws lite at least) are also in this bucket.
cc @rkoster

[beyhan]

The was some confusion which buckets are in scope for this that is why we posted the previous comment. We don't expect any production impact from this cost optimization. All candidate buckets of FI WG GCP account has been relocated to region europe-west2 were also the concourse workers are provisioned. With this no cross-region egress cost for CI should occur. Additionally, a retention period of 30 days has been configured for the buckets. We will observe what is the impact of this change to the costs and discuss further optimization based on our findings.