Skip to content

ci: optimize build pipeline with caching and parallelization#2974

Merged
bartlomieju merged 10 commits intomainfrom
ci/optimize-build-pipeline
Mar 14, 2026
Merged

ci: optimize build pipeline with caching and parallelization#2974
bartlomieju merged 10 commits intomainfrom
ci/optimize-build-pipeline

Conversation

@bartlomieju
Copy link
Member

@bartlomieju bartlomieju commented Mar 14, 2026

Summary

Optimizes the CI build pipeline from ~7 min to ~2.5 min (warm cache) or ~5 min (cold cache).

What changed

1. Cache reference types + docs (reference_gen/types/ + reference_gen/gen/)

  • On cache hit, skips type extraction (86s) and doc generation (162s) entirely
  • Keyed on hashFiles('reference_gen/**') — only regenerates when scripts, configs, or dependency versions change
  • First run after merge seeds the cache; all subsequent PRs hit it

2. Cache std-docs (runtime/reference/std/)

  • Skips 43 sequential JSR API fetches
  • Keyed on hashFiles('scripts/generate_std_docs.ts')

3. Split Lume into two builds

  • New _config-reference.ts: minimal config that renders only API reference pages (JSX only — no markdown-it, esbuild, tailwind, sitemap, toc, or OG images)
  • CI runs reference build first, then main build with SKIP_REFERENCE=1
  • Both write to _site/ with emptyDest: false
  • Main build no longer renders hundreds of API pages through the full plugin chain

4. Eliminate redundant reference generation

  • Build step runs deno task lume directly instead of deno task build which re-ran generate:reference

5. Parallelize I/O

  • afterBuild tasks (lint rule copy, LLM files, md source copy) run concurrently via Promise.all
  • Std-docs JSR fetches batched 10 at a time instead of 43 sequential

6. Fix SKIP_REFERENCE check

  • Use === "1" instead of env.has(), which incorrectly triggered on empty strings

7. Add build profiling (LUME_PROFILE=1)

  • Instruments build phases and individual processor timings
  • LUME_PROFILE=1 deno task build:light to use locally

Expected CI times

Step Before After (warm cache) After (cold cache)
Setup + fmt + test 20s 20s 20s
Reference types + docs 248s ~2s 248s
Std-docs 15s ~1s ~5s
Build: reference pages (in main build) ~30-40s ~30-40s
Build: main site 158s ~80s ~80s
Server + link checker 35s 35s 35s
Total ~7 min ~2.5 min ~5 min

New tasks

  • deno task lume:reference — run the reference-only Lume build
  • deno task build:split — full build using the split strategy

bartlomieju and others added 10 commits March 14, 2026 08:27
Skip expensive reference doc generation (~4 min) on content-only PRs by
caching reference_gen/gen/ output and detecting whether reference_gen/
files changed. Cache std-docs to avoid 43+ JSR API fetches per build.
Run the build step as `deno task lume` directly instead of `deno task
build` to avoid redundant reference generation.

Parallelize afterBuild tasks (lint rule copy, LLM file generation, md
source copy) and batch std-docs JSR fetches 10 at a time.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Instruments Lume build phases (scan, render, process, write) and
individual processor timings (esbuild, tailwind, sitemap, etc.).

Run with: LUME_PROFILE=1 deno task build:light

Local results show render phase (4.6s) and esbuild (1.7s) + tailwind
(1.1s) are the main time sinks within the Lume build.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cache _site/api/ (hundreds of rendered reference pages) keyed on the
generated JSON data + rendering templates + components. On cache hit,
set SKIP_REFERENCE=1 so Lume skips yielding and re-rendering all API
reference pages. Since emptyDest is false, the cached HTML in _site/api/
is preserved.

Also fix SKIP_REFERENCE checks to use value comparison (=== "1")
instead of env.has(), which would trigger on empty strings.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add _config-reference.ts — a minimal Lume config that only renders API
reference pages (JSX only, no markdown/esbuild/tailwind/sitemap). In CI,
run it in parallel with the main build (which sets SKIP_REFERENCE=1 to
skip reference pages), then merge outputs via cp.

This turns the single-threaded 158s Lume build into two parallel builds:
- Main (content pages): ~80-100s
- Reference (API pages): ~60-80s
Wall clock time is max(main, ref) instead of sum.

Also adds `deno task build:parallel` for local use.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Simpler to debug and the speedup comes from the stripped-down reference
config, not from parallelism. Both builds write to _site/ with
emptyDest:false so output merges naturally.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Include reference_gen/types/ in the cache so that both the type
extraction (86s) and doc generation (162s) are skipped on cache hit.
Previously only gen/ was cached, meaning types would still regenerate
on a cold gen cache miss.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The git diff change detection was redundant — the cache key already
encodes whether reference_gen/ changed via hashFiles. Always attempt
cache restore; on hit, skip both types (86s) and docs (162s). On miss,
regenerate and the cache/save action saves automatically.

Also removes fetch-depth:2 since git diff is no longer needed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bartlomieju bartlomieju merged commit 2c70b95 into main Mar 14, 2026
2 checks passed
@bartlomieju bartlomieju deleted the ci/optimize-build-pipeline branch March 14, 2026 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants