ci: optimize build pipeline with caching and parallelization#2974
Merged
bartlomieju merged 10 commits intomainfrom Mar 14, 2026
Merged
ci: optimize build pipeline with caching and parallelization#2974bartlomieju merged 10 commits intomainfrom
bartlomieju merged 10 commits intomainfrom
Conversation
Skip expensive reference doc generation (~4 min) on content-only PRs by caching reference_gen/gen/ output and detecting whether reference_gen/ files changed. Cache std-docs to avoid 43+ JSR API fetches per build. Run the build step as `deno task lume` directly instead of `deno task build` to avoid redundant reference generation. Parallelize afterBuild tasks (lint rule copy, LLM file generation, md source copy) and batch std-docs JSR fetches 10 at a time. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Instruments Lume build phases (scan, render, process, write) and individual processor timings (esbuild, tailwind, sitemap, etc.). Run with: LUME_PROFILE=1 deno task build:light Local results show render phase (4.6s) and esbuild (1.7s) + tailwind (1.1s) are the main time sinks within the Lume build. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cache _site/api/ (hundreds of rendered reference pages) keyed on the generated JSON data + rendering templates + components. On cache hit, set SKIP_REFERENCE=1 so Lume skips yielding and re-rendering all API reference pages. Since emptyDest is false, the cached HTML in _site/api/ is preserved. Also fix SKIP_REFERENCE checks to use value comparison (=== "1") instead of env.has(), which would trigger on empty strings. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add _config-reference.ts — a minimal Lume config that only renders API reference pages (JSX only, no markdown/esbuild/tailwind/sitemap). In CI, run it in parallel with the main build (which sets SKIP_REFERENCE=1 to skip reference pages), then merge outputs via cp. This turns the single-threaded 158s Lume build into two parallel builds: - Main (content pages): ~80-100s - Reference (API pages): ~60-80s Wall clock time is max(main, ref) instead of sum. Also adds `deno task build:parallel` for local use. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Simpler to debug and the speedup comes from the stripped-down reference config, not from parallelism. Both builds write to _site/ with emptyDest:false so output merges naturally. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Include reference_gen/types/ in the cache so that both the type extraction (86s) and doc generation (162s) are skipped on cache hit. Previously only gen/ was cached, meaning types would still regenerate on a cold gen cache miss. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The git diff change detection was redundant — the cache key already encodes whether reference_gen/ changed via hashFiles. Always attempt cache restore; on hit, skip both types (86s) and docs (162s). On miss, regenerate and the cache/save action saves automatically. Also removes fetch-depth:2 since git diff is no longer needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
thisisjofrank
approved these changes
Mar 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Optimizes the CI build pipeline from ~7 min to ~2.5 min (warm cache) or ~5 min (cold cache).
What changed
1. Cache reference types + docs (
reference_gen/types/+reference_gen/gen/)hashFiles('reference_gen/**')— only regenerates when scripts, configs, or dependency versions change2. Cache std-docs (
runtime/reference/std/)hashFiles('scripts/generate_std_docs.ts')3. Split Lume into two builds
_config-reference.ts: minimal config that renders only API reference pages (JSX only — no markdown-it, esbuild, tailwind, sitemap, toc, or OG images)SKIP_REFERENCE=1_site/withemptyDest: false4. Eliminate redundant reference generation
deno task lumedirectly instead ofdeno task buildwhich re-rangenerate:reference5. Parallelize I/O
Promise.all6. Fix
SKIP_REFERENCEcheck=== "1"instead ofenv.has(), which incorrectly triggered on empty strings7. Add build profiling (
LUME_PROFILE=1)LUME_PROFILE=1 deno task build:lightto use locallyExpected CI times
New tasks
deno task lume:reference— run the reference-only Lume builddeno task build:split— full build using the split strategy