Skip to content

Optimize CI: engine-grouped testing (42 jobs to 8)#1939

Merged
bpamiri merged 18 commits intodevelopfrom
peter/ci-engine-grouped-testing
Mar 13, 2026
Merged

Optimize CI: engine-grouped testing (42 jobs to 8)#1939
bpamiri merged 18 commits intodevelopfrom
peter/ci-engine-grouped-testing

Conversation

@bpamiri
Copy link
Collaborator

@bpamiri bpamiri commented Mar 12, 2026 •
edited
Loading

Summary

Restructures the GitHub Actions test matrix from 42 independent jobs (cfengine × dbengine) to 8 engine-grouped jobs that start each CF engine once and run all database test suites sequentially.

  • ~75% compute reduction — CF engine startup, CFPM install (Adobe), and container build happen once per engine instead of 6× per database
  • Oracle sleep 120 replaced with sqlplus health check loop (saves ~90s per occurrence)
  • Full test coverage preserved — all engine × database combinations still tested
  • JUnit XML output for PR annotations via EnricoMi/publish-unit-test-result-action
  • Engine × Database summary grid rendered in $GITHUB_STEP_SUMMARY and posted as PR comment
  • SQL Server memory reduced from 4GB to 2GB (fits all DBs in 7GB runner simultaneously)
  • Oracle memory reduced from 2GB to 1.5GB
  • BoxLang volumes re-enabled for local dev parity
  • Node.js 24 migration — bumped checkout@v5, labeler@v6, upload/download-artifact@v6
  • Container crash recovery — auto-restarts CF engine if it exits during startup (up to 3 retries)

How it works

Each engine job:

  1. Starts the CF engine + all compatible databases in parallel
  2. Waits for readiness (health checks, CFPM install, auto-restart on crash)
  3. Loops through each database, restarting the CF engine between runs for clean application state
  4. Converts JSON results to JUnit XML locally (no second HTTP request — runner.cfm doesn't cache)
  5. Continues even if one database fails (all DBs always run)
  6. Uploads per-database JUnit artifacts

Summary jobs download all artifacts and render a matrix grid + PR annotations.

Database exclusions (preserved)

Engine Excluded
adobe2018 sqlite, h2
adobe2021/2023/2025, boxlang h2
lucee5/6/7 (none)

Test plan

  • Verify all 8 engine jobs start and run correctly
  • Verify database health checks work (especially Oracle sqlplus loop)
  • Verify container restart between DB runs clears model cache
  • Verify JUnit XML artifacts are uploaded
  • Verify publish-results job creates PR annotations
  • Verify test-matrix-summary renders the grid (step summary + PR comment)
  • Verify memory budget: all DBs fit in 7GB runner simultaneously

🤖 Generated with Claude Code

@bpamiri @claude
docs: add implementation plan for CI engine-grouped testing optimization
96dacae 
Plan to restructure the GitHub Actions test matrix from 42 independent jobs
(cfengine x dbengine) to 8 engine-grouped jobs. Each CF engine starts once
and runs all database test suites sequentially, reducing total CI compute
by ~75%.

Key changes planned:
- Engine-only matrix (8 jobs vs 42)
- CFPM install once per Adobe engine instead of 6x
- Oracle sleep 120 replaced with sqlplus health check
- SQL Server memory 4GB->2GB, Oracle 2GB->1.5GB
- JUnit XML reporting via JSON-to-XML conversion
- Engine x database summary grid on workflow runs
- BoxLang volume mounts re-enabled for local dev parity

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions bot added the docs label Mar 12, 2026
bpamiri and others added 6 commits March 12, 2026 12:19
@bpamiri @claude
ci: reduce SQL Server memory from 4GB to 2GB for CI
ea27cee 
The test dataset is tiny (5 users, 8 authors, 40 photos). 4GB was excessive
and prevents running all databases simultaneously on a 7GB GitHub runner.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri @claude
ci: reduce Oracle memory from 2GB to 1.5GB for CI
305c641 
Allows running all databases simultaneously within 7GB GitHub runner budget.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri @claude
fix: re-enable BoxLang volume mounts for local dev parity
7e5d7ba 
Volumes were commented out in 9790880 when Dockerfile was changed to COPY
all code at build time. This broke local dev (code changes required rebuild).
Re-enabling volumes restores live-reload behavior matching other CF engines.
Docker gives volumes precedence over COPY, so CI still works correctly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri @claude
ci: restructure test matrix from 42 jobs to 8 engine-grouped jobs
01b2a2a 
Replaces cfengine x dbengine matrix (42 jobs) with cfengine-only matrix (8 jobs).
Each job starts one CF engine + all databases, runs test suites sequentially.

Key changes:
- CF engine starts once per job instead of 6x (saves 5 startups per engine)
- CFPM install runs once per Adobe engine instead of 6x (saves ~50 min compute)
- Oracle sleep 120 replaced with sqlplus health check loop
- All databases start in parallel during engine warmup
- Tests continue after individual database failures (all DBs always run)
- JUnit XML output uploaded for each engine-database pair
- Summary jobs render engine x database grid on workflow run page
- PR annotations via EnricoMi/publish-unit-test-result-action

Total compute reduction: ~75% (from ~840 min to ~200 min)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri @claude
ci: add checks and pull-requests write permissions for test reporting
56cd21d 
Required by EnricoMi/publish-unit-test-result-action to post test
result annotations and PR comments.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri @claude
ci: add permissions to snapshot workflow for test reporting
389cc11 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri bpamiri marked this pull request as ready for review March 12, 2026 19:26
bpamiri and others added 7 commits March 12, 2026 12:57
@bpamiri @claude
fix(ci): separate reload request from test request to handle 302 redi…
a6cc1d3 
…rect

Wheels returns HTTP 302 after reload=true completes. The test curl was
capturing this 302 as a failure instead of following the redirect.

Fix: issue the reload as a separate curl with -L (follow redirects),
then run the actual test request without reload.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri @claude
fix(ci): use curl -L to follow reload redirect in single request
466eb25 
The previous fix split reload and test into two requests. But the reload
request without format=json would run the full test suite in HTML format,
then the second request would run it again — doubling execution time.

Better approach: include reload=true AND format=json in one curl with -L.
Wheels' redirectAfterReload strips reload/password but preserves db= and
format=, so the redirect lands on ?db=X&format=json — exactly what we need.
Single request, single test execution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri @claude
fix(ci): remove reload=true — not needed and breaks Adobe CF
07be486 
The ?db= parameter switches application.wheels.dataSourceName directly
(runner.cfm:70-80). No reload needed because:
- $_setTestboxEnv() handles datasource switching per HTTP request
- populate.cfm creates identical table schemas across all databases
- The old 42-job workflow never used reload=true either

reload=true was breaking Adobe CF engines because $handleRestartAppRequest
passes "application" as a component reference to cfinvoke, which Adobe CF
can't resolve. Removing it fixes all Adobe engines and simplifies the curl.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri @claude
ci: bump actions/labeler v5 → v6 for Node.js 24 compatibility
b8c98a6 
v5 uses Node.js 20 which is deprecated and will be forced to Node.js 24
on June 2, 2026. v6 natively supports Node.js 24.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri @claude
fix(ci): bump actions to Node 24, restart engine between DB runs
b70a435 
Action version bumps (Node.js 24 support):
- actions/checkout v4 → v5
- actions/upload-artifact v4 → v5
- actions/download-artifact v4 → v5

Restart CF engine container between database runs to prevent cross-DB
contamination. Without restart, cached model metadata and association
methods from one database's test run leak into subsequent runs:
- BoxLang: shallow copy in runner.cfm causes datasource name to not
  actually switch (duplicate key errors on 2nd+ database)
- Adobe CF: stale association methods cause "method not found" errors
  on later databases (e.g. hasAuthor missing on 4th DB)

Container restart takes ~10-15s — much cheaper than a full rebuild,
and guarantees clean application state per database.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri @claude
Bump upload-artifact and download-artifact to v6 for Node.js 24
29c8870 
Fixes remaining Node.js 20 deprecation warnings in CI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri @claude
Fix JUnit XML int parsing and force Node.js 24 for download-artifact
1aa54ec 
- Cast count attributes to int() in JUnit XML generation (CFML returns
  floats for numeric values, causing ValueError in publish action)
- Set FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 on publish-results and
  test-matrix-summary jobs (download-artifact@v6 still ships Node 20)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Contributor

github-actions bot commented Mar 13, 2026 •
edited
Loading

Wheels Test Results

    42 files   3 696 suites   24m 34s ⏱️
76 902 tests 76 521 ✅ 381 💤 0 ❌
79 548 runs  79 167 ✅ 381 💤 0 ❌

Results for commit ddf142a.

♻️ This comment has been updated with latest results.

@bpamiri @claude
Post test matrix grid as PR comment
91297d7 
- Build matrix markdown into variable, write to both step summary
  and PR comment via gh CLI
- Update existing comment on subsequent pushes (no duplicates)
- Add pull-requests:write permission to test-matrix-summary job
- Move FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 to workflow-level env
- Include engine/db prefix in JUnit XML suite names for granularity
- Show failure count in matrix cells when tests fail

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Contributor

github-actions bot commented Mar 13, 2026 •
edited
Loading

Wheels Test Matrix

Engine MySQL PostgreSQL SQL Server H2 Oracle SQLite
lucee5 ✅ ✅ ✅ ✅ ✅ ✅
lucee6 ✅ ✅ ✅ ✅ ✅ ✅
lucee7 ✅ ✅ ✅ ✅ ✅ ✅
adobe2018 ✅ ✅ ✅ -- ✅ --
adobe2021 ✅ ✅ ✅ -- ✅ ✅
adobe2023 ✅ ✅ ✅ -- ✅ ✅
adobe2025 ✅ ✅ ✅ -- ✅ ✅
boxlang ✅ ✅ ✅ -- ✅ ✅

Results for commit 1bb880d.

bpamiri and others added 3 commits March 13, 2026 05:33
@bpamiri @claude
Fix json_test_case_results: use boolean not string
35ca3d2 
The publish-unit-test-result-action expects true/false, not "all".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri @claude
Use comma as thousands separator in test results comment
7f90943 
Changes "3 696 suites" to "3,696 suites" in the PR comment.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri @claude
Auto-restart CF engine container if it crashes during startup
ddf142a 
Adobe CF containers occasionally fail startup due to transient network
errors (CommandBox dependency clone failures). The previous wait logic
used docker ps which can't detect crashed containers, causing a 150s
timeout with no feedback.

Now uses docker inspect to detect exited/dead containers and retries
up to 3 times with docker compose up -d. Also adds proper error
annotations and container log output on failure.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@bpamiri bpamiri merged commit e5e2d59 into develop Mar 13, 2026
12 checks passed
@bpamiri bpamiri deleted the peter/ci-engine-grouped-testing branch March 13, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

docs

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bpamiri