test(orchestrator): increase test coverage, dev harness, and contributor docs

[lholmquist]

Hey, I just made a Pull Request!

This PR is part of the Backstage Stage Version Bump Trust work, this one focuses on the orchestrator plugins, https://redhat.atlassian.net/browse/RHIDP-14106

It also adds a contributor guide with steps for running the plugin dev/ harness, running the usual package/workspace validation commands, and a short manual smoke checklist.

✔️ Checklist

• A changeset describing the change and affected packages. (more info)
• Added or Updated documentation
• Tests for new functionality and regression tests for bug fixes
• Screenshots attached (for UI changes)

[rhdh-gh-app]

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@red-hat-developer-hub/backstage-plugin-orchestrator-backend-module-loki	workspaces/orchestrator/plugins/orchestrator-backend-module-loki	patch	v1.4.0
@red-hat-developer-hub/backstage-plugin-orchestrator-backend	workspaces/orchestrator/plugins/orchestrator-backend	patch	v8.11.0
@red-hat-developer-hub/backstage-plugin-scaffolder-backend-module-orchestrator	workspaces/orchestrator/plugins/scaffolder-backend-module-orchestrator	patch	v1.7.0

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.91%. Comparing base (e0d0986) to head (4405639).
⚠️ Report is 15 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4127      +/-   ##
==========================================
+ Coverage   54.89%   54.91%   +0.02%     
==========================================
  Files        2398     2399       +1     
  Lines       95836    95847      +11     
  Branches    26836    26813      -23     
==========================================
+ Hits        52610    52639      +29     
+ Misses      41734    41719      -15     
+ Partials     1492     1489       -3     

Flag	Coverage Δ		*Carryforward flag
adoption-insights	84.55% <ø> (ø)		Carriedforward from e0d0986
ai-integrations	69.71% <ø> (ø)		Carriedforward from e0d0986
app-defaults	69.79% <ø> (ø)		Carriedforward from e0d0986
augment	46.67% <ø> (ø)		Carriedforward from e0d0986
boost	76.77% <ø> (ø)		Carriedforward from e0d0986
bulk-import	72.56% <ø> (ø)		Carriedforward from e0d0986
cost-management	13.55% <ø> (ø)		Carriedforward from e0d0986
dcm	60.72% <ø> (ø)		Carriedforward from e0d0986
extensions	56.59% <ø> (ø)		Carriedforward from e0d0986
global-floating-action-button	71.18% <ø> (ø)		Carriedforward from e0d0986
global-header	68.56% <ø> (ø)		Carriedforward from e0d0986
homepage	47.50% <ø> (ø)		Carriedforward from e0d0986
install-dynamic-plugins	59.95% <ø> (ø)		Carriedforward from e0d0986
intelligent-assistant	74.53% <ø> (ø)		Carriedforward from e0d0986
konflux	91.98% <ø> (ø)		Carriedforward from e0d0986
lightspeed	69.02% <ø> (ø)		Carriedforward from e0d0986
mcp-integrations	83.40% <ø> (ø)		Carriedforward from e0d0986
orchestrator	67.16% <ø> (+0.30%)	⬆️
quickstart	62.72% <ø> (ø)		Carriedforward from e0d0986
sandbox	79.56% <ø> (ø)		Carriedforward from e0d0986
scorecard	85.34% <ø> (ø)		Carriedforward from e0d0986
theme	88.52% <ø> (ø)		Carriedforward from e0d0986
translations	5.12% <ø> (ø)		Carriedforward from e0d0986
x2a	12.40% <ø> (ø)		Carriedforward from e0d0986

*This pull request uses carry forward flags. Click here to find out more.

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0d0986...4405639. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[rhdh-qodo-merge]

PR Summary by Qodo

Orchestrator: add wiring tests, dev harnesses, and contributor guides

🧪 Tests 📝 Documentation ✨ Enhancement ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Add startTestBackend wiring tests for orchestrator backend and its extension modules.
• Introduce lightweight dev harnesses/config stubs for local backend validation.
• Publish per-package contributor guides and tighten key service unit-test assertions.

Diagram

graph TD
  A(["startTestBackend"]) --> B["orchestrator-backend"]
  A(["startTestBackend"]) --> C["scaffolder module"]
  A(["startTestBackend"]) --> D["loki module"]
  E(["dev/ harness"]) --> B["orchestrator-backend"]
  E(["dev/ harness"]) --> C["scaffolder module"]
  F[/"CONTRIBUTING docs"/] --> B["orchestrator-backend"]
  F[/"CONTRIBUTING docs"/] --> C["scaffolder module"]
  F[/"CONTRIBUTING docs"/] --> D["loki module"]

  subgraph Legend
    direction LR
    _t(["Test/harness runtime"]) ~~~ _p["Package/module"] ~~~ _d[/"Docs"/]
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Rely solely on unit tests (no startTestBackend wiring smokes)

• ➕ Faster tests with fewer moving parts
• ➕ Less coupling to Backstage backend initialization APIs
• ➖ Misses regressions in extension-point registration and auth-policy wiring during version bumps
• ➖ Lower confidence that modules load correctly together

2. Run full workspace app (packages/app + packages/backend) as the primary CI signal

• ➕ More realistic end-to-end coverage across UI + backend + integrations
• ➖ Significantly heavier, slower, and more brittle for bump-trust validation
• ➖ Requires more external services (SonataFlow, Loki, etc.), increasing CI flake risk

3. Add live-integration tests against real Loki/SonataFlow in CI

• ➕ Validates real client behavior and log/query semantics
• ➖ High operational cost (secrets, tenants, test data management) and higher flake rate
• ➖ Not aligned with the PR’s stated goal of lightweight bump-trust confidence

Recommendation: Keep the PR’s approach: lightweight startTestBackend wiring smokes + targeted unit assertions + optional local dev harnesses. It provides strong version-bump confidence for module/plugin registration and basic auth behavior without requiring a full Backstage app or live external dependencies.

Files changed (15) +732 / -8

Tests (4) +300 / -2

module.test.tsAdd startTestBackend wiring smoke for Loki module registration +56/-0

Add startTestBackend wiring smoke for Loki module registration

• Adds a test that boots a minimal backend and asserts the Loki module registers exactly one WorkflowLogProvider on the orchestrator workflow-logs extension point. Verifies the provider id is 'loki' using a minimal rootConfig setup.

workspaces/orchestrator/plugins/orchestrator-backend-module-loki/src/module.test.ts

plugin.test.tsAdd plugin-level startTestBackend smoke tests for health/auth behavior +188/-0

Add plugin-level startTestBackend smoke tests for health/auth behavior

• Introduces integration-style tests that boot the orchestrator backend plugin with mocked collaborators and permissions. Verifies unauthenticated health remains open and representative protected routes return 403 when permissions deny; includes an OpenAPI schema relaxation mock to avoid overly strict filter schemas in test runtime.

workspaces/orchestrator/plugins/orchestrator-backend/src/plugin.test.ts

OrchestratorService.test.tsStrengthen OrchestratorService assertions for abort/execute paths +11/-2

Strengthen OrchestratorService assertions for abort/execute paths

• Tightens tests to assert exact return values and that SonataFlowService methods are called with the expected parameter objects. Shifts away from weak 'called' / 'defined' assertions to bump-sensitive behavioral checks.

workspaces/orchestrator/plugins/orchestrator-backend/src/service/OrchestratorService.test.ts

module.test.tsAdd startTestBackend wiring smoke for scaffolder module action registration +45/-0

Add startTestBackend wiring smoke for scaffolder module action registration

• Adds a test that boots a minimal backend and asserts the scaffolder module registers both orchestrator actions on the scaffolderActions extension point. Verifies the expected action ids are present.

workspaces/orchestrator/plugins/scaffolder-backend-module-orchestrator/src/module.test.ts

Documentation (6) +375 / -0

CONTRIBUTING.mdAdd contributor guide for Loki backend module +91/-0

Add contributor guide for Loki backend module

• Documents prerequisites, expected testing scope, and local validation commands for the Loki workflow log provider module. Clarifies that development should be done via tests or by loading the module next to the host orchestrator backend rather than adding a standalone app.

workspaces/orchestrator/plugins/orchestrator-backend-module-loki/CONTRIBUTING.md

README.mdLink Loki module README to CONTRIBUTING guide +2/-0

Link Loki module README to CONTRIBUTING guide

• Adds a pointer to the new contributor guide for local development and validation workflows.

workspaces/orchestrator/plugins/orchestrator-backend-module-loki/README.md

CONTRIBUTING.mdAdd contributor guide for orchestrator backend plugin +136/-0

Add contributor guide for orchestrator backend plugin

• Documents how to run the plugin’s minimal dev harness, config stub expectations (including keeping SonataFlow autoStart disabled for CI/tests), and validation commands. Includes an optional smoke checklist emphasizing health endpoint behavior and permission-deny cases.

workspaces/orchestrator/plugins/orchestrator-backend/CONTRIBUTING.md

README.mdLink orchestrator backend README to CONTRIBUTING guide +2/-0

Link orchestrator backend README to CONTRIBUTING guide

• Adds a pointer to the new contributor guide for local development and contributor workflows.

workspaces/orchestrator/plugins/orchestrator-backend/README.md

CONTRIBUTING.MDAdd contributor guide for orchestrator scaffolder backend module +142/-0

Add contributor guide for orchestrator scaffolder backend module

• Documents a minimal local dev harness for scaffolder-module validation, including static token setup for authenticated curl calls. Defines validation commands, what automated tests cover, and an optional smoke checklist for verifying action registration.

workspaces/orchestrator/plugins/scaffolder-backend-module-orchestrator/CONTRIBUTING.MD

README.mdLink scaffolder module README to CONTRIBUTING guide +2/-0

Link scaffolder module README to CONTRIBUTING guide

• Adds a pointer to the new contributor guide for local development and contributor workflows.

workspaces/orchestrator/plugins/scaffolder-backend-module-orchestrator/README.md

Other (5) +57 / -6

itchy-flowers-smash.mdAdd changeset for patch releases across orchestrator packages +7/-0

Add changeset for patch releases across orchestrator packages

• Introduces a changeset marking patch bumps for orchestrator-backend, loki module, and scaffolder module. Describes the addition of wiring tests, a local dev harness, and contributor documentation.

workspaces/orchestrator/.changeset/itchy-flowers-smash.md

app-config.yamlAdd minimal dev-harness config for scaffolder module +12/-0

Add minimal dev-harness config for scaffolder module

• Adds a local-only backend config that binds to port 7007 and enables a static externalAccess token for development curl validation. Intended as a minimal config to run the scaffolder module harness without committing secrets.

workspaces/orchestrator/plugins/scaffolder-backend-module-orchestrator/app-config.yaml

index.tsAdd minimal dev backend harness for scaffolder module +26/-0

Add minimal dev backend harness for scaffolder module

• Introduces a small backend-defaults runner that loads the upstream scaffolder backend plugin and this orchestrator scaffolder module. Enables quick local verification of action registration in a running backend.

workspaces/orchestrator/plugins/scaffolder-backend-module-orchestrator/dev/index.ts

package.jsonAdd dev dependencies required for scaffolder module harness and wiring tests +3/-0

Add dev dependencies required for scaffolder module harness and wiring tests

• Adds backend-defaults, backend-test-utils, and plugin-scaffolder-backend as devDependencies to support the new dev harness and module wiring smoke tests.

workspaces/orchestrator/plugins/scaffolder-backend-module-orchestrator/package.json

yarn.lockUpdate lockfile for new scaffolder module dev dependencies +9/-6

Update lockfile for new scaffolder module dev dependencies

• Updates the workspace lockfile to include the newly added Backstage dev dependencies and related resolution/checksum changes. Aligns backend-defaults and scaffolder-backend plugin versions used by the new harness/test setup.

workspaces/orchestrator/yarn.lock

[rhdh-qodo-merge]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Context used

⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials

✅ Compliance rules (platform): 11 rules

✅ Cross-repo context

  Not relevant to this PR: redhat-developer/rhdh

  Not relevant to this PR: redhat-developer/rhdh-chart

  Not relevant to this PR: redhat-developer/rhdh-operator

  Not relevant to this PR: redhat-developer/rhdh-local

1. Broken CONTRIBUTING link 🐞 Bug ⚙ Maintainability

Description

scaffolder-backend-module-orchestrator/README.md links to ./CONTRIBUTING.md, but this PR adds
the contributor guide as CONTRIBUTING.MD (uppercase extension). On case-sensitive filesystems the
link will not resolve, breaking contributor navigation.

Code

workspaces/orchestrator/plugins/scaffolder-backend-module-orchestrator/README.md[7]

+For local development and contributor workflows, see [CONTRIBUTING.md](./CONTRIBUTING.md).

Relevance

●●● Strong

Team previously fixed broken/relative README links (PR #1407); likely to accept case-sensitive
CONTRIBUTING link fix.

PR-#1407

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The README explicitly links to CONTRIBUTING.md, while the added contributor guide file in this
package is named CONTRIBUTING.MD, which will not match on case-sensitive filesystems.

workspaces/orchestrator/plugins/scaffolder-backend-module-orchestrator/README.md[1-9]
workspaces/orchestrator/plugins/scaffolder-backend-module-orchestrator/CONTRIBUTING.MD[1-5]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`README.md` links to `./CONTRIBUTING.md`, but the actual file added is `CONTRIBUTING.MD`. This breaks the link on case-sensitive filesystems.

### Issue Context
Most Linux environments (including typical CI) treat `CONTRIBUTING.md` and `CONTRIBUTING.MD` as different paths.

### Fix Focus Areas
- workspaces/orchestrator/plugins/scaffolder-backend-module-orchestrator/README.md[7-7]
- workspaces/orchestrator/plugins/scaffolder-backend-module-orchestrator/CONTRIBUTING.MD[1-5]

### Suggested fix
Rename `CONTRIBUTING.MD` to `CONTRIBUTING.md` (preferred, consistent with other packages), or update the README link to match the actual filename.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

_{To customize comments, go to the Qodo configuration screen, or learn more in the docs.}