Skip to content

PDX-508: enforce mustContainArgument best-practice rules in the local validator#207

Merged
mrdailey99 merged 3 commits into
developfrom
feature/PDX-508-must-contain-argument
Jun 3, 2026
Merged

PDX-508: enforce mustContainArgument best-practice rules in the local validator#207
mrdailey99 merged 3 commits into
developfrom
feature/PDX-508-must-contain-argument

Conversation

@mrdailey99
Copy link
Copy Markdown
Collaborator

Summary

  • Implements the mustContainArgument check type in the local best-practices engine, unlocking 21 required-argument rules that shipped in provar_best_practices_rules.json but were silently skipped (no registered validator → silent-pass fallback). External/offline MCP users now get these load/exec-blocking checks without the Quality Hub back-end.
  • Scope item 1 of PDX-508 only (biggest single-ROI slice). Tiers 2–5, the vendored provar_test_step_schema.json, the Validation Rule Registry doc, and new MCP resources are follow-up slices.

What the validator does

  • For each rule, flags every apiCall whose apiId matches check.apiId when the required check.argument is absent or present only as an empty self-closing <argument/> with no <value>.
  • Reuses the existing getCallArguments / argumentHasValue helpers so semantics match the NitroX required-argument validators exactly (no drift).
  • Exact apiId match plus trailing :variant suffix (NitroX convention); disabled steps skipped; steps nested inside control-flow containers are checked (recursive). One aggregated violation per rule with count = offending step count (scored via the existing log2 formula).

Rules now enforced: control-flow (CONTROL-IF/WHILE/WAITFOR/FOREACH/SWITCH/SLEEP-*), assertions (ASSERT-COMPARISON-001, ASSERT-EXPECTED-001), BDD (BDD-GIVEN/WHEN/THEN-001), Apex/DB (SOQL-QUERY-001, DB-CONNECT-001/002, SQL-QUERY-001/002), web service (REST-CONN-001/002, REST-REQUEST-001).

Jira

https://provartesting.atlassian.net/browse/PDX-508

Test plan

  • yarn compile passes
  • yarn test:only passes (1356 tests; 7 new reproduce/cleared tests for mustContainArgument)
  • node scripts/mcp-smoke.cjs passes (60/60)
  • yarn lint passes

Changes

  • src/mcp/tools/bestPracticesEngine.ts: add validateMustContainArgument + register it in VALIDATOR_REGISTRY.
  • test/unit/mcp/bestPracticesEngine.test.ts: 7 tests — missing arg fires, populated clears, empty self-closing fires, count aggregation, disabled-step skip, non-matching apiId, nested-step recursion.
  • docs/mcp.md: document the newly-enforced mustContainArgument rule class under Warning rules.

Notes for reviewers / customer docs

  • Behaviour change flagged for public docs: test cases that previously scored clean may now surface these 21 rules. The Provar team maintains docs/provar-mcp-public-docs.md / the University course separately — this changes enforced offline behaviour and may warrant a mirror update.
  • Semantics decision to confirm: I treat an empty self-closing <argument/> (present but no <value>) as a violation, matching the sibling NitroX required-arg logic. If the QH back-end uses presence-only semantics for mustContainArgument, flag it and I'll align.
  • No new MCP tool/resource and no rule-count references in README, so no smoke TOTAL_EXPECTED or README changes were needed.

RCA: 21 required-argument rules (control-flow conditions, assertion comparison/expected values, BDD descriptions, SOQL/SQL/DB/REST connection+result args) shipped in provar_best_practices_rules.json with check.type "mustContainArgument", but the local engine never registered that check type — every one hit the silent-pass fallback and never fired, so external MCP users got none of these load/exec-blocking checks offline.
Fix: implement validateMustContainArgument in bestPracticesEngine.ts and register it; it flags steps whose apiId matches check.apiId when the required check.argument is absent or an empty self-closing <argument/>, reusing the existing argumentHasValue/getCallArguments helpers for parity with the NitroX required-arg validators. Exact apiId match (plus :variant suffix), disabled steps skipped, nested control-flow steps checked, aggregated one-violation-per-rule with count. Adds 7 reproduce/cleared unit tests and documents the newly-enforced rule class in docs/mcp.md.
Copilot AI review requested due to automatic review settings June 3, 2026 20:08
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 3, 2026

Quality Orchestrator

🟢 LOW · 2 / 100 · All changed files have mapped tests.


🧪 Tests to Run · Running 1 of 54 tests

  • unit/mcp/bestPracticesEngine.test.ts
▶ Run command
npx vitest run \
  unit/mcp/bestPracticesEngine.test.ts

⚡ quality-orchestrator  ·  /qo stub <file>  ·  qo analyze-local

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds local/offline enforcement for mustContainArgument best-practice checks so that required-argument rules in provar_best_practices_rules.json no longer silently pass when Quality Hub is unavailable.

Changes:

  • Implemented validateMustContainArgument and registered it in VALIDATOR_REGISTRY so mustContainArgument rules are executed locally.
  • Added unit tests covering missing/empty required arguments, aggregation/counting, disabled-step skipping, apiId non-matches, and nested-step recursion.
  • Documented the newly enforced mustContainArgument rule class in docs/mcp.md.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
src/mcp/tools/bestPracticesEngine.ts Adds and registers the mustContainArgument validator for local best-practice evaluation.
test/unit/mcp/bestPracticesEngine.test.ts Adds targeted tests to ensure mustContainArgument behavior matches expectations and handles nesting/aggregation.
docs/mcp.md Documents the offline/local enforcement of required-argument (mustContainArgument) rules and lists the currently enforced rule IDs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/mcp/tools/bestPracticesEngine.ts Outdated
Comment on lines +938 to +942
const requiredContainer = verdict.includes('UiWithRow')
? 'UiWithRow'
: verdict.includes('UiWithScreen')
? 'UiWithScreen'
: 'UiWithScreen';
? 'UiWithScreen'
: 'UiWithScreen';
RCA: Validated the local validator against quality-hub-agents MustContainArgumentValidator (best_practices_engine.py) and found four divergences: (1) empty <value/> or bare <value class="variable"/> was accepted but the backend treats present-but-empty as missing; (2) disabled steps were skipped but the backend checks them (a missing required arg is load-blocking regardless); (3) the legacy If/DoWhile condition-in-title format was not honored, risking false positives; (4) aggregating with count=N diverged the weighted-deduction score from the Lambda, which returns one violation per rule.
Fix: add argumentHasMeaningfulValue mirroring the backend content checks (variable needs path/text, funcCall and gt/lt/eq/ne/ge/le/and/or/not operator classes count, compound needs parts, simple needs text); stop skipping disabled steps; honor the condition-in-title legacy exception for If/DoWhile; emit a single violation per rule with no count so the score stays in parity (message still names every offender); use exact apiId match. Extract callSatisfiesRequiredArg/stepLabel helpers to keep complexity under 20. Updates tests (12 cases) and docs/mcp.md to the backend-faithful semantics.
RCA: Copilot review flagged a nested ternary in validateUiActionNestingStructure where both non-UiWithRow branches return 'UiWithScreen', so the inner verdict.includes('UiWithScreen') test is dead and the expression is harder to read. Prettier reformatting in this branch surfaced the pre-existing line in the diff.
Fix: collapse to verdict.includes('UiWithRow') ? 'UiWithRow' : 'UiWithScreen' — behavior-preserving (the eliminated branch returned the same value) and clearer. No test impact; the nesting message tests still pass.
@mrdailey99
Copy link
Copy Markdown
Collaborator Author

Backend parity validated against quality-hub-agents

Validated the empty-argument semantics (and the rest of the check) against the canonical MustContainArgumentValidator in lambda/src/validator/best_practices_engine.py. Found and corrected four divergences so the local/offline result matches the Lambda:

  1. Empty argument = missing (the open question) — backend uses present-AND-non-empty semantics. Now mirrored via argumentHasMeaningfulValue: a variable needs a <path> (or text), funcCall and the comparison/logic operator classes (gt/lt/eq/ne/ge/le/and/or/not) count, compound needs <parts>, simple needs text. A bare <value/> or <value class="variable"/> is a violation.
  2. Disabled steps — backend does not skip them; removed the local skip (a missing required arg is load-blocking regardless).
  3. Legacy condition-in-title — ported the backend exception where If/DoWhile may carry the condition in the step title (If: {expr}) instead of a condition argument.
  4. Score parity — backend returns one violation per rule; switched from aggregated count=N to a single violation with no count, so the weighted-deduction score stays in parity with the Lambda. The message still names every offending step.

Also: exact apiId match (matching the backend's @apiId="…"), and the mustContainArgument test suite expanded to 12 cases covering all of the above.

Copilot nit (redundant requiredContainer ternary): addressed in 4c408da — collapsed to verdict.includes('UiWithRow') ? 'UiWithRow' : 'UiWithScreen'. It was pre-existing logic that prettier reformatting pulled into this diff; the fix is behavior-preserving.

@mrdailey99 mrdailey99 merged commit 653aa09 into develop Jun 3, 2026
4 checks passed
@mrdailey99 mrdailey99 deleted the feature/PDX-508-must-contain-argument branch June 3, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants