Quality Orchestrator

AI-powered PR analysis tool. Fetches changed files from a GitHub PR (or your local working tree), maps them to test files, scores overall risk, and recommends the exact tests to run — or generates stubs for gaps.

---

Quick Start

pip install -r requirements.txt   # installs all deps
# or: pip install -e ".[all]"     # editable install with all extras

# Analyze a GitHub PR
qo analyze --pr 42 --repo owner/repo --token $GITHUB_TOKEN

# Analyze files you're about to commit (no token needed)
qo analyze-staged

# Analyze an explicit file list
qo analyze-local src/api/user.js src/lib/fraud.js

---

Commands

qo analyze

Fetch a GitHub PR's diff and analyze it.

qo analyze --pr <number> --repo <owner/repo> [options]

Option	Description
--pr, -p	PR number (required)
--repo, -r	GitHub repo in owner/repo form (required)
--token	GitHub token (or set GITHUB_TOKEN env var)
--format	Output format: rich (default) or markdown
--json	Emit raw JSON instead of formatted output
--tui	Interactive TUI with single-keypress run/stub actions
--generate-stubs, -g	Write test stub files for uncovered source files
--known-test-files	Explicit test file paths (pass once per file)
--test-dir, -t	Directory to scan for test files
--framework	Stub generator: auto (default), pytest, playwright

# Rich output (default)
qo analyze --pr 42 --repo acme/api

# Markdown for CI comment
qo analyze --pr 42 --repo acme/api --format markdown

# With test-dir to count "Running N of M tests"
qo analyze --pr 42 --repo acme/api --test-dir tests/

# Generate stubs for files with no coverage
qo analyze --pr 42 --repo acme/api --generate-stubs

---

qo analyze-local

Analyze a list of source files without a GitHub token.

qo analyze-local <file>... [options]

Accepts the same --format, --json, --tui, --generate-stubs, --test-dir, --framework flags as analyze.

qo analyze-local src/api/user.js src/lib/fraud.js
qo analyze-local src/api/user.js --test-dir tests/ --format markdown

---

qo analyze-staged

Auto-detect changed files from git and analyze them — no token required. Designed for pre-commit / pre-push workflows.

qo analyze-staged [options]

Option	Description
--staged / --all	Staged files only (default) vs all changes vs HEAD
--base, -b	Compare branch tip against this base (e.g. main)
--format	rich (default) or markdown
--json	Raw JSON output
--tui	Interactive TUI mode
--generate-stubs, -g	Write stubs for uncovered files
--test-dir, -t	Directory to scan for known test files
--framework	Stub framework: auto, pytest, playwright

# Staged files only (pre-commit)
qo analyze-staged

# All changes vs HEAD
qo analyze-staged --all

# Everything on this branch vs main (pre-push)
qo analyze-staged --base main

# Markdown output with total test count
qo analyze-staged --base main --test-dir tests/ --format markdown

---

qo install-hooks

Wire QO into git's pre-push or pre-commit hook. On HIGH risk + missing coverage the hook exits 1 and blocks the push/commit.

qo install-hooks [options]

Option	Description
--hook-type	pre-push (default) or pre-commit
--base	Base branch for pre-push comparison (default: main)
--force	Overwrite an existing non-QO hook
--uninstall	Remove a previously installed QO hook

# Install pre-push hook (blocks HIGH-risk pushes with gaps)
qo install-hooks

# Install pre-commit hook instead
qo install-hooks --hook-type pre-commit

# Remove a hook
qo install-hooks --uninstall

# Bypass the hook for a single push
git push --no-verify

---

qo list-tests

Discover and print all test files in a directory. Uses git ls-files (respects .gitignore) with a filesystem-walk fallback. Use this to verify test file discovery before running qo analyze.

qo list-tests [directory]

# List all test files in the current repo
qo list-tests

# List tests in a specific directory
qo list-tests packages/api/tests/

---

qo stub

Generate a test stub for a single source file without running a full analysis.

qo stub <file> [options]

Option	Description
--framework	Framework: auto (default), pytest, vitest, jest, playwright
--pr	PR number to embed in stub annotation
--write / --no-write	Write stub to disk (default) or print content to stdout

# Generate and write a stub for a source file
qo stub src/api/user.js

# Preview stub content without writing to disk
qo stub src/api/user.js --no-write

# Generate with an explicit framework
qo stub src/api/user.js --framework vitest

---

qo serve

Start the Quality Orchestrator REST API server.

qo serve [--host HOST] [--port PORT]

Defaults to 0.0.0.0:8000. Override via QO_HOST / QO_PORT env vars or CLI flags.

qo serve
qo serve --port 9000

Endpoints:

• POST /analyze-pr — same as qo analyze, accepts JSON body with pr, repo, token
• GET /health — liveness check

---

GitHub Actions

Add to .github/workflows/quality.yml:

name: Quality Orchestrator

on:
  pull_request:

jobs:
  analyze:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
      issues: write
    steps:
      - uses: actions/checkout@v4

      - uses: mrdailey99/QualityOrchestrator@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}

Inputs

Input	Default	Description
github-token	${{ github.token }}	Token for PR access and commenting
generate-stubs	'true'	Embed stub previews in the PR comment
framework	'auto'	Stub framework: auto, pytest, playwright
test-dir	''	Directory to scan for test files — enables Running N of M counter
fail-on-high	'false'	Exit 1 when risk is HIGH — use with required status checks to block merges
comment-on-low	'true'	Set 'false' to skip comments on LOW-risk PRs

Outputs

Output	Description
risk-tier	high, med, or low
risk-score	Numeric score 0–100

Use outputs to drive downstream steps:

- uses: mrdailey99/QualityOrchestrator@v1
  id: qo
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    test-dir: tests/
    fail-on-high: 'true'

- if: steps.qo.outputs.risk-tier == 'high'
  run: echo "High risk PR — notify team"

The action posts a Markdown analysis comment to the PR with risk tier, selected tests, and missing coverage gaps. On re-push the comment is updated in place — no duplicate comments.

PR Comment Bot (/qo stub)

Add .github/workflows/qo-bot.yml to enable the /qo stub comment command on PRs. When a repo collaborator posts a comment containing /qo stub <file>, the bot:

1. Validates the commenter has at least write permission.
2. Runs qo stub <file> --pr <number> to generate the stub.
3. Commits the stub file directly to the PR branch.
4. Replies with the committed path (or an Actions log link on failure).

name: QO Bot

on:
  issue_comment:
    types: [created]

jobs:
  qo-stub:
    if: >
      github.event.issue.pull_request != null &&
      startsWith(github.event.comment.body, '/qo stub ')
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
      issues: write
    steps:
      - uses: actions/checkout@v4
        with:
          ref: refs/pull/${{ github.event.issue.number }}/head

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - run: pip install -e ".[action,cli]" --quiet

      - name: Generate and commit stub
        env:
          GH_TOKEN: ${{ github.token }}
          COMMENT_BODY: ${{ github.event.comment.body }}
        run: |
          FILE="${COMMENT_BODY#/qo stub }"
          qo stub "$FILE" --pr ${{ github.event.issue.number }}

See .github/workflows/qo-bot.yml in this repo for the full workflow with permission checks, path validation, and failure replies.

---

Risk Scoring

Risk is scored 0–100 from three additive components:

Component	Weight	Description
Volume	0–20	files × 2, capped at 20
Category	0–50	Highest-weight category × 10 (payment/security=5, auth/migrations=4, middleware/models/api=3, config/lib=2, utils/components/ui/styles=1)
Coverage	0–30	Fraction of testable files with no mapped test × 30

Tiers: HIGH ≥ 70 · MED 40–69 · LOW < 40

---

Environment

Copy .env.example to .env:

GITHUB_TOKEN=ghp_...        # required for analyze and API
QO_HOST=0.0.0.0             # optional, default 0.0.0.0
QO_PORT=8000                # optional, default 8000

---

Development

# Install all deps
pip install -r requirements.txt
# or editable with all extras:
pip install -e ".[all]"

# Run tests
pytest tests/

# Run a single test module
pytest tests/engine/test_risk.py -v