[STG-1745] Add ui-test skill — adversarial UI testing with browse CLI#56
[STG-1745] Add ui-test skill — adversarial UI testing with browse CLI#56
Conversation
Builds on #52 with three key additions: 1. Local/remote mode selection — localhost uses local browser (no API key), deployed sites use Browserbase via cookie-sync for authenticated testing 2. Diff-driven testing — analyze git diff, generate targeted tests for what changed, execute with before/after snapshot comparison 3. Structured assertion protocol — STEP_PASS/STEP_FAIL markers with evidence, deterministic checks (axe-core, console errors, overflow detection), and adversarial testing patterns (XSS, empty submit, rapid click, keyboard-only) Smoke-tested against a local Next.js app: found real bugs (Escape not closing modals, undersized mobile touch targets) that confirmed the adversarial patterns work. Fixed browse eval recipes (no top-level await, console capture on-page not about:blank). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ssions Enables concurrent test execution by leveraging browse CLI's --session flag to spin up independent Browserbase browsers per test group, with fan-out via Agent tool and merged result reporting. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Documents how to add Bash(browse:*) to project or user settings so users don't get prompted on every browse snapshot/click/eval. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…t figure it out - Remove .ui-tests/suite.yml format and generation pipeline - Replace Workflow B (8-step codebase analysis) with lightweight exploratory testing - Simplify references/codebase-analysis.md to quick hints (framework detection, route finding) - Remove example YAML suite file - Update README to reflect no-artifacts philosophy - Drop Write tool from allowed-tools (no files to generate) The codegen/suite approach can ship as v2 later. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- XSS check: replace false-positive inline script count with input value check - Console capture: preserve original console.error in Examples 6 snippets - Form labels: use native i.labels API in browser-recipes.md (matches SKILL.md) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Also strengthens auto-select rule: localhost → browse env local, deployed URLs → browse env remote, applied consistently across all workflows including parallel sessions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove "remote only" restriction — named sessions work with local mode - Add BROWSE_SESSION=* permission pattern to avoid approval fatigue on parallel runs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
If references/design-system.md exists, use it as ground truth. Otherwise, screenshot 2-3 existing pages to establish baseline patterns (spacing, radii, colors, typography) and compare the changed page against them. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a test step fails, the skill now instructs the agent to take a screenshot and save it to .context/ui-test-screenshots/<step-id>.png, referenced in the STEP_FAIL marker and final report so developers can see exactly what went wrong. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extracted from browserbase-brand-guidelines skill: colors, typography, border radii, spacing grid, component patterns, and visual principles. The ui-test skill checks changed pages against this when it exists. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrite Budget & Limits section to use a coordinator/sub-agent model: main agent plans and delegates, sub-agents do the actual testing with a 20-step hard cap each. Wall clock target ~10 min for default runs.
e563ad6 to
dc1eedd
Compare
Integrates two external frameworks into the testing skill: - Judgement (Emil Kowalski + Josh Puckett + UI Wiki): 9 reference files covering animations, forms, touch/a11y, typography, polish, component design, marketing, performance, and 152 UI wiki rules. Adds deterministic eval checks for touch targets, iOS zoom, transition:all, z-index abuse, and form labels. Adds screenshot-based critique methodology. - Luck (soleio): Assembly Theory meta-evaluation lens — 7 facets adapted to UI (solvency, gradient coupling, compatibility, niche construction, circulation, integration, path sensitivity) for "will this UI thrive?" Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip out the Craft Quality Judgement section, Luck Lens meta-evaluation, and all references/judgement/ files. Keep the skill focused on functional testing, accessibility, and UX heuristics. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
0cfbc08 to
d94fa1b
Compare
Rename design-system.md → design-system.example.md with instructions for users to copy and fill in their own brand tokens. The skill reads design-system.md (user-created), not the example. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add browse wait timeout 3000 after axe-core script injection (SKILL.md, browser-recipes.md) - Fix form label check to include aria-label and aria-labelledby (SKILL.md) - Fix focus ring detection to check box-shadow too, not just outline (browser-recipes.md) - Fix window.__capturedErrors → window.__logs in Example 8 (EXAMPLES.md) - design-system.md already fixed in prior commit (renamed to .example.md) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add aria-labelledby to hasLabel check in browser-recipes.md - Add browse wait timeout 3000 after axe-core injection in Examples 4 and 7 - hasFocus box-shadow check was already fixed in prior commit Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix allowed-tools delimiter: commas → spaces per spec - Extract adversarial patterns → references/adversarial-patterns.md - Extract parallel testing → references/parallel-testing.md - Extract design consistency → references/design-consistency.md - Replace inline deterministic checks with summary table + link - Move rules/ux-heuristics.md → references/ (eliminate non-standard dir) - Add conditional loading triggers for all reference files - Update README.md file tree Zero content lost — all extracted sections live in reference files with "when to load" guidance for progressive disclosure. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Drop the "~30 seconds per browse command" claim and all derived wall clock estimates. Budget is now defined purely in steps/turns, which is the actual constraint that caps spend. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a self-contained HTML report that reviewers can open in a browser. Screenshots are base64-embedded so the file works offline as a single artifact. Failed tests render open by default with inline screenshots; passed tests are collapsed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Light theme with warm off-white (#F9F6F4) background, Inter font via Google Fonts, brand orange-red (#F03603) for failures, brand green (#90C94D) for passes, brand blue (#4DA9E4) for suggestions. Inline Browserbase logo SVG in header and footer. Borders over shadows per brand design system. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Restructure test runs into 2-3 sweeps from different angles: 1. Deterministic sweep: fixed checklist (axe-core, console errors, etc.) — same results every run 2. Exploratory-functional: interactions, edge cases, adversarial inputs 3. Exploratory-visual: responsive layout, keyboard nav, design consistency Also adds mandatory page discovery step so every route gets covered. This addresses the inconsistency where repeated runs surface different bugs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This reverts commit 379e4af.
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | ||
| <title>UI Test Report — {{TITLE}}</title> | ||
| <link rel="preconnect" href="https://fonts.googleapis.com"> | ||
| <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap" rel="stylesheet"> |
There was a problem hiding this comment.
Report template uses CDN violating offline requirement
Medium Severity
The report template loads Google Fonts from fonts.googleapis.com via two <link> tags, but SKILL.md explicitly states "The report must work offline — no CDN links, no external assets." This directly contradicts the offline requirement. Reports generated from this template will make external network requests and won't render the intended font when opened offline. The CSS fallback stack (Inter, -apple-system, ...) helps, but the external request itself violates the stated contract and may cause slow/blocked rendering in air-gapped environments.
Additional Locations (1)
…ning - Replace rigid budget tables (20 steps, 5 agents, quick/default/thorough modes) with guiding principles the agent uses to size effort itself - Add multi-angle planning flow: plan from 3 perspectives, deduplicate, execute once — produces broad coverage without re-running same checks - Keep 20-step safety valve as a runaway cap, not a target - Remove adjusting-the-budget modes entirely — one way of testing Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Make the three planning rounds mandatory and output-visible — the agent must write out all three rounds and the merged plan before it's allowed to launch any sub-agents. Prevents skipping straight to execution. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
|
||
| # Tab through elements one at a time | ||
| browse press Tab | ||
| browse eval "JSON.stringify({tag: document.activeElement?.tagName, text: document.activeElement?.textContent?.trim().slice(0,40), role: document.activeElement?.getAttribute('role'), ariaLabel: document.activeElement?.getAttribute('aria-label'), hasFocus: (() => { const s = window.getComputedStyle(document.activeElement); return s.outlineStyle !== 'none' || s.boxShadow !== 'none'; })()})" |
There was a problem hiding this comment.
Focus ring detection produces false positives from decorative shadows
Medium Severity
The hasFocus check uses s.boxShadow !== 'none' as an OR condition to detect visible focus rings. This returns true for any element with a decorative box-shadow (elevation, depth effects), which is extremely common in modern UIs. The agent is then told "hasFocus should be true for every element," so elements with decorative shadows will always pass the focus ring check even when no actual focus indicator exists. This masks real accessibility issues — the core purpose of this deterministic recipe.
Additional Locations (1)
- Planning rounds happen in the main agent's own response, never delegated - Sub-agents receive a specific numbered test list and run only those tests - Sub-agents do not explore or plan — execute assigned tests and stop - Add explicit step budget requirement in every sub-agent prompt - Step heuristics (25/40/75) are starting points, not rules Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Sub-agents must return only STEP_PASS/STEP_FAIL markers as text. Only the coordinating main agent generates the final HTML report. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…etrics
- Report header: Tests | Passed | Failed | Agents (not Total Steps)
- Replace {{TOTAL_STEPS}} with {{TOTAL_TESTS}}, add {{AGENT_COUNT}}
- Remove redundant {{TEST_COUNT}} placeholder
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- Sub-agents report STEP_SKIP|<id>|budget reached for unfinished tests - Main agent accepts partial results as-is, never re-runs sub-agents - Add 25/40/75 step heuristics to sub-agent budget guidance - Skipped tests appear in final report so developer knows coverage gaps Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
There are 4 total unresolved issues (including 2 from previous reviews).
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
|
|
||
| As a rough heuristic: ~25 steps for a few targeted checks, ~40 for a full page with functional + adversarial + a11y, ~75 for multiple pages or a broad category. **Adjust based on what the assigned tests actually require** — these are starting points, not rules. | ||
|
|
||
| As a rough heuristic: ~25 steps for a few targeted checks, ~40 for a full page with functional + adversarial + a11y, ~75 for multiple pages or a broad category. **Adjust based on what the assigned tests actually require** — these are starting points, not rules. |
There was a problem hiding this comment.
Duplicated paragraph in step budget guidance
Low Severity
The "As a rough heuristic: ~25 steps for a few targeted checks…" paragraph is duplicated verbatim on consecutive lines. This appears to be a copy-paste artifact. Since the SKILL.md is read by the agent at runtime to determine behavior, redundant instructions add noise and could confuse the agent's interpretation of the budget guidance.
| |-------------|-------| | ||
| | `{{TITLE}}` | Report title (e.g., "UI Test: PR #1234 — OAuth Settings") | | ||
| | `{{META}}` | One-line context: date, app URL, user, branch | | ||
| | `{{TOTAL_TESTS}}` | Total STEP_PASS + STEP_FAIL count | |
There was a problem hiding this comment.
HTML report definition excludes skipped tests from totals
Medium Severity
{{TOTAL_TESTS}} is defined as "Total STEP_PASS + STEP_FAIL count," explicitly excluding STEP_SKIP. But the text report format on line 81 counts total as pass + fail + skipped (20 = 14 + 4 + 2) and computes pass rate accordingly (70%). The HTML template also has no placeholder or stat card for skipped tests. This means the HTML report will show different totals and an inflated pass rate compared to the text report for the same test run.
Additional Locations (2)
The report h1 now uses {{TITLE_HTML}} which supports wrapping
PR references in <a> tags. Link styled in brand red with
underline on hover.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 3 previously only scanned ports. Now it also instructs the agent to: - Check out the correct branch when testing a PR - Install dependencies after branch switch - Verify the server actually renders content (not just HTTP 200 with a build error) Learned from a real failure: tested against a dev server running the wrong branch with a broken build, wasting the entire step budget on login attempts before discovering the app wasn't rendering. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>


Summary
Adds a
ui-testskill that uses thebrowseCLI to run AI-powered adversarial UI tests in a real browser. The agent analyzes git diffs to test only what changed (or explores the full app), checking functional correctness, accessibility, responsive layout, and UX heuristics.Key features:
git diff→ targeted tests for changed pages/componentsSTEP_PASS|id|evidence/STEP_FAIL|id|expected → actualwith screenshot evidenceFiles
Test plan
🤖 Generated with Claude Code
Note
Low Risk
Low risk since this PR primarily adds new markdown/HTML assets for a standalone skill and does not modify existing application/runtime code paths.
Overview
Introduces a new
skills/ui-testpackage defining aui-testskill for agent-driven UI QA using thebrowseCLI, including diff-driven, exploratory, and parallel testing workflows with structuredSTEP_PASS/STEP_FAILassertions and screenshot evidence.Adds extensive supporting materials: worked examples (
EXAMPLES.md), deterministic check recipes (axe-core, console errors, broken images, responsive/touch targets), adversarial test patterns, UX/design-consistency heuristics, and a reusablereferences/report-template.htmlfor generating a standalone HTML test report.Written by Cursor Bugbot for commit 9ffe8e4. This will update automatically on new commits. Configure here.