From f5f3e47e2ec1159cd2b4856b10805e0da01184da Mon Sep 17 00:00:00 2001 From: Parth Rohit Date: Sun, 14 Jun 2026 09:52:07 +0100 Subject: [PATCH] docs: add Azure scanner validation docs --- docs/VALIDATION_PLAN.md | 134 ++++++++++ docs/validation/AZURE_SCENARIOS.md | 348 ++++++++++++++++++++++++++ docs/validation/SCANNER_VALIDATION.md | 135 ++++++++++ docs/validation/TEST_RESULTS.md | 192 ++++++++------ 4 files changed, 727 insertions(+), 82 deletions(-) create mode 100644 docs/VALIDATION_PLAN.md create mode 100644 docs/validation/AZURE_SCENARIOS.md create mode 100644 docs/validation/SCANNER_VALIDATION.md diff --git a/docs/VALIDATION_PLAN.md b/docs/VALIDATION_PLAN.md new file mode 100644 index 0000000..663d9ff --- /dev/null +++ b/docs/VALIDATION_PLAN.md @@ -0,0 +1,134 @@ +# OpenShield Validation Plan + +This document defines the validation approach for OpenShield scanner behavior +and controlled Azure vulnerable scenarios. It is a planning document only. It +does not record completed validation and does not require real Azure validation +to run automatically in CI. + +All current test execution results are `Pending`. + +## Purpose + +The validation effort is intended to prove that OpenShield findings can move +through the product lifecycle in a reliable, contributor-verifiable way: + +1. The scanner detects an Azure misconfiguration. +2. The scan result can be persisted to PostgreSQL by the platform. +3. The Flask API can expose the persisted finding through the expected + endpoints. +4. The React frontend can display the issue and remediation guidance. + +The first version focuses on scanner validation and real Azure scenario +planning. It does not attempt to validate every rule at once. + +## Scope + +In scope: + +- Scanner entry point and rule-loading behavior. +- Rule metadata and finding shape validation. +- Low-cost Azure vulnerable scenario planning. +- Manual test result tracking. +- Cleanup and risk documentation for contributor-run validation. + +Out of scope for this documentation pass: + +- Frontend implementation changes. +- API implementation changes. +- Database schema changes. +- Real Azure CI automation. +- Credential, secret, or tenant-specific setup. +- Expensive Azure resource creation. + +## Validation Principles + +Validation should follow the same lifecycle that production data follows. + +### 1. Scanner + +The scanner is the first source of truth. A scenario is not considered valid +unless the expected rule produces a finding with the correct `rule_id`, +resource identity, severity, category, and remediation metadata. + +### 2. Database + +Findings should persist after a scan is triggered through the platform path. +Database validation should confirm that scan metadata and finding rows are +stored without changing the finding semantics produced by the scanner. + +### 3. API + +API validation should confirm that the persisted finding is available through +the expected read endpoints. Response payloads should preserve the rule ID, +resource name, resource ID, severity, category, description, remediation, and +playbook references needed by the dashboard. + +### 4. Frontend + +Frontend validation should confirm that the issue is visible to a user and +that remediation guidance can be reached from the dashboard. The frontend is +validated as a consumer of scanner/API data; this plan does not require +frontend source edits. + +## Phased Validation + +### Phase 1: Storage and Network + +Goal: Validate low-cost, easy-to-clean scenarios that are realistic in an Azure +Student subscription. + +Initial rules: + +- `AZ-STOR-001`: Public Blob Access Enabled on Storage Account +- `AZ-NET-001`: NSG Allows Unrestricted Inbound SSH from Any Source +- `AZ-NET-002`: NSG Allows Unrestricted Inbound RDP from Any Source + +### Phase 2: Key Vault and Identity + +Goal: Add Key Vault checks and identify which identity checks are safe to test +without tenant-wide risk. + +Initial Key Vault rules: + +- `AZ-KV-002`: Key Vault Allows Public Network Access Without Private Endpoint +- `AZ-KV-004`: Key Vault Purge Protection Disabled + +Identity rules should be reviewed individually because several require +Microsoft Graph or tenant-level permissions. + +### Phase 3: Database and Compute + +Goal: Add database and compute scenarios only after Phase 1 cleanup and +evidence collection are reliable. + +Database and compute resources may create more cost, quota, or cleanup risk +than Phase 1 resources. Each scenario must include a cost and cleanup review +before execution. + +### Phase 4: Full Rule Coverage + +Goal: Build toward full rule coverage after the low-cost path is proven. + +This phase should classify each rule as one of: + +- Safe for student subscription validation. +- Maintainer-only validation. +- Mock-only validation. +- Not safe or not practical to validate with real resources. + +## Non-Goals + +- Do not add Azure credentials or real subscription identifiers to the repo. +- Do not create expensive Azure resources as part of this plan. +- Do not run real Azure validation automatically in pull request CI. +- Do not claim validation has passed until results are manually recorded. +- Do not delete broad subscription resources during cleanup. + +## Documentation Map + +- `docs/validation/SCANNER_VALIDATION.md`: scanner entry point, rule format, + verified rule matrix, and scanner-specific notes. +- `docs/validation/AZURE_SCENARIOS.md`: low-cost vulnerable Azure scenario + plan with cleanup commands. +- `docs/validation/TEST_RESULTS.md`: manual status tracker and result + template. All rows start as `Pending`. diff --git a/docs/validation/AZURE_SCENARIOS.md b/docs/validation/AZURE_SCENARIOS.md new file mode 100644 index 0000000..47cb8c4 --- /dev/null +++ b/docs/validation/AZURE_SCENARIOS.md @@ -0,0 +1,348 @@ +# Azure Vulnerable Scenario Plan + +This document defines low-cost vulnerable Azure scenarios for manual +OpenShield validation. These scenarios are not CI automation. Do not run them +against production subscriptions. + +All test execution results are `Pending`. + +## General Requirements + +- Use a dedicated test subscription or isolated student subscription. +- Use a dedicated resource group for each validation run. +- Do not add real subscription IDs, tenant IDs, client IDs, secrets, or + resource names to the repository. +- Prefer Azure CLI commands that create only low-cost resources. +- Do not create virtual machines, VPN gateways, Azure Firewall, Application + Gateway, or other expensive resources in Phase 1. +- Clean up the validation resource group after each run. + +## Naming And Tagging + +Use placeholders in documentation and replace them locally during execution. + +```bash +export OSHIELD_LOCATION="" +export OSHIELD_RG="" +export OSHIELD_SUFFIX="" +export OSHIELD_DELETE_AFTER="" +``` + +Recommended tags: + +```bash +--tags purpose=openshield-validation owner=contributor delete-after="$OSHIELD_DELETE_AFTER" +``` + +Resource group creation: + +```bash +az group create \ + --name "$OSHIELD_RG" \ + --location "$OSHIELD_LOCATION" \ + --tags purpose=openshield-validation owner=contributor delete-after="$OSHIELD_DELETE_AFTER" +``` + +## Validation Scan Step + +Run the scanner after creating the vulnerable resource: + +```bash +python -c " +from dotenv import load_dotenv; load_dotenv() +import json, os +from scanner.engine import ScanEngine + +result = ScanEngine(os.environ['AZURE_SUBSCRIPTION_ID']).run_scan() +print(json.dumps(result, indent=2)) +" +``` + +Record only non-secret evidence in `docs/validation/TEST_RESULTS.md`. + +## Cleanup Strategy + +Primary cleanup should delete the validation resource group: + +```bash +az group delete --name "$OSHIELD_RG" --yes --no-wait +``` + +Before cleanup, verify that the resource group contains only validation +resources: + +```bash +az resource list --resource-group "$OSHIELD_RG" --output table +``` + +For Key Vault scenarios, remember that soft-deleted vault names may remain +reserved for the retention period. Purge only the explicitly created +validation vault, only if permitted, and only if purge protection was not +enabled. + +## Scenario Matrix + +| Test ID | Azure Service | Resource Group | Vulnerable Resource | Misconfiguration Introduced | Expected OpenShield Finding | Validation Command Or Scan Step | Expected Result | Actual Result | Pass/Fail | Cleanup Command | +|---|---|---|---|---|---|---|---|---|---|---| +| `VAL-STOR-001` | Storage Account | `$OSHIELD_RG` | Storage account named with `$OSHIELD_SUFFIX` | Blob public access enabled on the storage account | `AZ-STOR-001` on `Microsoft.Storage/storageAccounts` | Run `ScanEngine(...).run_scan()` and filter findings for `AZ-STOR-001` | Finding includes the validation storage account name and severity `HIGH` | Pending | Pending | `az group delete --name "$OSHIELD_RG" --yes --no-wait` | +| `VAL-NET-001` | Network Security Group | `$OSHIELD_RG` | NSG named with `$OSHIELD_SUFFIX` | Inbound allow rule for TCP `22` from internet | `AZ-NET-001` on `Microsoft.Network/networkSecurityGroups` | Run `ScanEngine(...).run_scan()` and filter findings for `AZ-NET-001` | Finding includes the validation NSG name and severity `HIGH` | Pending | Pending | `az group delete --name "$OSHIELD_RG" --yes --no-wait` | +| `VAL-NET-002` | Network Security Group | `$OSHIELD_RG` | NSG named with `$OSHIELD_SUFFIX` | Inbound allow rule for TCP `3389` from internet | `AZ-NET-002` on `Microsoft.Network/networkSecurityGroups` | Run `ScanEngine(...).run_scan()` and filter findings for `AZ-NET-002` | Finding includes the validation NSG name and severity `HIGH` | Pending | Pending | `az group delete --name "$OSHIELD_RG" --yes --no-wait` | +| `VAL-KV-002` | Key Vault | `$OSHIELD_RG` | Key Vault named with `$OSHIELD_SUFFIX` | Public network access enabled without a private endpoint | `AZ-KV-002` on `Microsoft.KeyVault/vaults` | Run `ScanEngine(...).run_scan()` and filter findings for `AZ-KV-002` | Finding includes the validation Key Vault name and severity `HIGH` | Pending | Pending | `az group delete --name "$OSHIELD_RG" --yes --no-wait` | +| `VAL-KV-004` | Key Vault | `$OSHIELD_RG` | Key Vault named with `$OSHIELD_SUFFIX` | Purge protection disabled | `AZ-KV-004` on `Microsoft.KeyVault/vaults` | Run `ScanEngine(...).run_scan()` and filter findings for `AZ-KV-004` | Finding includes the validation Key Vault name and severity `MEDIUM` | Pending | Pending | `az group delete --name "$OSHIELD_RG" --yes --no-wait` | + +## Scenario Details + +### VAL-STOR-001: Storage Account Public Blob Access + +Azure service: Storage Account + +Resource group: `$OSHIELD_RG` + +Vulnerable resource to create: one low-cost general purpose v2 storage account +with a globally unique name. + +Misconfiguration introduced: public blob access enabled at the account level. + +Expected OpenShield rule/finding: `AZ-STOR-001`. + +Example setup: + +```bash +az storage account create \ + --name "oshieldstor$OSHIELD_SUFFIX" \ + --resource-group "$OSHIELD_RG" \ + --location "$OSHIELD_LOCATION" \ + --sku Standard_LRS \ + --kind StorageV2 \ + --allow-blob-public-access true \ + --tags purpose=openshield-validation owner=contributor delete-after="$OSHIELD_DELETE_AFTER" +``` + +Validation command or scan step: run `ScanEngine(...).run_scan()` and search +the JSON output for `AZ-STOR-001`. + +Expected result: a `HIGH` severity finding for the validation storage account. + +Actual result: Pending + +Pass/fail: Pending + +Cleanup command: + +```bash +az group delete --name "$OSHIELD_RG" --yes --no-wait +``` + +### VAL-NET-001: NSG Allowing SSH From Internet + +Azure service: Network Security Group + +Resource group: `$OSHIELD_RG` + +Vulnerable resource to create: one NSG. + +Misconfiguration introduced: inbound allow rule for TCP port `22` from +`0.0.0.0/0`. + +Expected OpenShield rule/finding: `AZ-NET-001`. + +Example setup: + +```bash +az network nsg create \ + --name "oshield-nsg-ssh-$OSHIELD_SUFFIX" \ + --resource-group "$OSHIELD_RG" \ + --location "$OSHIELD_LOCATION" \ + --tags purpose=openshield-validation owner=contributor delete-after="$OSHIELD_DELETE_AFTER" + +az network nsg rule create \ + --resource-group "$OSHIELD_RG" \ + --nsg-name "oshield-nsg-ssh-$OSHIELD_SUFFIX" \ + --name "AllowSSHFromInternet" \ + --priority 100 \ + --direction Inbound \ + --access Allow \ + --protocol Tcp \ + --source-address-prefixes 0.0.0.0/0 \ + --source-port-ranges "*" \ + --destination-address-prefixes "*" \ + --destination-port-ranges 22 +``` + +Validation command or scan step: run `ScanEngine(...).run_scan()` and search +the JSON output for `AZ-NET-001`. + +Expected result: a `HIGH` severity finding for the validation NSG. + +Actual result: Pending + +Pass/fail: Pending + +Cleanup command: + +```bash +az group delete --name "$OSHIELD_RG" --yes --no-wait +``` + +### VAL-NET-002: NSG Allowing RDP From Internet + +Azure service: Network Security Group + +Resource group: `$OSHIELD_RG` + +Vulnerable resource to create: one NSG. + +Misconfiguration introduced: inbound allow rule for TCP port `3389` from +`0.0.0.0/0`. + +Expected OpenShield rule/finding: `AZ-NET-002`. + +Example setup: + +```bash +az network nsg create \ + --name "oshield-nsg-rdp-$OSHIELD_SUFFIX" \ + --resource-group "$OSHIELD_RG" \ + --location "$OSHIELD_LOCATION" \ + --tags purpose=openshield-validation owner=contributor delete-after="$OSHIELD_DELETE_AFTER" + +az network nsg rule create \ + --resource-group "$OSHIELD_RG" \ + --nsg-name "oshield-nsg-rdp-$OSHIELD_SUFFIX" \ + --name "AllowRDPFromInternet" \ + --priority 100 \ + --direction Inbound \ + --access Allow \ + --protocol Tcp \ + --source-address-prefixes 0.0.0.0/0 \ + --source-port-ranges "*" \ + --destination-address-prefixes "*" \ + --destination-port-ranges 3389 +``` + +Validation command or scan step: run `ScanEngine(...).run_scan()` and search +the JSON output for `AZ-NET-002`. + +Expected result: a `HIGH` severity finding for the validation NSG. + +Actual result: Pending + +Pass/fail: Pending + +Cleanup command: + +```bash +az group delete --name "$OSHIELD_RG" --yes --no-wait +``` + +### VAL-KV-002: Key Vault Public Network Access + +Azure service: Key Vault + +Resource group: `$OSHIELD_RG` + +Vulnerable resource to create: one Key Vault with a globally unique name. + +Misconfiguration introduced: public network access enabled without a private +endpoint. + +Expected OpenShield rule/finding: `AZ-KV-002`. + +Example setup: + +```bash +az keyvault create \ + --name "oshield-kv-$OSHIELD_SUFFIX" \ + --resource-group "$OSHIELD_RG" \ + --location "$OSHIELD_LOCATION" \ + --public-network-access Enabled \ + --enable-purge-protection false \ + --tags purpose=openshield-validation owner=contributor delete-after="$OSHIELD_DELETE_AFTER" +``` + +Validation command or scan step: run `ScanEngine(...).run_scan()` and search +the JSON output for `AZ-KV-002`. + +Expected result: a `HIGH` severity finding for the validation Key Vault. + +Actual result: Pending + +Pass/fail: Pending + +Cleanup command: + +```bash +az group delete --name "$OSHIELD_RG" --yes --no-wait +``` + +### VAL-KV-004: Key Vault Purge Protection Disabled + +Azure service: Key Vault + +Resource group: `$OSHIELD_RG` + +Vulnerable resource to create: one Key Vault with purge protection disabled. + +Misconfiguration introduced: purge protection disabled. + +Expected OpenShield rule/finding: `AZ-KV-004`. + +Example setup: + +```bash +az keyvault create \ + --name "oshield-kv-purge-$OSHIELD_SUFFIX" \ + --resource-group "$OSHIELD_RG" \ + --location "$OSHIELD_LOCATION" \ + --enable-purge-protection false \ + --tags purpose=openshield-validation owner=contributor delete-after="$OSHIELD_DELETE_AFTER" +``` + +Validation command or scan step: run `ScanEngine(...).run_scan()` and search +the JSON output for `AZ-KV-004`. + +Expected result: a `MEDIUM` severity finding for the validation Key Vault. + +Actual result: Pending + +Pass/fail: Pending + +Cleanup command: + +```bash +az group delete --name "$OSHIELD_RG" --yes --no-wait +``` + +## Known Risks + +### Azure Student Subscription Limits + +Student subscriptions have limited credit and quotas. Avoid resources that +consume compute, gateway, firewall, or premium capacity during initial +validation. + +### Resource Naming Uniqueness + +Storage account and Key Vault names are globally unique. Use a short local +suffix and do not commit the final names. + +### Cleanup Failures + +Resource group deletion can fail or take time. Always inspect the resource +group before and after cleanup. + +### Tenant-Level Permissions + +Identity rules may require tenant-wide permissions or Microsoft Graph access. +Do not test those rules in a shared tenant without explicit approval. + +### Unsafe Or Impractical Rules + +Rules that require VMs, VPN gateways, Azure Firewall, Application Gateway, SQL +servers, or production-like identity settings should be deferred until a +maintainer approves the scenario and cost. + +### False Positives And False Negatives + +Record unexpected findings as `Pending investigation`. Do not remove evidence +or mark a test as passed when the expected rule did not appear. diff --git a/docs/validation/SCANNER_VALIDATION.md b/docs/validation/SCANNER_VALIDATION.md new file mode 100644 index 0000000..10d9366 --- /dev/null +++ b/docs/validation/SCANNER_VALIDATION.md @@ -0,0 +1,135 @@ +# Scanner Validation + +This document describes how to validate the OpenShield scanner and its rule +surface. It is based on the current contents of `scanner/rules/*.py`. + +All execution status is `Pending` until a contributor records manual results +in `docs/validation/TEST_RESULTS.md`. + +## Scanner Entry Point + +The scanner entry point is: + +```python +from scanner.engine import ScanEngine + +result = ScanEngine(subscription_id).run_scan() +``` + +`ScanEngine` is defined in `scanner/engine.py`. + +## Rule Loading + +`ScanEngine.load_rules()` dynamically imports Python files from: + +```text +scanner/rules/*.py +``` + +Rule files are loaded in sorted path order. Any module with a callable +`scan()` function is added to the scan engine's rule list. Files without a +callable `scan()` function are skipped. + +During `run_scan()`, each loaded rule is called as: + +```python +rule.scan(azure_client, subscription_id) +``` + +The scanner expects each rule to return a list of finding dictionaries. If a +rule returns a non-list value, that rule's result is skipped. If a rule raises +an exception, the scan continues with the remaining rules. + +## Expected Rule Format + +Each rule module should define these module-level values: + +```python +RULE_ID = "AZ-STOR-001" +RULE_NAME = "Public Blob Access Enabled on Storage Account" +SEVERITY = "HIGH" +CATEGORY = "Storage" +FRAMEWORKS = {"CIS": "3.5", "NIST": "PR.AC-3", "ISO27001": "A.9.4.1"} +DESCRIPTION = "..." +REMEDIATION = "..." +PLAYBOOK = "playbooks/cli/fix_az_stor_001.sh" +``` + +Each rule should expose: + +```python +def scan(azure_client, subscription_id): + return [] +``` + +The expected finding fields are: + +| Field | Purpose | +|---|---| +| `rule_id` | Stable OpenShield rule ID, for example `AZ-STOR-001` | +| `rule_name` | Human-readable rule title | +| `severity` | Severity label such as `HIGH`, `MEDIUM`, `LOW`, or `INFO` | +| `category` | Rule category such as `Storage`, `Network`, or `Key Vault` | +| `resource_id` | Full Azure resource ID when available | +| `resource_name` | Azure resource name | +| `resource_type` | Azure provider type, for example `Microsoft.Storage/storageAccounts` | +| `description` | Security impact of the finding | +| `remediation` | Human-readable remediation guidance | +| `playbook` | Path to the matching remediation playbook | +| `frameworks` | Compliance mappings | +| `metadata` | Optional rule-specific context | + +`ScanEngine.run_scan()` adds `scan_id` and `detected_at` when a finding does +not already include them. + +## Verified Rule Matrix + +The following matrix was verified from actual files in `scanner/rules`. + +Total verified rule files: **44** + +| Category | Count | Rule IDs | +|---|---:|---| +| Compute | 4 | `AZ-CMP-001`, `AZ-CMP-002`, `AZ-CMP-003`, `AZ-CMP-004` | +| Database | 4 | `AZ-DB-001`, `AZ-DB-002`, `AZ-DB-003`, `AZ-DB-004` | +| Identity | 9 | `AZ-IDN-001`, `AZ-IDN-002`, `AZ-IDN-003`, `AZ-IDN-004`, `AZ-IDN-005`, `AZ-IDN-006`, `AZ-IDN-007`, `AZ-IDN-008`, `AZ-IDN-009` | +| Key Vault | 4 | `AZ-KV-002`, `AZ-KV-003`, `AZ-KV-004`, `AZ-KV-005` | +| KeyVault | 1 | `AZ-KV-001` | +| Network | 14 | `AZ-NET-001`, `AZ-NET-002`, `AZ-NET-003`, `AZ-NET-004`, `AZ-NET-005`, `AZ-NET-006`, `AZ-NET-007`, `AZ-NET-008`, `AZ-NET-009`, `AZ-NET-010`, `AZ-NET-011`, `AZ-NET-012`, `AZ-NET-013`, `AZ-NET-014` | +| PostQuantum | 3 | `AZ-PQC-001`, `AZ-PQC-002`, `AZ-PQC-003` | +| Storage | 5 | `AZ-STOR-001`, `AZ-STOR-002`, `AZ-STOR-003`, `AZ-STOR-004`, `AZ-STOR-005` | + +## Initial Live Validation Candidates + +These rules are the first candidates for real Azure validation because they +can be tested with low-cost resources and clear cleanup boundaries. + +| Rule ID | Rule Name | Reason for Initial Selection | Status | +|---|---|---|---| +| `AZ-STOR-001` | Public Blob Access Enabled on Storage Account | Uses a low-cost storage account configuration setting | Pending | +| `AZ-NET-001` | NSG Allows Unrestricted Inbound SSH from Any Source | Uses an NSG rule only; no VM required | Pending | +| `AZ-NET-002` | NSG Allows Unrestricted Inbound RDP from Any Source | Uses an NSG rule only; no VM required | Pending | +| `AZ-KV-002` | Key Vault Allows Public Network Access Without Private Endpoint | Uses a Key Vault network setting; no secret material needed | Pending | +| `AZ-KV-004` | Key Vault Purge Protection Disabled | Uses a Key Vault protection setting; must avoid enabling purge protection during cleanup testing | Pending | + +## Rules To Defer + +Defer the following groups until the low-cost path is reliable: + +- Identity rules that need tenant-level or Microsoft Graph permissions. +- Compute rules that require virtual machines. +- Network rules that require Application Gateway, VPN Gateway, Azure Firewall, + load balancers, or public IP resources. +- Database rules that require SQL or PostgreSQL resources. +- PostQuantum rules that require App Service, Key Vault keys, or certificates. + +## Known Scanner Notes + +- `AZ-KV-001` uses category `KeyVault`, while `AZ-KV-002` through + `AZ-KV-005` use `Key Vault`. Validation reports should preserve the current + scanner output but note the inconsistency. +- `AZ-NET-012` references `azure_client.get_nsg_flow_logs(...)`, but that + method is not currently present on `AzureClient`. Treat real Azure results + for `AZ-NET-012` as `Pending investigation` until the scanner surface is + reviewed. +- Real Azure validation must not be run automatically in PR CI. diff --git a/docs/validation/TEST_RESULTS.md b/docs/validation/TEST_RESULTS.md index 2354cf9..437dff5 100644 --- a/docs/validation/TEST_RESULTS.md +++ b/docs/validation/TEST_RESULTS.md @@ -1,86 +1,114 @@ -# Frontend/API Validation Results +# Validation Test Results -## Test Run Metadata +This file tracks manual validation status for scanner and low-cost Azure +scenario tests. + +No tests have been executed as part of this documentation update. Every result +starts as `Pending`. + +## Validation Status Tracking + +| Area | Status | Notes | +|---|---|---| +| Scanner entry point documented | Pending | Documentation exists; execution not yet validated | +| Rule loading behavior documented | Pending | Documentation exists; execution not yet validated | +| Rule metadata matrix verified from files | Pending | Rule IDs are listed from current files; no runtime validation claimed | +| Storage scenario execution | Pending | Not yet executed | +| Network scenario execution | Pending | Not yet executed | +| Key Vault scenario execution | Pending | Not yet executed | +| Database persistence confirmation | Pending | Not yet executed | +| API response confirmation | Pending | Not yet executed | +| Frontend display confirmation | Pending | Not yet executed | +| Cleanup confirmation | Pending | Not yet executed | + +## Environment + +Fill this section during manual validation. Do not add real secrets or +sensitive identifiers. | Field | Value | |---|---| -| Date | 2026-06-07 | -| Branch | `docs/issue-132-frontend-api-validation-dev` | -| Node Version | v22.x (npm 10.9.2) | -| Python Version | 3.13.1 | -| pytest Version | 9.0.3 | -| OS | Windows 11 | -| Tester | Automated validation run | - ---- - -## Frontend Build Validation - -| Test ID | Area | Command / Page Tested | Expected Result | Actual Result | Status | Evidence Notes | Follow-up Needed | -|---|---|---|---|---|---|---|---| -| FE-001 | Frontend | `npm install` | 0 exit code, packages installed | 243 packages added, 0 vulnerabilities | Pass | npm 10.9.2; noted npm 11.16.0 available | No | -| FE-002 | Frontend | `npm run lint` | 0 errors | 65 errors, 4 warnings | Fail | Mostly `no-unused-vars` for React imports (cosmetic in React 19). Also: unused vars in AILayer, DriftEventCard, RiskRanking, ActionItems; 1 `no-undef` in tailwind.config.js; 4 `react-hooks/exhaustive-deps` warnings | Non-blocking — does not affect build or runtime | -| FE-003 | Frontend | `npm run build` | Production bundle created | Built in 3.53s. Output: index.html (0.47 KB), CSS (31.16 KB gzip 6.36 KB), JS (782.15 KB gzip 218.81 KB) | Pass | Warning: JS chunk >500 KB — consider code splitting | No (cosmetic warning) | -| FE-004 | Frontend | `npm run dev` | Vite dev server starts | Vite v8.0.14 started in 697ms on http://localhost:5173 | Pass | Server confirmed running, then stopped | No | - ---- - -## Backend Test Validation - -| Test ID | Area | Command / Page Tested | Expected Result | Actual Result | Status | Evidence Notes | Follow-up Needed | -|---|---|---|---|---|---|---|---| -| API-001 | Backend | `pytest` (full suite) | All tests pass | Collection error: `KeyError: 'DATABASE_URL'` in `tests/test_jwt_config.py` | Fail | `api/app.py` calls `DatabaseManager()` at import time which requires `DATABASE_URL` env var. 73 tests collected but 1 collection error halted execution. | Yes — need DATABASE_URL set or test isolation for JWT config tests | - ---- - -## API Endpoint Validation - -> **Note:** API endpoint tests require a running backend with DATABASE_URL configured. These are marked Pending as the backend could not be started locally without a PostgreSQL instance. - -| Test ID | Area | Command / Page Tested | Expected Result | Actual Result | Status | Evidence Notes | Follow-up Needed | -|---|---|---|---|---|---|---|---| -| API-001 | API | `GET /health` without JWT | 200 `{"status":"ok"}` | Not tested (backend not running) | Pending | Endpoint registered in `app.py` — public, no DB needed | Run when backend available | -| API-002 | API | `GET /api/findings` without JWT | 200 with findings array | Not tested | Pending | Public GET per `_is_public_get()` in `app.py` | Run when backend available | -| API-003 | API | `GET /api/findings` with JWT | 200 with findings array | Not tested | Pending | Should behave identically to without JWT (GETs are public) | Run when backend available | -| API-004 | API | `GET /api/scans` with JWT | 200 with scans array | Not tested | Pending | Public GET endpoint | Run when backend available | -| API-005 | API | `GET /api/score` with JWT | 200 `{"score":N,"max_score":100}` | Not tested | Pending | Computed from findings count | Run when backend available | -| API-006 | API | `GET /api/compliance/cis` with JWT | 200 with framework + controls | Not tested | Pending | Requires rules table populated | Run when backend available | -| API-007 | API | `POST /api/scans/trigger` with JWT | 201 with scan result | Not tested | Pending | Requires valid JWT + Azure credentials | Run when backend available | -| API-008 | API | `GET /api/resources` with JWT | 200 with summary + resources | Not tested | Pending | Derived from findings | Run when backend available | -| API-009 | API | `GET /api/prioritization` with JWT | 200 with matrix + rankings | Not tested | Pending | Computed from findings | Run when backend available | -| API-010 | API | `GET /api/drift` with JWT | 200 with summary + events | Not tested | Pending | Requires 2+ scans with findings | Run when backend available | - ---- - -## Integration (Frontend-to-API Data Flow) Validation - -> **Note:** Integration tests require both frontend dev server and backend API running simultaneously. Marked Pending as backend could not be started. - -| Test ID | Area | Command / Page Tested | Expected Result | Actual Result | Status | Evidence Notes | Follow-up Needed | -|---|---|---|---|---|---|---|---| -| INT-001 | Integration | Monitoring page data flow | Page loads score gauge, trend chart, stat cards, findings distribution | Not tested | Pending | Requires running API with seeded data | Run when full stack available | -| INT-002 | Integration | Discovery page data flow | Page shows resource summary cards, filterable resource table | Not tested | Pending | Requires resources endpoint returning data | Run when full stack available | -| INT-003 | Integration | DetailedScan page data flow | Findings list renders, selecting a finding loads playbook | Not tested | Pending | Requires findings + playbook endpoints | Run when full stack available | -| INT-004 | Integration | Compliance page data flow | Framework cards show scores, controls table populates | Not tested | Pending | Requires all 4 compliance endpoints + scans | Run when full stack available | -| INT-005 | Integration | Drift page data flow | Summary cards + timeline renders drift events | Not tested | Pending | Requires 2+ scans to compute drift | Run when full stack available | -| INT-006 | Integration | Prioritization page data flow | Matrix chart + rankings + action items render | Not tested | Pending | Requires prioritization endpoint with data | Run when full stack available | -| INT-007 | Integration | AI Layer page data flow | Findings picker populates, chat accepts input, summary/CVE panels load | Not tested | Pending | Requires AI provider API key + findings data | Run when full stack available | - ---- - -## Summary - -| Category | Total | Pass | Fail | Pending | -|---|---|---|---|---| -| Frontend Build (FE-*) | 4 | 3 | 1 | 0 | -| Backend Tests | 1 | 0 | 1 | 0 | -| API Endpoints (API-*) | 10 | 0 | 0 | 10 | -| Integration (INT-*) | 7 | 0 | 0 | 7 | -| **Total** | **22** | **3** | **2** | **17** | - -### Key Findings - -1. **Frontend builds successfully** — production bundle compiles without errors despite lint warnings -2. **Lint failures are cosmetic** — 65 errors are predominantly unused `React` imports (safe to remove in React 19) -3. **Backend tests blocked** — `DATABASE_URL` is required at import time; tests cannot run without PostgreSQL -4. **API/Integration tests pending** — require a running backend with seeded database; document infrastructure needed for full validation +| Tester | Pending | +| Date | Pending | +| OpenShield branch or commit | Pending | +| Azure subscription type | Pending | +| Azure region | Pending | +| Validation resource group | Pending | +| Authentication method | Pending | +| Database used for persistence check | Pending | +| API base URL for API check | Pending | +| Frontend URL for display check | Pending | + +## Pre-Test Checklist + +| Check | Status | Notes | +|---|---|---| +| Azure CLI authenticated to the intended test subscription | Pending | | +| No production subscription selected | Pending | | +| Validation resource group name chosen | Pending | | +| Unique suffix chosen for global resource names | Pending | | +| Estimated resource cost reviewed | Pending | | +| Cleanup command reviewed | Pending | | +| OpenShield dependencies installed locally | Pending | | +| `AZURE_SUBSCRIPTION_ID` set locally, not committed | Pending | | +| No secrets added to repository files | Pending | | + +## Scenario Results + +| Test ID | Expected Rule | Scanner Result | Database Result | API Result | Frontend Result | Cleanup Result | Overall Status | Notes | +|---|---|---|---|---|---|---|---|---| +| `VAL-STOR-001` | `AZ-STOR-001` | Pending | Pending | Pending | Pending | Pending | Pending | Storage public blob access | +| `VAL-NET-001` | `AZ-NET-001` | Pending | Pending | Pending | Pending | Pending | Pending | NSG open SSH | +| `VAL-NET-002` | `AZ-NET-002` | Pending | Pending | Pending | Pending | Pending | Pending | NSG open RDP | +| `VAL-KV-002` | `AZ-KV-002` | Pending | Pending | Pending | Pending | Pending | Pending | Key Vault public network access | +| `VAL-KV-004` | `AZ-KV-004` | Pending | Pending | Pending | Pending | Pending | Pending | Key Vault purge protection disabled | + +## Evidence Checklist + +Record evidence without adding secrets, access tokens, real tenant IDs, or +other sensitive identifiers. + +| Evidence Item | Status | Notes | +|---|---|---| +| Scanner command recorded | Pending | | +| Scanner output captured with expected `rule_id` | Pending | | +| Scanner output contains expected validation resource name | Pending | | +| Finding severity matches expected severity | Pending | | +| Finding category matches current scanner output | Pending | | +| Finding persisted to PostgreSQL | Pending | | +| API response includes expected finding | Pending | | +| Frontend displays expected finding | Pending | | +| Remediation guidance visible | Pending | | +| Cleanup command executed | Pending | | +| Resource group deletion confirmed | Pending | | + +## Failure And Investigation Log + +Use this section to track failed or inconclusive validation. Do not mark a +scenario as passed until the expected rule appears and cleanup is confirmed. + +| Date | Test ID | Issue Type | Description | Follow-Up Owner | Status | +|---|---|---|---|---|---| +| Pending | Pending | False positive / false negative / cleanup / permissions / other | Pending | Pending | Pending | + +## Issue Type Definitions + +- False positive: OpenShield reports a finding when the resource is expected + to be compliant. +- False negative: OpenShield does not report the expected finding for an + intentionally vulnerable resource. +- Cleanup issue: Validation resources could not be deleted or remain in a + soft-deleted state. +- Permission issue: The scanner could not read the resource or tenant setting + required for the rule. +- Other: Any result that does not fit the categories above. + +## Post-Test Checklist + +| Check | Status | Notes | +|---|---|---| +| Validation resource group deleted or deletion started | Pending | | +| No unexpected resources left behind | Pending | | +| Key Vault soft-delete state reviewed if applicable | Pending | | +| No credentials written to docs or logs committed to git | Pending | | +| Results updated without claiming unverified pass status | Pending | |