Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .claude/launch.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"version": "0.0.1",
"configurations": [
{
"name": "web",
"runtimeExecutable": "npm",
"runtimeArgs": ["--prefix", "web", "run", "dev"],
"port": 3000,
"autoPort": true
}
]
}
21 changes: 2 additions & 19 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,29 +38,12 @@ jobs:
run: pytest tests/ -q -m "not browser"
- name: Pytest (reporting coverage gate)
run: |
pytest tests/test_categories_roadmap.py tests/test_report_categories_golden.py \
tests/test_categories_coverage.py tests/test_contrast_issues.py \
tests/test_indexation_coverage.py tests/test_crawl_segments.py \
tests/test_terminology.py tests/test_compare_payload.py \
tests/test_optional_audits.py tests/test_property_profile.py tests/test_reporting_gaps.py \
tests/test_text_content_analysis.py tests/test_builder_image_buckets.py \
tests/test_pipeline_report_pool_unit.py tests/test_reporting_builder_modules.py \
pytest tests/reporting/ \
--cov=website_profiling.reporting --cov-config=.coveragerc.reporting \
--cov-report=term-missing --cov-fail-under=100 -q -o addopts=
- name: Pytest (tools coverage gate)
run: |
pytest tests/test_alert_checker.py tests/test_schedule_runner.py tests/test_export_audit.py \
tests/test_export_audit_coverage.py tests/test_audit_tools.py tests/test_audit_tools_expanded.py \
tests/test_audit_tools_coverage.py tests/test_audit_tools_dispatch_coverage.py \
tests/test_audit_tools_links_extras.py tests/test_audit_tools_expansion.py \
tests/test_audit_tools_expansion_coverage.py tests/test_audit_tools_batch100_coverage.py tests/test_export_custom_coverage.py \
tests/test_export_artifacts_coverage.py tests/test_export_compare_coverage.py \
tests/test_export_tools_coverage.py tests/test_image_tools.py tests/test_export_custom.py \
tests/test_export_artifacts.py tests/test_export_compare.py tests/test_export_workbook.py \
tests/test_export_sitemap.py tests/test_mcp_registry.py tests/test_mcp_resources.py \
tests/test_router_tools.py tests/test_tool_selector.py \
tests/test_tools_gate100_coverage.py \
tests/test_tools_branch_coverage.py \
pytest tests/tools/ \
--cov=website_profiling.tools --cov-config=.coveragerc.tools \
--cov-report=term-missing --cov-fail-under=100 -q -o addopts=
- name: CLI smoke
Expand Down
10 changes: 5 additions & 5 deletions AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,12 +92,12 @@ These recur when adding features. Verify explicitly — do not assume tests caug
| Gate | Config | Source | Threshold | Test scope |
|------|--------|--------|-----------|------------|
| Core | `.coveragerc` | all packages **except** `tools/` and `reporting/` | 100% | `pytest tests/ -m "not browser"` |
| Reporting | `.coveragerc.reporting` | `website_profiling.reporting` | 100% | fixed test file list |
| Tools | `.coveragerc.tools` | `website_profiling.tools` | 100% | fixed test file list |
| Reporting | `.coveragerc.reporting` | `website_profiling.reporting` | 100% | `pytest tests/reporting/` |
| Tools | `.coveragerc.tools` | `website_profiling.tools` | 100% | `pytest tests/tools/` |
- **Symptom:** `./local-test` or core pytest passes at 100%, but CI fails on tools/reporting (e.g. 84% tools).
- **Causes:** (a) only ran core pytest, not reporting/tools gates; (b) added tests under `tests/test_<module>_coverage.py` but did not add the file to the tools gate list in **both** `scripts/local-test.sh`, `scripts/local-test.ps1`, and `.github/workflows/ci.yml`; (c) changed code under `website_profiling/tools/` without tests that hit those lines in the tools gate subset.
- **Do:** Run full `./local-test` before push. When adding tools coverage tests, name them `tests/test_<module>_coverage.py` (repo convention) and register the file in all three places above. Keep bash and PowerShell local-test scripts in sync.
- **Don't:** Assume `pytest tests/` alone matches CI. Don't rely on a single mega `test_tools_coverage_gaps.py`split by module.
- **Causes:** (a) only ran core pytest, not reporting/tools gates; (b) added reporting/tools tests outside `tests/reporting/` or `tests/tools/`; (c) changed code under `website_profiling/tools/` without tests that hit those lines in the tools gate subset.
- **Do:** Run full `./local-test` before push. Put reporting coverage tests in `tests/reporting/` and tools coverage tests in `tests/tools/` (one module per file, e.g. `test_<module>_coverage.py`). Keep bash and PowerShell local-test scripts in sync.
- **Don't:** Assume `pytest tests/` alone matches CI. Don't maintain long per-file lists in CIuse the directory gates above.

5. **Python — `runpy.run_module` / `__main__` guard tests**
- Tests that execute a module as `__main__` via `runpy.run_module(..., run_name="__main__")` emit:
Expand Down
42 changes: 42 additions & 0 deletions alembic/versions/017_content_drafts.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
"""Property-scoped content drafts for Content Studio.

Revision ID: 017_content_drafts
Revises: 016_competitor_keyword_gap
"""
from __future__ import annotations

from alembic import op

revision = "017_content_drafts"
down_revision = "016_competitor_keyword_gap"
branch_labels = None
depends_on = None


def upgrade() -> None:
op.execute("""
CREATE TABLE content_drafts (
id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
property_id BIGINT NOT NULL REFERENCES properties(id) ON DELETE CASCADE,
title TEXT NOT NULL DEFAULT 'Untitled draft',
target_keyword TEXT NOT NULL DEFAULT '',
landing_url TEXT,
status TEXT NOT NULL DEFAULT 'draft'
CHECK (status IN ('draft', 'ready', 'archived')),
body_html TEXT NOT NULL DEFAULT '',
title_tag TEXT NOT NULL DEFAULT '',
meta_description TEXT NOT NULL DEFAULT '',
grade_score SMALLINT,
grade_snapshot JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
)
""")
op.execute("""
CREATE INDEX content_drafts_property_updated_idx
ON content_drafts(property_id, updated_at DESC)
""")


def downgrade() -> None:
op.execute("DROP TABLE IF EXISTS content_drafts")
19 changes: 18 additions & 1 deletion docs/OPS.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,24 @@ POST /api/alerts/check?propertyId={id}

### Behavior

Evaluates health-score changes and stale GSC Links imports for the specified property. When `alert_webhook_url` is configured on the property, sends a POST notification to that URL.
Evaluates health-score changes and stale GSC Links imports for the specified property. When `alert_webhook_url` is configured on the property, sends a POST notification to that URL. When `alert_email` is set and SMTP is configured on the server, sends a plain-text email summary.

Response JSON includes `alerts`, `webhook_sent`, and `email_sent`.

### SMTP (optional, for alert email)

Set on the host running the web app (Docker: web service environment):

| Variable | Required | Default | Purpose |
|----------|----------|---------|---------|
| `SMTP_HOST` | Yes (with `SMTP_FROM`) | — | SMTP server hostname |
| `SMTP_FROM` | Yes (with `SMTP_HOST`) | — | From address |
| `SMTP_PORT` | No | `587` | SMTP port |
| `SMTP_USER` | No | — | Login user (if auth required) |
| `SMTP_PASS` | No | — | Login password |
| `SMTP_USE_TLS` | No | `true` | Use STARTTLS |

If SMTP is not configured, alert checks still succeed; `email_sent` is `false`.

### Example

Expand Down
42 changes: 2 additions & 40 deletions scripts/local-test.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -226,21 +226,7 @@ function Invoke-PytestCore {
function Invoke-PytestReporting {
Write-Log "Pytest (reporting coverage gate, 100%)"
& $VENV_PYTEST `
tests/test_categories_roadmap.py `
tests/test_report_categories_golden.py `
tests/test_categories_coverage.py `
tests/test_contrast_issues.py `
tests/test_indexation_coverage.py `
tests/test_crawl_segments.py `
tests/test_terminology.py `
tests/test_compare_payload.py `
tests/test_optional_audits.py `
tests/test_property_profile.py `
tests/test_reporting_gaps.py `
tests/test_text_content_analysis.py `
tests/test_builder_image_buckets.py `
tests/test_pipeline_report_pool_unit.py `
tests/test_reporting_builder_modules.py `
tests/reporting/ `
--cov=website_profiling.reporting `
--cov-config=.coveragerc.reporting `
--cov-report=term-missing `
Expand All @@ -253,31 +239,7 @@ function Invoke-PytestReporting {
function Invoke-PytestTools {
Write-Log "Pytest (tools coverage gate, 100%)"
& $VENV_PYTEST `
tests/test_alert_checker.py `
tests/test_schedule_runner.py `
tests/test_export_audit.py `
tests/test_export_audit_coverage.py `
tests/test_audit_tools.py `
tests/test_audit_tools_expanded.py `
tests/test_audit_tools_coverage.py `
tests/test_audit_tools_dispatch_coverage.py `
tests/test_audit_tools_links_extras.py `
tests/test_audit_tools_expansion.py `
tests/test_audit_tools_expansion_coverage.py `
tests/test_export_custom_coverage.py `
tests/test_export_artifacts_coverage.py `
tests/test_export_compare_coverage.py `
tests/test_export_tools_coverage.py `
tests/test_image_tools.py `
tests/test_export_custom.py `
tests/test_export_artifacts.py `
tests/test_export_compare.py `
tests/test_export_workbook.py `
tests/test_export_sitemap.py `
tests/test_mcp_registry.py `
tests/test_mcp_resources.py `
tests/test_tools_gate100_coverage.py `
tests/test_tools_branch_coverage.py `
tests/tools/ `
--cov=website_profiling.tools `
--cov-config=.coveragerc.tools `
--cov-report=term-missing `
Expand Down
45 changes: 2 additions & 43 deletions scripts/local-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -115,21 +115,7 @@ run_pytest_reporting() {
[[ "$PYTEST_NO_COV" -eq 1 ]] && return 0
log "Pytest (reporting coverage gate, 100%)"
"$VENV/bin/pytest" \
tests/test_categories_roadmap.py \
tests/test_report_categories_golden.py \
tests/test_categories_coverage.py \
tests/test_contrast_issues.py \
tests/test_indexation_coverage.py \
tests/test_crawl_segments.py \
tests/test_terminology.py \
tests/test_compare_payload.py \
tests/test_optional_audits.py \
tests/test_property_profile.py \
tests/test_reporting_gaps.py \
tests/test_text_content_analysis.py \
tests/test_builder_image_buckets.py \
tests/test_pipeline_report_pool_unit.py \
tests/test_reporting_builder_modules.py \
tests/reporting/ \
--cov=website_profiling.reporting \
--cov-config=.coveragerc.reporting \
--cov-report=term-missing \
Expand All @@ -142,34 +128,7 @@ run_pytest_tools() {
[[ "$PYTEST_NO_COV" -eq 1 ]] && return 0
log "Pytest (tools coverage gate, 100%)"
"$VENV/bin/pytest" \
tests/test_alert_checker.py \
tests/test_schedule_runner.py \
tests/test_export_audit.py \
tests/test_export_audit_coverage.py \
tests/test_audit_tools.py \
tests/test_audit_tools_expanded.py \
tests/test_audit_tools_coverage.py \
tests/test_audit_tools_dispatch_coverage.py \
tests/test_audit_tools_links_extras.py \
tests/test_audit_tools_expansion.py \
tests/test_audit_tools_expansion_coverage.py \
tests/test_audit_tools_batch100_coverage.py \
tests/test_export_custom_coverage.py \
tests/test_export_artifacts_coverage.py \
tests/test_export_compare_coverage.py \
tests/test_export_tools_coverage.py \
tests/test_image_tools.py \
tests/test_export_custom.py \
tests/test_export_artifacts.py \
tests/test_export_compare.py \
tests/test_export_workbook.py \
tests/test_export_sitemap.py \
tests/test_mcp_registry.py \
tests/test_mcp_resources.py \
tests/test_router_tools.py \
tests/test_tool_selector.py \
tests/test_tools_gate100_coverage.py \
tests/test_tools_branch_coverage.py \
tests/tools/ \
--cov=website_profiling.tools \
--cov-config=.coveragerc.tools \
--cov-report=term-missing \
Expand Down
56 changes: 56 additions & 0 deletions src/website_profiling/concurrency.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
"""Bounded parallel execution helpers shared by the agent loops and workflows.

When a model returns several independent, read-only tool calls in one turn we run
them concurrently with a bounded worker pool instead of one at a time — the same
"parallel tool execution" pattern Claude Code uses. Mirrors the ``ThreadPoolExecutor``
usage already in ``crawl/crawler.py``, ``llm/enrich.py`` and ``analysis/image_probe.py``,
and the env-int parsing in ``db/pool.py``.
"""
from __future__ import annotations

import os
from concurrent.futures import ThreadPoolExecutor
from typing import Callable, Sequence, TypeVar

T = TypeVar("T")
R = TypeVar("R")

DEFAULT_TOOL_CONCURRENCY = 6


def tool_concurrency(default: int = DEFAULT_TOOL_CONCURRENCY) -> int:
"""Max number of tool calls to dispatch concurrently within one agent turn.

Override with the ``WP_TOOL_CONCURRENCY`` env var; a value of ``1`` disables
parallelism (every dispatch runs sequentially). Empty or non-integer values fall
back to ``default``. The result is floored at 1.
"""
raw = (os.environ.get("WP_TOOL_CONCURRENCY") or "").strip()
if not raw:
return default
try:
return max(1, int(raw))
except ValueError:
return default


def map_parallel(
items: Sequence[T],
fn: Callable[[T], R],
*,
max_workers: int,
) -> list[R]:
"""Apply ``fn`` to each item, returning results in input order.

Runs sequentially when ``max_workers <= 1`` or there is at most one item; otherwise
uses a bounded thread pool. ``fn`` MUST NOT raise — callers wrap their work in
try/except and return an error value so one failure never sinks the batch.
"""
count = len(items)
if count == 0:
return []
workers = max(1, min(max_workers, count))
if workers == 1 or count == 1:
return [fn(item) for item in items]
with ThreadPoolExecutor(max_workers=workers) as pool:
return list(pool.map(fn, items))
7 changes: 7 additions & 0 deletions src/website_profiling/content_studio/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
"""Content Studio — draft writing and SEO scoring."""
from __future__ import annotations

from .score import score_content_draft
from .ai_suggest import analyze_content_draft

__all__ = ["score_content_draft", "analyze_content_draft"]
Loading
Loading