feat: MA lobbying data pipeline and dashboard charts

[nesanders]

This PR adds a complete pipeline for collecting and analyzing Massachusetts lobbying disclosure data. The MA Secretary of State publishes semi-annual lobbying filings that document which organizations hired lobbyists, how much they paid, and which specific bills their lobbyists worked on. By pairing this with bill metadata from the MA Legislature API and semantic environmental relevance scoring via Google Gemini embeddings, we can identify which industries are spending most on environmental legislation, track whether heavily-lobbied bills are more or less likely to pass, and correlate lobbying intensity with trends in DEP enforcement and budget.

The data integrates naturally with existing AMEND analyses: lobbying spend can be plotted alongside DEP staffing and enforcement actions to surface relationships between industry influence and regulatory outcomes. All four new database tables are exposed in the AI Analysis tool's semantic context, enabling natural-language queries like "which companies spent the most lobbying against clean water bills" or "has lobbying spend on climate legislation increased since 2015." The pipeline is fully incremental — weekly CI runs exit early when no new semi-annual filings have been posted, so the added runtime cost is near-zero on most weeks.

Summary

• Scraper (get_MA_lobbying.py): scrapes the MA SoS lobbying disclosure portal using an iPad User-Agent (bypasses Incapsula WAF without Selenium). Incremental via a disc_url set stored in MA_lobbying_summary_links.csv — weekly CI exits early when no new semi-annual filings are posted, so most runs touch only 2 search pages and exit.
• Legislature bills (get_MA_legislature_bills.py): fetches bill metadata (title, sponsor, committee, status, passed bool) from the MA Legislature OpenAPI for every unique (bill_number, general_court) pair in the lobbying data. JSON responses cached under MA_legislature_cache/ for incremental re-runs.
• Environmental scoring (score_lobbying_bills.py): scores each bill for environmental relevance using Gemini gemini-embedding-2 cosine similarity against 20 seed phrases (threshold 0.60). Only unscored bills are embedded per run.
• Dashboard charts (MA_lobbying_viz.py): 4 weekly-updated charts — annual spend trend, top 15 employers, bill intensity + pass rate, lobbying spend vs. enforcement actions. Plus 2 analysis-post charts.
• Pipeline wiring: all scripts added to update-data.yml CI, assemble_db.py (4 new tables: MA_Lobbying_Employers, MA_Lobbying_Bills, MA_Lobbying_Lobbyists, MA_Legislature_Bills), validate_data.py (lobbying tables in OPTIONAL_DATASETS — CI doesn't fail before first fetch), generate_semantic_context.py, and dashboard_charts.py.

Test plan

• End-to-end test with --year 2024 --limit 10: produced 18 disclosure rows, 56 employer rows, 539 bill rows with correct entity names, compensation amounts, bill numbers, titles, and positions
• Full 2024 fetch running in background — data CSVs will be committed as follow-up once complete
• Run get_MA_legislature_bills.py against live API after lobbying fetch completes
• Run score_lobbying_bills.py with real Google API key
• Run dashboard_charts.py to verify chart generation once DB is assembled

🤖 Generated with Claude Code

[github-actions]

✅ Semantic Eval Results

Each eval sends a natural-language question plus the relevant table schemas from db_semantic_context.txt to gpt-4o-mini, executes the generated SQL against AMEND.db, then scores it with a second LLM call using a per-case rubric. Hard pass = SQL ran and returned rows without hitting any known anti-patterns. Fatal = judge determined the query would return wrong or misleading results.

Metric	Value
Hard pass rate	10/10 (100%)
Fatal failures	0
Mean judge score	5.0/5
P50 judge score	5/5
Model	gpt-4o-mini
Semantic context hash	0b4c17034694

Per-case results

ID	Hard pass	Score	Fatal	Reason
cso_top_operator	✅	5/5	no	The query correctly aggregates total CSO discharge volume by operator for 2022, using the correct table and filters, and orders the results as required.
cso_monthly_rainfall	✅	5/5	no	The query correctly aggregates both CSO discharge volume and precipitation totals by month and joins them appropriately.
cso_by_watershed	✅	5/5	no	The query correctly joins the tables, groups by watershed, and applies the necessary filter for eventType.
enforcement_vs_budget	✅	5/5	no	The query correctly joins the enforcement actions with the budget data on the year, extracts the year from the EnforcementDate correctly, and applies the necessary filters.
staffing_trend	✅	5/5	no	The query correctly uses the MADEP_staff_Comptroller table, groups by year, and counts employees from 2005 to present.
303d_impaired_trend	✅	5/5	no	The query correctly counts listings from the EPA_303d_Impairments table, groups by reportingCycle, and orders the results correctly.
303d_named_waterbody	✅	5/5	no	The query correctly uses the EPA_303d_Impairments table, applies the maximum reportingCycle filter, and uses the LIKE pattern for the waterbody.
cso_to_impaired	✅	5/5	no	Both joins are correct and the reportingCycle filter is applied to EPA_303d_Impairments, which is appropriate.
all_caps_boston	✅	5/5	no	The query correctly uses UPPER() to filter the municipality for 'BOSTON' and checks for CSO event types using LIKE.
ecos_per_capita	✅	5/5	no	The query correctly uses the ECOS_budgets table, aggregates data for multiple states, and calculates per-capita spending accurately.

[github-actions]

✅ Semantic Eval Results

Each eval sends a natural-language question plus the relevant table schemas from db_semantic_context.txt to gpt-4o-mini, executes the generated SQL against AMEND.db, then scores it with a second LLM call using a per-case rubric. Hard pass = SQL ran and returned rows without hitting any known anti-patterns. Fatal = judge determined the query would return wrong or misleading results.

Metric	Value
Hard pass rate	10/10 (100%)
Fatal failures	0
Mean judge score	5.0/5
P50 judge score	5/5
Model	gpt-4o-mini
Semantic context hash	0b4c17034694

Per-case results

ID	Hard pass	Score	Fatal	Reason
cso_top_operator	✅	5/5	no	The query correctly aggregates total CSO discharge volume by operator for 2022, using the correct table and filters, and includes the necessary grouping and ordering.
cso_monthly_rainfall	✅	5/5	no	The query correctly aggregates both CSO discharge volume and total monthly rainfall, using the appropriate join pattern on aggregated months.
cso_by_watershed	✅	5/5	no	The query correctly joins the tables, groups by watershed, and applies the necessary filter for eventType.
enforcement_vs_budget	✅	5/5	no	The query correctly joins the enforcement actions with the budget data on the year, extracts the year from the EnforcementDate correctly, and applies the necessary filters.
staffing_trend	✅	5/5	no	The query correctly uses the MADEP_staff_Comptroller table, groups by year, and counts employees from 2005 to present.
303d_impaired_trend	✅	5/5	no	The query correctly counts listings from the EPA_303d_Impairments table, groups by reportingCycle, and orders the results correctly.
303d_named_waterbody	✅	5/5	no	The query correctly uses the EPA_303d_Impairments table, applies the maximum reportingCycle filter, and uses the LIKE pattern for the waterbody.
cso_to_impaired	✅	5/5	no	Both joins are correct and the reportingCycle filter is applied to EPA_303d_Impairments, which is appropriate.
all_caps_boston	✅	5/5	no	The query correctly uses UPPER() to filter the municipality for 'BOSTON' and checks for CSO event types using LIKE.
ecos_per_capita	✅	5/5	no	The query correctly uses the ECOS_budgets table, aggregates data for multiple states, and calculates per-capita spending accurately.

[github-actions]

✅ Semantic Eval Results

Each eval sends a natural-language question plus the relevant table schemas from db_semantic_context.txt to gpt-4o-mini, executes the generated SQL against AMEND.db, then scores it with a second LLM call using a per-case rubric. Hard pass = SQL ran and returned rows without hitting any known anti-patterns. Fatal = judge determined the query would return wrong or misleading results.

Metric	Value
Hard pass rate	10/10 (100%)
Fatal failures	0
Mean judge score	5.0/5
P50 judge score	5/5
Model	gpt-4o-mini
Semantic context hash	7dc233f87aac

Per-case results

ID	Hard pass	Score	Fatal	Reason
cso_top_operator	✅	5/5	no	The query correctly aggregates total CSO discharge volume by operator for 2022, using the correct table and filters, and orders the results as required.
cso_monthly_rainfall	✅	5/5	no	The query correctly aggregates both CSO discharge volume and total monthly rainfall, using the appropriate join pattern on aggregated months.
cso_by_watershed	✅	5/5	no	The query correctly joins the tables, groups by watershed, and applies the necessary filter for eventType.
enforcement_vs_budget	✅	5/5	no	The query correctly joins the enforcement actions and budget tables on the year, extracts the year from the EnforcementDate correctly, and applies the necessary filters.
staffing_trend	✅	5/5	no	The query uses the correct table, groups by year, and counts employees accurately from 2005 to present.
303d_impaired_trend	✅	5/5	no	The query correctly counts listings from EPA_303d_Impairments grouped by reportingCycle and ordered correctly.
303d_named_waterbody	✅	5/5	no	The query correctly uses the EPA_303d_Impairments table, applies the maximum reportingCycle filter, and uses the LIKE pattern for the waterbody.
cso_to_impaired	✅	5/5	no	Both joins are correct and the reportingCycle filter is applied to EPA_303d_Impairments, which is appropriate.
all_caps_boston	✅	5/5	no	The query correctly uses UPPER() to filter the municipality as required.
ecos_per_capita	✅	5/5	no	The query correctly uses the ECOS_budgets table, aggregates spending by state, and calculates per-capita spending for comparison across states.

[github-actions]

✅ Semantic Eval Results

Each eval sends a natural-language question plus the relevant table schemas from db_semantic_context.txt to gpt-4o-mini, executes the generated SQL against AMEND.db, then scores it with a second LLM call using a per-case rubric. Hard pass = SQL ran and returned rows without hitting any known anti-patterns. Fatal = judge determined the query would return wrong or misleading results.

Metric	Value
Hard pass rate	10/10 (100%)
Fatal failures	0
Mean judge score	5.0/5
P50 judge score	5/5
Model	gpt-4o-mini
Semantic context hash	7dc233f87aac

Per-case results

ID	Hard pass	Score	Fatal	Reason
cso_top_operator	✅	5/5	no	The query correctly aggregates total CSO discharge volume by operator for 2022, using the correct table and filters, and includes the necessary grouping and ordering.
cso_monthly_rainfall	✅	5/5	no	The query correctly aggregates both CSO discharge volume and total monthly rainfall, using the appropriate join pattern on aggregated months.
cso_by_watershed	✅	5/5	no	The query correctly joins the tables, groups by watershed, and applies the necessary filter for eventType.
enforcement_vs_budget	✅	5/5	no	The query correctly joins the enforcement actions and budget tables on the year, extracts the year from the EnforcementDate correctly, and applies the necessary filters.
staffing_trend	✅	5/5	no	The query uses the correct table, groups by year, and counts employees accurately from 2005 to present.
303d_impaired_trend	✅	5/5	no	The query correctly counts listings from EPA_303d_Impairments grouped by reportingCycle and ordered correctly.
303d_named_waterbody	✅	5/5	no	The query correctly uses the EPA_303d_Impairments table, applies the maximum reportingCycle filter, and uses the LIKE pattern for the waterbody.
cso_to_impaired	✅	5/5	no	Both joins are correct and the reportingCycle filter is applied to EPA_303d_Impairments, which is appropriate.
all_caps_boston	✅	5/5	no	The query correctly uses UPPER() to filter the municipality as required.
ecos_per_capita	✅	5/5	no	The query correctly uses the ECOS_budgets table, aggregates data for multiple states, and calculates per-capita spending accurately.

[github-actions]

✅ Semantic Eval Results

Each eval sends a natural-language question plus the relevant table schemas from db_semantic_context.txt to gpt-4o-mini, executes the generated SQL against AMEND.db, then scores it with a second LLM call using a per-case rubric. Hard pass = SQL ran and returned rows without hitting any known anti-patterns. Fatal = judge determined the query would return wrong or misleading results.

Metric	Value
Hard pass rate	10/10 (100%)
Fatal failures	0
Mean judge score	5.0/5
P50 judge score	5/5
Model	gpt-4o-mini
Semantic context hash	7dc233f87aac

Per-case results

ID	Hard pass	Score	Fatal	Reason
cso_top_operator	✅	5/5	no	The query correctly aggregates total CSO discharge volume by operator for 2022, using the correct table and filters, and orders the results as required.
cso_monthly_rainfall	✅	5/5	no	The query correctly aggregates both CSO discharge volume and total monthly rainfall, using the appropriate join pattern and date range filters.
cso_by_watershed	✅	5/5	no	The query correctly joins the tables, groups by watershed, and applies the necessary filter for eventType.
enforcement_vs_budget	✅	5/5	no	The query correctly joins the enforcement actions and budget tables on the year, extracts the year from the EnforcementDate correctly, and applies the necessary filters.
staffing_trend	✅	5/5	no	The query uses the correct table, groups by year, and counts employees accurately from 2005 to present.
303d_impaired_trend	✅	5/5	no	The query correctly counts listings from the EPA_303d_Impairments table, groups by reportingCycle, and orders the results correctly.
303d_named_waterbody	✅	5/5	no	The query correctly uses the EPA_303d_Impairments table, applies the maximum reportingCycle filter, and uses the LIKE pattern for the waterbody.
cso_to_impaired	✅	5/5	no	Both joins are correct and the reportingCycle filter is applied to EPA_303d_Impairments, which is appropriate.
all_caps_boston	✅	5/5	no	The query correctly uses UPPER() to filter the municipality and eventType, ensuring accurate results.
ecos_per_capita	✅	5/5	no	The query correctly uses the ECOS_budgets table, aggregates spending by state, and calculates per-capita spending for comparison across states.