diff --git a/.claude/projects/-home-nes-Documents-MAenvironmentaldata/memory/env_threshold_review.md b/.claude/projects/-home-nes-Documents-MAenvironmentaldata/memory/env_threshold_review.md new file mode 100644 index 0000000..f82f183 --- /dev/null +++ b/.claude/projects/-home-nes-Documents-MAenvironmentaldata/memory/env_threshold_review.md @@ -0,0 +1,14 @@ +--- +name: env-threshold-review +description: Pending task to revisit environmental relevance threshold after GC fix re-embed, with documented artifact +metadata: + type: project +--- + +After the GC formula fix and full re-embed (May 2026), the environmental bill count jumped from 329 → 654 at threshold=0.05. This needs a calibration review. + +**Task:** Re-run the threshold analysis — plot the score distribution, spot-check bills near the new boundary, and decide whether 0.05 is still correct or needs adjustment. Document the exercise in a written artifact (analysis page or data note) explaining: the differential cosine similarity method, the reference sets, how the threshold was chosen, and what the before/after counts were at various thresholds. + +**Why:** The doubling of env bill count is plausible (correct body text adds real signal) but should be verified with spot-checks. Some new bills at 0.05–0.08 may be genuine env bills the old wrong-GC embeddings missed; others may be false positives from body text that semantically resembles env topics without being env legislation. + +**Related:** [[project_data_pipeline]] — score_lobbying_bills.py ENV_THRESHOLD constant; [[ai_analysis_feature]] — env bill counts flow into the AMEND.db and dashboard. diff --git a/.claude/scheduled_tasks.lock b/.claude/scheduled_tasks.lock new file mode 100644 index 0000000..3e2ef1e --- /dev/null +++ b/.claude/scheduled_tasks.lock @@ -0,0 +1 @@ +{"sessionId":"d2fc4fd3-d9ff-44a4-a608-05ddeb65f47b","pid":1140968,"procStart":"103407381","acquiredAt":1779884724489} \ No newline at end of file diff --git a/.github/workflows/update-data.yml b/.github/workflows/update-data.yml index 4901052..cb26fab 100644 --- a/.github/workflows/update-data.yml +++ b/.github/workflows/update-data.yml @@ -15,6 +15,7 @@ jobs: contents: write # to push branch issues: write # to open failure issues pull-requests: write # to create PR + actions: write # to dispatch update-charts.yml after merge steps: - uses: actions/checkout@v4 @@ -54,6 +55,10 @@ jobs: python get_MA_precipitation.py python get_ATTAINS_303d.py python get_MS4_annual_reports.py --yes --skip-download + python get_MA_lobbying.py + python get_MA_legislature_bills.py + python score_lobbying_bills.py + python cluster_lobbying_bills.py --incremental - name: Validate data working-directory: get_data diff --git a/.gitignore b/.gitignore index 20a2b09..068ee71 100644 --- a/.gitignore +++ b/.gitignore @@ -103,6 +103,16 @@ get_data/MS4_annual_reports/ # Large files *EEADP_drinkingWater.csv +docs/data/MA_bill_embeddings.parquet +docs/data/MA_bill_embeddings.npy +get_data/MA_legislature_cache/ + +# MA lobbying large CSVs — stored in GCS, only samples committed +docs/data/MA_lobbying_bills.csv +docs/data/MA_lobbying_employers.csv +docs/data/MA_lobbying_summary_links.csv +docs/data/MA_lobbying_bills_scored.csv +docs/data/MA_legislature_bills.csv get_data/backup_AMEND.db get_data/EEADP_drinkingWater.csv @@ -117,3 +127,4 @@ get_data/backup_AMEND.db get_data/AMEND.db docs/assets/debug_screenshot.png +docs/data/MA_legislature_bills.csv.bak_wrong_gc diff --git a/CLAUDE.md b/CLAUDE.md index 402f316..9f06424 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -67,6 +67,13 @@ bash set_cors_gsutil.sh - **EPA NPDES page changes**: EPA changed JSON format and column names around 2025; both handled with `isinstance` checks and fallback column detection. - **EEA CSOAPI**: Requires `Referer` and `Origin` headers matching the portal URL; plain requests return HTTP 500. Pagination is 1-indexed. - **303(d) data (biennial)**: `get_ATTAINS_303d.py` fetches from MassGIS S3-hosted shapefiles (not the ATTAINS REST API, which times out on `/assessments`). Data updates only biennially (even years); the script exits early if all known cycles are already in the cached CSV. The 2020 cycle was never published by MassGIS. The 2024/2026 cycle is in draft as of April 2026 — the script will auto-detect it when MassGIS publishes the approved shapefile. `CSO_303d_Mapping` in `assemble_db.py` is a manually curated dict (35 verified matches of 56 CSO waterbodies); update it when a new cycle is added by reviewing new assessment unit names against CSO waterBody values. +- **MA lobbying portal (Incapsula WAF)**: The SoS portal (`sec.state.ma.us/LobbyistPublicSearch/`) is protected by Incapsula WAF. A Chrome User-Agent gets a 302 redirect to a JS challenge page. An **iPad User-Agent** bypasses it entirely with plain `requests` — no Selenium needed. The working UA is `Mozilla/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148`. +- **MA lobbying portal (search form)**: ASP.NET with ViewState. POST to `Default.aspx` with `drpType=L` (Lobbyist or Lobbying Entity — do NOT use `Z` which returns Client pages with different structure), `drpPageSize=20000`, `ddlYear=`. POST timeout must be 120s (response is ~2MB for ~1700 results). Data goes back to 2005 (22 years). +- **MA lobbying incremental fetch**: Disclosures are filed semi-annually per employer, so weekly CI almost always exits early. The incremental state is the `disc_url` set in `MA_lobbying_summary_links.csv` — only new `CompleteDisclosure.aspx` URLs get fetched. Always re-checks current and prior year for new filers. +- **MA lobbying full historical fetch**: At `REQUEST_DELAY=1.0s`, fetching a single year (~1700 registrants) takes ~10 hours. The full 22-year history (2005–2026) will take several days. To speed up the historical fetch (frozen past years), lower `REQUEST_DELAY` to 0.5s — safe since those filings never change. +- **Running lobbying scripts**: Do NOT use `conda run` with stdout redirect (`> file`) — `conda run` buffers all output through a pipe and the log file stays empty until the process exits. Run Python directly: `/home/nes/miniconda/envs/amend_python/bin/python -u get_MA_lobbying.py` (the `-u` flag ensures unbuffered output). +- **MA lobbying Gemini SDK**: Uses `google-genai` (new SDK, `google.genai`), NOT the old `google-generativeai` package. API: `client = genai.Client(api_key=...)`, then `client.models.embed_content(model='gemini-embedding-2', contents=text, config=types.EmbedContentConfig(output_dimensionality=768))`. +- **MA lobbying General Court formula bug (unfixed)**: `get_MA_lobbying.py` uses `FIRST_GC_START_YEAR = 2005` but the 183rd General Court started in 2003, not 2005. The correct constant is `2003`. As a result, every bill's `general_court` assignment is one session too low (year 2024 → GC192 instead of GC193, etc.), and `get_MA_legislature_bills.py` fetches bill text from the wrong legislative session. Fixing this constant and re-running the full pipeline would bring the title match rate from ~2% to ~65%. The remaining ~35% of mismatches are string-normalisation differences or genuinely wrong bill numbers in the SoS portal. See `get_data/NOTES_bill_embeddings.md` for full analysis. ## Running scripts diff --git a/analysis/MA_lobbying_tsne.py b/analysis/MA_lobbying_tsne.py new file mode 100644 index 0000000..43f21be --- /dev/null +++ b/analysis/MA_lobbying_tsne.py @@ -0,0 +1,236 @@ +"""Generate a UMAP scatter plot of MA lobbying bill embeddings. + +Visual design philosophy +───────────────────────── +MA legislative bill embeddings are semantically dense — all bills share heavy +regulatory language, so inter-cluster cosine distances are ~0.006 vs. +intra-cluster spread of ~0.53. Running t-SNE on all 25k bills produces a +featureless blob regardless of perplexity, because the structure simply doesn't +separate in 2-D. + +UMAP is used instead of t-SNE because it better preserves global structure, +pulling weakly-separated clusters apart more effectively than t-SNE's purely +local optimisation. Parameters: n_neighbors=30, min_dist=0.1, metric='cosine'. + +The chart shows TWO layers: + + Background (grey) — stratified sample of ~120 non-environmental bills per + cluster, rendered as tiny translucent grey dots. Provides + geographic context for the policy landscape. + + Signal (coloured) — all env-relevant bills (~654), one colour per cluster, + large outlined dots. These are what the visitor cares about. + +UMAP is computed on the combined ~3,650 point sample (all env + background), +which runs in ~30s and produces cleaner structure than t-SNE on this corpus. + +Run from the analysis/ directory: + /path/to/python -u MA_lobbying_tsne.py + +Outputs: + ../docs/_includes/charts/lobbying_bill_tsne.html +""" + +import sys +from pathlib import Path + +import numpy as np +import pandas as pd +import umap +from sklearn.preprocessing import normalize +import plotly.graph_objects as go + +sys.path.insert(0, str(Path(__file__).parent)) + +GCS_PARQUET = 'gs://openamend-data/MA_bill_embeddings.parquet' +LOCAL_PARQUET = Path('../docs/data/MA_bill_embeddings.parquet') +LABELS_CSV = Path('../docs/data/MA_bill_cluster_labels.csv') +OUT_HTML = Path('../docs/_includes/charts/lobbying_bill_tsne.html') + +# Non-env bills sampled per cluster for background context. +# 120 × 25 clusters ≈ 3 000 background points + ~329 env = ~3 300 total. +BG_PER_CLUSTER = 120 +RANDOM_STATE = 42 + +# UMAP hyperparameters +UMAP_N_NEIGHBORS = 30 # larger → more global structure +UMAP_MIN_DIST = 0.1 # smaller → tighter clusters +UMAP_METRIC = 'cosine' + +# 25-colour palette — qualitative, perceptually distinct, no cycling +PALETTE_25 = [ + '#1f77b4', '#ff7f0e', '#2ca02c', '#d62728', '#9467bd', + '#8c564b', '#e377c2', '#7f7f7f', '#bcbd22', '#17becf', + '#aec7e8', '#ffbb78', '#98df8a', '#ff9896', '#c5b0d5', + '#c49c94', '#f7b6d2', '#c7c7c7', '#dbdb8d', '#9edae5', + '#393b79', '#637939', '#8c6d31', '#843c39', '#7b4173', +] + + +def _load_parquet() -> pd.DataFrame: + try: + import gcsfs + fs = gcsfs.GCSFileSystem() + if fs.exists(GCS_PARQUET): + with fs.open(GCS_PARQUET, 'rb') as f: + df = pd.read_parquet(f) + print(f'Loaded {len(df)} rows from {GCS_PARQUET}') + return df + except Exception as e: + print(f'GCS load failed ({e}), trying local...') + if LOCAL_PARQUET.exists(): + df = pd.read_parquet(LOCAL_PARQUET) + print(f'Loaded {len(df)} rows from local Parquet') + return df + raise FileNotFoundError('No Parquet file found. Run score_lobbying_bills.py first.') + + +def main(): + parquet_df = _load_parquet() + + # Restrict to clustered bills + parquet_df = parquet_df[ + parquet_df['cluster_id'].notna() & (parquet_df['cluster_id'] != -1) + ].copy() + parquet_df['cluster_id'] = parquet_df['cluster_id'].astype(int) + + if 'is_environmental' not in parquet_df.columns: + parquet_df['is_environmental'] = False + parquet_df['is_environmental'] = parquet_df['is_environmental'].fillna(False).astype(bool) + + labels_df = pd.read_csv(LABELS_CSV, engine='python', on_bad_lines='skip') + # example_titles may contain unquoted commas that corrupt row parsing; + # keep only rows with a valid integer cluster_id. + labels_df = labels_df[ + pd.to_numeric(labels_df['cluster_id'], errors='coerce').notna() + ].copy() + labels_df['cluster_id'] = labels_df['cluster_id'].astype(int) + label_map = dict(zip(labels_df['cluster_id'].astype(int), labels_df['label'])) + nenv_map = dict(zip(labels_df['cluster_id'].astype(int), labels_df['n_env_bills'])) + + # ── Build subsample ────────────────────────────────────────────────────── + # Keep ALL env bills; sample BG_PER_CLUSTER non-env bills per cluster. + env_df = parquet_df[parquet_df['is_environmental']].copy() + non_env = parquet_df[~parquet_df['is_environmental']] + + rng = np.random.default_rng(RANDOM_STATE) + bg_parts = [] + for cid in sorted(non_env['cluster_id'].unique()): + sub = non_env[non_env['cluster_id'] == cid] + n = min(BG_PER_CLUSTER, len(sub)) + bg_parts.append(sub.sample(n=n, random_state=int(rng.integers(0, 2**31)))) + + bg_df = pd.concat(bg_parts, ignore_index=True) + sample = pd.concat([env_df, bg_df], ignore_index=True) + print(f'Subsample: {len(env_df)} env + {len(bg_df)} background = {len(sample)} total') + + # ── Embeddings ─────────────────────────────────────────────────────────── + emb = np.vstack(sample['embedding'].apply( + lambda v: np.array(v, dtype=np.float32) + ).values) + emb_norm = normalize(emb, norm='l2') + + # ── UMAP ───────────────────────────────────────────────────────────────── + print(f'Running UMAP (n={len(sample)}, n_neighbors={UMAP_N_NEIGHBORS}, ' + f'min_dist={UMAP_MIN_DIST}, metric={UMAP_METRIC})...') + reducer = umap.UMAP( + n_components=2, + n_neighbors=UMAP_N_NEIGHBORS, + min_dist=UMAP_MIN_DIST, + metric=UMAP_METRIC, + random_state=RANDOM_STATE, + low_memory=False, + ) + coords = reducer.fit_transform(emb_norm) + sample = sample.copy() + sample['x'] = coords[:, 0] + sample['y'] = coords[:, 1] + + # ── Build Plotly figure ────────────────────────────────────────────────── + fig = go.Figure() + + bg = sample[~sample['is_environmental']] + envs = sample[sample['is_environmental']] + + # Layer 1 — grey background (all non-env, single trace for performance) + fig.add_trace(go.Scatter( + x=bg['x'], y=bg['y'], + mode='markers', + marker=dict(color='#aaaaaa', size=4, opacity=0.20), + name='Non-environmental bills', + hovertext=[ + f'{row.get("bill_title", "")}
' + f'GC {int(row["general_court"])} · {label_map.get(int(row["cluster_id"]), "")}' + for _, row in bg.iterrows() + ], + hoverinfo='text', + showlegend=True, + legendgroup='bg', + legendgrouptitle=dict(text='Background'), + )) + + # Layer 2 — env bills, one trace per cluster that has any env bills + env_cluster_ids = sorted(envs['cluster_id'].unique()) + for i, cid in enumerate(env_cluster_ids): + sub = envs[envs['cluster_id'] == cid] + lbl = label_map.get(cid, f'Cluster {cid}') + nenv = nenv_map.get(cid, len(sub)) + color = PALETTE_25[cid % len(PALETTE_25)] + + fig.add_trace(go.Scatter( + x=sub['x'], y=sub['y'], + mode='markers', + marker=dict( + color=color, size=11, opacity=0.92, + line=dict(color='black', width=1.2), + ), + name=f'{lbl} ({nenv} env)', + hovertext=[ + f'{row.get("bill_title", "")}
' + f'GC {int(row["general_court"])} · 🌿 environmental
' + f'Cluster: {lbl}
' + f'Score: {row.get("env_relevance_score", ""):.3f}' + for _, row in sub.iterrows() + ], + hoverinfo='text', + showlegend=True, + legendgroup='env', + legendgrouptitle=dict(text='Environmental bills by cluster') if i == 0 else dict(text=''), + )) + + fig.update_layout( + title=dict( + text=( + 'MA Lobbying Bills — Environmental Bills in the Policy Landscape' + f'
Coloured = {len(envs)} environmentally-relevant bills · ' + f'grey = background sample ({len(bg):,} non-env) · ' + 'colour = topic cluster · hover for details · UMAP projection' + ), + font=dict(size=13), + ), + xaxis=dict(visible=False), + yaxis=dict(visible=False), + legend=dict( + font=dict(size=10), + itemsizing='constant', + tracegroupgap=8, + ), + margin=dict(l=10, r=10, t=70, b=10), + width=880, + height=600, + plot_bgcolor='#f8f8f8', + paper_bgcolor='white', + hovermode='closest', + ) + + OUT_HTML.parent.mkdir(parents=True, exist_ok=True) + html = fig.to_html(full_html=False, include_plotlyjs='cdn', config={'responsive': True}) + OUT_HTML.write_text( + '{% raw %}\n' + html + '\n{% endraw %}\n', + encoding='utf-8', + ) + print(f'Wrote {OUT_HTML}') + + +if __name__ == '__main__': + main() diff --git a/analysis/MA_lobbying_viz.py b/analysis/MA_lobbying_viz.py new file mode 100644 index 0000000..9c0b849 --- /dev/null +++ b/analysis/MA_lobbying_viz.py @@ -0,0 +1,1110 @@ +"""Generate charts for the MA environmental lobbying analysis. + +Dashboard charts (called with prefix='dash_' by dashboard_charts.py): + {prefix}lobbying_spend_trend — Annual lobbying spend on environmental bills, stacked by sector + {prefix}lobbying_top_employers — Top 15 employer spenders in most recent complete year + {prefix}lobbying_bill_intensity — Unique bills lobbied per year + pass rate + {prefix}lobbying_vs_enforcement — Dual-axis: lobbying spend vs. enforcement action count + +Analysis-post charts (no prefix, generate_post_charts): + lobbying_spend_vs_budget — Lobbying spend overlaid on DEP budget (dual-axis) + lobbying_bill_pass_by_spend_tier — Bill pass rate by lobbying intensity tier + lobbying_spend_vs_staff — Env lobbying spend vs. DEP FTE headcount (dual-axis) + lobbying_env_cluster_share — Env-bill lobbying spend by topic cluster, stacked over years + lobbying_top_env_employers — Top 20 employers ranked by total env-bill lobbying spend + lobbying_env_positions — Unique clients by Support/Oppose/Neutral position on env bills + lobbying_env_opponents — Top 20 clients by unique env bills opposed (all years) + lobbying_pass_by_position — Env bill pass rate by dominant lobbying position + lobbying_env_score_vs_clients — Scatter: env score vs. lobbying intensity, env + top-500 non-env + lobbying_cso_operators — Lobbying spend by known CSO operators (permittees), by year + +Data files written: + docs/data/facts_lobbying.yml — Key facts for Jekyll post templates +""" + +import sys +import os +from pathlib import Path +sys.path.insert(0, os.path.dirname(__file__)) + +import pandas as pd +import numpy as np +from sqlalchemy import create_engine +import chartjs + +BLUE = 'rgba(54, 110, 179, 0.85)' +RED = 'rgba(200, 60, 60, 0.85)' +ORANGE = 'rgba(230, 140, 40, 0.85)' +GREEN = 'rgba(60, 170, 80, 0.85)' +GREY = 'rgba(150, 150, 150, 0.6)' +PURPLE = 'rgba(130, 80, 200, 0.85)' +TEAL = 'rgba(30, 160, 160, 0.85)' +YELLOW = 'rgba(220, 180, 0, 0.85)' + +SECTOR_COLORS = [BLUE, ORANGE, GREEN, RED, PURPLE, TEAL, YELLOW, GREY] + +CHART_DIR = '../docs/_includes/charts' +FACTS_YML = '../docs/data/facts_lobbying.yml' + + +def _load_data(engine): + """Load lobbying/legislature tables from DB. Returns empty DataFrames if not yet populated.""" + def _safe_read(query): + try: + return pd.read_sql_query(query, engine) + except Exception: + return pd.DataFrame() + + employers = _safe_read('SELECT * FROM MA_Lobbying_Employers') + lobby_bills = _safe_read('SELECT * FROM MA_Lobbying_Bills') + # MA_Lobbying_Bills_Scored: is_environmental, env_relevance_score, cluster_id + # MA_Legislature_Bills: passed status + scored = _safe_read('SELECT * FROM MA_Lobbying_Bills_Scored') + leg_bills_raw = _safe_read( + 'SELECT bill_number, general_court, passed FROM MA_Legislature_Bills' + ) + if not scored.empty and not leg_bills_raw.empty: + leg_bills = scored.merge(leg_bills_raw, on=['bill_number', 'general_court'], how='left') + elif not scored.empty: + leg_bills = scored + else: + leg_bills = pd.DataFrame() + return employers, lobby_bills, leg_bills + + +def _env_bills(lobby_bills: pd.DataFrame, leg_bills: pd.DataFrame) -> pd.DataFrame: + """Return lobby_bills rows joined to environmentally relevant bills.""" + if leg_bills.empty or lobby_bills.empty or 'is_environmental' not in leg_bills.columns: + return pd.DataFrame() + env = leg_bills[leg_bills['is_environmental'] == 1][ + ['bill_number', 'general_court', 'passed'] + ].copy() + return lobby_bills.merge(env, on=['bill_number', 'general_court'], how='inner') + + +def _annual_env_spend(employers: pd.DataFrame, lobby_bills: pd.DataFrame, + leg_bills: pd.DataFrame) -> pd.DataFrame: + """Annual lobbying spend allocated to environmental bills (proportional). + + For each (entity, client, year) row in MA_Lobbying_Employers, computes + env_spend = compensation × (n_env_bills / n_all_bills) where both bill + counts are for that (entity, client, year) triple. Sums across all pairs + per year. + + Proportional allocation avoids inflating spend for clients who lobbied a + single env bill alongside hundreds of unrelated bills. + + Falls back to total env-client spend (non-proportional) if lobby_bills + has no year-level bill counts — but this should not occur in normal use. + + Excludes the legacy 'Total salaries received' aggregate rows. + """ + if employers.empty or lobby_bills.empty: + return pd.DataFrame() + env_lb = _env_bills(lobby_bills, leg_bills) + if env_lb.empty: + # No env scoring yet — fall back to total spend for clients with any bills + env_pairs = lobby_bills[['client_name', 'year']].drop_duplicates() + emp = employers[employers['client_name'] != 'Total salaries received'] + merged = emp.merge(env_pairs, on=['client_name', 'year'], how='inner') + return ( + merged.groupby('year')['compensation'] + .sum() + .reset_index() + .sort_values('year') + ) + + pair_keys = ['entity_name', 'client_name', 'year'] + # Count env bills per (firm, client, year) + env_counts = ( + env_lb.groupby(pair_keys)['bill_number'].nunique() + .reset_index(name='n_env') + ) + # Count all bills per (firm, client, year) + all_counts = ( + lobby_bills.groupby(pair_keys)['bill_number'].nunique() + .reset_index(name='n_all') + ) + fracs = env_counts.merge(all_counts, on=pair_keys, how='left') + fracs['env_frac'] = fracs['n_env'] / fracs['n_all'].replace(0, np.nan) + + emp = employers[employers['client_name'] != 'Total salaries received'] + merged = emp.merge(fracs, on=pair_keys, how='inner') + merged['env_spend'] = merged['compensation'] * merged['env_frac'].fillna(0) + return ( + merged.groupby('year')['env_spend'] + .sum() + .reset_index() + .rename(columns={'env_spend': 'compensation'}) + .sort_values('year') + ) + + +def generate_charts(engine, prefix=''): + """Generate dashboard lobbying charts. + + Parameters + ---------- + engine : sqlalchemy.engine.Engine + prefix : str + Filename prefix (e.g. 'dash_' for dashboard charts). + """ + employers, lobby_bills, leg_bills = _load_data(engine) + + if employers.empty: + print('MA lobbying data not yet available — skipping lobbying charts.') + return + + employers['year'] = pd.to_numeric(employers['year'], errors='coerce').astype('Int64') + if not lobby_bills.empty: + lobby_bills['year'] = pd.to_numeric(lobby_bills['year'], errors='coerce').astype('Int64') + + # ── Chart 1: Annual spend trend ─────────────────────────────────────────── + spend_trend = _annual_env_spend(employers, lobby_bills, leg_bills) + + if not spend_trend.empty: + years = spend_trend['year'].dropna().astype(int).tolist() + spend_m = (spend_trend['compensation'] / 1e6).tolist() + + c = chartjs.Chart( + 'Annual MA Lobbying Spend on Environmental Bills', + 'Bar', width=700, height=380, + ) + c.set_labels([str(y) for y in years]) + c.add_dataset(spend_m, 'Total spend ($M)', backgroundColor=f"'{BLUE}'") + c.set_params( + js_inline=False, + ylabel='Lobbying spend ($M)', + xlabel='Year', + ) + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_spend_trend.html') + print(f'Wrote {prefix}lobbying_spend_trend.html') + + # ── Chart 2: Top employers in most recent year ──────────────────────────── + most_recent_year = int(employers['year'].dropna().max()) + # Use second-most-recent year if most-recent looks like a partial year + # (fewer than half the employer count of the prior year) + year_counts = employers.groupby('year').size() + if len(year_counts) >= 2: + penultimate = int(sorted(year_counts.index)[-2]) + if year_counts[most_recent_year] < year_counts[penultimate] * 0.5: + most_recent_year = penultimate + + # Aggregate by client (paying entity), not by lobbying firm + emp_year = employers[ + (employers['year'] == most_recent_year) + & (employers['client_name'] != 'Total salaries received') + ] + top_employers = ( + emp_year.groupby('client_name')['compensation'].sum() + .nlargest(15) + .sort_values() # ascending for horizontal bar + .reset_index() + ) + + if not top_employers.empty: + c = chartjs.Chart( + f'Top 15 MA Lobbying Clients — {most_recent_year}', + 'HorizontalBar', width=700, height=440, + ) + c.set_labels(top_employers['client_name'].tolist()) + spend_k = (top_employers['compensation'] / 1e3).tolist() + c.add_dataset(spend_k, 'Spend ($K)', backgroundColor=f"'{ORANGE}'") + c.set_params(js_inline=False, xlabel='Lobbying spend ($K)') + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_top_employers.html') + print(f'Wrote {prefix}lobbying_top_employers.html') + + # ── Chart 3: Bill intensity — unique bills lobbied + pass rate ──────────── + if not lobby_bills.empty and not leg_bills.empty: + env_lb = _env_bills(lobby_bills, leg_bills) + if env_lb.empty: + env_lb = lobby_bills.copy() + env_lb['passed'] = np.nan + + bills_per_year = ( + env_lb.groupby('year')['bill_number'] + .nunique() + .reset_index(name='n_bills') + .sort_values('year') + ) + pass_rate_per_year = ( + env_lb.drop_duplicates(subset=['bill_number', 'general_court', 'year']) + .groupby('year')['passed'] + .mean() + .reset_index(name='pass_rate') + ) + bill_intensity = bills_per_year.merge(pass_rate_per_year, on='year', how='left') + + years_bi = bill_intensity['year'].dropna().astype(int).tolist() + n_bills = bill_intensity['n_bills'].tolist() + pass_pct = (bill_intensity['pass_rate'].fillna(0) * 100).tolist() + + c = chartjs.Chart( + 'Environmental Bills Lobbied per Year', + 'Bar', width=700, height=380, + ) + c.set_labels([str(y) for y in years_bi]) + c.add_dataset(n_bills, 'Unique bills lobbied', backgroundColor=f"'{TEAL}'", + yAxisID="'y'") + c.add_dataset(pass_pct, 'Pass rate (%)', backgroundColor=f"'{GREEN}'", + type="'line'", yAxisID="'y1'") + c.set_params( + js_inline=False, + ylabel='Bills lobbied', + xlabel='Year', + y2nd=1, + y2nd_title='Pass rate (%)', + ) + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_bill_intensity.html') + print(f'Wrote {prefix}lobbying_bill_intensity.html') + + # ── Chart 4: Lobbying spend vs. enforcement count (dual-axis) ───────────── + try: + enf = pd.read_sql_query( + "SELECT strftime('%Y', EnforcementDate) AS year, COUNT(*) AS n_actions " + "FROM MAEEADP_Enforcement " + "WHERE EnforcementType NOT IN (" + " 'Notice Of Non-Compliance','Field Notice Of Non Compliance'," + " 'BOIL ORDER','Federal Administrative Order Against PWS'," + " 'Federal Notice Of Noncompliance Against PWS'" + ") GROUP BY 1", + engine, + ) + enf['year'] = pd.to_numeric(enf['year'], errors='coerce').astype('Int64') + except Exception: + enf = pd.DataFrame() + + if not spend_trend.empty and not enf.empty: + merged = spend_trend.merge(enf, on='year', how='inner') + merged = merged.sort_values('year') + years_vs = merged['year'].astype(int).tolist() + spend_m_vs = (merged['compensation'] / 1e6).tolist() + n_enf = merged['n_actions'].tolist() + + c = chartjs.Chart( + 'MA Lobbying Spend vs. Enforcement Actions', + 'Bar', width=700, height=380, + ) + c.set_labels([str(y) for y in years_vs]) + c.add_dataset(spend_m_vs, 'Lobbying spend ($M)', backgroundColor=f"'{BLUE}'", + yAxisID="'y'") + c.add_dataset(n_enf, 'Enforcement actions', backgroundColor=f"'{RED}'", + type="'line'", yAxisID="'y1'") + c.set_params( + js_inline=False, + ylabel='Lobbying spend ($M)', + xlabel='Year', + y2nd=1, + y2nd_title='Enforcement actions', + ) + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_vs_enforcement.html') + print(f'Wrote {prefix}lobbying_vs_enforcement.html') + + # ── Chart 5: Lobbying spend by topic cluster (stacked bar by year) ─────────── + _chart_spend_by_cluster(engine, employers, lobby_bills, prefix) + + _write_facts(employers, spend_trend, most_recent_year) + + +def _chart_spend_by_cluster(engine, employers: pd.DataFrame, lobby_bills: pd.DataFrame, prefix: str): + """Stacked bar: annual employer spend broken down by bill topic cluster.""" + try: + scored = pd.read_sql_query( + 'SELECT bill_number, general_court, cluster_id FROM MA_Lobbying_Bills_Scored ' + 'WHERE cluster_id IS NOT NULL AND cluster_id != -1', + engine, + ) + cluster_labels = pd.read_sql_query( + 'SELECT cluster_id, label FROM MA_Bill_Cluster_Labels', engine, + ) + except Exception: + print(' Cluster data not yet in DB — skipping cluster spend chart.') + return + + if scored.empty: + print(' Cluster IDs not yet assigned — skipping cluster spend chart.') + return + + # Join cluster_id onto lobby_bills via bill_number + general_court + lb = lobby_bills.merge( + scored[['bill_number', 'general_court', 'cluster_id']], + on=['bill_number', 'general_court'], how='left' + ) + lb = lb.dropna(subset=['cluster_id']) + lb['cluster_id'] = lb['cluster_id'].astype(int) + + # Join client compensation: match (entity_name, client_name, year) + emp = employers[employers['client_name'] != 'Total salaries received'] + lb_emp = lb.merge(emp[['entity_name', 'client_name', 'year', 'compensation']], + on=['entity_name', 'client_name', 'year'], how='left') + + # Annual spend per cluster (divide compensation equally across clusters + # lobbied by each (firm, client) pair in that year to avoid double-counting) + clusters_per_pair_year = ( + lb_emp.groupby(['entity_name', 'client_name', 'year'])['cluster_id'] + .nunique() + .reset_index(name='n_clusters') + ) + lb_emp = lb_emp.merge(clusters_per_pair_year, + on=['entity_name', 'client_name', 'year']) + lb_emp['spend_share'] = lb_emp['compensation'] / lb_emp['n_clusters'] + + spend_by_cluster = ( + lb_emp.groupby(['year', 'cluster_id'])['spend_share'] + .sum() + .reset_index() + ) + + # Build cluster label map + label_map = dict(zip(cluster_labels['cluster_id'], cluster_labels['label'])) + spend_by_cluster['label'] = spend_by_cluster['cluster_id'].map(label_map).fillna('Other') + + years = sorted(spend_by_cluster['year'].dropna().astype(int).unique()) + # Top clusters by total spend across all years + top_clusters = ( + spend_by_cluster.groupby('cluster_id')['spend_share'] + .sum() + .nlargest(10) + .index.tolist() + ) + + c = chartjs.Chart( + 'MA Lobbying Spend by Topic Cluster', + 'Bar', width=700, height=420, + ) + c.set_labels([str(y) for y in years]) + + colors = SECTOR_COLORS + for i, cid in enumerate(top_clusters): + subset = spend_by_cluster[spend_by_cluster['cluster_id'] == cid] + year_spend = {int(r['year']): r['spend_share'] / 1e6 + for _, r in subset.iterrows()} + data = [year_spend.get(y, 0) for y in years] + label = label_map.get(cid, f'Cluster {cid}') + c.add_dataset(data, label, + backgroundColor=f"'{colors[i % len(colors)]}'", + stack="'topic'") + + c.set_params( + js_inline=False, + ylabel='Lobbying spend ($M)', + xlabel='Year', + stacked=True, + ) + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_spend_by_cluster.html') + print(f'Wrote {prefix}lobbying_spend_by_cluster.html') + + +def generate_post_charts(engine, prefix=''): + """Generate analysis-post lobbying charts (not suitable for weekly CI).""" + employers, lobby_bills, leg_bills = _load_data(engine) + + if employers.empty: + print('MA lobbying data not yet available — skipping post charts.') + return + + employers['year'] = pd.to_numeric(employers['year'], errors='coerce').astype('Int64') + + # ── Post chart 1: Lobbying spend vs. DEP budget ─────────────────────────── + try: + budget = pd.read_sql_query( + 'SELECT Year, DEPAdministration_inf FROM MassBudget_summary', engine + ) + budget['Year'] = pd.to_numeric(budget['Year'], errors='coerce').astype('Int64') + except Exception as e: + print(f' Budget query failed: {e}') + budget = pd.DataFrame() + + spend_trend = _annual_env_spend(employers, lobby_bills, leg_bills) + + if not spend_trend.empty and not budget.empty: + merged = spend_trend.merge(budget, left_on='year', right_on='Year', how='inner') + merged = merged.sort_values('year') + years_sb = merged['year'].astype(int).tolist() + spend_m = (merged['compensation'] / 1e6).tolist() + budget_m = (merged['DEPAdministration_inf'].astype(float) / 1e6).tolist() + + c = chartjs.Chart( + 'MA Lobbying Spend vs. DEP Budget (inflation-adjusted)', + 'Bar', width=700, height=400, + ) + c.set_labels([str(y) for y in years_sb]) + c.add_dataset(spend_m, 'Industry lobbying spend ($M)', backgroundColor=f"'{ORANGE}'", + yAxisID="'y'") + c.add_dataset(budget_m, 'DEP admin budget ($M, inflation-adj.)', + backgroundColor=f"'{BLUE}'", type="'line'", yAxisID="'y1'") + c.set_params( + js_inline=False, + ylabel='Lobbying spend ($M)', + xlabel='Year', + y2nd=1, + y2nd_title='DEP budget ($M)', + ) + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_spend_vs_budget.html') + print(f'Wrote {prefix}lobbying_spend_vs_budget.html') + + # ── Post chart 2: Bill pass rate by lobbying intensity tier ─────────────── + if not lobby_bills.empty and not leg_bills.empty: + env_lb = _env_bills(lobby_bills, leg_bills) + if not env_lb.empty and 'passed' in env_lb.columns: + employer_counts = ( + env_lb.groupby(['bill_number', 'general_court'])['client_name'] + .nunique() + .reset_index(name='employer_count') + ) + bill_info = leg_bills[['bill_number', 'general_court', 'passed']].drop_duplicates() + tc = employer_counts.merge(bill_info, on=['bill_number', 'general_court'], how='left') + + def _tier(n): + if n >= 10: + return '10+ clients' + elif n >= 3: + return '3–9 clients' + else: + return '1–2 clients' + + tc['tier'] = tc['employer_count'].apply(_tier) + tier_order = ['1–2 clients', '3–9 clients', '10+ clients'] + summary = ( + tc.groupby('tier')['passed'] + .agg(['mean', 'count']) + .reindex(tier_order) + .reset_index() + ) + + c = chartjs.Chart( + 'Environmental Bill Pass Rate by Lobbying Intensity', + 'Bar', width=500, height=360, + ) + c.set_labels(tier_order) + c.add_dataset( + (summary['mean'].fillna(0) * 100).tolist(), + 'Pass rate (%)', + backgroundColor=f"'{GREEN}'", + ) + c.set_params(js_inline=False, ylabel='Pass rate (%)', xlabel='Number of employer lobbiers') + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_bill_pass_by_spend_tier.html') + print(f'Wrote {prefix}lobbying_bill_pass_by_spend_tier.html') + + # ── Post chart 3: Lobbying spend vs. DEP FTE headcount ──────────────────── + try: + staff = pd.read_sql_query( + "SELECT year, COUNT(*) AS n_fte FROM MADEP_staff_Comptroller " + "WHERE pay_total_actual > 0 GROUP BY year", engine + ) + staff['year'] = pd.to_numeric(staff['year'], errors='coerce').astype('Int64') + except Exception: + staff = pd.DataFrame() + + if not spend_trend.empty and not staff.empty: + merged = spend_trend.merge(staff, on='year', how='inner').sort_values('year') + if not merged.empty: + years_s = merged['year'].astype(int).tolist() + spend_m = (merged['compensation'] / 1e6).tolist() + fte = merged['n_fte'].astype(int).tolist() + + c = chartjs.Chart( + 'Environmental Lobbying Spend vs. DEP Staff Headcount', + 'Bar', width=700, height=400, + ) + c.set_labels([str(y) for y in years_s]) + c.add_dataset(spend_m, 'Industry lobbying spend ($M)', + backgroundColor=f"'{ORANGE}'", yAxisID="'y'") + c.add_dataset(fte, 'DEP staff (FTE)', + backgroundColor=f"'{BLUE}'", type="'line'", yAxisID="'y1'") + c.set_params( + js_inline=False, + ylabel='Lobbying spend ($M)', + xlabel='Year', + y2nd=1, + y2nd_title='DEP staff (FTE)', + ) + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_spend_vs_staff.html') + print(f'Wrote {prefix}lobbying_spend_vs_staff.html') + + # ── Post chart 4: Env-bill lobbying spend by topic cluster, stacked ─────── + # Joins employers→lobby_bills→scored→cluster_labels. + if (not employers.empty and not lobby_bills.empty + and 'cluster_id' in leg_bills.columns): + try: + cluster_labels = pd.read_sql_query( + 'SELECT cluster_id, label FROM MA_Bill_Cluster_Labels', engine + ) + except Exception: + cluster_labels = pd.DataFrame() + + if not cluster_labels.empty and 'cluster_id' in cluster_labels.columns: + cluster_labels['cluster_id'] = pd.to_numeric( + cluster_labels['cluster_id'], errors='coerce' + ).astype('Int64') + + env_lb = _env_bills(lobby_bills, leg_bills) + if not env_lb.empty and not cluster_labels.empty: + # Attach cluster_id to each env lobby_bills row + scored_cluster = leg_bills[['bill_number', 'general_court', 'cluster_id']] + env_lb_c = env_lb.merge( + scored_cluster, on=['bill_number', 'general_court'], how='left' + ) + # Allocate (firm, client) compensation equally across env bills they + # lobbied that year, then sum by (year, cluster_id). + pair_year_bills = ( + env_lb_c.groupby(['entity_name', 'client_name', 'year']) + .size().reset_index(name='n_env_bills') + ) + emp = employers[employers['client_name'] != 'Total salaries received'] + emp_join = emp.merge( + pair_year_bills, on=['entity_name', 'client_name', 'year'], how='inner' + ) + emp_join['per_bill'] = emp_join['compensation'] / emp_join['n_env_bills'] + cluster_spend = env_lb_c.merge( + emp_join[['entity_name', 'client_name', 'year', 'per_bill']], + on=['entity_name', 'client_name', 'year'], how='left' + ).dropna(subset=['cluster_id', 'per_bill']) + cluster_spend['cluster_id'] = cluster_spend['cluster_id'].astype(int) + agg = ( + cluster_spend.groupby(['year', 'cluster_id'])['per_bill'] + .sum().reset_index() + ) + agg = agg.merge(cluster_labels, on='cluster_id', how='left') + pivot = agg.pivot_table( + index='year', columns='label', values='per_bill', aggfunc='sum' + ).fillna(0).sort_index() + # Keep top 8 clusters by total spend, group rest into "Other" + totals = pivot.sum(axis=0).sort_values(ascending=False) + top = totals.head(8).index.tolist() + other_cols = [c for c in pivot.columns if c not in top] + if other_cols: + pivot['Other'] = pivot[other_cols].sum(axis=1) + pivot = pivot[top + ['Other']] + else: + pivot = pivot[top] + + years = pivot.index.astype(int).tolist() + c = chartjs.Chart( + 'Environmental Lobbying Spend by Topic Cluster', + 'Bar', width=750, height=420, + ) + c.set_labels([str(y) for y in years]) + for i, col in enumerate(pivot.columns): + color = SECTOR_COLORS[i % len(SECTOR_COLORS)] + c.add_dataset( + (pivot[col] / 1e6).tolist(), col, + backgroundColor=f"'{color}'", stack="'a'", + ) + c.set_params( + js_inline=False, + ylabel='Allocated spend ($M)', + xlabel='Year', + stacked=1, + ) + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_env_cluster_share.html') + print(f'Wrote {prefix}lobbying_env_cluster_share.html') + + # ── Post chart 5: Top clients by cumulative environmental lobbying spend ── + if not employers.empty and not lobby_bills.empty: + env_lb = _env_bills(lobby_bills, leg_bills) + if not env_lb.empty: + # Per (firm, client, year): env share = env bills / total bills lobbied + pair_keys = ['entity_name', 'client_name', 'year'] + bill_counts = ( + lobby_bills.groupby(pair_keys).size() + .reset_index(name='n_all') + ) + env_counts = ( + env_lb.groupby(pair_keys).size() + .reset_index(name='n_env') + ) + shares = bill_counts.merge(env_counts, on=pair_keys, how='left') + shares['n_env'] = shares['n_env'].fillna(0) + shares['env_share'] = shares['n_env'] / shares['n_all'].replace(0, np.nan) + emp = employers[employers['client_name'] != 'Total salaries received'] + pair_year = emp.merge(shares, on=pair_keys, how='inner') + pair_year['env_spend'] = pair_year['compensation'] * pair_year['env_share'] + top_clients = ( + pair_year.groupby('client_name')['env_spend'] + .sum().sort_values(ascending=False).head(20) + ) + if not top_clients.empty: + # Reverse so largest is at top in horizontal bar (ascending order) + top_clients = top_clients.sort_values() + c = chartjs.Chart( + 'Top 20 Clients by Cumulative Environmental Lobbying Spend', + 'HorizontalBar', width=750, height=520, + ) + c.set_labels(top_clients.index.tolist()) + c.add_dataset( + (top_clients.values / 1e6).tolist(), + 'Total env-bill spend ($M, all years)', + backgroundColor=f"'{GREEN}'", + ) + c.set_params( + js_inline=False, + ylabel='', + xlabel='Cumulative env-bill spend ($M)', + ) + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_top_env_employers.html') + print(f'Wrote {prefix}lobbying_top_env_employers.html') + + # ── Post chart 6: Support/Oppose/Neutral trend on env bills ────────────── + _chart_env_position_trend(lobby_bills, leg_bills, prefix) + + # ── Post chart 7: Top opponents of env bills ────────────────────────────── + _chart_top_env_opponents(lobby_bills, leg_bills, prefix) + + # ── Post chart 8: Env bill pass rate by dominant lobbying position ──────── + _chart_pass_rate_by_position(lobby_bills, leg_bills, prefix) + + # ── Post chart 9: Env score vs. lobbying intensity scatter ─────────────── + _chart_env_score_vs_clients(engine, prefix) + + # ── Post chart 10: Lobbying spend by known CSO operators + proxies ──────── + # Cross-references MA_Lobbying_Employers.client_name with MAEEADP_CSO.permiteeName. + # Includes the Massachusetts Municipal Association as a proxy: it is the + # primary lobbyist for municipal CSO operators (most cities/towns lobby + # through MMA rather than directly). + try: + cso_permittees = pd.read_sql_query( + 'SELECT DISTINCT permiteeName FROM MAEEADP_CSO WHERE permiteeName IS NOT NULL', + engine, + ) + except Exception: + cso_permittees = pd.DataFrame() + + PROXY_LOBBYISTS = { + 'MASSACHUSETTS MUNICIPAL ASSOCIATION': 'Massachusetts Municipal Association (CSO proxy)', + } + + if not employers.empty and not cso_permittees.empty: + import re + def _norm(s): + # Collapse 'AND'/'&' to space, drop punctuation, collapse whitespace + t = re.sub(r'[&]', ' ', str(s).upper()) + t = re.sub(r'\bAND\b', ' ', t) + t = ''.join(ch if ch.isalnum() or ch == ' ' else ' ' for ch in t) + t = re.sub(r'\s+', ' ', t).strip() + return t + + operator_norms = {_norm(p): p for p in cso_permittees['permiteeName'].dropna()} + operator_norms = {k: v for k, v in operator_norms.items() if len(k) > 4} + + def _match_operator(name): + n = _norm(name) + for proxy_norm, label in PROXY_LOBBYISTS.items(): + if proxy_norm in n: + return label + for op_norm, op in operator_norms.items(): + if op_norm in n or n in op_norm: + return op + return None + + emp = employers[employers['client_name'] != 'Total salaries received'].copy() + emp['cso_operator'] = emp['client_name'].apply(_match_operator) + cso_emp = emp.dropna(subset=['cso_operator']) + if not cso_emp.empty: + yearly = ( + cso_emp.groupby(['year', 'cso_operator'])['compensation'] + .sum().reset_index() + ) + # Keep top 8 operators by total spend + top_ops = ( + yearly.groupby('cso_operator')['compensation'] + .sum().sort_values(ascending=False).head(8).index.tolist() + ) + yearly = yearly[yearly['cso_operator'].isin(top_ops)] + pivot = yearly.pivot_table( + index='year', columns='cso_operator', values='compensation', aggfunc='sum' + ).fillna(0).sort_index() + + years = pivot.index.astype(int).tolist() + c = chartjs.Chart( + 'Total Lobbying Spend by Known CSO Operators', + 'Bar', width=750, height=420, + ) + c.set_labels([str(y) for y in years]) + for i, col in enumerate(pivot.columns): + color = SECTOR_COLORS[i % len(SECTOR_COLORS)] + c.add_dataset( + (pivot[col] / 1e6).tolist(), col, + backgroundColor=f"'{color}'", stack="'a'", + ) + c.set_params( + js_inline=False, + ylabel='Annual lobbying spend ($M)', + xlabel='Year', + stacked=1, + ) + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_cso_operators.html') + print(f'Wrote {prefix}lobbying_cso_operators.html') + + +def _chart_env_position_trend(lobby_bills: pd.DataFrame, leg_bills: pd.DataFrame, + prefix: str): + """Stacked-area: unique clients taking Support/Oppose/Neutral positions on env bills by year. + + Note: "opposing an environmental bill" does not always mean opposing environmental + protection — some env advocates oppose bills they consider inadequate or harmful. + The chart shows industry engagement with env-relevant legislation, not ideology. + """ + if lobby_bills.empty or leg_bills.empty or 'is_environmental' not in leg_bills.columns: + return + + env_ids = leg_bills[leg_bills['is_environmental'] == 1][ + ['bill_number', 'general_court'] + ].copy() + env_lb = lobby_bills.merge(env_ids, on=['bill_number', 'general_court'], how='inner') + if env_lb.empty: + return + + pos_yr = ( + env_lb[env_lb['position'].isin(['Support', 'Oppose', 'Neutral'])] + .groupby(['year', 'position'])['client_name'] + .nunique() + .reset_index(name='n_clients') + ) + pivot = pos_yr.pivot_table( + index='year', columns='position', values='n_clients', fill_value=0 + ).sort_index() + for col in ['Support', 'Oppose', 'Neutral']: + if col not in pivot.columns: + pivot[col] = 0 + + # Drop sparse early years (fewer than 5 total clients across positions) + pivot = pivot[pivot[['Support', 'Oppose', 'Neutral']].sum(axis=1) >= 5] + if pivot.empty: + return + + years = pivot.index.astype(int).tolist() + c = chartjs.Chart( + 'Unique Clients by Position on Environmental Bills', + 'Bar', width=700, height=380, + ) + c.set_labels([str(y) for y in years]) + c.add_dataset(pivot['Support'].tolist(), 'Support', + backgroundColor=f"'{GREEN}'", stack="'pos'") + c.add_dataset(pivot['Neutral'].tolist(), 'Neutral', + backgroundColor=f"'{GREY}'", stack="'pos'") + c.add_dataset(pivot['Oppose'].tolist(), 'Oppose', + backgroundColor=f"'{RED}'", stack="'pos'") + c.set_params( + js_inline=False, + ylabel='Unique lobbying clients', + xlabel='Year', + stacked=True, + ) + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_env_positions.html') + print(f'Wrote {prefix}lobbying_env_positions.html') + + +def _chart_top_env_opponents(lobby_bills: pd.DataFrame, leg_bills: pd.DataFrame, + prefix: str): + """Horizontal bar: clients ranked by unique env bills opposed (all years). + + "Opposing" an env-relevant bill can reflect either industry opposition to + new regulation, or an env group opposing a bill it considers harmful. + Top opponents are labelled accordingly where known. + """ + if lobby_bills.empty or leg_bills.empty or 'is_environmental' not in leg_bills.columns: + return + + env_ids = leg_bills[leg_bills['is_environmental'] == 1][ + ['bill_number', 'general_court'] + ].copy() + env_lb = lobby_bills.merge(env_ids, on=['bill_number', 'general_court'], how='inner') + if env_lb.empty: + return + + oppose = ( + env_lb[env_lb['position'] == 'Oppose'] + .groupby('client_name')[['bill_number', 'general_court']] + .apply(lambda g: g.drop_duplicates().shape[0]) + .reset_index(name='n_bills_opposed') + .sort_values('n_bills_opposed', ascending=False) + .head(20) + .sort_values('n_bills_opposed') # ascending for horizontal bar + ) + if oppose.empty: + return + + c = chartjs.Chart( + 'Top 20 Clients Opposing Environmental Bills (all years)', + 'HorizontalBar', width=750, height=520, + ) + c.set_labels(oppose['client_name'].tolist()) + c.add_dataset( + oppose['n_bills_opposed'].tolist(), + 'Unique env bills opposed', + backgroundColor=f"'{RED}'", + ) + c.set_params( + js_inline=False, + ylabel='', + xlabel='Unique environmental bills opposed', + ) + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_env_opponents.html') + print(f'Wrote {prefix}lobbying_env_opponents.html') + + +def _chart_pass_rate_by_position(lobby_bills: pd.DataFrame, leg_bills: pd.DataFrame, + prefix: str): + """Grouped bar: env bill pass rate by dominant lobbying position. + + Classifies each env bill as 'Mostly supported', 'Mostly opposed', or + 'Contested/Neutral' based on which position has the most unique clients. + Shows pass rate and bill count per category. + """ + if lobby_bills.empty or leg_bills.empty or 'is_environmental' not in leg_bills.columns: + return + if 'passed' not in leg_bills.columns: + return + + env_scored = leg_bills[leg_bills['is_environmental'] == 1][ + ['bill_number', 'general_court', 'passed'] + ].drop_duplicates() + if env_scored.empty: + return + + env_lb = lobby_bills.merge( + env_scored[['bill_number', 'general_court']], + on=['bill_number', 'general_court'], how='inner' + ) + pos_counts = ( + env_lb[env_lb['position'].isin(['Support', 'Oppose'])] + .groupby(['bill_number', 'general_court', 'position'])['client_name'] + .nunique() + .unstack(fill_value=0) + .reset_index() + ) + for col in ['Support', 'Oppose']: + if col not in pos_counts.columns: + pos_counts[col] = 0 + + def _category(row): + if row['Support'] > row['Oppose']: + return 'Mostly supported' + if row['Oppose'] > row['Support']: + return 'Mostly opposed' + return 'Contested / Neutral' + + pos_counts['category'] = pos_counts.apply(_category, axis=1) + tc = pos_counts.merge(env_scored, on=['bill_number', 'general_court'], how='left') + + cat_order = ['Mostly supported', 'Mostly opposed', 'Contested / Neutral'] + summary = ( + tc.groupby('category')['passed'] + .agg(pass_rate='mean', n_bills='count') + .reindex(cat_order) + .fillna(0) + .reset_index() + ) + + c = chartjs.Chart( + 'Environmental Bill Pass Rate by Lobbying Position', + 'Bar', width=520, height=360, + ) + c.set_labels(cat_order) + c.add_dataset( + (summary['pass_rate'] * 100).round(1).tolist(), + 'Pass rate (%)', + backgroundColor=[f"'{GREEN}'", f"'{RED}'", f"'{GREY}'"], + ) + c.set_params( + js_inline=False, + ylabel='Pass rate (%)', + xlabel='Dominant lobbying position', + ) + c.jekyll_write(f'{CHART_DIR}/{prefix}lobbying_pass_by_position.html') + print(f'Wrote {prefix}lobbying_pass_by_position.html') + + +def _chart_env_score_vs_clients(engine, prefix: str, top_n_nonenv: int = 500): + """Scatter: environmental relevance score (x) vs. unique lobbying clients (y, log scale). + + Three groups: + Environmental — all env-relevant bills (green, outlined) + Appropriations — annual budget / line-item bills (purple); separated because + they attract hundreds of clients for budget reasons unrelated + to the bill's policy topic — they dominate the y-axis and + are a distinct lobbying mechanism + Non-env policy — top-N most-lobbied non-appropriations, non-env bills (grey) + + Y-axis is log-scaled: most bills have 1–10 clients, appropriations have 300+, + so linear scale compresses the interesting region. + Marginal histograms show the density distribution of each group along both axes. + Threshold line (x = 0.05) only on main scatter and top (x) marginal. + """ + import plotly.express as px + + try: + scored = pd.read_sql_query( + 'SELECT bill_number, general_court, bill_id, bill_title, ' + ' env_relevance_score, is_environmental ' + 'FROM MA_Lobbying_Bills_Scored', + engine, + ) + counts = pd.read_sql_query( + 'SELECT bill_number, general_court, ' + ' COUNT(DISTINCT client_name) AS n_clients ' + 'FROM MA_Lobbying_Bills ' + 'GROUP BY bill_number, general_court', + engine, + ) + except Exception as e: + print(f' env_score_vs_clients: DB query failed ({e}) — skipping') + return + + df = scored.merge(counts, on=['bill_number', 'general_court'], how='left') + df['n_clients'] = df['n_clients'].fillna(0).astype(int) + df['bill_title'] = df['bill_title'].fillna('').astype(str) + + # Classify appropriations by title pattern — these are the annual budget bills + # and their line-item amendments, which attract 100–350 clients purely because + # they're the vehicle for all state spending decisions. + _approp_re = ( + r'(?i)making appropriations|appropriation.*fiscal year' + r'|line item \d|amendment.*\d{4}-\d{4}' + ) + df['is_approp'] = df['bill_title'].str.contains(_approp_re, regex=True, na=False) + + env = df[df['is_environmental'] == 1].copy() + approp = df[(df['is_environmental'] == 0) & df['is_approp']].copy() + policy_nonenv = ( + df[(df['is_environmental'] == 0) & ~df['is_approp']] + .nlargest(top_n_nonenv, 'n_clients') + .copy() + ) + + def _group(row): + if row['is_environmental'] == 1: + return 'Environmental' + if row['is_approp']: + return 'Appropriations bill' + return f'Non-env policy (top {top_n_nonenv})' + + plot_df = pd.concat([env, approp, policy_nonenv], ignore_index=True) + plot_df['group'] = plot_df.apply(_group, axis=1) + # log1p for y so bills with 0 clients don't vanish; displayed as n_clients + plot_df['n_clients_log'] = np.log1p(plot_df['n_clients']) + # Short title for hover name (shown bold at top) + plot_df['title_short'] = plot_df['bill_title'].str.slice(0, 90) + + color_map = { + 'Environmental': '#2ca02c', + 'Appropriations bill': '#9467bd', + f'Non-env policy (top {top_n_nonenv})': '#aaaaaa', + } + + fig = px.scatter( + plot_df, + x='env_relevance_score', + y='n_clients', + color='group', + color_discrete_map=color_map, + hover_name='title_short', + hover_data={ + 'title_short': False, + 'bill_title': False, + 'is_environmental': False, + 'is_approp': False, + 'group': False, + 'n_clients_log': False, + 'env_relevance_score': ':.3f', + 'n_clients': True, + 'bill_id': True, + 'general_court': True, + }, + marginal_x='histogram', + marginal_y='histogram', + labels={ + 'env_relevance_score': 'Environmental relevance score', + 'n_clients': 'Unique lobbying clients', + 'bill_id': 'Bill ID', + 'general_court': 'General Court', + 'group': '', + }, + title=( + 'Environmental Relevance vs. Lobbying Intensity
' + f'All env bills · top {top_n_nonenv} non-env policy bills · ' + 'appropriations bills shown separately · hover for title' + ), + opacity=0.72, + width=820, + height=620, + ) + + # Env dots: slightly larger, outlined + fig.update_traces( + selector=dict(type='scatter', name='Environmental'), + marker=dict(size=8, line=dict(color='black', width=0.8)), + ) + fig.update_traces( + selector=dict(type='scatter', name='Appropriations bill'), + marker=dict(size=5), + ) + fig.update_traces( + selector=dict(type='scatter', name=f'Non-env policy (top {top_n_nonenv})'), + marker=dict(size=5), + ) + + # Threshold line on main scatter and top x-marginal. + # Plain add_vline without row/col — plotly draws it at x=0.05 on each subplot's + # own x-axis. The right marginal histogram's x-axis is in units of bill count + # (0–300+), so x=0.05 lands at the invisible left edge there. No row/col + # specification avoids the axis-matching infinite-loop bug in plotly express + # marginal figures. + fig.add_vline( + x=0.05, line_dash='dot', line_color='#2ca02c', line_width=1.2, + annotation_text='env threshold (0.05)', + annotation_position='top right', + annotation_font_size=10, + ) + + # Log scale on main scatter y-axis. + # Must NOT use log_y=True in px.scatter — it transforms a shared axis in a + # way that breaks marginal histogram rendering. + # In px.scatter with marginal_x + marginal_y, yaxis is the main scatter y-axis + # and yaxis2 (right marginal) has matches='y', so both get log together which + # is correct: the marginal histogram's n_clients axis stays in sync. + fig.update_layout(yaxis=dict( + type='log', + tickmode='array', + tickvals=[1, 2, 5, 10, 20, 50, 100, 200, 350], + ticktext=['1', '2', '5', '10', '20', '50', '100', '200', '350'], + )) + + fig.update_layout( + legend=dict(orientation='h', yanchor='bottom', y=1.02, xanchor='left', x=0), + plot_bgcolor='#f8f8f8', + paper_bgcolor='white', + ) + + out = Path(CHART_DIR) / f'{prefix}lobbying_env_score_vs_clients.html' + html = fig.to_html(full_html=False, include_plotlyjs='cdn', config={'responsive': True}) + out.write_text('{% raw %}\n' + html + '\n{% endraw %}\n', encoding='utf-8') + print(f'Wrote {prefix}lobbying_env_score_vs_clients.html') + + +def _write_facts(employers: pd.DataFrame, spend_trend: pd.DataFrame, most_recent_year: int): + facts = {} + if not employers.empty: + emp_year = employers[ + (employers['year'] == most_recent_year) + & (employers['client_name'] != 'Total salaries received') + ] + facts['lobbying_most_recent_year'] = most_recent_year + facts['lobbying_n_employers'] = int(emp_year['client_name'].nunique()) + facts['lobbying_n_firms'] = int(emp_year['entity_name'].nunique()) + if not spend_trend.empty: + latest_spend = spend_trend[spend_trend['year'] == most_recent_year]['compensation'] + if not latest_spend.empty: + facts['lobbying_total_spend_latest'] = int(latest_spend.iloc[0]) + + with open(FACTS_YML, 'w') as f: + for k, v in facts.items(): + f.write(f'{k}: {v}\n') + + +if __name__ == '__main__': + _db = Path(__file__).parent.parent / 'get_data' / 'AMEND.db' + engine = create_engine(f'sqlite:///{_db}') + generate_charts(engine, prefix='') + generate_post_charts(engine, prefix='') diff --git a/analysis/dashboard_charts.py b/analysis/dashboard_charts.py index 74388a9..04e1295 100644 --- a/analysis/dashboard_charts.py +++ b/analysis/dashboard_charts.py @@ -12,6 +12,7 @@ import ECOS_budgets_viz import EPA_303d_viz import MS4_compliance_viz +import MA_lobbying_viz from EEA_DP_CSO_map import CSOAnalysisEEADP PREFIX = 'dash_' @@ -56,3 +57,6 @@ # --- MS4 stormwater compliance charts (3 charts) --- MS4_compliance_viz.generate_charts(engine, prefix=PREFIX) + +# --- Lobbying charts (4 charts; gracefully skipped if data not yet fetched) --- +MA_lobbying_viz.generate_charts(engine, prefix=PREFIX) diff --git a/docs/_includes/charts/dash_lobbying_bill_intensity.html b/docs/_includes/charts/dash_lobbying_bill_intensity.html new file mode 100644 index 0000000..c2a53b1 --- /dev/null +++ b/docs/_includes/charts/dash_lobbying_bill_intensity.html @@ -0,0 +1,73 @@ +{% raw %} + + + Environmental Bills Lobbied per Year + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/dash_lobbying_spend_by_cluster.html b/docs/_includes/charts/dash_lobbying_spend_by_cluster.html new file mode 100644 index 0000000..064f0c0 --- /dev/null +++ b/docs/_includes/charts/dash_lobbying_spend_by_cluster.html @@ -0,0 +1,73 @@ +{% raw %} + + + MA Lobbying Spend by Topic Cluster + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/dash_lobbying_spend_trend.html b/docs/_includes/charts/dash_lobbying_spend_trend.html new file mode 100644 index 0000000..1828b8a --- /dev/null +++ b/docs/_includes/charts/dash_lobbying_spend_trend.html @@ -0,0 +1,73 @@ +{% raw %} + + + Annual MA Lobbying Spend on Environmental Bills + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/dash_lobbying_top_employers.html b/docs/_includes/charts/dash_lobbying_top_employers.html new file mode 100644 index 0000000..3c44a5f --- /dev/null +++ b/docs/_includes/charts/dash_lobbying_top_employers.html @@ -0,0 +1,74 @@ +{% raw %} + + + Top 15 MA Lobbying Employers — 2024 + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/dash_lobbying_vs_enforcement.html b/docs/_includes/charts/dash_lobbying_vs_enforcement.html new file mode 100644 index 0000000..84adde9 --- /dev/null +++ b/docs/_includes/charts/dash_lobbying_vs_enforcement.html @@ -0,0 +1,73 @@ +{% raw %} + + + MA Lobbying Spend vs. Enforcement Actions + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_bill_intensity.html b/docs/_includes/charts/lobbying_bill_intensity.html new file mode 100644 index 0000000..eebf181 --- /dev/null +++ b/docs/_includes/charts/lobbying_bill_intensity.html @@ -0,0 +1,73 @@ +{% raw %} + + + Environmental Bills Lobbied per Year + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_bill_pass_by_spend_tier.html b/docs/_includes/charts/lobbying_bill_pass_by_spend_tier.html new file mode 100644 index 0000000..f18e9f2 --- /dev/null +++ b/docs/_includes/charts/lobbying_bill_pass_by_spend_tier.html @@ -0,0 +1,73 @@ +{% raw %} + + + Environmental Bill Pass Rate by Lobbying Intensity + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_bill_tsne.html b/docs/_includes/charts/lobbying_bill_tsne.html new file mode 100644 index 0000000..ff36a0a --- /dev/null +++ b/docs/_includes/charts/lobbying_bill_tsne.html @@ -0,0 +1,4 @@ +{% raw %} +
+
+{% endraw %} diff --git a/docs/_includes/charts/lobbying_bill_umap_env.html b/docs/_includes/charts/lobbying_bill_umap_env.html new file mode 100644 index 0000000..dd37fe2 --- /dev/null +++ b/docs/_includes/charts/lobbying_bill_umap_env.html @@ -0,0 +1,4 @@ +{% raw %} +
+
+{% endraw %} diff --git a/docs/_includes/charts/lobbying_bill_umap_summary.html b/docs/_includes/charts/lobbying_bill_umap_summary.html new file mode 100644 index 0000000..370390a --- /dev/null +++ b/docs/_includes/charts/lobbying_bill_umap_summary.html @@ -0,0 +1,4 @@ +{% raw %} +
+
+{% endraw %} diff --git a/docs/_includes/charts/lobbying_cso_operators.html b/docs/_includes/charts/lobbying_cso_operators.html new file mode 100644 index 0000000..99303d2 --- /dev/null +++ b/docs/_includes/charts/lobbying_cso_operators.html @@ -0,0 +1,73 @@ +{% raw %} + + + Total Lobbying Spend by Known CSO Operators + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_env_cluster_share.html b/docs/_includes/charts/lobbying_env_cluster_share.html new file mode 100644 index 0000000..2e12346 --- /dev/null +++ b/docs/_includes/charts/lobbying_env_cluster_share.html @@ -0,0 +1,73 @@ +{% raw %} + + + Environmental Lobbying Spend by Topic Cluster + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_env_opponents.html b/docs/_includes/charts/lobbying_env_opponents.html new file mode 100644 index 0000000..7522442 --- /dev/null +++ b/docs/_includes/charts/lobbying_env_opponents.html @@ -0,0 +1,74 @@ +{% raw %} + + + Top 20 Clients Opposing Environmental Bills (all years) + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_env_positions.html b/docs/_includes/charts/lobbying_env_positions.html new file mode 100644 index 0000000..137c6f1 --- /dev/null +++ b/docs/_includes/charts/lobbying_env_positions.html @@ -0,0 +1,73 @@ +{% raw %} + + + Unique Clients by Position on Environmental Bills + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_env_score_vs_clients.html b/docs/_includes/charts/lobbying_env_score_vs_clients.html new file mode 100644 index 0000000..e43cda4 --- /dev/null +++ b/docs/_includes/charts/lobbying_env_score_vs_clients.html @@ -0,0 +1,4 @@ +{% raw %} +
+
+{% endraw %} diff --git a/docs/_includes/charts/lobbying_pass_by_position.html b/docs/_includes/charts/lobbying_pass_by_position.html new file mode 100644 index 0000000..9252f95 --- /dev/null +++ b/docs/_includes/charts/lobbying_pass_by_position.html @@ -0,0 +1,73 @@ +{% raw %} + + + Environmental Bill Pass Rate by Lobbying Position + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_spend_by_cluster.html b/docs/_includes/charts/lobbying_spend_by_cluster.html new file mode 100644 index 0000000..238d339 --- /dev/null +++ b/docs/_includes/charts/lobbying_spend_by_cluster.html @@ -0,0 +1,73 @@ +{% raw %} + + + MA Lobbying Spend by Topic Cluster + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_spend_trend.html b/docs/_includes/charts/lobbying_spend_trend.html new file mode 100644 index 0000000..955de63 --- /dev/null +++ b/docs/_includes/charts/lobbying_spend_trend.html @@ -0,0 +1,73 @@ +{% raw %} + + + Annual MA Lobbying Spend on Environmental Bills + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_spend_vs_budget.html b/docs/_includes/charts/lobbying_spend_vs_budget.html new file mode 100644 index 0000000..de8f7ec --- /dev/null +++ b/docs/_includes/charts/lobbying_spend_vs_budget.html @@ -0,0 +1,73 @@ +{% raw %} + + + MA Lobbying Spend vs. DEP Budget (inflation-adjusted) + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_spend_vs_staff.html b/docs/_includes/charts/lobbying_spend_vs_staff.html new file mode 100644 index 0000000..c70fdfd --- /dev/null +++ b/docs/_includes/charts/lobbying_spend_vs_staff.html @@ -0,0 +1,73 @@ +{% raw %} + + + Environmental Lobbying Spend vs. DEP Staff Headcount + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_top_employers.html b/docs/_includes/charts/lobbying_top_employers.html new file mode 100644 index 0000000..074a9de --- /dev/null +++ b/docs/_includes/charts/lobbying_top_employers.html @@ -0,0 +1,74 @@ +{% raw %} + + + Top 15 MA Lobbying Clients — 2025 + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_top_env_employers.html b/docs/_includes/charts/lobbying_top_env_employers.html new file mode 100644 index 0000000..0322716 --- /dev/null +++ b/docs/_includes/charts/lobbying_top_env_employers.html @@ -0,0 +1,74 @@ +{% raw %} + + + Top 20 Clients by Cumulative Environmental Lobbying Spend + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_includes/charts/lobbying_vs_enforcement.html b/docs/_includes/charts/lobbying_vs_enforcement.html new file mode 100644 index 0000000..af64ad8 --- /dev/null +++ b/docs/_includes/charts/lobbying_vs_enforcement.html @@ -0,0 +1,73 @@ +{% raw %} + + + MA Lobbying Spend vs. Enforcement Actions + + + +
+ + + + +
+ + +{% endraw %} diff --git a/docs/_posts/2026-05-22-ma-environmental-lobbying.md b/docs/_posts/2026-05-22-ma-environmental-lobbying.md new file mode 100644 index 0000000..5db5b11 --- /dev/null +++ b/docs/_posts/2026-05-22-ma-environmental-lobbying.md @@ -0,0 +1,107 @@ +--- +layout: post +title: "(DRAFT) Who lobbies the Massachusetts Legislature on environmental policy?" +ancillary: 0 +--- + +*This post is in DRAFT status. Numbers and narrative claims will be filled in once the full historical lobbying dataset (2005–present) finishes scraping. Charts currently render against the partial dataset and will update automatically.* + +Every employer that hires a lobbyist in Massachusetts is required to file public disclosures with the Secretary of State: who they retained, how much they paid, and which bills they tried to influence. The Secretary of State publishes these filings on the [Lobbyist Public Search portal](https://www.sec.state.ma.us/LobbyistPublicSearch/) going back to 2005. Across {{ site.data.facts_lobbying.lobbying_most_recent_year }}, **{{ site.data.facts_lobbying.lobbying_n_employers }} employers** disclosed lobbying activity totalling roughly **${{ site.data.facts_lobbying.lobbying_total_spend_latest | divided_by: 1000000 }}M**. + +But what fraction of that money is spent on environmental policy? Which bills are environmentally relevant in the first place? And does lobbying intensity correlate with regulatory capacity — DEP staffing, enforcement actions, agency budget? + +This post is the first systematic analysis of the MA environmental lobbying landscape. The underlying dataset is documented on the [MA lobbying data page]({{ site.baseurl }}/data/MA_lobbying.html); the [analysis code](https://github.com/nesanders/MAenvironmentaldata/blob/master/analysis/MA_lobbying_viz.py) and [scraping pipeline](https://github.com/nesanders/MAenvironmentaldata/blob/master/get_data/README_lobbying.md) are on GitHub. + +--- + +## What counts as an "environmental" bill? + +Filer-reported subject tags (e.g. "Energy & Environment") are unreliable: a utility lobbying a wastewater bill may tag it "Utilities & Energy"; a developer opposing wetlands reform may tag it "Land Use." Rather than trust the tags, we **embed the full text of each bill** using Google's `gemini-embedding-2` model and score it by **differential cosine similarity** against two reference sets of 20 real MA bills each — one set known to be environmental (PFAS, stormwater, clean energy, etc.), one set known not to be (health, labor, education). A bill is flagged `is_environmental = True` when its similarity to env examples exceeds its similarity to non-env examples by at least 0.05. + +Methodology details and the cluster labelling pipeline are documented on the [data page]({{ site.baseurl }}/data/MA_lobbying.html#environmental-relevance-scoring). + +To visualise the result, every lobbied bill is projected into 2-D via t-SNE and coloured by its k-means topic cluster. Environmental bills are shown as large outlined dots; non-environmental bills are smaller and muted. Hover for the title. + +{% include charts/lobbying_bill_tsne.html %} + +*Caveat:* topic clusters reflect the dominant topic across all bills in the cluster. No cluster is purely environmental — environmentally-relevant bills are scattered across multiple topic clusters, including a heavy concentration in the "Health, Climate, and Community" cluster. + +--- + +## How much does environmental lobbying cost? + +The next chart shows total annual lobbying spend by employers who lobbied at least one environmentally-relevant bill, broken down by topic cluster. Stacked bar height represents allocated spend ($M); allocation distributes each employer's annual total proportionally across the bills they lobbied that year, so a single employer who lobbied 10 bills in different clusters contributes 1/10 of its total to each. + +{% include charts/lobbying_env_cluster_share.html %} + +*[Once full historical data lands: which cluster dominates? Are clean-energy bills lobbied at the same intensity as health-environment bills? How has the mix shifted from 2005 to 2024?]* + +--- + +## Who are the biggest environmental lobbying spenders? + +The cumulative spend below is calculated as `(employer total compensation that year) × (share of that employer's bills flagged environmental)`. This share-weighted measure means an employer that lobbied 100 bills, 5 of which were environmental, contributes 5% of their annual spend to the env total. + +{% include charts/lobbying_top_env_employers.html %} + +*[Once full historical data lands: are these primarily utilities, industry trade groups, environmental NGOs, or municipalities? What share of the top 20 are regulated entities vs. public-interest advocates?]* + +--- + +## Lobbying spend vs. DEP capacity over time + +The strongest cross-dataset comparison this dataset enables is between **industry lobbying spend on environmental bills** and the **regulatory capacity of the Department of Environmental Protection (DEP)** — both its budget and its staffing levels — over time. + +### Lobbying spend vs. DEP administrative budget + +{% include charts/lobbying_spend_vs_budget.html %} + +The DEP administrative budget is inflation-adjusted (2024 dollars) and sourced from the MA Comptroller's CTHRU system back to FY2005, with earlier years from MassBudget's historical archive. See the [DEP budget data page]({{ site.baseurl }}/data/EEA_budget.html) for methodology. + +### Lobbying spend vs. DEP staffing + +{% include charts/lobbying_spend_vs_staff.html %} + +DEP headcount is the annual count of unique employees with non-zero payroll in the MA Comptroller's payroll dataset (CTHRU-based). The y-axis is a raw FTE count; the lobbying spend axis is in millions of dollars. + +*[Once full historical data lands: do these series correlate? Does lobbying ramp up in years when DEP staffing is cut? Or is there a lag — does lobbying intensity precede or follow regulatory rollbacks?]* + +--- + +## Does lobbying intensity predict bill passage? + +For each environmental bill, we count the number of distinct employers who lobbied it. Bills with more lobbyers tend to be higher-stakes — but is heavily-lobbied legislation more or less likely to pass? + +{% include charts/lobbying_bill_pass_by_spend_tier.html %} + +*[Once full historical data lands: interpret the tier comparison. A higher pass rate among heavily-lobbied bills could indicate either successful industry influence or simply that important/well-supported bills attract more attention from all sides.]* + +--- + +## CSO operators and lobbying + +The Combined Sewer Overflow (CSO) dataset identifies {{ site.data.facts_EEA_CSO.n_operators }} permitted operators discharging untreated sewage into MA waterways. Some are municipalities (cities and towns); some are regional authorities (MWRA, Springfield Water and Sewer Commission). When CSO-related bills come before the Legislature, do these operators lobby — and how aggressively? + +In practice, **most municipal CSO operators do not lobby directly** — they lobby through the [Massachusetts Municipal Association (MMA)](https://www.mma.org/), which represents nearly all 351 MA cities and towns on Beacon Hill. The chart below shows direct lobbying spend by known CSO permittees alongside MMA as a proxy for the municipal sector. MMA totals reflect *all* of its lobbying activity (not only CSO-related bills), so it is best read as a ceiling on potential municipal-sector engagement on CSO policy rather than a measure of CSO-specific lobbying intensity. + +{% include charts/lobbying_cso_operators.html %} + +*[Once full historical data lands: which operators show consistent lobbying activity year over year? Do spend trends correlate with CSO enforcement events at the operator's facilities?]* + +--- + +## Caveats and limitations + +- **Spend allocation is approximate.** The SoS portal reports a single per-employer-per-period compensation figure, not per-bill spend. We allocate proportionally across the bills the employer disclosed lobbying. An employer who spends most of their effort on one priority bill but mentions ten others will have spend over-distributed to the secondary bills. +- **Environmental scoring is a similarity score, not a label.** A bill flagged `is_environmental = True` is more textually similar to known env bills than to known non-env bills — it is not a domain-expert classification. We expose `env_relevance_score` so downstream analysts can choose their own threshold; the current default (0.05 differential) is calibrated for high recall and accepts some false positives. +- **Legacy filings (pre-~2013) are coarser.** The pre-2013 portal format reports total compensation across all clients in one figure (no per-client breakdown) and sometimes omits bill titles. Bills with empty titles are present in the dataset but have zero embeddings (and thus `is_environmental = False`). +- **CSO operator matching is fuzzy.** Direct substring matching captures operators that lobby in their own name (e.g. MWRA, large independent water districts). MMA is included as an explicit proxy for the municipal sector, but its totals reflect *all* of its lobbying — not only CSO-related work. Operators that retain commercial lobbying firms cannot be attributed back to their underlying client from the SoS disclosure data alone. +- **The Legislature API does not serve pre-2009 bills.** For lobbying activity 2005–2008, bill titles come only from the SoS portal (often blank in the legacy format), so environmental scoring quality is reduced for those years. + +--- + +## Reproducibility + +All charts on this page are generated by [`analysis/MA_lobbying_viz.py`](https://github.com/nesanders/MAenvironmentaldata/blob/master/analysis/MA_lobbying_viz.py), which reads exclusively from the assembled SQLite database (`AMEND.db`). The scoring pipeline is in [`get_data/score_lobbying_bills.py`](https://github.com/nesanders/MAenvironmentaldata/blob/master/get_data/score_lobbying_bills.py); the clustering pipeline is in [`get_data/cluster_lobbying_bills.py`](https://github.com/nesanders/MAenvironmentaldata/blob/master/get_data/cluster_lobbying_bills.py). See [`get_data/README_lobbying.md`](https://github.com/nesanders/MAenvironmentaldata/blob/master/get_data/README_lobbying.md) for the full pipeline. + +The complete bill embeddings (768-dimensional vectors plus full text) are persisted to `gs://openamend-data/MA_bill_embeddings.parquet` (~100 MB; not committed to the repo). A lightweight scored CSV without embeddings is committed at [`docs/data/MA_lobbying_bills_scored.csv`]({{ site.baseurl }}/data/MA_lobbying_bills_scored.csv). diff --git a/docs/assets/db_semantic_context.txt b/docs/assets/db_semantic_context.txt index d0dd5d4..dc8e03a 100644 --- a/docs/assets/db_semantic_context.txt +++ b/docs/assets/db_semantic_context.txt @@ -1,5 +1,5 @@ # AMEND Database Semantic Context -Generated: 2026-04-29 +Generated: 2026-05-29 ## Global Data Notes @@ -81,6 +81,58 @@ Generated: 2026-04-29 FROM MS4_AnnualReports m JOIN MAEEADP_CSO c ON m.municipality_normalized = c.municipality WHERE m.extraction_confidence != 'low' AND c.eventType LIKE 'CSO%' GROUP BY 1 +- **Lobbying spend on environmental bills by year**: + SELECT l.year, SUM(e.total_expenditure) AS total_spend + FROM MA_Lobbying_Employers e + JOIN MA_Lobbying_Bills l ON e.employer_name = l.employer_name AND e.year = l.year + JOIN MA_Lobbying_Bills_Scored s ON l.bill_number = s.bill_number AND l.general_court = s.general_court + WHERE s.is_environmental = 1 + GROUP BY l.year ORDER BY l.year + NOTE: total_expenditure is in MA_Lobbying_Employers; MA_Lobbying_Bills has NO spending column. + NOTE: is_environmental is in MA_Lobbying_Bills_Scored, NOT in MA_Legislature_Bills. + +- **Top employers on environmental bills** (most recent year): + SELECT e.employer_name, e.total_expenditure + FROM MA_Lobbying_Employers e + WHERE e.year = (SELECT MAX(year) FROM MA_Lobbying_Employers) + AND EXISTS ( + SELECT 1 FROM MA_Lobbying_Bills l + JOIN MA_Lobbying_Bills_Scored s ON l.bill_number = s.bill_number AND l.general_court = s.general_court + WHERE l.employer_name = e.employer_name AND l.year = e.year AND s.is_environmental = 1 + ) + ORDER BY e.total_expenditure DESC LIMIT 15 + +- **Lobbying activity by topic cluster** (how many employers per cluster): + SELECT c.label, COUNT(DISTINCT l.employer_name) AS n_employers, COUNT(DISTINCT l.bill_number) AS n_bills + FROM MA_Lobbying_Bills l + JOIN MA_Lobbying_Bills_Scored s ON l.bill_number = s.bill_number AND l.general_court = s.general_court + JOIN MA_Bill_Cluster_Labels c ON s.cluster_id = c.cluster_id + GROUP BY c.label ORDER BY n_employers DESC + +- **Lobbying spend vs. enforcement count by year** (dual-axis): + WITH spend AS ( + SELECT year, SUM(total_expenditure) AS lobbying_spend FROM MA_Lobbying_Employers GROUP BY year + ), enforcement AS ( + SELECT strftime('%Y', EnforcementDate) AS year, COUNT(*) AS n_actions FROM MAEEADP_Enforcement GROUP BY 1 + ) + SELECT s.year, s.lobbying_spend, e.n_actions FROM spend s LEFT JOIN enforcement e ON s.year = e.year ORDER BY s.year + +- **Bill passage rate by lobbying spend tier**: + SELECT + CASE WHEN employer_count >= 10 THEN 'Heavily lobbied (10+ employers)' + WHEN employer_count >= 3 THEN 'Moderately lobbied (3–9 employers)' + ELSE 'Lightly lobbied (1–2 employers)' END AS lobby_tier, + AVG(CAST(b.passed AS FLOAT)) AS pass_rate, + COUNT(*) AS n_bills + FROM ( + SELECT l.bill_number, l.general_court, COUNT(DISTINCT l.employer_name) AS employer_count + FROM MA_Lobbying_Bills l GROUP BY l.bill_number, l.general_court + ) counts + JOIN MA_Lobbying_Bills_Scored s ON counts.bill_number = s.bill_number AND counts.general_court = s.general_court + JOIN MA_Legislature_Bills b ON counts.bill_number = b.bill_number AND counts.general_court = b.general_court + WHERE s.is_environmental = 1 + GROUP BY lobby_tier + - **CSO discharges to 303(d) impaired waters** (two-step join): MAEEADP_CSO JOIN CSO_303d_Mapping ON MAEEADP_CSO.waterBody = CSO_303d_Mapping.csoWaterBody JOIN EPA_303d_Impairments ON CSO_303d_Mapping.waterbody303d = EPA_303d_Impairments.waterbody @@ -1116,6 +1168,399 @@ EEA00PES00000000L8 KENNETH FIALKOSKY AL-0028553 AL PESTICIDE CREDENTIAL PESTIC --- +### MA_Bill_Cluster_Labels (26 rows) +Topic cluster labels for MA lobbying bill clusters derived from k-means on Gemini embeddings. One row per cluster_id (0–14). Key fields: cluster_id (join key to MA_Lobbying_Bills_Scored.cluster_id), label (3–5 word topic description, e.g. "Clean Energy Policy"), n_bills (total bills in cluster), n_env_bills (environmental bills in cluster). IMPORTANT: this table has NO bill_number or general_court column — it is a lookup table only. To get bills by topic, join MA_Lobbying_Bills_Scored on cluster_id, then join MA_Lobbying_Bills. + +```sql +CREATE TABLE "MA_Bill_Cluster_Labels" ( + "index" BIGINT, + cluster_id TEXT, + label TEXT, + n_bills FLOAT, + n_env_bills FLOAT, + example_titles TEXT +) +``` + +/* 5 rows from MA_Bill_Cluster_Labels: +cluster_id label n_bills n_env_bills example_titles +0 Municipal Environmental Governance 1983.0 50.0 An Act promoting healthy communities | An Act to allow simple majority voting | An Act to allow simple majority voting | An Act to reduce duplicative permitting in septic system upgrades | An Act empowering towns and cities to protect residents and the environment from harmful pesticides +1 Patient Safety, Staff, and Transparency 1289.0 1.0 An Act ensuring access to qualified health professionals | An Act relative to hospital medical staffs | An Act regulating surgical assistants | An Act to protect the independence of clinical decision making | An Act To Provide For The Autonomy Of Hospital Medical Staffs +2 Waste Reduction and Recycling 463.0 181.0 An Act relative to enhancing circularity in recycling | Rep. Kaufman, Jay (D) - An Act to reduce solid waste in the Commonwealth | An Act to reduce packaging waste | An Act to reduce plastics | An Act to reduce plastic packaging waste +3 Comprehensive Student Education and Wellness 1572.0 7.0 An Act implementing the Every Student Succeeds Act | An Act empowering students and schools to thrive | An Act regarding breakfast after the bell | An Act regarding breakfast after the bell | An Act to remodel public school athletics through social-emotional learning +4 Infrastructure Investment for Growth 1124.0 8.0 An Act relative to job creation, workforce development and infrastructure investment | An act financing the general governmental infrastructure of the Commonwealth | An Act financing the general governmental infrastructure of the Commonwealth | An Act financing improvements to the Commonwealth's transportation system | An Act relative to open spaces and tax incentives +*/ + +**Column notes:** +- `cluster_id`: Integer cluster ID (0–14). Join key to MA_Lobbying_Bills_Scored.cluster_id. +- `label`: 3–5 word topic label generated by Gemini 2.5 Flash (e.g. "Clean Energy Policy"). +- `n_bills`: Total number of bills assigned to this cluster. +- `n_env_bills`: Number of bills in this cluster flagged is_environmental=1. +- `example_titles`: Resolve providing for an investigation and study by a special commission relative to robotic surgery in the Commonwealth | Resolve establishing a commission to consider the inclusion of nuclear energy in Massachusetts' future energy planning | An Act relative to establishing a food justice frontline | Resolve providing for a commission to study the affects of offshore wind projects on Massachusetts fisheries | Resolve Providing for a "Special Commission on Peace Officer Standards and Training" to Study and Make Recommendations Concerning the Implementation of a Peace Officers Standards and Training (POST) System, An act relative to the protection fo children | An Act to protect the commonwealth from dangerous persons | An Act to protect victims of crimes and the public | An Act clarifying consent laws for adolescents | An Act relative to consensual adolescent sexual activity, An Act transitioning Massachusetts to clean electricity, heating and transportation | An Act transitioning Massachusetts to clean electricity, heating and transportation | An Act transitioning the Commonwealth to 100% clean electricity, heating, and transportation | An Act transitioning the Commonwealth to 100% clean electricity, heating, and transportation | An Act investing in a prosperous, clean commonwealth by 2030, An Act to prevent wage theft, promote employer accountability, and enhance public enforcement | An Act to prevent wage theft, promote employer accountability, and enhance public enforcement | An Act to prevent wage theft, promote employer accountability, and enhance public enforcement | An Act to prevent wage theft, promote employer accountability, and enhance public enforcement | An Act relative to independent contractors, An Act to improve access to behavioral health services | An Act to improve access to behavioral health services | An Act improving access to behavioral health services | An Act providing equitable access to behavioral health services for MassHealth consumers | An Act expanding access to mental health services, An Act to empower communities to transition to renewable energy | An Act relative to municipal lighting authorities | An Act promoting local energy investment and infrastructure modernization | An Act promoting local energy investment and infrastructure modernization | An Act promoting local energy investment and infrastructure modernization, An Act relative to the safe operation of motor vehicles | An Act relative to the safe operation of motor vehicles | An Act prohibiting video recording or broadcasting while driving | An Act relative to the use of video screens while operating a motor vehicle | An Act relative to the safe operation of motor vehicles, An Act relative to the lawful sale of handguns | An Act relative to the lawful sale of handguns | An Act relative to the lawful sale of handguns | An Act relative to waterfowl hunting | An Act relative to unloaded rifles or shotguns, An Act relative to pharmaceutical price transparency and cost sharing | An Act relative to pharmaceutical price transparency and cost sharing. | An Act relative to consumer protection for prescription drug purchases | An Act creating patient choice and education in the dispensing of opiate drugs | An act creating patient choice and education in the dispensing of opiate drugs, An Act relative to motor vehicle surcharges | An Act Relative to Motor Vehicle Surcharges | An Act relative to home owners insurance and funding repairs | An Act relative to automobile insurance companies | An Act Relative to Automobile Insurance Companies, An Act relative to job creation, workforce development and infrastructure investment | An act financing the general governmental infrastructure of the Commonwealth | An Act financing the general governmental infrastructure of the Commonwealth | An Act financing improvements to the Commonwealth's transportation system | An Act relative to open spaces and tax incentives, An Act relative to ensuring treatment for genetic craniofacial conditions | An Act relative to ensuring treatment for genetic craniofacial conditions | An Act relative to ensuring treatment for genetic craniofacial conditions | An Act relative to ensuring treatment for genetic craniofacial conditions | An Act relative to ensuring treatment for genetic craniofacial conditions, An Act relative to enhancing circularity in recycling | Rep. Kaufman, Jay (D) - An Act to reduce solid waste in the Commonwealth | An Act to reduce packaging waste | An Act to reduce plastics | An Act to reduce plastic packaging waste, An Act relative to assisted living and the frail elder waiver | An Act relative to assist families care for elders | An Act relative to intensive case management for clinically complex older adults | An Act relative to stabilizing the Commonwealth?s nursing facilities | An Act promoting affordability of home care services, An Act relative to Mass Port police officers | An Act relative to Mass Port police officers | An Act relative to Mass Port police officers | An Act relative to Mass Port police officers | An Act relative to the retirement of MassPort officers, An Act regarding right of first refusal in the event of foreclosure and short sales and deeds in lieu | An Act for a right of first refusal for foreclosed property (Foreclosure TOPA) | An Act supporting affordable housing with a local option for a fee to be applied to certain real estate transactions | An Act to guarantee a tenant’s first right of refusal | An Act authorizing municipalities to protect low and moderate income tenants and units of governmentally involved housing, An Act promoting healthy communities | An Act to allow simple majority voting | An Act to allow simple majority voting | An Act to reduce duplicative permitting in septic system upgrades | An Act empowering towns and cities to protect residents and the environment from harmful pesticides, An Act making appropriations for fiscal year 2025 to provide for supplementing certain existing appropriations and for certain other activities and projects | An Act making appropriations for fiscal year 2025 to provide for supplementing certain existing appropriations and for certain other activities and projects | An Act making appropriations for the fiscal year 2025 to provide for supplementing certain existing appropriations and for certain other activities and projects | An Act making appropriations for the fiscal year 2025 to provide for supplementing certain existing appropriations and for certain other activities and projects | An Act making appropriations for the fiscal year 2025 to provide for supplementing certain existing appropriations and for certain other activities and projects, An Act implementing the Every Student Succeeds Act | An Act empowering students and schools to thrive | An Act regarding breakfast after the bell | An Act regarding breakfast after the bell | An Act to remodel public school athletics through social-emotional learning, An Act ensuring access to qualified health professionals | An Act relative to hospital medical staffs | An Act regulating surgical assistants | An Act to protect the independence of clinical decision making | An Act To Provide For The Autonomy Of Hospital Medical Staffs, An Act eliminating telecommunications tax exemptions | AN ACT RELATIVE TO CERTAIN TAX EXEMPTIONS ., An Act clarifying the meal break law to allow for private enforcement +HB757 An Act for a competitive economy through safer alternatives to toxic chemicals +HB4952 An Act increasing the experience and background necessary to operate low pressure processed steam boilers +HB4865 An Act relative to a competitive economy through safer alternatives to toxic chemicals +SB684 An Act relative to contract compliance with homeland security +SB474 An Act relative to establishing family and employment security through temporary disability insurance and family temporary disability insurance +HB3124 An Act relative to expanding energy efficiency in the commonwealth +HB4825 An Act relative to expanding energy efficiency in the Commonwealth +SB1524 An Act relative to expanding energy efficiency in the Commonwealth +HB4558 An Act relative to increasing the experience and background necessary to operate low pressure processed steam boilers +HB3809 An Act relative to insurance coverage for autism +HB4935 An Act relative to insurance coverage for autism +HB3912 An Act relative to patient safety +HB4668 An Act relative to post-partum depression +HB4859 An Act relative to post-partum depression +HB4909 An Act relative to providing incentives for productive workers compensation audits +HB1853 An Act relative to the competitive determination of workers' compensation insurance rates +HB1864 An Act relative to the competitive determination of workers' compensation insurance rates +HB3881 An Act relative to toxic fire retardant prevention +SB389 An Act to create environmental justice +SB730 An Act to establish a self sufficiency standard in the Commonwealth +SB2517 An Act to protect motor vehicle owners' and independent repairers' right to repair +HB1846 An Act to provide incentives for productive workers compensation audits | HB1188 +HB1202 +HB1218 +HB1219 +HB1224 +HB1227 +HB1450 +HB1455 +HB1566 +HB1568HB1665 +HB1695 +HB1697 +HB1700 +HB1701 +HB1736 +HB1778HB1779 +HB1981 +HB1982 +HB2003 +HB2004 +HB2140 +HB2668 +HB2692 +HB2702 +HB2726 +HB2765 +HB2778 +HB2780 +HB2822 +HB2881 +HB2933 +HB2938 +HB2960HB3014 +HB3037 +HB3061 +HB3061 +HB3065 +HB3065 +HB3083HB3095 +HB310 +HB3124 +HB3308 +HB3362 +HB3364 +HB337 +HB3411 +HB3412 +HB3446 +HB3462HB3491 +HB3570 +HB3572 +HB3575 +HB3583 +HB3615 +HB3646 +HB3647 +HB3670 +HB3672 +HB3673 +HB3674 +HB3675 +HB3699 +HB3760 +HB3762 +HB3951 +HB4072HB702 +HB750 +HB756 +HB775HB782 +HB806 +HB808 +HB811 +HB818 +HB834 +HB849 +HB852 +HB853 +HB88 +HB960 +HD2622 +SB103 +SB104 +SB105 +SB1323 +SB1387 +SB1504 +SB1508 +SB1577SB1607 +SB168 +SB1912 +SB193SB1953 +SB1963 +SB2019 +SB205 +SB206 +SB2061 +SB2101 +SB373 +SB382 +SB386 +SB387 +SB388 +SB389 +SB394 +SB396 +SB411 +SB412 +SB436 +SB441 +SB6 +SB623 +SB625 +SB636 +SB653 +SB663 +SB765 +SB767 +SB770 +SB783 +SB817 +SB88 +SB882 +SB90SB96 +SB97 +SD1336 +Greenhouse gas policy +Energy conservation and decoupling +Building Code +Stormwater +Ch. 21E +Ch. 91 +MEPA +Highway Access Permit +MESA +Transportation reform | An Act relative to financial services contracts for dental benefits corporations; +SB2363 An Act relative to oral cancer therapy; +SB2359: An Act to establish standards for long term care insurance; +SB2134: An Act to establish standards for long term care insurance (SB411: An Act to establish standards for long term care insurance); +HB4464: An Order relative to authorizing the committee on Financial Services to make an investigation and study of certain Senate and House documents concerning health care insurance and insurance coverage issues; +HB295: An Act relative to increasing consumer access to licensed marriage and family therapy; +HB304: An Act to define the use of observation services; +HB3928: An Act relative to the treatment of cleft palate and cleft lip; +HB4070: An Act relative to Health Care Quality Improvement and Cost Reduction Act of 2012; +SB2362: An Act establishing the Massachusetts Childhood Vaccine Program; +SB2120: An Act establishing the Massachusetts Childhood Vaccine Program; +HB585: An Act to protect patients from breakthrough seizures; +HB 4420: An Act relative to miscellaneous insurance and banking issues; +HB4257: An Act to limit retroactive denials of health insurance (HB1204 An Act to limit retroactive denials of health insurance claims); +HB2137: An Act providing for the equitable apportionment of certain liens; +HB1216: An Act relative to manufacturers' coupons and rebates; +SB69: An Act providing for insurance coverage for vision screening for children; +SB72: An Act relative to increasing consumer access to licensed marriage and family therapists; +SB285: An Act establishing a special commission to promote prescription pharmaceutical affordability in the Commonwealth; +SB519: An Act relative to health care affordability; +SB527: An Act defining the use of observational services; +SB2121: An Act to provide for tobacco cessation benefits; +SB821: An Act establishing accountability for managed care organizations; +SB2440: Order relative to authorizing the joint committee on the Judiciary to make an investigation and study of certain current Senate documents relative to the Judiciary (SB825 An Act revising the charitable immunity cap); +SB1070: An Act relative to oral cancer therapy; +SB498 - An Act relative to student health insurance; +HB3916: An Act relative to financial services contracts for dental benefits corporations; +HB3917: An Act relative to payment for use of ambulance services; +HB3928: An Act relative to the treatment of cleft palate and cleft lip; +HB3981: Text of a further amendment to the Senate amendment of the engrossed Bill relative to tiered and selective network health plans; +HB4200 (Sections 129, 130, 131): Committee of Conference on the disagreeing votes of the two branches of the House Bill making appropriations for the fiscal year 2013; +SB2270: Printed as amended version of the engrossed Bill relative to improving the quality of health care and reducing costs through increased transparency, efficiency and innovation. SB2400: An act improving the quality of health care and reducing costs through increased transparency, efficiency and innovation. | - An Act Establishing A Board Of Casting Agents +HB1270- An Act Relative To The Protection Of Talented Children And Their Families +HB2511-An Act Relative To Increasing Economic Development And Business Opportunity In The Design And Manufacture Of Video Games Through The Use Of Tax Credits +HB2526-An Act Relative To Film Tax Credit Percentage +HB2629-An Act To Promote Sales Tax Fairness For Main Street Retailers +HB2630-An Act Relative To The Tax Expenditure Budget +HB2662-An Act Relative To Tax Credits, Education And Training For The Design And Development Of Video Game And Interactive Digital Media Products +HB2738-An Act Relative To Targeted Investment Of Taxpayer Dollars +HB3382-An Act Relative To Transportation Finance +HB3526- An Act To Ensure Compliance With Federal Marketplace Fairness +HB3535-An Act Relative To Transportation Finance +SB1441-An Act Relative To Eligible Production Expenses For The Film Tax Credit +SB1630-An Act Protecting The Commercial Value Of Artists, Entertainers, And Other Notable Personalities +HB 1 - An Act making appropriations for FY2014 +HB3382 - An Act relative to transportation finance +HB3415 - An Act relative to transportation finance +SB1766 - An Act relative to transportation finance +SB1770 - An Act relative to transportation finance +HB3400 - An Act making appropriations for the fiscal year 2014 for the maintenance of the departments, boards, commissions, institutions and certain activities of the Commonwealth, for interest, sinking fund and serial bond requirements and for certain permanent improvements +HB3401 - Printed as amended version of the Bill making appropriations for the fiscal year 2014 for the maintenance of departments, boards, commissions, institutions and certain activities of the Commonwealth, for interest, sinking fund and serial bond requirements and for certain permanent improvements +H3538 - An Act making appropriations for the Fiscal Year 2014 for the maintenance of the departments, boards, commissions, institutions and certain activities of the commonwealth, for interest, sinking fund and serial bond requirements and for certain permanent improvements +SB3 - Text of the Senate amendments to the House Bill making appropriations for the fiscal year 2014 for the maintenance of the departments, boards, commissions, institutions and certain activities of the Commonwealth, for interest, sinking fund and serial bond requirements and for certain permanent improvements +SB1800 - Text of the Senate amendments to the House Bill making appropriations for the fiscal year 2014 for the maintenance of the departments, boards, commissions, institutions and certain activities of the Commonwealth, for interest, sinking fund and serial bond requirements and for certain permanent improvements +Government Relations Group is paid a monthly retainer which encompasses all activity and is not paid on an hourly or per issue basis. | An Act Increasing The School Choice Cap +HB225 An Act Protecting The Commonwealth's Lakes And Ponds +HB1596 An Act Relative To Creditable Service For Teachers In Charter Schools +HB1925 An Act Establishing A Charter School Working Group +HB1926 An Act Establishing A Charter School Commission +HB1935 An Act Relative To Charter School Renewal +HB1951 An Act To Improve Access To Charter Schools For Students With Disabilities And Limited English Proficiency +HB2721 An Act Relative To Unused District School Buildings +SB231 An Act Establishing A Charter School Working Group. +SB246 An Act Establishing A Charter School Working Group., An Act authorizing the town of Tewksbury to grant 8 additional licenses for the sale of alcoholic beverages to be drunk on the premises | An Act authorizing the city of Fitchburg to grant 1 additional license for the sale of all alcoholic beverages not to be drunk on the premises | An Act authorizing the town of Walpole to grant an additional license for the sale of wines and malt beverages not to be drunk on the premises | An Act authorizing the town of Dedham to grant four additional licenses for the sale of all alcoholic beverages, four wine and malt licenses, all to be drunk on the premises, and one wine and malt package store license | An Act relative to special one-day licenses to sell or dispense alcoholic beverages, An Act Relative to State Personnel | An Act Relative to State Personnel | An Act relative to modernize municipal finance and government | An Act relative to Protecting Senior Health | Senate Ways and Means - An Act establishing a sick leave bank for Christine Chrzanowski, an employee of the Department of Mental Health, An Act Establishing the Massachusetts Data Privacy Act | Establishing the Massachusetts Consumer Data Privacy Act | An Act relative to consumer data privacy | An Act advancing the economic development of the commonwealth through comprehensive data privacy | An Act establishing the Massachusetts Information Privacy Act + +--- + +### MA_Legislature_Bills (31,511 rows) +MA Legislature OpenAPI: one row per (bill_number, general_court session). Key fields: bill_number (e.g. H.1234), general_court (session number), title (bill title), sponsor_name, status (final status text), passed (1 if signed/enacted, 0 otherwise). Foreign key: bill_number + general_court → MA_Lobbying_Bills. NOTE: environmental relevance scoring lives in MA_Lobbying_Bills_Scored, not here. + +```sql +CREATE TABLE "MA_Legislature_Bills" ( + "index" BIGINT, + bill_id TEXT, + bill_number BIGINT, + bill_prefix TEXT, + general_court BIGINT, + title TEXT, + sponsor_name TEXT, + status TEXT, + passed BIGINT +) +``` + +/* 5 rows from MA_Legislature_Bills: +bill_id bill_number bill_prefix general_court title sponsor_name status passed +H4999 4999 H 193 An Act promoting access to midwifery care and out-of-hospital birth options Midwifery Care and Out-of-Hospital options Signed by the Governor, Chapter 186 of the Acts of 2024 1 +H4773 4773 H 193 An Act promoting access to midwifery care and out-of-hospital birth options House Committee on Ways and Means Published as amended, see H4785 0 +S1405 1405 S 193 An Act relative to newborn screenings for congenital cytomegalovirus Joan B. Lovely No further action taken 0 +S607 607 S 193 An Act to increase access to nurse-midwifery services Brendan P. Crighton No further action taken 0 +H1069 1069 H 193 An Act to increase access to nurse-midwifery services Kay Khan Senate concurred 0 +*/ + +**Column notes:** +- `bill_number`: MA bill number (e.g. H.1234). Join key to MA_Lobbying_Bills and MA_Lobbying_Bills_Scored. +- `general_court`: Session number. 185=2009-10, 186=2011-12, ..., 192=2021-22, 193=2023-24, 194=2025-26. +- `passed`: 1 if the bill was signed into law or enacted; 0 if it died or is still pending. +- `bill_prefix`: ALL CAPS. H, S, HD, SD + +--- + +### MA_Lobbying_Bills (228,046 rows) +MA SoS lobbying disclosures: fact table linking employers to bills they lobbied. One row per (bill_number, general_court, employer_name, year). Key fields: bill_number (e.g. H.1234, S.5678), general_court (MA legislative session number, e.g. 193 for the 2023–2024 session), employer_name, year. Join to MA_Legislature_Bills on (bill_number, general_court) for bill title, sponsor, passed status, and env_relevance_score. Join to MA_Lobbying_Employers on (employer_name, year) for employer total expenditure. IMPORTANT: this table has NO spending amount column — spending lives in MA_Lobbying_Employers. + +```sql +CREATE TABLE "MA_Lobbying_Bills" ( + "Unnamed: 0" BIGINT, + entity_name TEXT, + client_name TEXT, + year BIGINT, + general_court BIGINT, + chamber TEXT, + bill_number BIGINT, + bill_title TEXT, + position TEXT, + amount FLOAT, + entity_name_norm TEXT, + client_name_norm TEXT +) +``` + +/* 5 rows from MA_Lobbying_Bills: +Unnamed: 0 entity_name client_name year general_court chamber bill_number bill_title position amount entity_name_norm client_name_norm +0 Aaron Agulnek BAY STATE BIRTH COALITION, INC. 2024 193 House Bill 4999 An Act promoting access to midwifery care and out-of-hospital birth options Neutral 0.0 AARON AGULNEK BAY STATE BIRTH COALITION +1 Aaron Agulnek BAY STATE BIRTH COALITION, INC. 2024 193 House Bill 4773 An Act promoting access to midwifery care and out-of-hospital birth options Support 0.0 AARON AGULNEK BAY STATE BIRTH COALITION +2 Aaron Agulnek Vanessa Colleran 2024 193 Senate Bill 1405 An Act relative to newborn screenings for congenital cytomegalovirus Support 0.0 AARON AGULNEK VANESSA COLLERAN +3 Aaron Agulnek Vanessa Colleran 2024 193 House Bill 4999 An Act promoting access to midwifery care and out-of-hospital birth options Neutral 0.0 AARON AGULNEK VANESSA COLLERAN +4 Aaron Agulnek Massachusetts Affiliate of ACNM, Inc 2024 193 House Bill 4999 An Act promoting access to midwifery care and out-of-hospital birth options Neutral 0.0 AARON AGULNEK MASSACHUSETTS AFFILIATE OF ACNM +*/ + +**Column notes:** +- `bill_number`: MA bill number (e.g. H.1234, S.5678, HD.1234). Joins to MA_Legislature_Bills.bill_number. +- `general_court`: MA legislative session number (e.g. 193 = 2023–2024). Joins to MA_Legislature_Bills.general_court. +- `employer_name`: Lobbying employer. Joins to MA_Lobbying_Employers.employer_name. +- `year`: Filing year. IMPORTANT: This table has NO spending amount — total expenditure is in MA_Lobbying_Employers. +- `chamber`: House Bill, Senate Bill, Executive, HB, SB, House Docket, Senate Docket, FY, ITT, C, SH, HR, CMR, B +- `position`: Neutral, Support, Oppose + +--- + +### MA_Lobbying_Bills_Scored (26,245 rows) +Environmental relevance scores for MA lobbying bills, derived from Gemini embeddings. One row per unique (bill_number, general_court). Key fields: bill_number, general_court, bill_title, bill_id (e.g. H1234), env_relevance_score (float; differential cosine similarity to env vs non-env example bills; higher = more environmentally relevant), is_environmental (1 if env_relevance_score >= 0.05), cluster_id (integer topic cluster 0–14; -1 = unassigned). Join to MA_Lobbying_Bills on (bill_number, general_court) to find which employers lobbied env bills. Join to MA_Bill_Cluster_Labels on cluster_id for human-readable cluster topic labels. IMPORTANT: is_environmental is per-bill; no cluster is purely environmental. + +```sql +CREATE TABLE "MA_Lobbying_Bills_Scored" ( + "index" TEXT, + bill_number BIGINT, + general_court BIGINT, + bill_title TEXT, + bill_id TEXT, + env_relevance_score FLOAT, + is_environmental BIGINT, + cluster_id BIGINT +) +``` + +/* 5 rows from MA_Lobbying_Bills_Scored: +bill_number general_court bill_title bill_id env_relevance_score is_environmental cluster_id +537 186 An Act To Assure Equitable Heath Coverage For All Children S537 -0.0040966272354125 0 18 +549 186 An Act Relative to Health Care Affordability S549 -0.0452432036399841 0 18 +873 186 An Act Strengthening Health Reform S873 -0.0194134712219238 0 1 +974 186 An Act to regulate health insurance contracts/policies. NULL NULL NULL NULL +NULL NULL 0 21 NULL NULL NULL +*/ + +**Column notes:** +- `bill_number`: MA bill number. Join key to MA_Lobbying_Bills and MA_Legislature_Bills. +- `general_court`: MA legislative session number. Join key to MA_Lobbying_Bills. +- `bill_id`: Chamber-prefixed bill ID (e.g. H1234, S567). Used for Legislature API lookups. +- `env_relevance_score`: Differential cosine similarity: max_sim(env_examples) - max_sim(non_env_examples). Positive = more env-like than non-env. Threshold for is_environmental is 0.05. +- `is_environmental`: 1 if env_relevance_score >= 0.05. Use this to filter to environmentally relevant bills. +- `cluster_id`: Topic cluster integer (0–14). -1 means unassigned. Join to MA_Bill_Cluster_Labels for label. + +--- + +### MA_Lobbying_Employers (12,244 rows) +MA Secretary of State lobbying disclosures: one row per (employer, year). An "employer" is an organization that retains lobbyists to lobby the MA Legislature. Data covers 2007–present; updated annually as the SoS portal publishes prior-year filings. Key fields: employer_name, year, total_expenditure (dollars), subject_tags (filer-reported; unreliable for environmental filtering — use MA_Legislature_Bills.is_environmental instead). Join to MA_Lobbying_Bills on (employer_name, year) to see which bills an employer lobbied. Join to MA_Lobbying_Lobbyists on (employer_name, year) to see which lobbyists they retained. + +```sql +CREATE TABLE "MA_Lobbying_Employers" ( + "index" BIGINT, + entity_name TEXT, + client_name TEXT, + year BIGINT, + reg_type TEXT, + compensation FLOAT, + entity_name_norm TEXT, + client_name_norm TEXT +) +``` + +/* 5 rows from MA_Lobbying_Employers: +entity_name client_name year reg_type compensation entity_name_norm client_name_norm +21c, LLC EDUCATION TRUST, INC. (THE) 2024 Lobbyist Entity 7500.0 21C EDUCATION TRUST (THE) +21c, LLC Amplify Education, Inc. 2024 Lobbyist Entity 22500.0 21C AMPLIFY EDUCATION +21c, LLC The Massachusetts Alliance for Early College, Inc. 2024 Lobbyist Entity 15000.0 21C MASSACHUSETTS ALLIANCE FOR EARLY COLLEGE +21c, LLC Benjamin Franklin Cummings Institute of Technology 2024 Lobbyist Entity 12500.0 21C BENJAMIN FRANKLIN CUMMINGS INSTITUTE OF TECHNOLOGY +21c, LLC Total salaries received 2024 Lobbyist Entity 57500.0 21C TOTAL SALARIES RECEIVED +*/ + +**Column notes:** +- `employer_name`: Organization name as filed with the SoS. Spelling varies across years; use LIKE for fuzzy matching. +- `year`: Calendar year of the filing (e.g. 2022). Filings for year Y are typically published mid-year Y+1. +- `total_expenditure`: Annual lobbying expenditure in dollars (NULL if not reported). +- `subject_tags`: Filer-assigned subject area tags (e.g. "Energy & Environment"). UNRELIABLE for environmental filtering — use MA_Legislature_Bills.is_environmental instead. +- `reg_type`: Lobbyist, Lobbyist Entity + +--- + ### MA_precipitation_daily (9,862 rows) Daily precipitation averages across Massachusetts weather stations (NOAA ACIS). Key fields: date (YYYY-MM-DD), precip_in_avg (inches), n_stations. diff --git a/docs/dashboard.md b/docs/dashboard.md index aff491e..c9456fd 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -20,6 +20,7 @@ For full analysis and narrative context, follow the links in each section. [CSO Discharge Trends](#cso) · [303(d) Impaired Waters](#303d) · [MS4 Stormwater Compliance](#ms4) · +[Lobbying Spend](#lobbying) · [CSO Data Quality Indicator](#data-quality) · [Discharge by Watershed](#watershed) @@ -313,4 +314,38 @@ discharge over the full reporting period to date. --- + + +## Lobbying Spend on Environmental Bills + +Data: [MA Secretary of State lobbying disclosures]({{ site.url }}{{ site.baseurl }}/data/MA_lobbying.html), 2005–present. + +Lobbying disclosures filed semi-annually with the MA Secretary of State identify which organizations hired lobbyists, how much clients paid, and which specific bills were lobbied. Bills are scored for environmental relevance using Google Gemini embeddings; charts show only employers that lobbied at least one environmentally relevant bill. Compensation figures reflect total payments from each client to each lobbying entity per year. + +Data last updated **{{ site.data.ts_update_MA_lobbying.updated | date: "%-d %B %Y" }}**. Refreshed automatically on a weekly basis; exits early when no new semi-annual filings are posted (filings are submitted twice yearly, so most weeks see no change). + +### Annual lobbying spend on environmental bills + +{% include charts/dash_lobbying_spend_trend.html %} + +### Top employers — most recent complete year + +{% include charts/dash_lobbying_top_employers.html %} + +### Environmental bills lobbied per year + +{% include charts/dash_lobbying_bill_intensity.html %} + +### Lobbying spend vs. enforcement actions + +{% include charts/dash_lobbying_vs_enforcement.html %} + +### Lobbying spend by topic cluster + +*Lobbying spend allocated across bill topic clusters (k-means on Gemini embeddings, labeled by Gemini Flash). Cluster labels are assigned once and updated manually when new years of data are added.* + +{% include charts/dash_lobbying_spend_by_cluster.html %} + +--- + *Charts regenerated weekly from the latest available data. Last update visible in the [Actions log](https://github.com/nesanders/MAenvironmentaldata/actions/workflows/update-charts.yml).* diff --git a/docs/data/MA_bill_cluster_labels.csv b/docs/data/MA_bill_cluster_labels.csv new file mode 100644 index 0000000..251af1c --- /dev/null +++ b/docs/data/MA_bill_cluster_labels.csv @@ -0,0 +1,245 @@ +cluster_id,label,n_bills,n_env_bills,example_titles +0,Municipal Environmental Governance,1983,50,An Act promoting healthy communities | An Act to allow simple majority voting | An Act to allow simple majority voting | An Act to reduce duplicative permitting in septic system upgrades | An Act empowering towns and cities to protect residents and the environment from harmful pesticides +1,"Patient Safety, Staff, and Transparency",1289,1,An Act ensuring access to qualified health professionals | An Act relative to hospital medical staffs | An Act regulating surgical assistants | An Act to protect the independence of clinical decision making | An Act To Provide For The Autonomy Of Hospital Medical Staffs +2,Waste Reduction and Recycling,463,181,"An Act relative to enhancing circularity in recycling | Rep. Kaufman, Jay (D) - An Act to reduce solid waste in the Commonwealth | An Act to reduce packaging waste | An Act to reduce plastics | An Act to reduce plastic packaging waste" +3,Comprehensive Student Education and Wellness,1572,7,An Act implementing the Every Student Succeeds Act | An Act empowering students and schools to thrive | An Act regarding breakfast after the bell | An Act regarding breakfast after the bell | An Act to remodel public school athletics through social-emotional learning +4,Infrastructure Investment for Growth,1124,8,"An Act relative to job creation, workforce development and infrastructure investment | An act financing the general governmental infrastructure of the Commonwealth | An Act financing the general governmental infrastructure of the Commonwealth | An Act financing improvements to the Commonwealth's transportation system | An Act relative to open spaces and tax incentives" +5,Data Privacy and Security,1358,4,An Act Establishing the Massachusetts Data Privacy Act | Establishing the Massachusetts Consumer Data Privacy Act | An Act relative to consumer data privacy | An Act advancing the economic development of the commonwealth through comprehensive data privacy | An Act establishing the Massachusetts Information Privacy Act +6,Special Commission Studies,1013,15,"Resolve providing for an investigation and study by a special commission relative to robotic surgery in the Commonwealth | Resolve establishing a commission to consider the inclusion of nuclear energy in Massachusetts' future energy planning | An Act relative to establishing a food justice frontline | Resolve providing for a commission to study the affects of offshore wind projects on Massachusetts fisheries | Resolve Providing for a ""Special Commission on Peace Officer Standards and Training"" to Study and Make Recommendations Concerning the Implementation of a Peace Officers Standards and Training (POST) System" +7,"Tax Policy, Exemptions, and Relief",1270,1,An Act eliminating telecommunications tax exemptions | AN ACT RELATIVE TO CERTAIN TAX EXEMPTIONS . | An Act providing for local property tax relief | An Act to invest in our communities | An Act to Invest in Our Communities +8,Local Affordable Housing & Tenant Purchase,1148,7,An Act regarding right of first refusal in the event of foreclosure and short sales and deeds in lieu | An Act for a right of first refusal for foreclosed property (Foreclosure TOPA) | An Act supporting affordable housing with a local option for a fee to be applied to certain real estate transactions | An Act to guarantee a tenant’s first right of refusal | An Act authorizing municipalities to protect low and moderate income tenants and units of governmentally involved housing +9,Public Safety and Youth Justice,1175,5,An act relative to the protection fo children | An Act to protect the commonwealth from dangerous persons | An Act to protect victims of crimes and the public | An Act clarifying consent laws for adolescents | An Act relative to consensual adolescent sexual activity +10,Improving Behavioral Health Access,629,0,An Act to improve access to behavioral health services | An Act to improve access to behavioral health services | An Act improving access to behavioral health services | An Act providing equitable access to behavioral health services for MassHealth consumers | An Act expanding access to mental health services +11,Gun Laws and Safety,668,7,An Act relative to the lawful sale of handguns | An Act relative to the lawful sale of handguns | An Act relative to the lawful sale of handguns | An Act relative to waterfowl hunting | An Act relative to unloaded rifles or shotguns +12,Public Safety Retirement Benefits,1256,0,An Act relative to Mass Port police officers | An Act relative to Mass Port police officers | An Act relative to Mass Port police officers | An Act relative to Mass Port police officers | An Act relative to the retirement of MassPort officers +13,Economic and Regulatory Policy,651,13,"An Act clarifying the meal break law to allow for private enforcement +HB757 An Act for a competitive economy through safer alternatives to toxic chemicals +HB4952 An Act increasing the experience and background necessary to operate low pressure processed steam boilers +HB4865 An Act relative to a competitive economy through safer alternatives to toxic chemicals +SB684 An Act relative to contract compliance with homeland security +SB474 An Act relative to establishing family and employment security through temporary disability insurance and family temporary disability insurance +HB3124 An Act relative to expanding energy efficiency in the commonwealth +HB4825 An Act relative to expanding energy efficiency in the Commonwealth +SB1524 An Act relative to expanding energy efficiency in the Commonwealth +HB4558 An Act relative to increasing the experience and background necessary to operate low pressure processed steam boilers +HB3809 An Act relative to insurance coverage for autism +HB4935 An Act relative to insurance coverage for autism +HB3912 An Act relative to patient safety +HB4668 An Act relative to post-partum depression +HB4859 An Act relative to post-partum depression +HB4909 An Act relative to providing incentives for productive workers compensation audits +HB1853 An Act relative to the competitive determination of workers' compensation insurance rates +HB1864 An Act relative to the competitive determination of workers' compensation insurance rates +HB3881 An Act relative to toxic fire retardant prevention +SB389 An Act to create environmental justice +SB730 An Act to establish a self sufficiency standard in the Commonwealth +SB2517 An Act to protect motor vehicle owners' and independent repairers' right to repair +HB1846 An Act to provide incentives for productive workers compensation audits | HB1188 +HB1202 +HB1218 +HB1219 +HB1224 +HB1227 +HB1450 +HB1455 +HB1566 +HB1568HB1665 +HB1695 +HB1697 +HB1700 +HB1701 +HB1736 +HB1778HB1779 +HB1981 +HB1982 +HB2003 +HB2004 +HB2140 +HB2668 +HB2692 +HB2702 +HB2726 +HB2765 +HB2778 +HB2780 +HB2822 +HB2881 +HB2933 +HB2938 +HB2960HB3014 +HB3037 +HB3061 +HB3061 +HB3065 +HB3065 +HB3083HB3095 +HB310 +HB3124 +HB3308 +HB3362 +HB3364 +HB337 +HB3411 +HB3412 +HB3446 +HB3462HB3491 +HB3570 +HB3572 +HB3575 +HB3583 +HB3615 +HB3646 +HB3647 +HB3670 +HB3672 +HB3673 +HB3674 +HB3675 +HB3699 +HB3760 +HB3762 +HB3951 +HB4072HB702 +HB750 +HB756 +HB775HB782 +HB806 +HB808 +HB811 +HB818 +HB834 +HB849 +HB852 +HB853 +HB88 +HB960 +HD2622 +SB103 +SB104 +SB105 +SB1323 +SB1387 +SB1504 +SB1508 +SB1577SB1607 +SB168 +SB1912 +SB193SB1953 +SB1963 +SB2019 +SB205 +SB206 +SB2061 +SB2101 +SB373 +SB382 +SB386 +SB387 +SB388 +SB389 +SB394 +SB396 +SB411 +SB412 +SB436 +SB441 +SB6 +SB623 +SB625 +SB636 +SB653 +SB663 +SB765 +SB767 +SB770 +SB783 +SB817 +SB88 +SB882 +SB90SB96 +SB97 +SD1336 +Greenhouse gas policy +Energy conservation and decoupling +Building Code +Stormwater +Ch. 21E +Ch. 91 +MEPA +Highway Access Permit +MESA +Transportation reform | An Act relative to financial services contracts for dental benefits corporations; +SB2363 An Act relative to oral cancer therapy; +SB2359: An Act to establish standards for long term care insurance; +SB2134: An Act to establish standards for long term care insurance (SB411: An Act to establish standards for long term care insurance); +HB4464: An Order relative to authorizing the committee on Financial Services to make an investigation and study of certain Senate and House documents concerning health care insurance and insurance coverage issues; +HB295: An Act relative to increasing consumer access to licensed marriage and family therapy; +HB304: An Act to define the use of observation services; +HB3928: An Act relative to the treatment of cleft palate and cleft lip; +HB4070: An Act relative to Health Care Quality Improvement and Cost Reduction Act of 2012; +SB2362: An Act establishing the Massachusetts Childhood Vaccine Program; +SB2120: An Act establishing the Massachusetts Childhood Vaccine Program; +HB585: An Act to protect patients from breakthrough seizures; +HB 4420: An Act relative to miscellaneous insurance and banking issues; +HB4257: An Act to limit retroactive denials of health insurance (HB1204 An Act to limit retroactive denials of health insurance claims); +HB2137: An Act providing for the equitable apportionment of certain liens; +HB1216: An Act relative to manufacturers' coupons and rebates; +SB69: An Act providing for insurance coverage for vision screening for children; +SB72: An Act relative to increasing consumer access to licensed marriage and family therapists; +SB285: An Act establishing a special commission to promote prescription pharmaceutical affordability in the Commonwealth; +SB519: An Act relative to health care affordability; +SB527: An Act defining the use of observational services; +SB2121: An Act to provide for tobacco cessation benefits; +SB821: An Act establishing accountability for managed care organizations; +SB2440: Order relative to authorizing the joint committee on the Judiciary to make an investigation and study of certain current Senate documents relative to the Judiciary (SB825 An Act revising the charitable immunity cap); +SB1070: An Act relative to oral cancer therapy; +SB498 - An Act relative to student health insurance; +HB3916: An Act relative to financial services contracts for dental benefits corporations; +HB3917: An Act relative to payment for use of ambulance services; +HB3928: An Act relative to the treatment of cleft palate and cleft lip; +HB3981: Text of a further amendment to the Senate amendment of the engrossed Bill relative to tiered and selective network health plans; +HB4200 (Sections 129, 130, 131): Committee of Conference on the disagreeing votes of the two branches of the House Bill making appropriations for the fiscal year 2013; +SB2270: Printed as amended version of the engrossed Bill relative to improving the quality of health care and reducing costs through increased transparency, efficiency and innovation. SB2400: An act improving the quality of health care and reducing costs through increased transparency, efficiency and innovation. | - An Act Establishing A Board Of Casting Agents +HB1270- An Act Relative To The Protection Of Talented Children And Their Families +HB2511-An Act Relative To Increasing Economic Development And Business Opportunity In The Design And Manufacture Of Video Games Through The Use Of Tax Credits +HB2526-An Act Relative To Film Tax Credit Percentage +HB2629-An Act To Promote Sales Tax Fairness For Main Street Retailers +HB2630-An Act Relative To The Tax Expenditure Budget +HB2662-An Act Relative To Tax Credits, Education And Training For The Design And Development Of Video Game And Interactive Digital Media Products +HB2738-An Act Relative To Targeted Investment Of Taxpayer Dollars +HB3382-An Act Relative To Transportation Finance +HB3526- An Act To Ensure Compliance With Federal Marketplace Fairness +HB3535-An Act Relative To Transportation Finance +SB1441-An Act Relative To Eligible Production Expenses For The Film Tax Credit +SB1630-An Act Protecting The Commercial Value Of Artists, Entertainers, And Other Notable Personalities +HB 1 - An Act making appropriations for FY2014 +HB3382 - An Act relative to transportation finance +HB3415 - An Act relative to transportation finance +SB1766 - An Act relative to transportation finance +SB1770 - An Act relative to transportation finance +HB3400 - An Act making appropriations for the fiscal year 2014 for the maintenance of the departments, boards, commissions, institutions and certain activities of the Commonwealth, for interest, sinking fund and serial bond requirements and for certain permanent improvements +HB3401 - Printed as amended version of the Bill making appropriations for the fiscal year 2014 for the maintenance of departments, boards, commissions, institutions and certain activities of the Commonwealth, for interest, sinking fund and serial bond requirements and for certain permanent improvements +H3538 - An Act making appropriations for the Fiscal Year 2014 for the maintenance of the departments, boards, commissions, institutions and certain activities of the commonwealth, for interest, sinking fund and serial bond requirements and for certain permanent improvements +SB3 - Text of the Senate amendments to the House Bill making appropriations for the fiscal year 2014 for the maintenance of the departments, boards, commissions, institutions and certain activities of the Commonwealth, for interest, sinking fund and serial bond requirements and for certain permanent improvements +SB1800 - Text of the Senate amendments to the House Bill making appropriations for the fiscal year 2014 for the maintenance of the departments, boards, commissions, institutions and certain activities of the Commonwealth, for interest, sinking fund and serial bond requirements and for certain permanent improvements +Government Relations Group is paid a monthly retainer which encompasses all activity and is not paid on an hourly or per issue basis. | An Act Increasing The School Choice Cap +HB225 An Act Protecting The Commonwealth's Lakes And Ponds +HB1596 An Act Relative To Creditable Service For Teachers In Charter Schools +HB1925 An Act Establishing A Charter School Working Group +HB1926 An Act Establishing A Charter School Commission +HB1935 An Act Relative To Charter School Renewal +HB1951 An Act To Improve Access To Charter Schools For Students With Disabilities And Limited English Proficiency +HB2721 An Act Relative To Unused District School Buildings +SB231 An Act Establishing A Charter School Working Group. +SB246 An Act Establishing A Charter School Working Group." +14,Massachusetts Clean Energy Transition,799,238,"An Act transitioning Massachusetts to clean electricity, heating and transportation | An Act transitioning Massachusetts to clean electricity, heating and transportation | An Act transitioning the Commonwealth to 100% clean electricity, heating, and transportation | An Act transitioning the Commonwealth to 100% clean electricity, heating, and transportation | An Act investing in a prosperous, clean commonwealth by 2030" +15,Pharmaceutical Pricing and Access,781,3,An Act relative to pharmaceutical price transparency and cost sharing | An Act relative to pharmaceutical price transparency and cost sharing. | An Act relative to consumer protection for prescription drug purchases | An Act creating patient choice and education in the dispensing of opiate drugs | An act creating patient choice and education in the dispensing of opiate drugs +16,Wage Protection and Enforcement,1281,2,"An Act to prevent wage theft, promote employer accountability, and enhance public enforcement | An Act to prevent wage theft, promote employer accountability, and enhance public enforcement | An Act to prevent wage theft, promote employer accountability, and enhance public enforcement | An Act to prevent wage theft, promote employer accountability, and enhance public enforcement | An Act relative to independent contractors" +17,Home and Community Elder Care,921,0,An Act relative to assisted living and the frail elder waiver | An Act relative to assist families care for elders | An Act relative to intensive case management for clinically complex older adults | An Act relative to stabilizing the Commonwealth?s nursing facilities | An Act promoting affordability of home care services +18,Healthcare Coverage and Access,1142,0,An Act relative to ensuring treatment for genetic craniofacial conditions | An Act relative to ensuring treatment for genetic craniofacial conditions | An Act relative to ensuring treatment for genetic craniofacial conditions | An Act relative to ensuring treatment for genetic craniofacial conditions | An Act relative to ensuring treatment for genetic craniofacial conditions +19,State Budget Appropriations,862,12,An Act making appropriations for fiscal year 2025 to provide for supplementing certain existing appropriations and for certain other activities and projects | An Act making appropriations for fiscal year 2025 to provide for supplementing certain existing appropriations and for certain other activities and projects | An Act making appropriations for the fiscal year 2025 to provide for supplementing certain existing appropriations and for certain other activities and projects | An Act making appropriations for the fiscal year 2025 to provide for supplementing certain existing appropriations and for certain other activities and projects | An Act making appropriations for the fiscal year 2025 to provide for supplementing certain existing appropriations and for certain other activities and projects +20,Empowering Local Clean Energy,1058,54,An Act to empower communities to transition to renewable energy | An Act relative to municipal lighting authorities | An Act promoting local energy investment and infrastructure modernization | An Act promoting local energy investment and infrastructure modernization | An Act promoting local energy investment and infrastructure modernization +21,Automobile Insurance Rates and Coverage,745,12,An Act relative to motor vehicle surcharges | An Act Relative to Motor Vehicle Surcharges | An Act relative to home owners insurance and funding repairs | An Act relative to automobile insurance companies | An Act Relative to Automobile Insurance Companies +22,State Financial and Service Initiatives,1084,29,"An Act Relative to State Personnel | An Act Relative to State Personnel | An Act relative to modernize municipal finance and government | An Act relative to Protecting Senior Health | Senate Ways and Means - An Act establishing a sick leave bank for Christine Chrzanowski, an employee of the Department of Mental Health" +23,Granting Additional Alcohol Licenses,521,0,"An Act authorizing the town of Tewksbury to grant 8 additional licenses for the sale of alcoholic beverages to be drunk on the premises | An Act authorizing the city of Fitchburg to grant 1 additional license for the sale of all alcoholic beverages not to be drunk on the premises | An Act authorizing the town of Walpole to grant an additional license for the sale of wines and malt beverages not to be drunk on the premises | An Act authorizing the town of Dedham to grant four additional licenses for the sale of all alcoholic beverages, four wine and malt licenses, all to be drunk on the premises, and one wine and malt package store license | An Act relative to special one-day licenses to sell or dispense alcoholic beverages" +24,Road Safety and Vehicle Technology,1123,5,An Act relative to the safe operation of motor vehicles | An Act relative to the safe operation of motor vehicles | An Act prohibiting video recording or broadcasting while driving | An Act relative to the use of video screens while operating a motor vehicle | An Act relative to the safe operation of motor vehicles diff --git a/docs/data/MA_bill_emb_mean.npy b/docs/data/MA_bill_emb_mean.npy new file mode 100644 index 0000000..ee4346d Binary files /dev/null and b/docs/data/MA_bill_emb_mean.npy differ diff --git a/docs/data/MA_bill_kmeans.joblib b/docs/data/MA_bill_kmeans.joblib new file mode 100644 index 0000000..63a6d30 Binary files /dev/null and b/docs/data/MA_bill_kmeans.joblib differ diff --git a/docs/data/MA_legislature_bills_sample.csv b/docs/data/MA_legislature_bills_sample.csv new file mode 100644 index 0000000..552eed6 --- /dev/null +++ b/docs/data/MA_legislature_bills_sample.csv @@ -0,0 +1,104 @@ +,bill_id,bill_number,bill_prefix,general_court,title,sponsor_name,status,passed +0,H4999,4999,H,193,An Act promoting access to midwifery care and out-of-hospital birth options,Midwifery Care and Out-of-Hospital options,"Signed by the Governor, Chapter 186 of the Acts of 2024",1 +1,H4773,4773,H,193,An Act promoting access to midwifery care and out-of-hospital birth options,House Committee on Ways and Means,"Published as amended, see H4785",0 +2,S1405,1405,S,193,An Act relative to newborn screenings for congenital cytomegalovirus,Joan B. Lovely,No further action taken,0 +3,S607,607,S,193,An Act to increase access to nurse-midwifery services,Brendan P. Crighton,No further action taken,0 +4,H1069,1069,H,193,An Act to increase access to nurse-midwifery services,Kay Khan,Senate concurred,0 +5,H1202,1202,H,193,An Act to preserve special needs trusts for disabled seniors,Kate Hogan,"New draft substituted, see H4841",0 +6,S108,108,S,193,An Act establishing guardians as providers of medical care to support the rights of incapacitated persons,Joan B. Lovely,"Accompanied a study order, see S2795",0 +7,H5033,5033,H,193,An Act to improve quality and oversight of long-term care,Long Term Care,"Signed by the Governor, Chapter 197 of the Acts of 2024",1 +8,H4800,4800,H,193,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements","General Appropriation Bill, FY25",Item 7066-0036 passed over veto - 155 YEAS to 2 NAYS (See YEA and NAY No. 191),0 +9,H2007,2007,H,193,An Act enabling physician assistants to authorize psychiatric holds and ensure adequate training on their use,Edward R. Philips,"Accompanied a study order, see H4897",0 +10,H4058,4058,H,193,An Act expanding access to mental health services,Marjorie C. Decker,"Accompanied a study order, see H4879",0 +11,H5100,5100,H,193,An Act relative to strengthening Massachusetts’ economic leadership,Economic Development,"For actions on the sections returned with recommendations of amendment, see H5111",0 +12,H2135,2135,H,193,An Act relative to removing barriers to care for physician assistants,Christine P. Barber,Accompanied S1354,0 +13,S1267,1267,S,193,An Act expanding access to mental health services,Jacob R. Oliveira,"Accompanied a study order, see S2642",0 +14,H2217,2217,H,193,An Act relative to limited service clinics,Kay Khan,"Accompanied a study order, see H4739",0 +15,H1950,1950,H,193,An Act relative to physician assistant non-competes,Jon Santiago,"Accompanied a study order, see H4570",0 +16,H2812,2812,H,193,An Act establishing fairness for agricultural laborers,Carlos González,"Accompanied a study order, see H5045",0 +17,S1837,1837,S,193,An Act establishing fairness for agricultural laborers,Adam Gomez,"Accompanied a study order, see H5045",0 +18,S2673,2673,S,193,An Act relative to social work uplifting practices and exam removal.,Joint Committee on Higher Education,No further action taken,0 +19,H4451,4451,H,193,An Act relative to social work uplifting practices and exam removal,Joint Committee on Higher Education,No further action taken,0 +20,S2227,2227,S,193,An Act to end debt-based driving restrictions and remove economic roadblocks,Julian Cyr,"Accompanied a study order, see S2796",0 +21,H3314,3314,H,193,An Act to end debt-based driving restrictions and remove economic roadblocks,Brandy Fluker Oakley,"Accompanied a study order, see H4928",0 +22,H1434,1434,H,193,An Act reducing barriers to employment through credit discrimination,Josh S. Cutler,No further action taken,0 +23,S673,673,S,193,An Act ensuring support groups are part of the pathway of care following the annual mental health wellness exam,Joan B. Lovely,"Accompanied a study order, see H4634",0 +24,H1075,1075,H,193,An Act ensuring support groups are part of the pathway of care following the annual mental health wellness exam,Michael P. Kushmerek,"Accompanied a study order, see H4854",0 +25,H946,946,H,193,An Act to increase access to lactation care and services,Christine P. Barber,"Accompanied a study order, see H4691",0 +26,S691,691,S,193,An Act to increase access to lactation care and services,Susan L. Moran,"Accompanied a study order, see H4634",0 +27,H5077,5077,H,193,making appropriations for the fiscal year 2024 to provide for supplementing certain existing appropriations and for certain other activities and projects,House Committee on Ways and Means,"Signed by the Governor, Chapter 248 of the Acts of 2024",1 +28,H1918,1918,H,193,An Act relative to employment protections for victims of abusive behavior,Jay D. Livingstone,No further action taken,0 +29,S1152,1152,S,193,An Act relative to employment protections for victims of abusive behavior,Cynthia Stone Creem,Accompanied H1918,0 +30,H2216,2216,H,193,An Act relative to dental hygienists,Kay Khan,"Accompanied a study order, see H4739",0 +31,S1429,1429,S,193,An Act relative to dental hygienists,Michael O. Moore,"Accompanied a study order, see S2648",0 +32,H3880,3880,H,193,An Act relative to dental hygienist reciprocal licensure,Tackey Chan,"New draft substituted, see H4842",0 +33,H2241,2241,H,193,An Act enabling trained dental hygienists to administer dermal fillers and botox,Frank A. Moran,"Accompanied a study order, see H4739",0 +34,S664,664,S,193,An Act relative to the sustainability of public health dental hygienists through adequate reimbursements,Robyn K. Kennedy,"Accompanied a study order, see H4634",0 +35,H2255,2255,H,193,An Act relative to the sustainability of public health dental hygienists through adequate reimbursements,Smitty Pignatelli,"Accompanied a study order, see H4739",0 +36,S1379,1379,S,193,An Act to improve oral health for all Massachusetts residents,Adam Gomez,"Accompanied a new draft, see S2740",0 +37,S1357,1357,S,193,An Act enabling trained dental hygienists to administer dermal fillers and botox,Julian Cyr,"Accompanied a study order, see S2648",0 +38,H2215,2215,H,193,An Act protecting children from harmful diet pills and muscle-building supplements,Kay Khan,No further action taken,0 +39,H5159,5159,H,193,An Act enhancing the market review process,Market Review Process,"Signed by the Governor, Chapter 343 of the Acts of 2024",1 +40,H4799,4799,H,193,An Act making appropriations for the fiscal year 2024 to provide for supplementing certain existing appropriations and for certain other activities and projects,,"Signed by the Governor, Chapter 206 of the Acts of 2024",1 +41,H4442,4442,H,193,An Act creating a special legislative commission to study barriers to job retention for low-income workers,Joint Committee on Labor and Workforce Development,No further action taken,0 +42,S2475,2475,S,193,An Act relative to the licensure of behavior analysts,Joint Committee on Consumer Protection and Professional Licensure,No further action taken,0 +43,S376,376,S,193,An Act relative to Massachusetts home care eligibility,Patricia D. Jehlen,No further action taken,0 +44,S1244,1244,S,193,An Act building resiliency in communities through mental health,Julian Cyr,Accompanied H1985,0 +45,H1985,1985,H,193,An Act building resiliency in communities through mental health,Brandy Fluker Oakley,No further action taken,0 +46,S345,345,S,193,An Act to update accuracy of membership to the state advisory council on early education and care,Rebecca L. Rausch,No further action taken,0 +47,H135,135,H,193,An Act establishing basic needs assistance for Massachusetts immigrant residents ,Antonio F. D. Cabral,No further action taken,0 +48,S75,75,S,193,An Act to lift kids out of deep poverty,Sal N. DiDomenico,No further action taken,0 +49,H144,144,H,193,An Act to lift kids out of deep poverty,Marjorie C. Decker,No further action taken,0 +50,S1890,1890,S,193,An Act excluding the Segal AmeriCorps Education Award from taxable income,Susan L. Moran,No further action taken,0 +51,H1748,1748,H,193,"An Act relative to access to a decedent's electronic mail accounts + +",Jeffrey N. Roy,"Accompanied a study order, see H4517",0 +52,S930,930,S,193,An Act relative to access to a decedent's electronic mail accounts,Cynthia Stone Creem,"Accompanied a study order, see S2612",0 +53,S1362,1362,S,193,An Act relative to the Massachusetts lead law and promoting equal access to lead-free housing,James B. Eldridge,"Accompanied a study order, see S2648",0 +54,H2223,2223,H,193,An Act relative to the Massachusetts lead law and promoting equal access to lead-free housing,David Henry Argosky LeBoeuf,"Accompanied a study order, see H4739",0 +55,S1156,1156,S,193,An Act protecting an employee's right to rebuttal of personnel records,Sal N. DiDomenico,No further action taken,0 +56,H1881,1881,H,193,An Act clarifying the wage act to protect hospital workers from being deprived of their wages,Denise C. Garlick,"Accompanied a study order, see H5014",0 +57,H1891,1891,H,193,An Act clarifying the prevailing wage act to protect actions by the attorney general,Kenneth I. Gordon,No further action taken,0 +58,H1088,1088,H,193,An Act relative to greater fairness in insurance,Jay D. Livingstone,"Accompanied a study order, see H4705",0 +59,S1161,1161,S,193,An Act clarifying the prevailing wage act to protect actions by the attorney general,Sal N. DiDomenico,No further action taken,0 +60,S1177,1177,S,193,An Act clarifying the wage act to protect hospital workers from being deprived of their wages,Paul R. Feeney,Accompanied H1881,0 +61,H4282,4282,H,193,An Act relative to defense against abusive waivers,Joint Committee on Labor and Workforce Development,No further action taken,0 +62,H1892,1892,H,193,An Act relative to anti-retaliation protections for public employees,Kenneth I. Gordon,"Accompanied a study order, see H5043",0 +63,H1886,1886,H,193,An Act protecting an employee's right to rebuttal of personnel records,Kenneth I. Gordon,Accompanied S1156,0 +64,H1709,1709,H,193,An Act to further protect employees through a private right of action,Tram T. Nguyen,"Accompanied a study order, see H4517",0 +65,H4970,4970,H,193,An Act to ensure legal parentage equality,Legal Parentage Equality,"Signed by the Governor, Chapter 166 of the Acts of 2024",1 +66,S99,99,S,193,An Act creating a special commission to study the current refugee resettlement infrastructure and ensure the successful integration of refugees in Massachusetts ,Robyn K. Kennedy,No further action taken,0 +67,H1350,1350,H,193,An Act to guarantee a tenant’s first right of refusal,Jay D. Livingstone,No further action taken,0 +68,S880,880,S,193,An Act to guarantee a tenant’s first right of refusal,Patricia D. Jehlen,No further action taken,0 +69,S864,864,S,193,"An Act promoting access to counsel and housing stability in Massachusetts +",Sal N. DiDomenico,No further action taken,0 +70,H4977,4977,H,193,An Act relative to the Affordable Homes Act,Affordable Homes Act,"Signed by the Governor, Chapter 150 of the Acts of 2024",1 +71,H4360,4360,H,193,An Act promoting access to counsel and housing stability in Massachusetts,Joint Committee on the Judiciary,No further action taken,0 +72,H4429,4429,H,193,An Act relative to fairness in debt collection,Joint Committee on Financial Services,No further action taken,0 +73,S2713,2713,S,193,An Act relative to fairness in debt collection,Senate Committee on Ways and Means,No further action taken,0 +74,H1308,1308,H,193,An Act relative to the use of credit reporting in housing,Manny Cruz,No further action taken,0 +75,S877,877,S,193,An Act relative to the use of credit reporting in housing,Adam Gomez,No further action taken,0 +76,H1302,1302,H,193,An Act relative to manufactured housing,Tackey Chan,No further action taken,0 +77,S1354,1354,S,193,An Act relative to removing barriers to care for physician assistants,Julian Cyr,No further action taken,0 +78,H4459,4459,H,193,An Act relative to strengthening Massachusetts' economic leadership,Maura T. Healey,"Accompanied a new draft, see H4722",0 +79,H4601,4601,H,193,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",,"Reported, in part, by H4800",0 +80,H2,2,H,193,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the Commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Maura T. Healey,No further action taken,0 +81,H1253,1253,H,193,An Act relative to social work uplifting practices and exam removal ,Christine P. Barber,"Accompanied a new draft, see H4451",0 +82,S2800,2800,S,193,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",,See H4601,0 +83,S1335,1335,S,193,An Act updating the regulations governing licensed birth centers in Massachusetts,Joanne M. Comerford,"Accompanied a new draft, see H4566",0 +84,H4566,4566,H,193,An Act promoting access to midwifery care and out-of-hospital birth options,Joint Committee on Public Health,"New draft substituted, see H4773",0 +85,S1457,1457,S,193,An Act promoting access to midwifery care and out-of-hospital birth options,Rebecca L. Rausch,"Accompanied a new draft, see H4566",0 +86,S1415,1415,S,193,An Act relative to birthing justice in the Commonwealth,Liz Miranda,"Accompanied a new draft, see S2734",0 +87,H1415,1415,H,193,An Act to increase the penalty for school tardiness,Tackey Chan,"Accompanied a study order, see H4517",0 +88,H188,188,H,193,An Act establishing guardians as providers of medical care to support the rights of incapacitated persons,Sally P. Kerans,Accompanied S108,0 +89,S756,756,S,193,An Act preserving special needs trusts for disabled seniors,Patricia D. Jehlen,Accompanied H1202,0 +90,S1465,1465,S,193,An Act protecting children from harmful diet pills and muscle-building supplements,Michael F. Rush,Accompanied H2215,0 +91,S2835,2835,S,193,An Act relative to adoptions,Cynthia Stone Creem,"Accompanied a new draft, see S3009",0 +92,H4709,4709,H,193,An Act relative to adoptions,Lindsay N. Sabadosa,"Accompanied a new draft, see S3009",0 +93,H4496,4496,H,193,An Act making appropriations for the fiscal year 2024 to provide for supplementing certain existing appropriations and for certain other activities and projects,Maura T. Healey,No further action taken,0 +94,H2253,2253,H,193,An Act to improve oral health for all Massachusetts residents,Smitty Pignatelli,"Accompanied a new draft, see S2740",0 +95,H197,197,H,193,An Act creating a special commission to study the current refugee resettlement infrastructure and ensure the successful integration of refugees in Massachusetts ,David Henry Argosky LeBoeuf,Accompanied S99,0 +96,H2777,2777,H,193,An Act excluding the Segal AmeriCorps Educational Award from taxable income,"William J. Driscoll, Jr.",Accompanied S1890,0 +97,S76,76,S,193,An Act establishing basic needs assistance for Massachusetts immigrant residents,Sal N. DiDomenico,Accompanied H135,0 +98,H542,542,H,193,An Act to promote racially inclusive curriculum in schools,Tram T. Nguyen,"Accompanied a new draft, see H4519",0 +99,H3052,3052,H,193,An Act promoting the civil rights and inclusion of American Muslims in the Commonwealth,Vanna Howard,No further action taken,0 diff --git a/docs/data/MA_lobbying.md b/docs/data/MA_lobbying.md new file mode 100644 index 0000000..4ba15e6 --- /dev/null +++ b/docs/data/MA_lobbying.md @@ -0,0 +1,81 @@ +--- +title: MA Lobbying Disclosures +author: NES +layout: data_listing +ancillary: 0 +--- + +## Data source + +The [MA Secretary of State](https://www.sec.state.ma.us/LobbyistPublicSearch/) publishes semi-annual lobbying disclosure filings for all registered lobbyists and lobbying entities in Massachusetts. Filers report which clients hired them, how much each client paid, and which specific bills they lobbied on behalf of each client (with chamber, bill number, title, and position — Support, Oppose, or Neutral). + +Data is available from 2005 (184th General Court) through the present, spanning 22 years across 11 legislative sessions. Each two-year legislative session is identified by a General Court number (GC 184 = 2005–2006, GC 194 = 2025–2026, etc.). + +The data from this source has been archived on this site, last updated on **{{ site.data.ts_update_MA_lobbying.updated | date: "%-d %B %Y %I:%M %P" }}**. +Filings are refreshed automatically on a weekly basis; the script exits early when no new semi-annual filings have been posted. + +## Environmental relevance scoring + +To identify which bills are environmentally relevant, each bill's full text (fetched from the [MA Legislature OpenAPI](https://malegislature.gov/api/swagger)) is preprocessed by stripping repeated legislative scaffolding (e.g. "Chapter X of the General Laws, as appearing in the 2020 Official Edition, is hereby amended by inserting after…") that appears identically across thousands of bills regardless of topic. The bill title is then prepended to the cleaned body, and the combined text is truncated to 3,000 characters before embedding. Bills for which no full text is available fall back to the title alone. The cleaned text is embedded using Google's **Gemini Embedding model** (`gemini-embedding-2`, 768-dimensional vectors). + +Environmental relevance is scored using **differential cosine similarity**: for each bill, the maximum cosine similarity to a set of 42 known environmental bills is computed, and the maximum cosine similarity to a set of 42 known non-environmental bills is subtracted. Bills with a differential score above **0.05** are flagged as `is_environmental` (~1.3% of all uniquely lobbied bills). The non-environmental reference set spans eight policy domains — labor, criminal justice, healthcare, education, housing, municipal licensing, digital/media, and LGBTQ/social — to prevent cross-domain false positives. + +**Data coverage note:** Bills from the two oldest legislative sessions (GC 183–184, 2005–2008) have no full text in the Legislature API and are often missing titles in the lobbying portal as well. These ~1,500 bills embed as zero vectors and are excluded from topic clustering (assigned `cluster_id = -1`). They are retained in the lobbying activity data but do not appear in the t-SNE visualization. + +Embeddings are stored in a Parquet file on Google Cloud Storage (`gs://openamend-data/MA_bill_embeddings.parquet`) alongside bill full text. The lightweight scored CSV (scores and cluster IDs only, no embeddings) is committed to this repository. + +## Topic clustering + +All lobbied bills with valid embeddings (~24,400 bills) are clustered into **25 topic groups** using the **k-means clustering** algorithm on the L2-normalised Gemini embeddings (cosine-space clustering). Each cluster is labelled using **Gemini 2.5 Flash**, which receives the 20 most central bill titles in the cluster and returns a 3–5 word topic label. Clustering is a one-time operation re-run manually when the historical data changes significantly. + +| Cluster | Label | Bills | Env. bills | +|---------|-------|------:|----------:|{% for row in site.data.MA_bill_cluster_labels %} +| {{ row.cluster_id }} | {{ row.label }} | {{ row.n_bills }} | {{ row.n_env_bills }} |{% endfor %} + +### Bill embedding space (t-SNE) + +The plot below shows environmental bills projected into the policy landscape using [t-SNE](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding). **Coloured, outlined dots** are the 329 environmentally-relevant bills, coloured by topic cluster; **grey dots** are a stratified background sample (~120 per cluster, ~3,000 total) providing geographic context. Hover over any point for the bill title. + +Note: MA legislative bill embeddings are semantically dense — even after boilerplate stripping, mean inter-cluster cosine distance is only ~0.006 vs. mean intra-cluster spread of ~0.53. Visualising all 25,000+ bills produces a featureless blob because the underlying high-dimensional structure does not project cleanly to two dimensions. The subsample approach makes the environmentally-relevant bills legible without misrepresenting the cluster separation. + +{% include charts/lobbying_bill_tsne.html %} + +## Download archive + +Full CSVs are stored in Google Cloud Storage (too large for the repository). +These links will be active once the initial full-history scrape is complete and uploaded: + +* Lobbying employers (entity–client–year) — `gs://openamend-data/MA_lobbying_employers.csv` +* Lobbying bills (entity–client–bill–year) — `gs://openamend-data/MA_lobbying_bills.csv` +* Lobbying bills scored (env relevance + cluster) — `gs://openamend-data/MA_lobbying_bills_scored.csv` +* Legislature bill metadata — `gs://openamend-data/MA_legislature_bills.csv` +* [Bill embeddings (768-dim Parquet)](https://storage.googleapis.com/openamend-data/MA_bill_embeddings.parquet) + +## Data tables + +### Lobbying Employers + +One row per (entity, client, year). Records how much each client paid each lobbying entity in a given year. + +| Entity Name | Client Name | Year | Reg Type | Compensation | +| --- | --- | --- | --- | --- |{% for row in site.data.MA_lobbying_employers_sample limit:10 %} +| {{ row.entity_name }} | {{ row.client_name }} | {{ row.year }} | {{ row.reg_type }} | {{ row.compensation }} |{% endfor %} +{: .sortable} + +### Lobbying Bills + +One row per (entity, client, bill, session). Records which bills each entity lobbied on behalf of each client, with the lobbying position. + +| Entity Name | Client Name | Year | Chamber | Bill Number | Bill Title | Position | +| --- | --- | --- | --- | --- | --- | --- |{% for row in site.data.MA_lobbying_bills_sample limit:10 %} +| {{ row.entity_name }} | {{ row.client_name }} | {{ row.year }} | {{ row.chamber }} | {{ row.bill_number }} | {{ row.bill_title }} | {{ row.position }} |{% endfor %} +{: .sortable} + +### Legislature Bills + +Bill metadata fetched from the [MA Legislature OpenAPI](https://malegislature.gov/api/swagger). Includes sponsor, final status, and derived `passed` boolean. Environmental relevance scores and cluster IDs are stored separately in `MA_lobbying_bills_scored.csv` (see above). + +| Bill Number | General Court | Title | Sponsor | Status | Passed | +| --- | --- | --- | --- | --- | --- |{% for row in site.data.MA_legislature_bills_sample limit:10 %} +| {{ row.bill_number }} | {{ row.general_court }} | {{ row.title }} | {{ row.sponsor_name }} | {{ row.status }} | {{ row.passed }} |{% endfor %} +{: .sortable} diff --git a/docs/data/MA_lobbying_bills_sample.csv b/docs/data/MA_lobbying_bills_sample.csv new file mode 100644 index 0000000..a3595ef --- /dev/null +++ b/docs/data/MA_lobbying_bills_sample.csv @@ -0,0 +1,101 @@ +Unnamed: 0,entity_name,client_name,year,general_court,chamber,bill_number,bill_title,position,amount,entity_name_norm,client_name_norm +0,Aaron Agulnek,"BAY STATE BIRTH COALITION, INC.",2024,193,House Bill,4999,An Act promoting access to midwifery care and out-of-hospital birth options,Neutral,0.0,AARON AGULNEK,BAY STATE BIRTH COALITION +1,Aaron Agulnek,"BAY STATE BIRTH COALITION, INC.",2024,193,House Bill,4773,An Act promoting access to midwifery care and out-of-hospital birth options,Support,0.0,AARON AGULNEK,BAY STATE BIRTH COALITION +2,Aaron Agulnek,Vanessa Colleran,2024,193,Senate Bill,1405,An Act relative to newborn screenings for congenital cytomegalovirus,Support,0.0,AARON AGULNEK,VANESSA COLLERAN +3,Aaron Agulnek,Vanessa Colleran,2024,193,House Bill,4999,An Act promoting access to midwifery care and out-of-hospital birth options,Neutral,0.0,AARON AGULNEK,VANESSA COLLERAN +4,Aaron Agulnek,"Massachusetts Affiliate of ACNM, Inc",2024,193,House Bill,4999,An Act promoting access to midwifery care and out-of-hospital birth options,Neutral,0.0,AARON AGULNEK,MASSACHUSETTS AFFILIATE OF ACNM +5,Aaron Agulnek,"Massachusetts Affiliate of ACNM, Inc",2024,193,Senate Bill,607,An Act to increase access to nurse-midwifery services,Support,0.0,AARON AGULNEK,MASSACHUSETTS AFFILIATE OF ACNM +6,Aaron Agulnek,"Massachusetts Affiliate of ACNM, Inc",2024,193,House Bill,1069,An Act to increase access to nurse-midwifery services,Support,0.0,AARON AGULNEK,MASSACHUSETTS AFFILIATE OF ACNM +7,Aaron Agulnek,"Guardian Community Trust, Inc.",2024,193,House Bill,1202,An Act to preserve special needs trusts for disabled seniors,Support,0.0,AARON AGULNEK,GUARDIAN COMMUNITY TRUST +8,Aaron Agulnek,"Guardian Community Trust, Inc.",2024,193,Senate Bill,108,An Act establishing guardians as providers of medical care to support the rights of incapacitated persons,Support,0.0,AARON AGULNEK,GUARDIAN COMMUNITY TRUST +9,Aaron Agulnek,"Guardian Community Trust, Inc.",2024,193,House Bill,5033,An Act to improve quality and oversight of long-term care,Neutral,0.0,AARON AGULNEK,GUARDIAN COMMUNITY TRUST +10,Aaron Agulnek,"Guardian Community Trust, Inc.",2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,GUARDIAN COMMUNITY TRUST +11,Aaron Agulnek,"MASSACHUSETTS ASSOCIATION OF PHYSICIAN ASSOCIATES, INC.",2024,193,House Bill,2007,An Act enabling physician assistants to authorize psychiatric holds and ensure adequate training on their use,Support,0.0,AARON AGULNEK,MASSACHUSETTS ASSOCIATION OF PHYSICIAN +12,Aaron Agulnek,"MASSACHUSETTS ASSOCIATION OF PHYSICIAN ASSOCIATES, INC.",2024,193,House Bill,4058,An Act expanding access to mental health services,Support,0.0,AARON AGULNEK,MASSACHUSETTS ASSOCIATION OF PHYSICIAN +13,Aaron Agulnek,"MASSACHUSETTS ASSOCIATION OF PHYSICIAN ASSOCIATES, INC.",2024,193,House Bill,5100,An Act relative to strengthening Massachusetts' economic leadership,Neutral,0.0,AARON AGULNEK,MASSACHUSETTS ASSOCIATION OF PHYSICIAN +14,Aaron Agulnek,"MASSACHUSETTS ASSOCIATION OF PHYSICIAN ASSOCIATES, INC.",2024,193,House Bill,2135,An Act relative to removing barriers to care for physician assistants,Support,0.0,AARON AGULNEK,MASSACHUSETTS ASSOCIATION OF PHYSICIAN +15,Aaron Agulnek,"MASSACHUSETTS ASSOCIATION OF PHYSICIAN ASSOCIATES, INC.",2024,193,Senate Bill,1267,An Act expanding access to mental health services,Support,0.0,AARON AGULNEK,MASSACHUSETTS ASSOCIATION OF PHYSICIAN +16,Aaron Agulnek,"MASSACHUSETTS ASSOCIATION OF PHYSICIAN ASSOCIATES, INC.",2024,193,House Bill,2217,An Act relative to limited service clinics,Support,0.0,AARON AGULNEK,MASSACHUSETTS ASSOCIATION OF PHYSICIAN +17,Aaron Agulnek,"MASSACHUSETTS ASSOCIATION OF PHYSICIAN ASSOCIATES, INC.",2024,193,House Bill,1950,An Act relative to physician assistant non-competes,Support,0.0,AARON AGULNEK,MASSACHUSETTS ASSOCIATION OF PHYSICIAN +18,Aaron Agulnek,"Massachusetts Law Reform Institute, Inc.",2024,193,House Bill,5100,An Act relative to strengthening Massachusetts' economic leadership,Neutral,0.0,AARON AGULNEK,MASSACHUSETTS LAW REFORM INSTITUTE +19,Aaron Agulnek,"Massachusetts Law Reform Institute, Inc.",2024,193,House Bill,2812,An Act establishing fairness for agricultural laborers,Support,0.0,AARON AGULNEK,MASSACHUSETTS LAW REFORM INSTITUTE +20,Aaron Agulnek,"Massachusetts Law Reform Institute, Inc.",2024,193,Senate Bill,1837,An Act establishing fairness for agricultural laborers,Support,0.0,AARON AGULNEK,MASSACHUSETTS LAW REFORM INSTITUTE +21,Aaron Agulnek,"National Association of Social Workers, Inc.",2024,193,Senate Bill,2673,An Act relative to social work uplifting practices and exam removal.,Support,0.0,AARON AGULNEK,NATIONAL ASSOCIATION OF SOCIAL WORKERS +22,Aaron Agulnek,"National Association of Social Workers, Inc.",2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,NATIONAL ASSOCIATION OF SOCIAL WORKERS +23,Aaron Agulnek,"National Association of Social Workers, Inc.",2024,193,House Bill,4451,An Act relative to social work uplifting practices and exam removal,Support,0.0,AARON AGULNEK,NATIONAL ASSOCIATION OF SOCIAL WORKERS +24,Aaron Agulnek,"National Association of Social Workers, Inc.",2024,193,House Bill,5100,An Act relative to strengthening Massachusetts' economic leadership,Neutral,0.0,AARON AGULNEK,NATIONAL ASSOCIATION OF SOCIAL WORKERS +25,Aaron Agulnek,National Consumer Law Center Inc.,2024,193,Senate Bill,2227,An Act to end debt-based driving restrictions and remove economic roadblocks,Support,0.0,AARON AGULNEK,NATIONAL CONSUMER LAW CENTER +26,Aaron Agulnek,National Consumer Law Center Inc.,2024,193,House Bill,3314,An Act to end debt-based driving restrictions and remove economic roadblocks,Support,0.0,AARON AGULNEK,NATIONAL CONSUMER LAW CENTER +27,Aaron Agulnek,National Consumer Law Center Inc.,2024,193,House Bill,1434,An Act reducing barriers to employment through credit discrimination,Support,0.0,AARON AGULNEK,NATIONAL CONSUMER LAW CENTER +28,Aaron Agulnek,"Greater Boston Legal Services, Inc.",2024,193,House Bill,1434,An Act reducing barriers to employment through credit discrimination,Support,0.0,AARON AGULNEK,GREATER BOSTON LEGAL SERVICES +29,Aaron Agulnek,GPS Group Peer Support LLC,2024,193,Senate Bill,673,An Act ensuring support groups are part of the pathway of care following the annual mental health wellness exam,Support,0.0,AARON AGULNEK,GPS GROUP PEER SUPPORT +30,Aaron Agulnek,GPS Group Peer Support LLC,2024,193,House Bill,1075,An Act ensuring support groups are part of the pathway of care following the annual mental health wellness exam,Support,0.0,AARON AGULNEK,GPS GROUP PEER SUPPORT +31,Aaron Agulnek,Healthy Children Project Inc DBA ALPP,2024,193,House Bill,946,An Act to increase access to lactation care and services,Support,0.0,AARON AGULNEK,HEALTHY CHILDREN PROJECT DBA ALPP +32,Aaron Agulnek,Healthy Children Project Inc DBA ALPP,2024,193,Senate Bill,691,An Act to increase access to lactation care and services,Support,0.0,AARON AGULNEK,HEALTHY CHILDREN PROJECT DBA ALPP +33,Aaron Agulnek,"Bottom Line, Inc.",2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,BOTTOM LINE +34,Aaron Agulnek,ParentChild+ Inc.,2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,PARENTCHILD+ +35,Aaron Agulnek,Jewish Community Relations Council of Boston,2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,JEWISH COMMUNITY RELATIONS COUNCIL OF BOSTON +36,Aaron Agulnek,"MA CAPE, Inc",2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,MA CAPE +37,Aaron Agulnek,"MA CAPE, Inc",2024,193,House Bill,5077,making appropriations for the fiscal year 2024 to provide for supplementing certain existing appropriations and for certain other activities and projects,Neutral,0.0,AARON AGULNEK,MA CAPE +38,Aaron Agulnek,Samaritans Inc.,2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,SAMARITANS +39,Aaron Agulnek,Veterans Inc.,2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,VETERANS +40,Aaron Agulnek,OutstandingLife,2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,OUTSTANDINGLIFE +41,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,193,House Bill,1918,An Act relative to employment protections for victims of abusive behavior,Support,0.0,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +42,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,193,Senate Bill,1152,An Act relative to employment protections for victims of abusive behavior,Support,0.0,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +43,Aaron Agulnek,Roman Catholic Archbishop of Boston,2024,193,House Bill,5077,making appropriations for the fiscal year 2024 to provide for supplementing certain existing appropriations and for certain other activities and projects,Neutral,0.0,AARON AGULNEK,ROMAN CATHOLIC ARCHBISHOP OF BOSTON +44,Aaron Agulnek,"MASSACHUSETTS DENTAL HYGIENISTS' ASSOCIATION , INC.",2024,193,House Bill,2216,An Act relative to dental hygienists,Support,0.0,AARON AGULNEK,MASSACHUSETTS DENTAL HYGIENISTS ASSOCIATION +45,Aaron Agulnek,"MASSACHUSETTS DENTAL HYGIENISTS' ASSOCIATION , INC.",2024,193,Senate Bill,1429,An Act relative to dental hygienists,Support,0.0,AARON AGULNEK,MASSACHUSETTS DENTAL HYGIENISTS ASSOCIATION +46,Aaron Agulnek,"MASSACHUSETTS DENTAL HYGIENISTS' ASSOCIATION , INC.",2024,193,House Bill,3880,An Act relative to dental hygienist reciprocal licensure,Neutral,0.0,AARON AGULNEK,MASSACHUSETTS DENTAL HYGIENISTS ASSOCIATION +47,Aaron Agulnek,"MASSACHUSETTS DENTAL HYGIENISTS' ASSOCIATION , INC.",2024,193,House Bill,2241,An Act enabling trained dental hygienists to administer dermal fillers and botox,Support,0.0,AARON AGULNEK,MASSACHUSETTS DENTAL HYGIENISTS ASSOCIATION +48,Aaron Agulnek,"MASSACHUSETTS DENTAL HYGIENISTS' ASSOCIATION , INC.",2024,193,Senate Bill,664,An Act relative to the sustainability of public health dental hygienists through adequate reimbursements,Support,0.0,AARON AGULNEK,MASSACHUSETTS DENTAL HYGIENISTS ASSOCIATION +49,Aaron Agulnek,"MASSACHUSETTS DENTAL HYGIENISTS' ASSOCIATION , INC.",2024,193,House Bill,2255,An Act relative to the sustainability of public health dental hygienists through adequate reimbursements,Support,0.0,AARON AGULNEK,MASSACHUSETTS DENTAL HYGIENISTS ASSOCIATION +50,Aaron Agulnek,"MASSACHUSETTS DENTAL HYGIENISTS' ASSOCIATION , INC.",2024,193,Senate Bill,1379,An Act to improve oral health for all Massachusetts residents,Support,0.0,AARON AGULNEK,MASSACHUSETTS DENTAL HYGIENISTS ASSOCIATION +51,Aaron Agulnek,"MASSACHUSETTS DENTAL HYGIENISTS' ASSOCIATION , INC.",2024,193,House Bill,5100,An Act relative to strengthening Massachusetts' economic leadership,Neutral,0.0,AARON AGULNEK,MASSACHUSETTS DENTAL HYGIENISTS ASSOCIATION +52,Aaron Agulnek,"MASSACHUSETTS DENTAL HYGIENISTS' ASSOCIATION , INC.",2024,193,Senate Bill,1357,An Act enabling trained dental hygienists to administer dermal fillers and botox,Support,0.0,AARON AGULNEK,MASSACHUSETTS DENTAL HYGIENISTS ASSOCIATION +53,Aaron Agulnek,Bryn Austin,2024,193,House Bill,2215,An Act protecting children from harmful diet pills and muscle-building supplements,Support,0.0,AARON AGULNEK,BRYN AUSTIN +54,Aaron Agulnek,Bryn Austin,2024,193,House Bill,5159,An Act enhancing the market review process,Neutral,0.0,AARON AGULNEK,BRYN AUSTIN +55,Aaron Agulnek,Southwest Boston Senior Services,2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,SOUTHWEST BOSTON SENIOR SERVICES +56,Aaron Agulnek,"The Mass Mentoring Partnership, Inc.",2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,MASS MENTORING PARTNERSHIP +57,Aaron Agulnek,Mass Home Care Association,2024,193,House Bill,4799,An Act making appropriations for the fiscal year 2024 to provide for supplementing certain existing appropriations and for certain other activities and projects,Neutral,0.0,AARON AGULNEK,MASS HOME CARE ASSOCIATION +58,Aaron Agulnek,Mass Home Care Association,2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,MASS HOME CARE ASSOCIATION +59,Aaron Agulnek,Mass Home Care Association,2024,193,House Bill,5077,making appropriations for the fiscal year 2024 to provide for supplementing certain existing appropriations and for certain other activities and projects,Neutral,0.0,AARON AGULNEK,MASS HOME CARE ASSOCIATION +60,Aaron Agulnek,"Metropolitan Council for Educational Opportunity (METCO), Inc.",2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,METROPOLITAN COUNCIL FOR EDUCATIONAL OPPORTUNITY (METCO) +61,Aaron Agulnek,"Ascentria Care Alliance, Inc.",2024,193,House Bill,4442,An Act creating a special legislative commission to study barriers to job retention for low-income workers,Support,0.0,AARON AGULNEK,ASCENTRIA CARE ALLIANCE +62,Aaron Agulnek,"Ascentria Care Alliance, Inc.",2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,ASCENTRIA CARE ALLIANCE +63,Aaron Agulnek,"Beacon ABA Services, Inc.",2024,193,Senate Bill,2475,An Act relative to the licensure of behavior analysts,Support,0.0,AARON AGULNEK,BEACON ABA SERVICES +64,Aaron Agulnek,"Criterion Child Enrichment, Inc.",2024,193,Senate Bill,2475,An Act relative to the licensure of behavior analysts,Support,0.0,AARON AGULNEK,CRITERION CHILD ENRICHMENT +65,Aaron Agulnek,"Evergreen Center, Inc.",2024,193,Senate Bill,2475,An Act relative to the licensure of behavior analysts,Support,0.0,AARON AGULNEK,EVERGREEN CENTER +66,Aaron Agulnek,Southwest Boston Senior Services,2024,193,Senate Bill,376,An Act relative to Massachusetts home care eligibility,Support,0.0,AARON AGULNEK,SOUTHWEST BOSTON SENIOR SERVICES +67,Aaron Agulnek,Jewish Community Relations Council of Boston,2024,193,House Bill,4799,An Act making appropriations for the fiscal year 2024 to provide for supplementing certain existing appropriations and for certain other activities and projects,Neutral,0.0,AARON AGULNEK,JEWISH COMMUNITY RELATIONS COUNCIL OF BOSTON +68,Aaron Agulnek,Jewish Community Relations Council of Boston,2024,193,House Bill,5077,making appropriations for the fiscal year 2024 to provide for supplementing certain existing appropriations and for certain other activities and projects,Neutral,0.0,AARON AGULNEK,JEWISH COMMUNITY RELATIONS COUNCIL OF BOSTON +69,Aaron Agulnek,Jewish Community Relations Council of Boston,2024,193,Senate Bill,1244,An Act building resiliency in communities through mental health,Support,0.0,AARON AGULNEK,JEWISH COMMUNITY RELATIONS COUNCIL OF BOSTON +70,Aaron Agulnek,Jewish Community Relations Council of Boston,2024,193,House Bill,1985,An Act building resiliency in communities through mental health,Support,0.0,AARON AGULNEK,JEWISH COMMUNITY RELATIONS COUNCIL OF BOSTON +71,Aaron Agulnek,Jewish Community Relations Council of Boston,2024,193,Senate Bill,345,An Act to update accuracy of membership to the state advisory council on early education and care,Support,0.0,AARON AGULNEK,JEWISH COMMUNITY RELATIONS COUNCIL OF BOSTON +72,Aaron Agulnek,"Massachusetts Law Reform Institute, Inc.",2024,193,House Bill,135,An Act establishing basic needs assistance for Massachusetts immigrant residents,Support,0.0,AARON AGULNEK,MASSACHUSETTS LAW REFORM INSTITUTE +73,Aaron Agulnek,"Massachusetts Law Reform Institute, Inc.",2024,193,Senate Bill,75,An Act to lift kids out of deep poverty,Support,0.0,AARON AGULNEK,MASSACHUSETTS LAW REFORM INSTITUTE +74,Aaron Agulnek,"Massachusetts Law Reform Institute, Inc.",2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,MASSACHUSETTS LAW REFORM INSTITUTE +75,Aaron Agulnek,"Massachusetts Law Reform Institute, Inc.",2024,193,House Bill,144,An Act to lift kids out of deep poverty,Support,0.0,AARON AGULNEK,MASSACHUSETTS LAW REFORM INSTITUTE +76,Aaron Agulnek,"Massachusetts Service Alliance, Inc.",2024,193,House Bill,5077,making appropriations for the fiscal year 2024 to provide for supplementing certain existing appropriations and for certain other activities and projects,Neutral,0.0,AARON AGULNEK,MASSACHUSETTS SERVICE ALLIANCE +77,Aaron Agulnek,"Massachusetts Service Alliance, Inc.",2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,MASSACHUSETTS SERVICE ALLIANCE +78,Aaron Agulnek,"Massachusetts Service Alliance, Inc.",2024,193,Senate Bill,1890,An Act excluding the Segal AmeriCorps Education Award from taxable income,Support,0.0,AARON AGULNEK,MASSACHUSETTS SERVICE ALLIANCE +79,Aaron Agulnek,"Massachusetts Service Alliance, Inc.",2024,193,House Bill,4799,An Act making appropriations for the fiscal year 2024 to provide for supplementing certain existing appropriations and for certain other activities and projects,Neutral,0.0,AARON AGULNEK,MASSACHUSETTS SERVICE ALLIANCE +80,Aaron Agulnek,"Massachusetts Service Alliance, Inc.",2024,193,House Bill,5100,An Act relative to strengthening Massachusetts' economic leadership,Neutral,0.0,AARON AGULNEK,MASSACHUSETTS SERVICE ALLIANCE +81,Aaron Agulnek,Person_not_a_Company,2024,193,House Bill,1748,An Act relative to access to a decedent's electronic mail accounts,Support,0.0,AARON AGULNEK,PERSON_NOT_A_COMPANY +82,Aaron Agulnek,Person_not_a_Company,2024,193,Senate Bill,930,An Act relative to access to a decedent's electronic mail accounts,Support,0.0,AARON AGULNEK,PERSON_NOT_A_COMPANY +83,Aaron Agulnek,"Massachusetts Fair Housing Center, Inc.",2024,193,Senate Bill,1362,An Act relative to the Massachusetts lead law and promoting equal access to lead-free housing,Support,0.0,AARON AGULNEK,MASSACHUSETTS FAIR HOUSING CENTER +84,Aaron Agulnek,"Massachusetts Fair Housing Center, Inc.",2024,193,House Bill,2223,An Act relative to the Massachusetts lead law and promoting equal access to lead-free housing,Support,0.0,AARON AGULNEK,MASSACHUSETTS FAIR HOUSING CENTER +85,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,193,Senate Bill,1156,An Act protecting an employee's right to rebuttal of personnel records,Support,0.0,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +86,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,193,House Bill,1881,An Act clarifying the wage act to protect hospital workers from being deprived of their wages,Support,0.0,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +87,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,193,House Bill,5100,An Act relative to strengthening Massachusetts' economic leadership,Neutral,0.0,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +88,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,193,House Bill,1891,An Act clarifying the prevailing wage act to protect actions by the attorney general,Support,0.0,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +89,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,193,House Bill,1088,An Act relative to greater fairness in insurance,Support,0.0,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +90,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,193,Senate Bill,1161,An Act clarifying the prevailing wage act to protect actions by the attorney general,Support,0.0,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +91,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,193,Senate Bill,1177,An Act clarifying the wage act to protect hospital workers from being deprived of their wages,Support,0.0,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +92,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,193,House Bill,4282,An Act relative to defense against abusive waivers,Support,0.0,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +93,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,193,House Bill,1892,An Act relative to anti-retaliation protections for public employees,Support,0.0,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +94,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,193,House Bill,1886,An Act protecting an employee's right to rebuttal of personnel records,Support,0.0,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +95,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,193,House Bill,1709,An Act to further protect employees through a private right of action,Support,0.0,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +96,Aaron Agulnek,The Boston Debate League Incorporated,2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,BOSTON DEBATE LEAGUEORPORATED +97,Aaron Agulnek,"International Institute of New England, Inc",2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,INTERNATIONAL INSTITUTE OF NEW ENGLAND +98,Aaron Agulnek,"Massachusetts Association for Community Action, Inc.",2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,MASSACHUSETTS ASSOCIATION FOR COMMUNITY ACTION +99,Aaron Agulnek,"OPERATION ABLE OF GREATER BOSTON, INC.",2024,193,House Bill,4800,"An Act making appropriations for the fiscal year 2025 for the maintenance of the departments, boards, commissions, institutions, and certain activities of the commonwealth, for interest, sinking fund, and serial bond requirements, and for certain permanent improvements",Neutral,0.0,AARON AGULNEK,OPERATION ABLE OF GREATER BOSTON diff --git a/docs/data/MA_lobbying_bills_scored_sample.csv b/docs/data/MA_lobbying_bills_scored_sample.csv new file mode 100644 index 0000000..8a23302 --- /dev/null +++ b/docs/data/MA_lobbying_bills_scored_sample.csv @@ -0,0 +1,1167 @@ +,bill_number,general_court,bill_title,bill_id,env_relevance_score,is_environmental,cluster_id +0,537,186,An Act To Assure Equitable Heath Coverage For All Children,S537,-0.0040966272354125,0,18 +1,549,186,An Act Relative to Health Care Affordability,S549,-0.0452432036399841,0,18 +2,873,186,An Act Strengthening Health Reform,S873,-0.0194134712219238,0,1 +3,974,186,An Act to regulate health insurance contracts/policies.,,,, +,,,0,21,,, +4,1,186,"A message from His Excellency the Governor submitting the annual budget of the Commonwealth for the fiscal year beginning July first, two thousand nine",H1,-0.0527363419532775,0,4 +5,10,186,"and H4047 ""Transportation reform bills""",S10,-0.0087350606918334,0,13 +6,100,186,"HB3797 +SB92 +SB554 +SB555 +SB579 +Authority management issues +Ch. 27 of 2009",H100,-0.0107224583625793,0,13 +7,1000,186,"An Act Relative To Banks And Banking +HB1050 An Act Relative To The Safety Of Public Deposits +HB4800 An Act Making Appropriations For The Fiscal Year 2011 +SB513 An Act Relative To The Safety Of Public Deposits +HB4841 An Act Relative to Making Appropriations for the Fiscal years 2010 & 2011 +HB4868 An Act Relative to Making Appropriations for the Fiscal years 2010 & 2011 +HB5028 An Act Relative to Making Appropriations for the Fiscal years 2010 & 2011 +SB520 An Act To Include Credit Unions For Deposit Of Public Monies +SB1783 An Act Providing For The Uniform Prudent Management Of Institutional Funds",S1000,-0.0248819589614868,0,13 +8,1002,186,An Act relative to policies of insurance.,,,, +,,,0,21,,, +9,1003,186,An Act relative to viatical settlements .,,,, +,,,0,21,,, +10,1004,186,An Act relative to medical examinations.,,,, +,,,0,1,,, +11,101,186,"HB102 HB104 HB106 HB1179 HB1194 HB1195 HB119 +6 HB1197 HB1198 HB1201 HB1202 HB1208 HB1219 HB1223 +HB1226 HB1227 HB1231 HB1232 HB1236 HB1238 HB1239 +HB1242 HB1245 HB1310 HB1425 HB1455 HB1649 HB1654 H +B1697 HB1700 HB1728 HB1779 HB1794 HB1809 HB1844 HB +1868 HB1982 HB2001 HB2218 HB2263 HB2668 HB2712 HB2 +728 HB2755 HB2767 HB277 HB2835 HB2847 HB2933 HB294 +0 HB2958 HB2962 HB3083 HB309 HB310 HB3124 HB3452 H +B3454 HB3457 HB3459 HB3474 HB3569 HB3572 HB3575 HB +3583 HB3588 HB3623 HB3651 HB3683 HB3684 HB3686 HB3 +689 HB3781 HB3810 HB3812 HB3817 HB3821 HB3823 HB38 +56 HB3859 HB3921 HB395 HB3956 HB4043 HB4072 HB4084 HB4088 HB4100 HB4101 +HB4106 HB787 HB811 HB858 HD2026 HD2267 HB1221 HB75 +0 HB2940 SB100 SB121 SB1316 SB1528 SB1533 SB1546 +SB1609 SB168 SB1719 SB173 SB1791 SB1813 SB1847 SB1 +87 SB2052 SB226 SB3 SB455 SB6 SB634 SB648 SB651 SB +667 SB670 SB694 SB718 SB751 SB765 SB766 SB768 SB77 +7 SB90 SD632 SB343 SB441 SB2060 SB1373",H101,-0.0451977252960205,0,13 +12,1012,186,"HB1035 +HB1044 +HB1063 +HB1067 +HB1076 +HB1077 +HB1079 +HB108 +HB1085 +HB1087HB1088 +HB1092 +HB1093 +HB1259 +HB1332 +HB1388 +HB1389HB1452 +HB1520 +HB1521 +HB1696 +HB1813 +HB1844 +HB188 +HB1923 +HB1928 +HB1932 +HB1939HB1940 +HB1950 +HB2054 +HB2059 +HB2069 +HB2073 +HB2085HB2086 +HB2100 +HB2102 +HB2111 +HB2126 +HB2128 +HB2181 +HB22 +HB2806 +HB2807 +HB3021HB3595 +HB3597 +HB3599 +HB3614 +HB3629 +HB3717 +HB3760 +HB3779 +HB3920 +HB4258 +HB4271 +HB4275 +HB4438 +HB4441 +HB4459 +HB4513 +HB4744 +HB4747HB656 +HB972 +HB974 +HB978SB1262 +SB1299 +SB1615 +SB1752 +SB1755 +SB1779 +SB1780 +SB1783 +SB1794 +SB18 +SB2028SB2208 +SB2257 +SB2314 +SB2322 +SB2383 +SB313 +SB317 +SB318 +SB39 +SB457 +SB458 +SB480SB521 +SB534 +SB537 +SB545SB551 +SB553 +SB554 +SB556 +SB557 +SB558 +SB560 +SB561 +SB571 +SB573 +SB574 +SB577 +SB588 +SB609 +SB670 +SB807 +SB810 +SB811 +SB813 +SB815 +SB833SB834 +SB835 +SB837 +SB852SB853 +SB854 +SB866 +SB871 +SB873 +SB876 +SB897 +SB950",H1012,-0.0149142146110534,0,13 +13,1020,186,"HB1028 +HB1031 +HB1032 +HB1033 +HB1034 +HB1035 +HB1036 +HB1038 +HB1044 +HB1056 +HB1064 +HB1065 +HB1077 +HB1079 +HB109 +HB1091 +HB1092HB1093 +HB1097 +HB110 +HB1101 +HB1102 +HB1107 +HB1526 +HB1924 +HB1926 +HB1949 +HB1950HB1956 +HB2111 +HB2185 +HB2509 +HB3451 +HB3558 +HB3559HB3563 +HB3598 +HB3809 +HB3889 +HB3897 +HB4138 +HB4477 +HB4490 +HB860 +HB865 +HB879 +HB885 +HB898 +HB905 +HB908 +HB910 +HB914 +HB915 +HB921 +HB923 +HB930 +HB931 +HB932 +HB933 +HB936 +HB937 +HB938 +HB939HB939 +HB948 +HB950 +HB962HB972 +HB974 +HB976 +HB977 +Line Item: 4100-0060 +Line Item: 4513-1020 +SB1038 +SB1674 +SB1675 +SB17 +SB1737 +SB1738SB1745 +SB1747 +SB1752 +SB19SB2099 +SB2156 +SB2223 +SB2271 +SB2437 +SB316 +SB324 +SB327 +SB329 +SB330 +SB445 +SB448 +SB449 +SB451 +SB457 +SB458 +SB463 +SB465 +SB467 +SB469SB470 +SB473 +SB478 +SB482SB485 +SB493 +SB496 +SB501 +SB508 +SB516 +SB518 +SB523 +SB528 +SB529 +SB532 +SB541 +SB547 +SB549 +SB551 +SB553 +SB558 +SB560 +SB563 +SB585 +SB588SB77 +SB835 +SB859 +SB861 +Line item: 4580-1000 +DOI Small Group Regulations +Medicaid Pharmacy Carve-out",H1020,-0.0142633318901062,0,13 +14,1024,186,An Act banning the use of certain socio-economic factors for insurance underwriting and rating of motor vehicle liability insurance.,,,, +,,,0,21,,, +15,1028,186,An Act to improve asthma management,H1028,0.0095524191856384,0,18 +16,103,186,An Act transferring county sheriffs to the Commonwealth.,,,, +,,,0,4,,, +17,1032,186,"HB1044 +HB1046 +HB1056 +HB1186 +HB1255 +HB1265 +HB1273 +HB1323 +HB1324 +HB1327 +HB133 +HB1338 +HB1348 +HB1360 +HB1372 +HB1373 +HB1404HB1405 +HB1434 +HB1440 +HB1446 +HB1456 +HB1487 +HB1506 +HB1581 +HB1590 +HB1592 +HB1659 +HB168 +HB1689 +HB17 +HB1712 +HB1780 +HB1794 +HB1795 +HB1797 +HB1799 +HB18 +HB1803 +HB1805 +HB1807 +HB1808 +HB1814 +HB1815 +HB1816 +HB1817 +HB1822 +HB1828 +HB1830 +HB1831HB1833 +HB1834 +HB1835 +HB1836 +HB1838 +HB1839 +HB1842HB1846 +HB1847 +HB1848 +HB1853 +HB1859 +HB1860 +HB1861 +HB1863 +HB1864 +HB1865 +HB1866 +HB1867 +HB1868 +HB1870 +HB1880 +HB19 +HB1926 +HB1936 +HB1937 +HB1941 +HB1942 +HB1948 +HB1952 +HB1953 +HB1963 +HB198 +HB199 +HB1992 +HB20 +HB200 +HB2005 +HB202 +HB207 +HB208 +HB21 +HB212 +HB2189 +HB2208 +HB2239 +HB2240 +HB2270 +HB228 +HB2297 +HB2474 +HB2592HB2666 +HB2667 +HB2673 +HB2674 +HB2676 +HB2678 +HB2694HB2700 +HB2702 +HB2706 +HB2707 +HB2708 +HB2721 +HB2731 +HB2735 +HB2742 +HB2743 +HB2744 +HB2745 +HB2746 +HB2748 +HB2751 +HB2754 +HB2763 +HB2777HB2778 +HB2792 +HB2793 +HB2799 +HB2807 +HB2812 +HB2818HB282 +HB2823 +HB2826 +HB2836 +HB2837 +HB2846 +HB2848 +HB2852 +HB2864 +HB2884 +HB2886HB2891 +HB2893 +HB2900 +HB2901 +HB2902 +HB2903 +HB2904HB2905 +HB2918 +HB2920 +HB2931 +HB2943 +HB2960 +HB2989 +HB2998 +HB3001 +HB3021 +HB3033 +HB3034 +HB3079 +HB31 +HB3124 +HB3125 +HB3188 +HB3215 +HB3249 +HB325 +HB3313 +HB3323HB3324 +HB3330 +HB3387 +HB3440 +HB3443 +HB3484 +HB3485HB3515 +HB3517 +HB3523 +HB3526 +HB3534 +HB3544 +HB3545 +HB3547 +HB3556 +HB3563 +HB3578 +HB3579 +HB3580 +HB3581 +HB3582 +HB3583 +HB3586 +HB3587HB3588 +HB3607 +HB3615 +HB3648 +HB3649 +HB3675 +HB3693HB3694 +HB3727 +HB3743 +HB3750 +HB3754 +HB3760 +HB3809 +HB3825 +HB3842 +HB3881 +HB3973 +HB3990 +HB4056 +HB4072 +HB4076 +HB4090 +HB4094 +HB4107HB4123 +HB4127 +HB4129 +HB4133 +HB4138 +HB4142 +HB4143HB4159 +HB4163 +HB4164 +HB4166 +HB4167 +HB4181 +HB4201 +HB4207 +HB4253 +HB4266 +HB4271 +HB4288 +HB4293 +HB4359 +HB4373 +HB4374 +HB4410 +HB66 +HB665 +HB713 +HB730 +HB757 +HB759 +HB778 +HB785 +HB803 +HB808 +HB831 +HB833 +HB837 +HB879 +HB893 +HB905 +HB907 +HB913HB92 +HB921 +HB931 +HB932 +HB938 +HB950 +HB951 +HB971 +HB974 +HB976 +HB977 +HB978 +HD4442 +HD4467 +SB103 +SB1048 +SB1083 +SB1089 +SB1225 +SB123SB1237 +SB1238 +SB124 +SB1240 +SB1241 +SB1245 +SB1246 +SB1247 +SB1265 +SB1279 +SB1299 +SB1304 +SB1309 +SB133 +SB1333 +SB1344 +SB1347 +SB1357SB1373 +SB1376 +SB1377 +SB1382 +SB1383 +SB1385 +SB1387 +SB1404 +SB1412 +SB1426 +SB1435 +SB1448 +SB1453 +SB1464 +SB1487 +SB1489 +SB1524 +SB1529SB155 +SB1552 +SB1571 +SB1591 +SB1595 +SB1608 +SB1611 +SB1652 +SB1659 +SB1671 +SB1677 +SB1678 +SB1679 +SB1686 +SB1709 +SB1721 +SB1723 +SB1729SB174 +SB1741 +SB1762 +SB1776 +SB1860 +SB1885 +SB1939 +SB1964 +SB1994 +SB2028 +SB2042 +SB2061 +SB2117 +SB2170 +SB2183 +SB2195 +SB2201 +SB2203SB2210 +SB2216 +SB313 +SB316 +SB323 +SB404 +SB42 +SB432SB442 +SB445 +SB449 +SB457 +SB463 +SB465 +SB469 +SB473 +SB478 +SB496 +SB498 +SB501 +SB51 +SB516 +SB517 +SB521 +SB523SB53 +SB554 +SB58 +SB609 +SB625 +SB659 +SB668 +SB671 +SB673 +SB677 +SB678 +SB680 +SB681 +SB682 +SB687 +SB688 +SB689SB690 +SB695 +SB696 +SB698SB706 +SB710 +SB711 +SB718 +SB719 +SB720 +SB724 +SB725 +SB726 +SB727 +SB729 +SB732 +SB733 +SB735 +SB758 +SB77 +SB779SB787 +SB788 +SB792 +SB86 +SB87 +SB915 +SB924 +SB940 +SB962 +SB963 +SB98 +SB988",H1032,-0.0183512568473815,0,13 +18,1034,186,An Act Relative To Insurance Coverage For Craniofacial Disorders,H1034,-0.0319668054580688,0,18 +19,1035,186,An Act relative to pre-authorization of medical and health care services.,,,, +,,,0,18,,, +20,1038,186,An Act to include dental benefits in all Commonwealth Care Plans,S1038,-0.0615882873535156,0,18 +21,104,186,An Act strengthening the Commonwealth’s partnership with municipalities.,,,, +,,,0,5,,, +22,1042,186,An Act relative to dental insurance for certain retired teachers.,S1042,-0.0265950560569763,0,12 +23,1044,186,- Widows,S1044,-0.0340147018432617,0,12 +24,1046,186,"H168 +H1079 +S446, H3452 +S529 +S478, H3809, S501, H923 +H2024 +H4129, H4139, H4101, S2060, H4100, H1, S3, S2053 +S2054, S2055, S2060 +H1091 +S757, H3586 +S868 +Connector Small Group Pilot +S867 +H1107 +Cost Trend Hearings +H2118 +S810 +Disparities Council +S476, S475 +EHR Implementation +H1106, S526, S542, S591 +S1299, H2807 +S485 +Intermediate Level of Care bulletin +S541 +S514, H1042, H3678, H1047, H3679 +956 CMR 5.00 +H991, S456 +H104, H3486 +H879, H908, H910, H933, S469, S473, H977, S496 +S518, S451, S523, S921, H951, H1031, S465, H939 +S448, H932, S449, H1032, S463, H950, H1056, H898 +S445, S516 +S455, H992 +H919, H920 +211 CMR 52 +S553 +H108, S18 +payment reform +S528, S968 +S869, H2109 +H974 +Provider Termination Consumer Advisory Bulletin +129 CMR 3.00 +114.6 CMR 3.00 +129 CMR 2.00 +Reserves Assessment +Reserves Study +H1080 +Small Group Rates +S495 +105 CMR 130, 105 CMR 140 +H1077 +H971 +H3453, 114.6 CMR 14.00 +H2343",H1046,-0.0173439979553222,0,13 +25,1047,186,An Act relative to health reimbursement accounts.,S1047,-0.0095735788345336,0,18 +26,1054,186,"HB3240 +HB3639 +HB3640 +SB1935",H1054,-0.0089887380599975,0,13 +27,1059,186,An Act relative to structured settlements and reserves.,,,, +,,,0,21,,, +28,106,186,HB274 HB294 HB302 HB322 HB330 HB334 HB1856 HB2234 HB3015 HB3016 HB4062 HB4063 HB4064 HB4065 HB4066 HB4068 HB4081 HB4082 HD1850 SB11 SB85 SB140 SB145 SB157 SB164 SB174 SB184 SB1361 SB2035 SB2037 SB2038 SB2039 SB2040 SB2041 SB2042,,-0.0289129018783569,0,13 +29,1060,186,An Act relative to annuities.,,,, +,,,0,0,,, +30,1061,186,"An Act reforming the process of appraising motor vehicle damage covered by auto insurance policies +HB3207 An Act providing additional consumer protections in the repair of damaged motor vehicles +SB122 An Act relative to auto body labor rates +SB124 An Act creating a civil legal action to be enforced by the attorney general of the Commonwealth to protect an individual's right to repair an automobile +SB185 An Act to create a board of registration of automotive mechanics +SB191 An Act providing protections against fraud in the repair of damaged vehicles +SB459 An Act to facilitate the resolution of auto insurance claims in emergencies +SB460 An Act to further define unfair and deceptive acts and practices in the business of insurance +SB522 An Act reducing automobile insurance costs and premiums in the Commonwealth +SB528 An Act relative to insurance companies and quality measures + +""Attorney Martin Corry, Jr., & Corry Associates LLC has been retained by Alliance of Automotive Service Providers MA/RI to represent the interests of Alliance of Automotive Service Providers MA/RI for the months January-June, 2010. This retainer is for services rendered and has been disclosed as required by law. No payment has been made for any individual legislative petition, regardless of the position taken by the client & Attorney Martin Corry, Jr., & Corry Associates LLC on behalf of said client.""",H1061,-0.0183841586112976,0,13 +31,1063,186,An Act making strategic Investments in the commonwealth’s chronically distressed hospitals program.,,,, +,,,0,4,,, +32,1064,186,"HB1816 + +HB2020 + +HB3583 + +HB4129 + +SB455 + +SB457 + +SB475 + +SB557 + +SB562 + +SB568 + +SB1299 + +SB1467 + +SB1772",H1064,0.0004434585571289,0,4 +33,107,186,"HB1188 +HB1202 +HB1218 +HB1219 +HB1224 +HB1227 +HB1450 +HB1455 +HB1566 +HB1568HB1665 +HB1695 +HB1697 +HB1700 +HB1701 +HB1736 +HB1778HB1779 +HB1981 +HB1982 +HB2003 +HB2004 +HB2140 +HB2668 +HB2692 +HB2702 +HB2726 +HB2765 +HB2778 +HB2780 +HB2822 +HB2881 +HB2933 +HB2938 +HB2960HB3014 +HB3037 +HB3061 +HB3061 +HB3065 +HB3065 +HB3083HB3095 +HB310 +HB3124 +HB3308 +HB3362 +HB3364 +HB337 +HB3411 +HB3412 +HB3446 +HB3462HB3491 +HB3570 +HB3572 +HB3575 +HB3583 +HB3615 +HB3646 +HB3647 +HB3670 +HB3672 +HB3673 +HB3674 +HB3675 +HB3699 +HB3760 +HB3762 +HB3951 +HB4072HB702 +HB750 +HB756 +HB775HB782 +HB806 +HB808 +HB811 +HB818 +HB834 +HB849 +HB852 +HB853 +HB88 +HB960 +HD2622 +SB103 +SB104 +SB105 +SB1323 +SB1387 +SB1504 +SB1508 +SB1577SB1607 +SB168 +SB1912 +SB193SB1953 +SB1963 +SB2019 +SB205 +SB206 +SB2061 +SB2101 +SB373 +SB382 +SB386 +SB387 +SB388 +SB389 +SB394 +SB396 +SB411 +SB412 +SB436 +SB441 +SB6 +SB623 +SB625 +SB636 +SB653 +SB663 +SB765 +SB767 +SB770 +SB783 +SB817 +SB88 +SB882 +SB90SB96 +SB97 +SD1336 +Greenhouse gas policy +Energy conservation and decoupling +Building Code +Stormwater +Ch. 21E +Ch. 91 +MEPA +Highway Access Permit +MESA +Transportation reform",H107,0.0110623836517333,0,13 +34,1073,186,An Act Relative to a Temporary Suspension of the Minimum Creditable Coverage Requirements.,,,, +,,,0,18,,, +35,1077,186,An Act Relative to Minimizing Unnecessary Health Care Costs by Streamlining Administrative Requirements.,,,, +,,,0,18,,, +36,1079,186,An Act to further define adverse determinations by insurers.,,,, +,,,0,10,,, +37,108,186,HB241 HB1063 HB1067 HB1076 HB1686 HB1696 HB1869 HB1931 HB2026 HB2030 HB2031 HB2032 HB2069 HB2085 HB2086 HB2100 HB2109 HB2123 HB2164 HB2165 HB2173 HB2358 HB2481 HB2518 HB2574 HB3591 HB3912 SB18 SB79 SB231 SB259 SB508 SB531 SB557 SB561 SB569 SB571 SB578 SB579 SB587 SB615 SB798 SB799 SB803 SB814 SB831 SB838 SB841 SB844 SB845 SB854 SB855 SB856 SB857 SB866 SB867 SB869 SB872 SB876 SB886 SB887 SB888 SB890 SB898 SB899 SB900 SB937 SB988 SB1087 SB1093 SB1095 SB1120 SB1183 SB1218 SB1615 SB1753 SB1755 SB1757 SB1794 SB1937 SB2028 SD1956,H108,0.0008912682533264,0,13 +38,1080,186,An Act relative to manufacturers coupons and rebates.,,,, +,,,0,15,,, +39,1081,186,manufacturers coupons & rebates,H1081,-0.0412923693656921,0,15 +40,1082,186,"An Act improving MassHealth +HB1104 +An Act improving access to coverage for Medicaid beneficiaries",S1082,-0.0210204720497131,0,12 +41,1084,186,An Act relative to shared responsibility in health reform.,,,, +,,,0,18,,, +42,1087,186,An Act To Ensure Equitable Health Coverage For All Children,H1087,-0.0071159601211547,0,18 +43,1088,186,An Act relative to providing for review of proposed changes to health care provider mandates by the Division of Health Care Finance and Policy.,,,, +,,,0,18,,, +44,109,186,"An Act Prohibiting The Use Of Health Data For Marketing Purposes +HB110 An Act To Prevent The Sharing Of Prescription Data +SB17 An Act Relative To Data Mining +SB19 An Act Protecting The Confidentiality Of Patient Prescription Records",H109,0.0010385513305664,0,15 +45,1090,186,An Act relative to the retirement options of certain educational personnel.,S1090,-0.009455382823944,0,12 +46,1091,186,An Act relative to co-payments and deductibles for chronic diseases.,,,, +,,,0,18,,, +47,1092,186,An Act to extend patient protections to recipients of MassHealth .,,,, +,,,0,18,,, +48,1093,186,An Act authorizing health care professionals to negotiate with health care insurers and providing for the powers and duties of the attorney general.,,,, +,,,0,1,,, +49,1095,186,An Act relative to credible service for school nurses.,S1095,-0.0203610658645629,0,12 +50,1098,186,An Act Relative to prescription drugs.,,,, +,,,0,15,,, +51,11,186,Budget Line Item 4800-0038,,-0.0197784900665283,0,4 +52,110,186,"An Act relative to continuing education requirements for operators of hoisting machinery. +",S110,0.0093894004821777,0,0 +53,1100,186,An Act Relative To Equitable Dental Reimbursement Rates For Services Provided To Publicly Aided Patients,H1100,-0.042680025100708,0,18 +54,1101,186,An Act To Include Dental Benefits In All Commonwealth Care Plans,H1101,-0.0575893521308898,0,18 +55,1102,186,An Act relative to health care affordability.,H1102,-0.0501240491867065,0,18 +56,1103,186,"An Act requiring public notice prior to restricting MassHealth coverage +HB1950 +An Act to require equitable payment from the Commonwealth +BUDGET Language contained in line item 4000-0500 HB 2, HB 4600, HB 4601, HB 4800, SB 4 , SB 2470",H1103,-0.0158981084823608,0,18 +57,1104,186,"An Act improving access to coverage for Medicaid beneficiaries +HB3679 An Act repealing mandated benefits that are no longer effective +HB4799 An Act relative to increasing access to affordable health insurance coverage +HB4911 An Act relative to improving access to coverage for Medicaid beneficiaries +SB455 An Act increasing access to affordable health insurance coverage + +Bill #: HB978 An Act to streamline the physician credentialing process +Bill #: HB992 An Act increasing access to affordable health insurance coverage +Bill #: HB1082 An Act improving MassHealth +Bill #: HB1083 An Act improving the insurance partnership program +Bill #: HB1089 An Act relative to strengthening the determination of need process +Bill #: HB1104 An Act improving access to coverage for Medicaid beneficiaries +Bill #: HB1105 An Act relative to administrative simplification for MassHealth +Bill #: HB2954 An Act imposing a moratorium on mandated benefits +Bill #: HB3679 An Act repealing mandated benefits that are no longer effective +Bill #: HB3779 An Act relative to medical malpractice reform +Bill #: SB462 An Act to establish standards for long term care insurance +Bill #: SB494 An Act requiring standard credentialing of physicians by health insurers +Bill #: SB564 An act relating to equitable provider reimbursement +Bill #: SB565 An act relative to strengthening the DoN Program +Bill #: SB566 An act to promote electronic transmission of health care transactions",H1104,-0.0155457854270935,0,13 +58,1106,186,"An Act Relative to Prescription Drug Coverage. +",H1106,-0.0005543828010559,0,15 +59,1107,186,An Act to Promote Health Insurance Transparency .,,,, +,,,0,18,,, +60,1110,186,An Act to maintain faculty and college excellence in the Commonwealth.,,,, +,,,0,16,,, +61,1117,186,An Act limiting termination retirement allowances.,,,, +,,,0,12,,, +62,1141,186,A message from His Excellency the Governor (under Section 7H of Chapter 29 of the General Laws) submitting to the General Court recommended corrective amendments to his original budget submission,H1141,-0.0384519696235656,0,4 +63,1148,186,An Act Relative to the treatment of chronic lyme disease,H1148,0.0099892616271972,0,1 +64,115,186,"HB132 +HB135 +HB136 +HB137 +HB143 +HB144 +HB153 +HB154 +HB155 +HB156 +HB165 +HB177 +HB181 +HB182 +HB183 +HB346 +HB349 +HB379 +HB387 +HB388HB391 +HB392 +HB393 +HB405HB406 +HB411 +HB414 +HB415 +HB416 +HB418 +HB420 +HB422 +HB423 +HB425 +HB426 +HB436 +HB438 +HB445 +HB447 +HB452 +HB454 +HB458 +HB459 +HB468 +HB481HB484 +HB487 +HB489 +HB491HB492 +HB493 +HB496 +HB500 +HB501 +HB504 +HB505 +HB508 +HB512 +HB519 +HB521 +HB940 +HB1027 +HB1055 +HB1070 +HB1416HB1433 +HB1482 +HB1589 +HB1613 +HB1641 +HB1768 +HB1780 +HB1860 +HB1934 +HB1953 +HB2061 +HB2063 +HB2116 +HB2203 +HB2301 +HB2342 +HB2439 +HB2484HB2624 +HB2641 +HB2926 +HB3286 +HB3307 +HB3656 +HB3658HB3661 +HB3739 +HB3785 +HB3803 +HB3808 +HB4079 +HD1732 +HD2830 +SB45 +SB46 +SB47 +SB65SB70 +SB213 +SB222 +SB224 +SB227 +SB229 +SB230 +SB239 +SB251 +SB254 +SB263 +SB265 +SB267 +SB270 +SB278 +SB280 +SB292 +SB293 +SB501 +SB652 +SB716SB764 +SB779 +SB957 +SB976SB1039 +SB1453 +SB1608 +SB1741 +SB1948 +SB1959 +SD107",H115,-0.0162057280540466,0,13 +65,1151,186,H4664,H1151,0.0035618543624877,0,6 +66,1152,186,An Act Establishing a Renewable Portfolio Standard for New Baseload Low Cost Renewable Energy.,,,, +,,,0,20,,, +67,116,186,"HB129 +HB1255 +HB3857HB4047 +HB4129 +HB4155 +HB4160 +HB4181 +SB68 +SB81 +SB1671 +SB1741",H116,-0.0039823651313781,0,13 +68,1162,186,An Act relative to student fees.,,,, +,,,0,3,,, +69,1170,186,An Act making faculty sabbatical provisions at state colleges consistent with other higher education institutions.,,,, +,,,0,0,,, +70,1173,186,An Act allowing the use of credit cards to pay tuition; HB4768 An Act relative to the use of credit cards for payment to the Commonwealth; SB2275 An Act concerning financial literacy in schools;,S1173,-0.0194217562675476,0,13 +71,1174,186,An Act relative to the John and Abigail Adams Scholarship Program. ,,,, +,,,0,3,,, diff --git a/docs/data/MA_lobbying_employers_sample.csv b/docs/data/MA_lobbying_employers_sample.csv new file mode 100644 index 0000000..d06ee57 --- /dev/null +++ b/docs/data/MA_lobbying_employers_sample.csv @@ -0,0 +1,101 @@ +,entity_name,client_name,year,reg_type,compensation,entity_name_norm,client_name_norm +0,"21c, LLC","EDUCATION TRUST, INC. (THE)",2024,Lobbyist Entity,7500.0,21C,EDUCATION TRUST (THE) +1,"21c, LLC","Amplify Education, Inc.",2024,Lobbyist Entity,22500.0,21C,AMPLIFY EDUCATION +2,"21c, LLC","The Massachusetts Alliance for Early College, Inc.",2024,Lobbyist Entity,15000.0,21C,MASSACHUSETTS ALLIANCE FOR EARLY COLLEGE +3,"21c, LLC",Benjamin Franklin Cummings Institute of Technology,2024,Lobbyist Entity,12500.0,21C,BENJAMIN FRANKLIN CUMMINGS INSTITUTE OF TECHNOLOGY +4,"21c, LLC",Total salaries received,2024,Lobbyist Entity,57500.0,21C,TOTAL SALARIES RECEIVED +5,"21c, LLC","Artists For Humanity, Inc.",2024,Lobbyist Entity,12500.0,21C,ARTISTS FOR HUMANITY +6,"27 South Strategies, LLC","The Massachusetts International Festival of the Arts, Inc.",2024,Lobbyist Entity,24000.0,27 SOUTH STRATEGIES,MASSACHUSETTS INTERNATIONAL FESTIVAL OF ARTS +7,"27 South Strategies, LLC","Veterinary Emergency Group, LLC",2024,Lobbyist Entity,27000.0,27 SOUTH STRATEGIES,VETERINARY EMERGENCY GROUP +8,"27 South Strategies, LLC",Thrive Scholars,2024,Lobbyist Entity,21600.0,27 SOUTH STRATEGIES,THRIVE SCHOLARS +9,"27 South Strategies, LLC",Boston Center for Independent Living INC,2024,Lobbyist Entity,2500.0,27 SOUTH STRATEGIES,BOSTON CENTER FOR INDEPENDENT LIVING +10,"27 South Strategies, LLC","Guardian Community Trust, Inc.",2024,Lobbyist Entity,36000.0,27 SOUTH STRATEGIES,GUARDIAN COMMUNITY TRUST +11,"27 South Strategies, LLC","Gobrands, Inc",2024,Lobbyist Entity,30000.0,27 SOUTH STRATEGIES,GOBRANDS +12,"27 South Strategies, LLC",Total salaries received,2024,Lobbyist Entity,141100.0,27 SOUTH STRATEGIES,TOTAL SALARIES RECEIVED +13,Anthony Arthur Abdelahad,"Massachusetts Association of Vocational Administrators, Inc.",2024,Lobbyist,34999.98,ANTHONY ARTHUR ABDELAHAD,MASSACHUSETTS ASSOCIATION OF VOCATIONAL ADMINISTRATORS +14,Anthony Arthur Abdelahad,"Massachusetts Association of Insurance Agents, Inc.",2024,Lobbyist,67500.0,ANTHONY ARTHUR ABDELAHAD,MASSACHUSETTS ASSOCIATION OF INSURANCE AGENTS +15,Anthony Arthur Abdelahad,"USA Hauling & Recycling, Inc.",2024,Lobbyist,0.0,ANTHONY ARTHUR ABDELAHAD,USA HAULING AND RECYCLING +16,Anthony Arthur Abdelahad,"Lindy Farms of Connecticut, LLC",2024,Lobbyist,0.0,ANTHONY ARTHUR ABDELAHAD,LINDY FARMS OF CONNECTICUT +17,Anthony Arthur Abdelahad,"Greathorse, LLC",2024,Lobbyist,0.0,ANTHONY ARTHUR ABDELAHAD,GREATHORSE +18,Anthony Arthur Abdelahad,Massachusetts Municipal Wholesale Electric Company,2024,Lobbyist,70000.0,ANTHONY ARTHUR ABDELAHAD,MASSACHUSETTS MUNICIPAL WHOLESALE ELECTRIC COMPANY +19,Anthony Arthur Abdelahad,Construction Industries of Massachusetts Massachusetts Aggregate and Asphalt Pavement Division,2024,Lobbyist,24000.0,ANTHONY ARTHUR ABDELAHAD,CONSTRUCTION INDUSTRIES OF MASSACHUSETTS MASSACHUSETTS AGGREGATE AND ASPHALT PAVEMENT DIVISION +20,Anthony Arthur Abdelahad,"AMERICAN MEDICAL RESPONSE OF MASSACHUSETTS, INC.",2024,Lobbyist,59641.86,ANTHONY ARTHUR ABDELAHAD,AMERICAN MEDICAL RESPONSE OF MASSACHUSETTS +21,Anthony Arthur Abdelahad,"MANAFORT BROTHERS, INCORPORATED",2024,Lobbyist,14000.0,ANTHONY ARTHUR ABDELAHAD,MANAFORT BROTHERSORPORATED +22,Anthony Arthur Abdelahad,"Peckham Industries, Inc.",2024,Lobbyist,51000.0,ANTHONY ARTHUR ABDELAHAD,PECKHAM INDUSTRIES +23,Anthony Arthur Abdelahad,"MGM Resorts International Operations, Inc.",2024,Lobbyist,13125.0,ANTHONY ARTHUR ABDELAHAD,MGM RESORTS INTERNATIONAL OPERATIONS +24,Anthony Arthur Abdelahad,Massachusetts Alliance of Boys & Girls Clubs,2024,Lobbyist,60000.0,ANTHONY ARTHUR ABDELAHAD,MASSACHUSETTS ALLIANCE OF BOYS AND GIRLS CLUBS +25,Anthony Arthur Abdelahad,"Massachusetts Urgent Care Association, Inc",2024,Lobbyist,45833.35,ANTHONY ARTHUR ABDELAHAD,MASSACHUSETTS URGENT CARE ASSOCIATION +26,Anthony Arthur Abdelahad,Western New England University,2024,Lobbyist,22500.0,ANTHONY ARTHUR ABDELAHAD,WESTERN NEW ENGLAND UNIVERSITY +27,Anthony Arthur Abdelahad,"The Massachusetts Propane Association,Inc",2024,Lobbyist,60000.0,ANTHONY ARTHUR ABDELAHAD,MASSACHUSETTS PROPANE ASSOCIATIONINC +28,Anthony Arthur Abdelahad,"Community Music School of Springfield, Inc.",2024,Lobbyist,24999.96,ANTHONY ARTHUR ABDELAHAD,COMMUNITY MUSIC SCHOOL OF SPRINGFIELD +29,Anthony Arthur Abdelahad,Total salaries received,2024,Lobbyist,547600.15,ANTHONY ARTHUR ABDELAHAD,TOTAL SALARIES RECEIVED +30,Paul G. Afonso,"Eastern Salt Company, Inc.",2024,Lobbyist,35000.0,PAUL G AFONSO,EASTERN SALT COMPANY +31,Paul G. Afonso,"The Next Street MA, LLC",2024,Lobbyist,15000.0,PAUL G AFONSO,NEXT STREET MA +32,Paul G. Afonso,GardaWorld Federal Services LLC,2024,Lobbyist,30000.0,PAUL G AFONSO,GARDAWORLD FEDERAL SERVICES +33,Paul G. Afonso,PPL Services Corporation,2024,Lobbyist,30000.0,PAUL G AFONSO,PPL SERVICES CORPORATION +34,Paul G. Afonso,"American Physical Therapy Association of Massachusetts, Inc.",2024,Lobbyist,20000.0,PAUL G AFONSO,AMERICAN PHYSICAL THERAPY ASSOCIATION OF MASSACHUSETTS +35,Paul G. Afonso,Massachusetts Association of Nurse Anesthesiology,2024,Lobbyist,39000.0,PAUL G AFONSO,MASSACHUSETTS ASSOCIATION OF NURSE ANESTHESIOLOGY +36,Paul G. Afonso,"Massachusetts Association of Health Plans, Inc",2024,Lobbyist,57000.0,PAUL G AFONSO,MASSACHUSETTS ASSOCIATION OF HEALTH PLANS +37,Paul G. Afonso,Arbella Mutual Insurance Company,2024,Lobbyist,36000.0,PAUL G AFONSO,ARBELLA MUTUAL INSURANCE COMPANY +38,Paul G. Afonso,"ChargePoint, Inc.",2024,Lobbyist,21000.0,PAUL G AFONSO,CHARGEPOINT +39,Paul G. Afonso,"Form Energy, Inc.",2024,Lobbyist,39000.0,PAUL G AFONSO,FORM ENERGY +40,Paul G. Afonso,Applied Materials Inc.,2024,Lobbyist,45000.0,PAUL G AFONSO,APPLIED MATERIALS +41,Paul G. Afonso,Massachusetts Coalition for Suicide Prevention,2024,Lobbyist,7500.0,PAUL G AFONSO,MASSACHUSETTS COALITION FOR SUICIDE PREVENTION +42,Paul G. Afonso,Hazel Health Inc.,2024,Lobbyist,21000.0,PAUL G AFONSO,HAZEL HEALTH +43,Paul G. Afonso,"MGM Resorts International Operations, Inc.",2024,Lobbyist,30000.0,PAUL G AFONSO,MGM RESORTS INTERNATIONAL OPERATIONS +44,Paul G. Afonso,Tripadvisor Inc.,2024,Lobbyist,45000.0,PAUL G AFONSO,TRIPADVISOR +45,Paul G. Afonso,"Uber Technologies, Inc.",2024,Lobbyist,45000.0,PAUL G AFONSO,UBER TECHNOLOGIES +46,Paul G. Afonso,"Boehringer Ingelheim Pharmaceuticals, Inc.",2024,Lobbyist,27000.0,PAUL G AFONSO,BOEHRINGER INGELHEIM PHARMACEUTICALS +47,Paul G. Afonso,AbbVie Inc.,2024,Lobbyist,35000.0,PAUL G AFONSO,ABBVIE +48,Paul G. Afonso,Moderna Inc.,2024,Lobbyist,33000.0,PAUL G AFONSO,MODERNA +49,Paul G. Afonso,Vivid Seats,2024,Lobbyist,30000.0,PAUL G AFONSO,VIVID SEATS +50,Paul G. Afonso,The Hertz Corporation,2024,Lobbyist,3250.0,PAUL G AFONSO,HERTZ CORPORATION +51,Paul G. Afonso,JetBlue Airways Corporation,2024,Lobbyist,24000.0,PAUL G AFONSO,JETBLUE AIRWAYS CORPORATION +52,Paul G. Afonso,"GPR Product Development, LLC",2024,Lobbyist,24000.0,PAUL G AFONSO,GPR PRODUCT DEVELOPMENT +53,Paul G. Afonso,"Massachusetts Association of Public Insurance Adjusters, Inc.",2024,Lobbyist,20000.0,PAUL G AFONSO,MASSACHUSETTS ASSOCIATION OF PUBLIC INSURANCE ADJUSTERS +54,Paul G. Afonso,"Onyx Springfield Crossing, LLC",2024,Lobbyist,26225.0,PAUL G AFONSO,ONYX SPRINGFIELD CROSSING +55,Paul G. Afonso,Total salaries received,2024,Lobbyist,737975.0,PAUL G AFONSO,TOTAL SALARIES RECEIVED +56,Aaron Agulnek,"Massachusetts Affiliate of ACNM, Inc",2024,Lobbyist,0.0,AARON AGULNEK,MASSACHUSETTS AFFILIATE OF ACNM +57,Aaron Agulnek,"Adoptions With Love, Inc.",2024,Lobbyist,18725.0,AARON AGULNEK,ADOPTIONS WITH LOVE +58,Aaron Agulnek,Roman Catholic Archbishop of Boston,2024,Lobbyist,14000.0,AARON AGULNEK,ROMAN CATHOLIC ARCHBISHOP OF BOSTON +59,Aaron Agulnek,"Ascentria Care Alliance, Inc.",2024,Lobbyist,16321.5,AARON AGULNEK,ASCENTRIA CARE ALLIANCE +60,Aaron Agulnek,"BAY STATE BIRTH COALITION, INC.",2024,Lobbyist,0.0,AARON AGULNEK,BAY STATE BIRTH COALITION +61,Aaron Agulnek,"Beacon ABA Services, Inc.",2024,Lobbyist,3300.0,AARON AGULNEK,BEACON ABA SERVICES +62,Aaron Agulnek,The Boston Debate League Incorporated,2024,Lobbyist,9000.0,AARON AGULNEK,BOSTON DEBATE LEAGUEORPORATED +63,Aaron Agulnek,"Bottom Line, Inc.",2024,Lobbyist,12600.0,AARON AGULNEK,BOTTOM LINE +64,Aaron Agulnek,Bryn Austin,2024,Lobbyist,8000.0,AARON AGULNEK,BRYN AUSTIN +65,Aaron Agulnek,"Criterion Child Enrichment, Inc.",2024,Lobbyist,7620.0,AARON AGULNEK,CRITERION CHILD ENRICHMENT +66,Aaron Agulnek,Southwest Boston Senior Services,2024,Lobbyist,13200.0,AARON AGULNEK,SOUTHWEST BOSTON SENIOR SERVICES +67,Aaron Agulnek,"Evergreen Center, Inc.",2024,Lobbyist,7400.0,AARON AGULNEK,EVERGREEN CENTER +68,Aaron Agulnek,GPS Group Peer Support LLC,2024,Lobbyist,0.0,AARON AGULNEK,GPS GROUP PEER SUPPORT +69,Aaron Agulnek,"Guardian Community Trust, Inc.",2024,Lobbyist,37800.0,AARON AGULNEK,GUARDIAN COMMUNITY TRUST +70,Aaron Agulnek,Healthy Children Project Inc DBA ALPP,2024,Lobbyist,12250.0,AARON AGULNEK,HEALTHY CHILDREN PROJECT DBA ALPP +71,Aaron Agulnek,"International Institute of New England, Inc",2024,Lobbyist,16387.5,AARON AGULNEK,INTERNATIONAL INSTITUTE OF NEW ENGLAND +72,Aaron Agulnek,Jewish Community Relations Council of Boston,2024,Lobbyist,30000.0,AARON AGULNEK,JEWISH COMMUNITY RELATIONS COUNCIL OF BOSTON +73,Aaron Agulnek,"MASSACHUSETTS ASSOCIATION OF PHYSICIAN ASSOCIATES, INC.",2024,Lobbyist,16200.0,AARON AGULNEK,MASSACHUSETTS ASSOCIATION OF PHYSICIAN +74,Aaron Agulnek,MA Coalition to Prevent Gun Violence,2024,Lobbyist,5818.75,AARON AGULNEK,MA COALITION TO PREVENT GUN VIOLENCE +75,Aaron Agulnek,"Massachusetts Fair Housing Center, Inc.",2024,Lobbyist,12300.0,AARON AGULNEK,MASSACHUSETTS FAIR HOUSING CENTER +76,Aaron Agulnek,Mass Home Care Association,2024,Lobbyist,25376.43,AARON AGULNEK,MASS HOME CARE ASSOCIATION +77,Aaron Agulnek,"The Mass Mentoring Partnership, Inc.",2024,Lobbyist,13300.0,AARON AGULNEK,MASS MENTORING PARTNERSHIP +78,Aaron Agulnek,"Massachusetts Service Alliance, Inc.",2024,Lobbyist,19128.0,AARON AGULNEK,MASSACHUSETTS SERVICE ALLIANCE +79,Aaron Agulnek,"MASSACHUSETTS DENTAL HYGIENISTS' ASSOCIATION , INC.",2024,Lobbyist,15600.0,AARON AGULNEK,MASSACHUSETTS DENTAL HYGIENISTS ASSOCIATION +80,Aaron Agulnek,"MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION, INC.",2024,Lobbyist,14276.1,AARON AGULNEK,MASSACHUSETTS EMPLOYMENT LAWYERS ASSOCIATION +81,Aaron Agulnek,"Metropolitan Council for Educational Opportunity (METCO), Inc.",2024,Lobbyist,16800.0,AARON AGULNEK,METROPOLITAN COUNCIL FOR EDUCATIONAL OPPORTUNITY (METCO) +82,Aaron Agulnek,"Massachusetts Immigrant and Refugee Advocacy Coalition, Inc.",2024,Lobbyist,17500.0,AARON AGULNEK,MASSACHUSETTS IMMIGRANT AND REFUGEE ADVOCACY COALITION +83,Aaron Agulnek,"National Association of Social Workers, Inc.",2024,Lobbyist,6000.0,AARON AGULNEK,NATIONAL ASSOCIATION OF SOCIAL WORKERS +84,Aaron Agulnek,National Consumer Law Center Inc.,2024,Lobbyist,8000.0,AARON AGULNEK,NATIONAL CONSUMER LAW CENTER +85,Aaron Agulnek,"Boston Women's Health Book Collective, Inc",2024,Lobbyist,9106.24,AARON AGULNEK,BOSTON WOMENS HEALTH BOOK COLLECTIVE +86,Aaron Agulnek,OutstandingLife,2024,Lobbyist,10000.0,AARON AGULNEK,OUTSTANDINGLIFE +87,Aaron Agulnek,ParentChild+ Inc.,2024,Lobbyist,15900.0,AARON AGULNEK,PARENTCHILD+ +88,Aaron Agulnek,American Red Cross,2024,Lobbyist,12000.0,AARON AGULNEK,AMERICAN RED CROSS +89,Aaron Agulnek,Veterans Inc.,2024,Lobbyist,21000.0,AARON AGULNEK,VETERANS +90,Aaron Agulnek,Vanessa Colleran,2024,Lobbyist,0.0,AARON AGULNEK,VANESSA COLLERAN +91,Aaron Agulnek,"Greater Boston Legal Services, Inc.",2024,Lobbyist,4800.0,AARON AGULNEK,GREATER BOSTON LEGAL SERVICES +92,Aaron Agulnek,Samaritans Inc.,2024,Lobbyist,15000.0,AARON AGULNEK,SAMARITANS +93,Aaron Agulnek,"Massachusetts Association for Community Action, Inc.",2024,Lobbyist,28605.0,AARON AGULNEK,MASSACHUSETTS ASSOCIATION FOR COMMUNITY ACTION +94,Aaron Agulnek,"MA CAPE, Inc",2024,Lobbyist,12000.0,AARON AGULNEK,MA CAPE +95,Aaron Agulnek,Massachusetts Association of Criminal Defense Attorneys,2024,Lobbyist,5000.0,AARON AGULNEK,MASSACHUSETTS ASSOCIATION OF CRIMINAL DEFENSE ATTORNEYS +96,Aaron Agulnek,"Boston Arts Summer Institute, Inc",2024,Lobbyist,30000.0,AARON AGULNEK,BOSTON ARTS SUMMER INSTITUTE +97,Aaron Agulnek,"Massachusetts Law Reform Institute, Inc.",2024,Lobbyist,35000.0,AARON AGULNEK,MASSACHUSETTS LAW REFORM INSTITUTE +98,Aaron Agulnek,Person_not_a_Company,2024,Lobbyist,0.0,AARON AGULNEK,PERSON_NOT_A_COMPANY +99,Aaron Agulnek,"The Massachusetts International Festival of the Arts, Inc.",2024,Lobbyist,24000.0,AARON AGULNEK,MASSACHUSETTS INTERNATIONAL FESTIVAL OF ARTS diff --git a/docs/data/MA_lobbying_summary_links_sample.csv b/docs/data/MA_lobbying_summary_links_sample.csv new file mode 100644 index 0000000..22f6030 --- /dev/null +++ b/docs/data/MA_lobbying_summary_links_sample.csv @@ -0,0 +1,101 @@ +entity_name,year,summary_url,disc_url +"21c, LLC",2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToCIgVktzTowTHDxMh68dzxw=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgV37dyqQoUDisAp0HQj3O1pvbVeMcoDoacAMaRr2Mf99 +"21c, LLC",2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToCIgVktzTowTHDxMh68dzxw=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgXAU1mOSIPF3KjdioJzQK7GTBYZqIBleIjHEgHlgIwyR +"27 South Strategies, LLC",2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToLJGaPu8imTWWjtAcbtwXKw=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgcoLYtPytS6Q7w2M3NiYM53n7demZYWm18Dr0xTUMf5r +"27 South Strategies, LLC",2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToLJGaPu8imTWWjtAcbtwXKw=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgSXfpEfLke3lETpI5Y7C2e2ZM0l4wjjv9/832arPlMVR +Anthony Arthur Abdelahad,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToIZaI0RYElFBc0rF4Rmp3l0=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgaWNEJe/apgcdJzXHHB7EAFtcbyEbYn5gedl83dcomhZ +Anthony Arthur Abdelahad,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToIZaI0RYElFBc0rF4Rmp3l0=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgX4ZlWvTdlv0vC9lWfSD8xBev6weUSYRembZkGXq/wAw +Viviana Maria Abreu-Hernandez,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToM2HWp6YmFE+YgO96aP/A5I=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgZSPqXbo58KwzgYFGPnBNPd/utUZ2MOz/4DydO/+etzT +Nicholas Acquavito,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToHfiM0QaH5N/SlsPCxInyYY=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgdXzTBWuPOKsWr88jvo1uqpbSCKVcja8IQNZ21oepaN2 +Nicholas Acquavito,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToHfiM0QaH5N/SlsPCxInyYY=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgeAhpJRceda9+GC5qh19Tu3NgpsKYj5zIi3vP8/7SQk8 +Nicholas Adamopoulos,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToPlZbcNrYW4d+NHy8foCdug=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgTa0rHR+xuAgTsnk7DvffCxWlwjehga1XRda6Xkd8INn +Nicholas Adamopoulos,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToPlZbcNrYW4d+NHy8foCdug=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgfEQS+TMIy9RZrDGQAXzkTda3kZjhYTzYl8QH0K2sHDI +Susan L. Adams,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToM8kt7PGk1rFfzSJr+3NjaI=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgeI2+Z6mVkQfGSvtx36ohJ5mD2ESK451cWsr9Ics4LDr +Susan L. Adams,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToM8kt7PGk1rFfzSJr+3NjaI=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgar/Enj7muUqXMFxtT6DGy4MT+l3WzC4Hv9QMVN+eWSN +Danielle Adams,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToAsmQp3G5lbjlFjEDlXNctg=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgarrGBNmmP+XKS5LPYC209FsZyO17aECxP2LfYK98SBk +Danielle Adams,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToAsmQp3G5lbjlFjEDlXNctg=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXga9jSL+3c6bSmcJxArWt8Hx0p7Q3TTZRYQArqGrjbtf7 +DDC Advocacy LLC,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToCzVvJSzOSmwUD8ek+H6jaQ=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgR71KMTsVPocmGsLQ58jVmuBc9Z0+8EHhwq4ON2Oj4dD +Paul G. Afonso,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToBKdxfxjWqUjuQgrnj88sIc=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgQtUAPSKwYocfdm1SB+dnZK/fZ8BhNL8e24lWW+jN5vp +Paul G. Afonso,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToBKdxfxjWqUjuQgrnj88sIc=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgVTwhl+nT0vO3Bh9T5GPvd430+NfeAMeJx+JQ+HwKv1A +Vishakha Agarwal,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToPX5KwCTnjlwpzILeNTJt/E=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgU9C6Gm1BoXXC7h3Qi3Ytw7H2cL/71tD3BryDVEkrd7c +Vishakha Agarwal,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToPX5KwCTnjlwpzILeNTJt/E=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgZfJp7bKMEwozzt/7gnP3r9pI2zQKIHgUKxCaFq7S2NG +Aaron Agulnek,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToBo1TUa+fA7XnuVN/pIRGKY=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgY2hfOrszDnaRhjPXYGdJxw510I3FVGKqEw1T4GL8Nl3 +Aaron Agulnek,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToBo1TUa+fA7XnuVN/pIRGKY=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgc93PzPTcWSPkEUiMGcsYQVGhpWmJgUf7gax90rMDuiZ +Josie Ahlberg,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToJ4z+gvXdkqwiVHiQyEKPpw=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgYCwnYvAmsla/TJGEeQZuohPZTOoXQLZOM35T2Gt9Yb/ +Josie Ahlberg,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToJ4z+gvXdkqwiVHiQyEKPpw=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgXKQCiVIHYOkczE7g2snzpYzmxqm/lvuxk3Nm7Uknbqf +Kola Akindele,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToIvwrY/ijqeAMyTwUp/GBWA=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgeEErSGDW0Qwz/kLMSnq9EecApg8xFxPXqCykT5OFOkF +Kola Akindele,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToIvwrY/ijqeAMyTwUp/GBWA=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgVMUMB0NFesWJrFmbuQZja4YSbAyqsJ1q76Je+KtjeCb +Charles Thomas Alagero,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToJ8SQ5YJG/0pdBeauVXAb8E=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgV0/d3Bqm515tZVLQ+1LsUnusdDSVtkZmf34YrcIXDwZ +Charles Thomas Alagero,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToJ8SQ5YJG/0pdBeauVXAb8E=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgen0U1aXFk432L4xIzpqyaOx+vXQk75pFX2SwO+YXaNO +Allison Alaimo,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToKqx7iVbUonMeNELMPTPN64=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgXOzjJXTVaJxnfyGn7fQQ6ai4dllO9n2+LEIX2rBAi65 +Allison Alaimo,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToKqx7iVbUonMeNELMPTPN64=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgW5TYH6dhY6McJQ/gCnbyTB/r1iYH9Ky0IOBF6hFN4l0 +Pisano Alan,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToMHkRz163FG2iW5jwomK8oU=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgZY284YF/hnTGb5+XngeO6TlkzxDS18D9qwxSS6Rpaf1 +Pisano Alan,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToMHkRz163FG2iW5jwomK8oU=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgeAEsbQjH0Drr0ms+ZjelCX4ms0Pg6UaQhpw+lZtBY2S +David Albright,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToHdgt9z6H7lWNo/k23rXq0c=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgb+rNi8qlwHTnAYUkWRQtW+Y+0J6+j0SwtgEwp2kbs4q +David Albright,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToHdgt9z6H7lWNo/k23rXq0c=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgU0eZ/crnRS53avfwCANo1Y3dcmbJ3RlFXE9Ug9xNmnS +"Theodore J Aleixo, Jr.",2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToJUyyzPImGRoZCEexDHGqjE=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgSHXgRAHQTP3G5CBWUIJImJOMOFWIJQkPJEhGmkDrjkD +"Theodore J Aleixo, Jr.",2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToJUyyzPImGRoZCEexDHGqjE=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgU+IqBUhNGhT5ierubEulTqUTF78SfejLzUBBqMWZ7l5 +Elizabeth Palmer Alfred,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToLOWLCpNpc0QfDs5NxAGfiM=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgYB9llZ40wcDm1J5UhIYPvgaiH2BCkoJ0idvS8aJoeN+ +Elizabeth Palmer Alfred,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToLOWLCpNpc0QfDs5NxAGfiM=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgV7ecvFf27EuT780s7skOjSelfsOGrmZIgFlXVXW+Ikw +Raza Ali,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToCo1v5brs/2Dq8PR8rFYHzU=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgVZyrGNKyBjX/IGJy4kd6fXssQWXp9RttjmVcuS1Rk+M +Raza Ali,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToCo1v5brs/2Dq8PR8rFYHzU=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgUY4PoSASSl1EQ7ooJRCnxtsrW37cZkWmBg2Qhy3Ijom +Gabrielle Kate Alicante,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToCr2v3+LX+7BrPAtR5JOsMA=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgRTycP8VQ2CZCnO9JyRBziEdtkW1EhLNLM3JqbP9xTxk +Gabrielle Kate Alicante,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToCr2v3+LX+7BrPAtR5JOsMA=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgXcm54zaupP9mWYYylq5DGSVBXqVKpsUkV+y58N6LM/i +Travis H Allen,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToG9Wr58n7/6aHdb1RhLeW2E=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgayhKc2rBd6bXjcG4xihbvraWcwMJ6VO2YPwADsOJIt/ +Travis H Allen,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToG9Wr58n7/6aHdb1RhLeW2E=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgaxGWNR5rarXJcixhUzVtzgxDjCEPyaCk3FvwFG32nJJ +Nancy Leah Allen Scannell,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToDKXRJtuT6QroDp6IHe6AUw=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgXCBeTqYHrN8ijVU/6Tpvr+hIKKrPr3ZVW9T26DCXRul +Nancy Leah Allen Scannell,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToDKXRJtuT6QroDp6IHe6AUw=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgQMCvODRr+WrM/3FttvCpVG+xpUvfIY37o7rV7VSMQBb +La-Brina Almeida,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToPKH8fNtmIONxjbhvUh5meM=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgcGXvwuUlPSjhrgH0hTW7gaGUOIlVl72vYF1uSaPWHb1 +Altria Client Services LLC,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToHrpAFq+hemD3/GGTrFLIKY=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgUU9hz/A/ameT7cAUsr6+MCD9snxaV/A3cysRq/GSJc4 +Altria Client Services LLC,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToHrpAFq+hemD3/GGTrFLIKY=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgUU9hz/A/ameT7cAUsr6+MC+b+7YAGcSPCXG3v9mPzVZ +Altria Client Services LLC,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToHrpAFq+hemD3/GGTrFLIKY=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgZSZQrllROM5b/1WYQRXDV3Cx0q/RN+sppjcqhnEQyX3 +Jason Aluia,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToO4ZzT3kiJ+OiaHsRgsMHOA=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgUN3EzpEE1mCOMs5YBHc9ktY1djYtF5bbPsLXFhKy1NY +Jason Aluia,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToO4ZzT3kiJ+OiaHsRgsMHOA=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgR+b3O5M3tJ3hLG5MeCqG3Sa6SrwOx/ieHrPT1NlMeHx +John James Alvarez,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToKFyX8ebGM0CLBB1OCY3dsY=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgVBmmMRPFLsp9ANvjW/Hyb+LzSSzVPH8FI48mqY4idXS +John James Alvarez,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToKFyX8ebGM0CLBB1OCY3dsY=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgRIL5t5Il4xGBMm2eT1p3D/wZXvOu2440m/Ui+NkVruK +Oamshri Amarasingham,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToGZUCaD088YXmopCl08K22s=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgRSHPfwPmspptBsOE6YGNJ4/zl8PjEA9DXmA3Aa/+ETq +Oamshri Amarasingham,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToGZUCaD088YXmopCl08K22s=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgf29pjNH+VVqYd13eqUszEGM9H7wmZFGmeAcUgCSU61L +Robert J Ambrogi,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToH5CL/fyPNxKgduNVp7IEqk=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgaCq6vP8sqSQuk5ObozY+p5qpz5FAc+7hPko9c4cVyaU +Robert J Ambrogi,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToH5CL/fyPNxKgduNVp7IEqk=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgeyzdnfYpOubL9grjh7jJQq9c6Iyh4IWtUkHU8rqV9of +Samuel Anderson,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToMzX6vi49N4eq0sPtNMC0yU=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgR0wkSHfiSjO3uDv5YTXv0Xhow/ZMUsLBi1oCLW9TsGS +Samuel Anderson,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToMzX6vi49N4eq0sPtNMC0yU=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgbQLET5N+ipPLAEG8GWqBCvxDIWoqnIvBu2HDNR677Ko +Christopher Robert Anderson,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToMwRDFW4GtQOXOhflD6ipng=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgZ3VSlmxu+H63IYGdz1vkSWk1dfhg7pprRJzA0AdnCBp +Christopher Robert Anderson,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToMwRDFW4GtQOXOhflD6ipng=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgTEPbK5kfOGzdw2KuzADBqMEPm0uxqL66GFniMO8rsll +Leda Anderson,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToLvSqLf3CTfrUJ4DeqqrJ5c=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgfNXzWu8eYrdtvgqkBUQ/NYNCtHgXvOGMENaubKFXIcu +Leda Anderson,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToLvSqLf3CTfrUJ4DeqqrJ5c=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgTf7eXWlbtWTfltacRM2dv5UlZMo12JZ07qxJQ3ZQaok +Kathleen R Anderson,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToE+r+46nBFC4UmuMAHonrDU=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgZQVaPABGRCuy0rOG4IeDyWa5/z9T5gUMU/H9eOWEdH/ +Kathleen R Anderson,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToE+r+46nBFC4UmuMAHonrDU=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgWqhTu4jTJBVDQ0lba3g4EZ63/Vkh9shAjaQ79eC+Hpy +Lisa Marie Andoscia,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToAGaAwMQyVRKp1FqsmTZXn0=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgdPIuanfK+xqJcJmgVKN6s4wjl2uX3zLlrQI9XEmhBv0 +Lisa Marie Andoscia,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToAGaAwMQyVRKp1FqsmTZXn0=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgRQtZPZ8hRVfLorYR65NcKWFnUsjM5zjf8MLWAmSNgQz +Holly Andreozzi,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToLRAU6t2DgQUVtT7gqJyvqc=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgYoq4mcdp0TPgFKKI0XUoicKm9+FWioT4XK6g39mfi1M +Holly Andreozzi,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToLRAU6t2DgQUVtT7gqJyvqc=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgeZ8Q0l1iXoOAU2zC9wZSNs4GqMdYZCujNfFhoXyqjET +Cole Keelan Angley,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToAurA9WBH+4DrHMV5Gk8le0=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgXSa4n1+RgMBw5n1/3fv/rXWpz254u0mqqb0g2ssVyS8 +Cole Keelan Angley,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToAurA9WBH+4DrHMV5Gk8le0=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgWeSlBObnloghnnb9mhn93qwjlxIx4wQDmXTTr290rmY +Rafael Antigua,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToJ7skATIfA2bsBEViEpKGUw=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgapPvXCrM9Xj0I6pxkZY5x3WXEShzrWxTNNOw3BFzlsf +Rafael Antigua,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToJ7skATIfA2bsBEViEpKGUw=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgZ9vWetxuHHyrcnLeG7+ajpQFUdJfM7DayyToWdjQkHT +Chelsea Elizabeth Aquino-Fenstermaker,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToE3uODVYDn91Qe0H6LJm2Uc=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgTikU7RESJpFH5Iq4bNBOP1aC8tB4dIJohR+BBI8p+G1 +Chelsea Elizabeth Aquino-Fenstermaker,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToE3uODVYDn91Qe0H6LJm2Uc=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgWylDIDUdKz80hFbJf1J51GRa7JvI4HYtyLlatvjka3F +Luz A Arevalo,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToPY58KS4ndJxBEYkVkD0WQk=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgeHw4y3BR5V0EBXHfJeGKAMYRKyEIAac7uduI/v8WrYP +Luz A Arevalo,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToPY58KS4ndJxBEYkVkD0WQk=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgXTCliPkcw9ihv+op+XWL1DMf73aKm12CzNzG/xQ6N2S +Derek Armstrong,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToHEfluu0/yz74vvm/zVxr2Y=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgbiHn5kwvbJQK46TxWj7CSi8UemxdBsyRzRnHCXMw70T +Derek Armstrong,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToHEfluu0/yz74vvm/zVxr2Y=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgfKv+kWEEas+4fySlysewY8GTBP8Z84p0+xCMTNZTQt7 +Matthew Paul Arsenault,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToMgNmDPSWwPWyDgmQeVACHc=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgZkJmU8UQCojgaaGP7FfmUgNSoAzMX/y5BCn2Qi6MCVm +Christina Ascolillo,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToNhkFsXobuXyl3nDOcMw48A=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgZNAykwKKMFLSRbQRyWLDHMkRRhDxITK9gJmy76gtOYy +Christina Ascolillo,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToNhkFsXobuXyl3nDOcMw48A=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgbPpnbybiOPgd8sZ6ucBIqFDQqc2vV4f26hrf/zb/aS0 +"Robert Ash, Jr.",2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToOcNafpIPfzXwdDuSn8mcAI=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgcBMMVzaag9A5myBl5Yf7wcDF0aKH7oO/7dVCjFkMJe6 +"Robert Ash, Jr.",2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToOcNafpIPfzXwdDuSn8mcAI=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgTw2PGUdf0PGL2BBmbsdsMV33Wm51mCGAAK4CQHMKkDJ +Sarah Selamawit Assefa,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToM41C7ZgkglKFtXNKWRmSvo=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgcMjQn9L1a8xFJ1QSLfvOtNe4PthU3J8pGzXTGoAOYnz +Sarah Selamawit Assefa,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToM41C7ZgkglKFtXNKWRmSvo=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgY5TBBA1XZk4NAfgWamuR2TvOwg6Ex+lDq0I/uGVu0nz +Ellen Attaliades,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToH9nbhFdO7lR5y2ZFMgKFQI=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgWCVxmDm+cm69E0RnTwMpdGShKryoxHqVwrNVjiGFWDq +Ellen Attaliades,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToH9nbhFdO7lR5y2ZFMgKFQI=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgazvCIgm5EbrgxICTIw8gJAvYKV20qGe1WRJs/ya85C1 +Melissa Kraus Augenbraun,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToGh56xu8sS0wMjqRyARjrw8=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgbCZZwVi2gJsWTDGy9Lncfp/HfJPTOn8wAh1y41Qs0mo +Melissa Kraus Augenbraun,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToGh56xu8sS0wMjqRyARjrw8=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgbKgrgXelnbhJdGHzwpBzBldeWp0ZKTIyhKASfxiqxZC +John Augustine,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToB6HQ1VwPkhToH4pJzdV57k=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgeG7ujwsMr0juJHTRUOC/LJYC9Zipxl37/CqeZlZfZwn +John Augustine,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToB6HQ1VwPkhToH4pJzdV57k=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgaOvwA0ZVX+VqeUhNzulhHes1F7sivgcuSbwyiy1+NMh +Jessica Avery,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToHMlP3E4cFXs+4GhWQqyOso=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgTiDiv0J1IF2QA2lPo/Qhap0TcO3JFMM0YDtmPlQYcZ1 +Jessica Avery,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToHMlP3E4cFXs+4GhWQqyOso=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgQTdylX7S/vxSrF05IcU+iL3xl+SToMx5ovzvv3Lam2S +Michael Avitzur,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToG1p8N+dGu7rF/IMh8XZalM=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgVTvTAox6YK1NomGHC0WXzASSpuL/D7a44buX6BQLMA8 +Michael Avitzur,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToG1p8N+dGu7rF/IMh8XZalM=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgYsNEfROmEkGnYjfv/99wl8JYWhlRa+ngLIg9yx6ZB57 +Steven Baddour,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToBDVe49iQMg9INjK5+/DjjQ=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgd657H4cdXnrpg67QNIA8ulQgfoBxbWNJq2pJBN0y/TE +Steven Baddour,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToBDVe49iQMg9INjK5+/DjjQ=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgZ/DPahXVB++JJbwJqoN2EhUEIw7VLeyfxqt/jxE1B3A +Omaira Badillo,2024,https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=vaOOB4xLr17uOlDTEexToORQXGKX+F0KZV/Tu524F+g=,https://www.sec.state.ma.us/LobbyistPublicSearch/CompleteDisclosure.aspx?sysvalue=hTpfuAXMhicRvD5o4yuXgTqggfbwXp0fIqlsWQAZqjjNsNER6O2kBpFGFWYFbRN8 diff --git a/docs/data/embedding_diagnostics.png b/docs/data/embedding_diagnostics.png new file mode 100644 index 0000000..4925c69 Binary files /dev/null and b/docs/data/embedding_diagnostics.png differ diff --git a/docs/data/facts_lobbying.yml b/docs/data/facts_lobbying.yml new file mode 100644 index 0000000..b4290b1 --- /dev/null +++ b/docs/data/facts_lobbying.yml @@ -0,0 +1,4 @@ +lobbying_most_recent_year: 2025 +lobbying_n_employers: 1381 +lobbying_n_firms: 144 +lobbying_total_spend_latest: 723619 diff --git a/docs/data/ts_update_MA_legislature.yml b/docs/data/ts_update_MA_legislature.yml new file mode 100644 index 0000000..abf7590 --- /dev/null +++ b/docs/data/ts_update_MA_legislature.yml @@ -0,0 +1 @@ +updated: 2026-05-28 20:21:37 diff --git a/docs/data/ts_update_MA_lobbying.yml b/docs/data/ts_update_MA_lobbying.yml new file mode 100644 index 0000000..6439e3b --- /dev/null +++ b/docs/data/ts_update_MA_lobbying.yml @@ -0,0 +1 @@ +updated: 2026-05-26 22:06:37 diff --git a/get_data/NOTES_bill_embeddings.md b/get_data/NOTES_bill_embeddings.md new file mode 100644 index 0000000..ee44784 --- /dev/null +++ b/get_data/NOTES_bill_embeddings.md @@ -0,0 +1,376 @@ +# Bill Embedding and Clustering — Design Notes + +## Current approach (as of May 2026) + +**Model:** Google Gemini `gemini-embedding-2`, 768-dimensional output. + +**Text preparation per bill:** +1. Strip legislative scaffolding from body text using `_SCAFFOLD_RE` — the repeated "Chapter X of the General Laws…is hereby amended by inserting after…" boilerplate that appears identically in thousands of bills regardless of topic. +2. Prepend the bill title (always specific and clean signal). +3. Truncate the combined string to 3,000 characters (~750 tokens). +4. Fall back to title alone when no body text is available. + +Bills from GC 183–184 (2005–2008) frequently have no full text in the Legislature API and embed as near-zero vectors; these are excluded from clustering (`cluster_id = -1`). + +**Clustering:** K-means, k=25, on L2-normalised (cosine-space) embeddings. +Cluster labels generated by Gemini 2.5 Flash from the 20 most central bill titles per cluster. + +**Environmental scoring:** Differential cosine similarity — max cosine similarity to 42 known env bills minus max cosine similarity to 42 known non-env bills. Threshold: 0.05. Results in ~329 flagged bills (~1.3% of all uniquely lobbied bills with valid embeddings). + +### Parquet schema note + +The `full_text` column in `MA_bill_embeddings.parquet` stores the **constructed embed text** (SoS title prepended to the stripped legislature body), not the raw legislature API text. The docstring in `score_lobbying_bills.py` is incorrect on this point (says "from MA Legislature API"). See line 408–410 of that file. + +--- + +## ⚠️ Data integrity issue: wrong General Court in Legislature API lookups + +### Root cause: off-by-one bug in `_year_to_general_court` + +`get_MA_lobbying.py` computes `general_court = 183 + (year - 2005) // 2`. This is wrong: the 183rd General Court convened in **January 2003**, not 2005. The correct formula is `183 + (year - 2003) // 2`. The result is that every bill lookup since the scraper was written has been sent to the wrong legislative session — one GC too early. + +Correct mapping (GC183 = 2003–2004): + +| Year filed | Wrong GC (bug) | Correct GC | +|-----------|---------------|-----------| +| 2005–2006 | GC183 | **GC184** | +| 2009–2010 | GC185 | **GC186** | +| 2013–2014 | GC187 | **GC188** | +| 2021–2022 | GC191 | **GC192** | +| 2023–2024 | GC192 | **GC193** | +| 2025–2026 | GC193 | **GC194** | + +**Confirmed experimentally:** `H3111` year=2024 with wrong GC (192) returns "An Act relative to the open meeting law". With correct GC (193) returns "An Act relative to DCR skating rinks" — matching the SoS title exactly. Verified for 5+ additional bills. + +**Scale of impact (60-bill random sample with live API checks):** + +| GC version | Title match rate | +|-----------|-----------------| +| Wrong GC (bug) | 2% (1/60) | +| **Corrected GC (+1)** | **65% (39/60)** | +| Still wrong after fix | 35% (21/60) | + +The corrected GC raises the match rate from ~2% to ~65%. The remaining 35% of mismatches split into: +1. **String normalisation:** SoS sometimes prefixes the bill number to the title (e.g., "H. 2324, An Act relative to residency" vs API "An Act relative to residency") — the bill is the same but the string comparison fails +2. **Cross-session lobbying:** a small number of lobbyists file disclosures in year X for bills introduced in the following session (GC+2 from the wrong formula, not just +1) +3. **Wrong numbers in SoS portal:** a residual fraction where the lobbyist recorded an incorrect bill number; no GC adjustment recovers these + +This bug also affects the `general_court` column in `MA_lobbying_bills.csv` — all session assignments are wrong by one GC throughout the corpus. + +### Fix + +In `get_MA_lobbying.py`, change: +```python +FIRST_GC_START_YEAR = 2005 +``` +to: +```python +FIRST_GC_START_YEAR = 2003 +``` + +Then re-run the full pipeline: `get_MA_lobbying.py` → `get_MA_legislature_bills.py` → `score_lobbying_bills.py --reembed` → `cluster_lobbying_bills.py` → `assemble_db.py`. + +The legislature API cache files are named `bill_{gc}_{bill_id}.json` using the wrong GC, so the corrected lookups will mostly be cache misses and re-fetch from the API. Estimated re-fetch time: ~3–4 hours for ~25,000 bills at 1 req/s. + +Also update `CLAUDE.md`: the note "GC 183 = 2005–2006" is wrong; it should read "GC 183 = 2003–2004, GC 184 = 2005–2006". + +### Impact on embeddings and scoring + +With wrong GC, the legislature body text belongs to a completely different bill ~98% of the time. The bill **title** (from the SoS portal, stored in `bill_title`) is always correct regardless of the GC bug. + +**This is why title-only embeddings significantly outperform title+body embeddings** (silhouette 0.043 vs 0.023 — see comparison below): the body text is nearly pure noise under the wrong-GC regime. After fixing the GC bug and re-embedding with correct body text, the title+body method improved from 0.023 → 0.028 (+21%), but title-only remained best at 0.041 — a ~48% lead. Body text still dilutes topic signal even when it belongs to the correct bill, due to structural boilerplate in MA legislative language. + +--- + +## Embedding space structure + +MA legislative bill embeddings are **semantically dense** — even after boilerplate stripping, bills cluster tightly regardless of topic because they share heavy structural language. + +Empirical measurements (May 2026, 24,612 valid bills after mean-centering + L2 normalisation): + +| Metric | Value | +|--------|-------| +| Mean inter-cluster cosine distance (k=25) | ~0.006 | +| Mean intra-cluster cosine spread | ~0.53 | +| Ratio (separation / spread) | ~0.011 | + +For comparison, a well-separated topic model typically has inter/intra ratios >0.5. The dense structure is likely partly an artifact of the body-text mismatch problem (body text is unrelated to bill topic). + +--- + +## Embedding method comparison (2,000-bill sample, k=25) + +Tested twice: once with **wrong GC** (body text is ~98% from unrelated bills) and again after the GC fix with **correct body text**. Same 2,000-bill sample each time. + +### Wrong GC (body = unrelated bills ~98% of the time) + +| Method | Silhouette↑ | DB↓ | Notes | +|--------|-------------|-----|-------| +| A. Original (SoS title + API body combined) | 0.023 | 4.76 | Current production method at time of test | +| B. **Title only (SoS)** | **0.043** | **3.71** | Best; 2× silhouette improvement | +| C. Body only (API) | 0.024 | 4.61 | Nearly as bad as combined | +| D. Concat [L2(title) \| L2(body)] | 0.025 | 4.76 | No improvement over original | + +### Correct GC (body text matches the actual lobbied bill — run May 2026 after fix) + +| Method | Silhouette↑ | DB↓ | vs. wrong-GC | +|--------|-------------|-----|--------------| +| A. Original (SoS title + API body combined) | 0.0278 | 4.46 | +21% | +| B. **Title only (SoS)** | **0.0412** | **3.83** | -4% (unchanged) | +| C. Body only (API) | 0.0278 | 4.37 | +16% | +| D. Concat [L2(title) \| L2(body)] | 0.0296 | 4.54 | +18% | + +**Title-only is still the best embedding strategy even with correct body text.** Fixing the GC improved A, C, D by 16–21%, but title-only (B) leads by ~48% silhouette over A and remained essentially unchanged. The body text appears to dilute topic signal even when it belongs to the correct bill — MA legislative body text is structurally noisy (boilerplate, amendments-to-existing-law syntax, cross-references) even after scaffold stripping. + +Switching to title-only embeddings would require a full re-embed of ~25,000 bills. The current parquet stores title+body (method A). + +--- + +## LLM classification spot-check (Gemini 2.5 Flash, May 2026) + +42 bills sampled across 6 score buckets around the 0.05 threshold; classified with Gemini 2.5 Flash (SoS title + first 800 chars of stored body text). + +**Overall agreement with current embedding classification: 64% (27/42).** + +| Score range | Bills | Agreement | Key finding | +|-------------|-------|-----------|-------------| +| < −0.05 | 1,419 | — (not tested) | | +| −0.05 to −0.02 | 7,149 | 80% | 1/5 false negative (climate adaptation bill) | +| −0.02 to 0.0 | 10,243 | 100% in sample of 8 | | +| 0.0 to 0.02 | 4,997 | **25%** | 6/8 are env per LLM — major false negatives | +| 0.02 to 0.05 | 1,795 | **12%** | 7/8 are env per LLM — major false negatives | +| 0.05 to 0.08 | 301 | 88% | 1/8 false positive | +| > 0.08 | 28 | — (not tested) | | + +**Examples of false negatives the LLM catches:** +- "An Act to expand the bottle bill" (waste management) — score +0.010 +- "An Act preventing gas expansion to protect climate, community health and safety" — score +0.019 +- "An Act relative to net metering" — score +0.023 +- "An Act requiring local approval for battery storage facility permitting" — score +0.011 +- "An Act relative to fair and stable electricity pricing" — score +0.032 +- "An Act relative to environmental justice in the commonwealth" — score +0.032 +- "Communication from Department of Energy Resources re: renewable portfolio standards" — score +0.038 +- "An Act to expand the bottle bill" — score +0.010 + +The LLM also correctly flags title/body mismatches (e.g., "An Act to strengthen notifications in the Wetlands Protection Act" whose stored body is actually eyewitness identification law), explicitly noting the discrepancy and weighting the title signal. + +**Cost of hybrid LLM approach (Gemini 2.5 Flash):** + +| Scope | Bills | One-time cost | +|-------|-------|--------------| +| Just the positive-score borderline (0.0–0.08) | 7,093 | ~$0.27 | +| Including near-zero negative band (−0.02–0.08) | 17,336 | ~$0.67 | +| Incremental weekly CI run (~20–30 new borderline bills) | ~25 | <$0.001 | + +--- + +## Clustering algorithm evaluation + +### K-means silhouette sweep (8k subsample, mean-centered + L2) + +| k | Silhouette↑ | DB↓ | +|---|-------------|-----| +| 4 | 0.018 | 6.07 | +| 8 | 0.022 | 5.53 | +| 10 | 0.023 | 5.32 | +| 15 | 0.025 | 4.95 | +| 20 | 0.023 | 4.96 | +| 25 | 0.024 | 4.76 | +| 30 | 0.026 | 4.67 | +| 40 | 0.027 | 4.60 | + +**Conclusion:** The silhouette curve is completely flat across k=4..40 with no elbow. k=25 was chosen on domain grounds (produces ~1,000 bills/cluster, yields coherent topic labels). + +Mean-centering before L2 normalisation gives a small but consistent improvement (silhouette +0.007 at k=25). + +### HDBSCAN sweep (full set, mean-centered + L2) + +| min_size | min_samples | Clusters | Noise% | Silhouette↑ | DB↓ | +|----------|-------------|----------|--------|-------------|-----| +| 10 | 5 | 203 | 84.4% | 0.162 | 2.06 | +| 10 | 10 | 61 | 90.8% | 0.157 | 1.82 | +| 15 | 5 | 100 | 88.7% | 0.154 | 2.23 | +| 20 | 5 | 52 | 87.9% | 0.097 | 2.52 | +| 30 | 5 | 23 | 88.9% | 0.101 | 2.74 | +| 40 | 5 | 18 | 89.2% | 0.107 | 2.70 | +| 50 | 5 | 10 | 90.6% | 0.116 | 2.31 | + +**Conclusion:** HDBSCAN consistently pushes 84–91% of bills into noise. Not suitable for this corpus — the data has no meaningful density gaps between policy domains. + +--- + +## Diagnostic plots (May 2026, post-GC fix) + +![Embedding diagnostics](../docs/data/embedding_diagnostics.png) + +Four panels generated from the full 25,932-bill scored corpus and a 2,000-bill embedding sample after the GC formula fix. + +### Score distribution + +The distribution is sharply right-skewed around a mean of −0.011. Nearly all bills score below zero; the right tail is thin but real. Key counts at the current threshold (0.05): + +| Band | Bills | Notes | +|------|-------|-------| +| < 0 (clearly non-env) | 18,923 | 73% of corpus | +| 0–0.02 (borderline negative) | 4,164 | Spot-check: ~80%+ non-env | +| 0.02–0.05 (borderline positive) | 2,190 | Spot-check: ~75% env per LLM | +| 0.05–0.08 (just above threshold) | 590 | Spot-check: ~88% env | +| ≥ 0.08 (strong env) | 65 | High confidence | +| **Total flagged (≥ 0.05)** | **654** | Up from 329 before GC fix | + +There is no natural density gap between 0.02 and 0.05, suggesting the threshold is somewhat arbitrary in this zone. The LLM spot-check found large false-negative rates in the 0.02–0.05 band. + +### Embedding method comparison + +Title-only embeddings lead in both the wrong-GC and correct-GC conditions: + +| Method | Wrong GC | Correct GC | Δ | +|--------|----------|------------|---| +| A. Title + body (production) | 0.023 | 0.028 | +21% | +| **B. Title only** | **0.043** | **0.041** | −4% (stable) | +| C. Body only | 0.024 | 0.028 | +16% | +| D. Concat [L2(t)‖L2(b)] | 0.025 | 0.030 | +18% | + +Fixing the GC improved all body-containing methods but did not close the gap with title-only. Body text adds structural noise (boilerplate, amendment scaffolding, cross-references) even when it belongs to the correct bill. + +### Cluster env density + +Environmental bills concentrate strongly in two clusters: + +| Cluster | Label | Env bills | % env | +|---------|-------|-----------|-------| +| 14 | Massachusetts Clean Energy Transition | 238 | 30% | +| 2 | Waste Reduction and Recycling | 181 | 39% | +| 20 | Empowering Local Clean Energy | 54 | 5% | +| 0 | Municipal Environmental Governance | 50 | 3% | + +The remaining 21 clusters each contain ≤29 env bills. This concentration is a positive signal: the k=25 topology is correctly separating the clean-energy and waste policy space from unrelated legislation. + +--- + +## t-SNE visualisation + +**[→ Interactive t-SNE (Plotly)](../docs/_includes/charts/lobbying_bill_tsne.html)** + +## t-SNE design note + +Projecting all 25,000+ bills into 2D produces a featureless blob because the inter-cluster cosine distance (~0.006) is ~100× smaller than intra-cluster spread (~0.53). + +**Current approach:** Plot only the 329 environmental bills (coloured by cluster) over a stratified background sample of ~120 non-env bills per cluster (~3,000 total in grey). This makes environmental bills legible within the policy landscape without misrepresenting cluster separation. + +--- + +## Recommended next steps + +1. **Switch to title-only embeddings** for both classification and clustering. Confirmed best strategy after two rounds of testing — title-only wins even with correct body text. Requires a full re-embed (~25k bills). Update `score_lobbying_bills.py` to use `title` instead of the combined `full_text`. Current model files (committed and on GCS) are title+body; a full re-cluster would follow the re-embed. + +2. **Add LLM classification for borderline bills** (score 0.0–0.08, ~7,100 bills). One-time cost ~$0.27 with Gemini 2.5 Flash. Store `llm_is_environmental` and `llm_reason` as additional columns. Use as override when LLM confidence ≥ 0.80. Particularly important given the env threshold review needed (see below). + +3. **Environmental threshold review**: After the GC fix, the env bill count jumped 329 → 654 at threshold=0.05. Needs calibration review — plot the score distribution, spot-check bills near the boundary, and decide whether 0.05 is still correct. Document in an analysis page or data note explaining the method, reference sets, and before/after counts. + +4. **Legislature API lookup quality** (lower priority): match bills by title similarity rather than bill number, or investigate whether the SoS portal exposes docket numbers that map to the API's DocketNumber field. Less urgent now that body text quality is confirmed as structurally noisy regardless. + +--- + +## Re-running clustering + +```bash +cd get_data +/path/to/python -u cluster_lobbying_bills.py +``` + +Then re-run `assemble_db.py` and `MA_lobbying_viz.py` to propagate new cluster labels. + +--- + +## LLM summary + taxonomy pilot diagnostics (495-bill sample, gemini-2.5-flash) + +**Run date:** May 2026 **Cost:** $0.0525 for 495 bills (LLM only — see corrected full-corpus estimate below) + +### 1. Env classification — reference set precision/recall + +20 known-env titles and 36 known-non-env titles from the reference sets in `score_lobbying_bills.py` were classified by the LLM. + +| Metric | Value | +|--------|-------| +| Recall (20 known env titles) | **100%** (20/20) | +| Specificity (36 known non-env) | **97%** (35/36) | +| False negatives | 0 | +| False positives | 1 — "An Act limiting autonomous driving capabilities to zero emission and electric vehicles" (borderline ZEV/EV policy) | + +### 2. LLM vs embedding disagreement (495-bill pilot) + +| | Count | +|---|---| +| Both env (agreement) | 18 | +| LLM env only (embedding false negatives) | 63 | +| Embedding env only (embedding false positives) | 0 | + +Bills LLM classifies env but embedding misses (10 examples): +- An Act Relative to Pollution Health Effects Mitigation +- An Act relative to the water resources funding act +- An Act establishing a wildlife management commission +- An Act Relative to the Reduction of Particulate Emissions From Diesel Engines +- An Act further regulating the Pesticide Control Act +- An Act relative to the moose population in the Commonwealth +- An Act further regulating above ground tanks used for the storage of certain fluids +- Prohibit Sale or Use of Polystyrene Packaging +- Environmentally Preferable Cleaning Supplies +- An Act promoting incremental hydropower improvements + +### 3. Structured output quality + +| Metric | Value | +|--------|-------| +| Avg tags per bill | 3.14 | +| Bills with 0 valid tags | 1 | +| Bills with 0 valid categories | 0 | + +### 4. Silhouette comparison (k=25 clustering) + +| Method | Silhouette↑ | Δ | +|--------|-------------|---| +| Original title+body | -0.0387 | — | +| **Summary embed** | **-0.0492** | −27% | + +Both scores are negative, indicating bills are on average closer to the nearest *other* cluster's centroid than to their own. The 27% degradation with summary embeddings is statistically real but both values are in the same "no useful cluster separation" regime — summaries do not improve clustering geometry. However, the **primary value of summaries is not clustering but classification**: LLM `is_environmental` shows 100% recall / 97% specificity on reference titles vs. ~4.5% env rate with the embedding classifier alone (large false-negative rate). + +**Implication for full run:** Proceed for taxonomy tags and `is_environmental` signal; do not expect summary embeddings to improve the UMAP/cluster visualisations. + +### 5. Cost actuals + +**Pricing (as of May 2026):** + +| Model | Operation | Price | +|-------|-----------|-------| +| gemini-2.5-flash | Input (uncached) | $0.075 / 1M tokens | +| gemini-2.5-flash | Input (cached) | $0.01875 / 1M tokens | +| gemini-2.5-flash | Output | $0.300 / 1M tokens | +| gemini-2.5-flash | Thinking tokens | $3.50 / 1M tokens | +| gemini-embedding-2 | Input | $0.20 / 1M tokens | + +**Pilot run (495 bills, LLM only — embedding not tracked initially):** + +| Metric | Value | +|--------|-------| +| Input tokens (total) | 981,288 | +| Cached tokens | 792,990 (80.8%) | +| Avg input / bill | 1,982 tokens | +| Avg output / bill | 152 tokens | +| LLM cost (495 bills) | $0.0525 | +| Savings from prompt caching | ~46% | + +**Full-corpus cost estimate (all 25,932 bills, corrected):** + +| Component | Tokens | Cost | +|-----------|--------|------| +| LLM summarize (generate_content) | ~52M input, ~4M output | ~$2.77 | +| Summary embed (gemini-embedding-2, ~100 tok/summary) | ~2.6M | ~$0.52 | +| Bill embed in score_lobbying.py (~700 tok/bill) | ~18.2M | ~$3.64 | +| **Total all-in** | | **~$6.93** | + +Note: the initial estimate of "$2.76 projected" only captured LLM generate_content calls. The `gemini-embedding-2` model at $0.20/1M tokens adds ~$4.16 across both the score_lobbying embedding pass and inline summary embedding. Thinking tokens (budget=0) confirmed zero in tests; tracking added to script to verify in production runs. + +### 6. UMAP with summary embeddings + +**[→ Interactive UMAP (summary embeddings)](../docs/_includes/charts/lobbying_bill_umap_summary.html)** diff --git a/get_data/README_lobbying.md b/get_data/README_lobbying.md new file mode 100644 index 0000000..3f325a7 --- /dev/null +++ b/get_data/README_lobbying.md @@ -0,0 +1,225 @@ +# MA Lobbying pipeline + +Tracks which bills are being lobbied in the Massachusetts legislature, how much is +spent, and whether those bills are environmentally relevant. Four scripts, the first +three of which run in weekly CI: + +``` +get_MA_lobbying.py + ↓ MA_lobbying_employers.csv, MA_lobbying_bills.csv, MA_lobbying_summary_links.csv +get_MA_legislature_bills.py + ↓ MA_legislature_bills.csv + MA_legislature_cache/.json +score_lobbying_bills.py + ↓ MA_lobbying_bills_scored.csv + gs://openamend-data/MA_bill_embeddings.parquet +cluster_lobbying_bills.py ← manual / one-time + ↓ MA_bill_cluster_labels.csv (cluster_id column also written back to scored CSV) +``` + +All scripts must be run from `get_data/`. Use the `amend_python` conda env, and run +Python directly (not `conda run`) for live log output: + +```bash +/home/nes/miniconda/envs/amend_python/bin/python -u