feat: Add SCB client/context modules and improve news generation test coverage by Copilot · Pull Request #732 · Hack23/riksdagsmonitor

Copilot · 2026-03-02T11:43:45Z

Adds a native TypeScript client for Statistics Sweden (SCB) PxWeb API and a context module mapping SCB indicators to Riksdag committees, enabling news generation workflows to cross-reference World Bank and SCB data. Also fills significant test coverage gaps in policy-analysis.ts and article-quality-enhancer.ts.

New modules

scripts/scb-client.ts — REST client for SCB PxWeb API with 15 policy domain→table mappings, retry logic, and relevance detection
scripts/scb-context.ts — Maps SCB indicators to all 15 Riksdag committees with i18n section headings (14 languages), parallel to world-bank-context.ts

Enhancements

scripts/world-bank-context.ts — Added detectStatisticalSources() that combines World Bank and SCB source detection in a single call, enabling workflows to identify which data sources are relevant for a given article
scripts/article-quality-enhancer.ts — Recommends SCB statistical context via a dedicated recommendSCBContext threshold flag, independent of the World Bank recommendEconomicContext option
scripts/world-bank-client.ts — Fixed typo rulOfLaw → ruleOfLaw in INDICATOR_IDS export
scripts/statistical-claims-detector.ts — GDP-share claims now use SCB-only verification (GDP growth ≠ GDP share)

Bug fixes

scripts/scb-context.ts — Fixed Unicode word boundary issue in hasSCBContext(): \bbrå\b replaced with substring match for reliable Swedish text detection
scripts/scb-client.ts — Removed unused AbortController in callTool(); non-JSON MCP responses now throw errors instead of returning raw text to typed callers
tests/generate-sitemap-html.test.ts — Removed file existence check for sitemap HTML files (sitemaps are now generated at deploy time, not committed to repo)

Tests (+163 new)

tests/scb-client.test.ts — 28 tests (constructor, domains, API calls, retries)
tests/scb-context.test.ts — 32 tests (indicators, headings, committee mapping, detection)
tests/policy-analysis.test.ts — 25 tests (confidence scoring, domain detection, framing, SCB tables)
tests/article-quality-enhancer.test.ts — 30 tests (depth analysis, party detection, reference quality, HTML nesting)
tests/world-bank-context.test.ts — 7 new tests for detectStatisticalSources()

Usage

import { detectStatisticalSources } from './world-bank-context';
import { SCBClient } from './scb-client';
import { getSCBContextHeading, hasSCBContext } from './scb-context';

// Detect which data sources are relevant for article text
const sources = detectStatisticalSources(articleText);
// → { hasWorldBank: true, hasSCB: true, domains: ['economy', 'labour'] }

// Fetch SCB data for a policy domain
const client = new SCBClient();
const data = await client.fetchDomainData('economy');

// Get localized section heading for article enrichment
const heading = getSCBContextHeading('economy', 'sv');
// → "Ekonomisk statistik från SCB"

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

…mmittee coverage - scripts/scb-client.ts: SCB MCP client with 15 policy domains covering all 15 Riksdag committees - scripts/scb-context.ts: SCB context provider with localized headings (14 languages), committee mappings - scripts/statistical-claims-detector.ts: Fact-check politicians' statistical claims against World Bank/SCB data - scripts/world-bank-client.ts: Added Tax Revenue, Rule of Law, Voice & Accountability, Gov Effectiveness indicators - scripts/world-bank-context.ts: Added indicator contexts for KU, SkU committees (fills coverage gaps) - scripts/article-quality-enhancer.ts: Enhanced scoring for SCB context and statistical claims detection - scripts/types/validation.ts: Added hasSCBContext, hasStatisticalClaims to QualityMetrics - scripts/data-transformers/types.ts: Added statisticalClaims to ArticleContentData - tests/scb-client.test.ts: 25 tests for SCB client - tests/scb-context.test.ts: 42 tests for SCB context - tests/statistical-claims-detector.test.ts: 58 tests for claims detector Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>

tests/scb-client.test.ts

+ * @license Apache-2.0
+ */
+
+import { describe, it, expect, vi, beforeEach } from 'vitest';


… workflow enrichment - world-bank-client.ts: Added 4 new indicators (Tax Revenue, Rule of Law, Voice & Accountability, Gov Effectiveness) - world-bank-context.ts: Added 4 new indicator contexts covering SkU and KU committees - scb-client.ts: Added 3 new domains (taxation, culture, governance) for full committee coverage - scb-context.ts: Added 3 new domain contexts for SkU, KrU, KU committees - Updated all 10 agentic workflow .md files with enhanced SCB/World Bank enrichment + fact-checking guidance - Updated tests: WB client (16 indicators), SCB client (15 domains), SCB context (15 contexts) - Full test suite: 2550 pass, 1 pre-existing failure (sitemap.html not found) Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>

Co-authored-by: pethers <1726836+pethers@users.noreply.github.com> Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>

github-actions · 2026-03-02T12:43:03Z

🔍 Lighthouse Performance Audit

Category	Score	Status
Performance	85/100	🟡
Accessibility	95/100	🟢
Best Practices	90/100	🟢
SEO	95/100	🟢

📥 Download full Lighthouse report

Budget Compliance: Performance budgets enforced via budget.json

Copilot

Pull request overview

Adds Statistics Sweden (SCB) enrichment and statistical-claim fact-checking capabilities to the news-generation pipeline, alongside updates to World Bank indicator mappings and expanded unit test coverage.

Changes:

Introduces SCBClient + scb-context to map SCB domains/tables to Riksdag committees and support localized headings.
Adds a statistical claims detector with 14-language fact-check headings, plus wiring into article quality metrics/warnings.
Extends World Bank indicator IDs/context mappings and updates/expands Vitest coverage accordingly.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
tests/world-bank-client.test.ts	Updates expected indicator count after new World Bank IDs were added.
tests/statistical-claims-detector.test.ts	Adds unit tests for statistical claim detection, assessment, and localization.
tests/scb-context.test.ts	Adds unit tests for SCB domain/committee mappings, headings, and detection.
tests/scb-client.test.ts	Adds unit tests for SCB client configuration, domain lookups, and mocked network behavior.
scripts/world-bank-context.ts	Adds World Bank indicator contexts for taxation and governance-related indicators.
scripts/world-bank-client.ts	Extends `INDICATOR_IDS` with additional World Bank indicators.
scripts/types/validation.ts	Extends quality metrics to include SCB-context and statistical-claims flags.
scripts/statistical-claims-detector.ts	Adds claim extraction + verdict assessment + localized fact-check headings.
scripts/scb-context.ts	Adds SCB domain→committee mappings and SCB-context detection/headings utilities.
scripts/scb-client.ts	Adds an MCP-backed SCB client with retry logic and domain/table mappings.
scripts/data-transformers/types.ts	Extends article content type to include detected statistical claims.
scripts/article-quality-enhancer.ts	Integrates SCB-context + statistical-claims detection into quality metrics and warnings.
.github/workflows/news-weekly-review.md	Updates workflow guidance to use SCB + World Bank enrichment and claims detection.
.github/workflows/news-week-ahead.md	Adds guidance for pre-fetching committee-mapped SCB/WB indicators.
.github/workflows/news-propositions.md	Updates enrichment/fact-check guidance for propositions workflow.
.github/workflows/news-motions.md	Updates enrichment/fact-check guidance for motions workflow.
.github/workflows/news-monthly-review.md	Expands monthly-review enrichment/fact-check guidance with concrete indicators/tables.
.github/workflows/news-month-ahead.md	Adds guidance for SCB/WB prefetching for upcoming calendar events.
.github/workflows/news-evening-analysis.md	Documents committee-mapped indicators and introduces fact-checking guidance.
.github/workflows/news-committee-reports.md	Updates enrichment/fact-check guidance based on reporting committee mappings.
.github/workflows/news-article-generator.md	Adds detailed tables/instructions for SCB/WB enrichment and statistical-claims fact-checking.

Copilot · 2026-03-02T12:48:30Z

scripts/article-quality-enhancer.ts

    warnings.push('Recommended: Add economic context (World Bank indicators, GDP, unemployment data)');
  }

+  if (options.recommendEconomicContext && !metrics.hasSCBContext) {


recommendEconomicContext is documented as a World Bank recommendation (see QualityThresholds), but it’s also used to gate the SCB recommendation. This makes it impossible to recommend World Bank context without also recommending SCB context (and the option name becomes misleading). Consider introducing a separate threshold flag (e.g., recommendSCBContext) or renaming the existing option to reflect that it covers both data sources, then use the appropriate flag here.

Suggested change

if (options.recommendEconomicContext && !metrics.hasSCBContext) {

if (metrics.hasStatisticalClaims && !metrics.hasSCBContext) {

Added recommendSCBContext as a separate boolean in QualityThresholds (defaults to true). The SCB warning now uses options.recommendSCBContext while World Bank keeps options.recommendEconomicContext. Fixed in 41b317c.

Copilot · 2026-03-02T12:48:31Z

scripts/scb-context.ts

+}
+
+// ---------------------------------------------------------------------------
+// SCB indicator mappings (12 domains → committees)


The section header comment says “12 domains → committees”, but SCB_INDICATOR_CONTEXTS contains 15 entries below. This mismatch makes the file misleading for future maintainers; update the comment to reflect the actual number (or avoid hard-coding the count).

Suggested change

// SCB indicator mappings (12 domains → committees)

// SCB indicator mappings (domains → committees)

Updated to // SCB indicator mappings (domains → committees) without hard-coding the count. Fixed in 41b317c.

Copilot · 2026-03-02T12:48:31Z

scripts/scb-context.ts

+    /\bbefolkningsstatistik/i, // Population statistics
+    /\bbostadsbyggande/i, // Housing construction
+    /\butrikeshandel/i, // Foreign trade
+    /\bbrå\b/i, // Crime statistics authority


The regex \bbrå\b won’t reliably match “Brå/BRÅ” because JavaScript \b word boundaries don’t treat å as a word character. This makes hasSCBContext() miss BRÅ references in Swedish text. Prefer a Unicode-aware pattern (with the u flag and explicit boundaries) or a simpler case-insensitive substring match for “brå”.

Suggested change

/\bbrå\b/i, // Crime statistics authority

/brå/i, // Crime statistics authority (substring match to avoid Unicode \b issues)

Replaced with /brå/i (substring match) to avoid Unicode \b issues. Fixed in 41b317c.

Copilot · 2026-03-02T12:48:32Z

scripts/scb-client.ts

+  private async callTool<T>(toolName: string, params: Record<string, unknown>): Promise<T | null> {
+    const controller = new AbortController();
+    const timeoutId = setTimeout(() => controller.abort(), this.timeout);
+
+    try {
+      const response = await this.fetchWithRetry(toolName, params);
+      return response as T;


callTool() creates an AbortController and timeout, but the controller’s signal is never used because fetchWithRetry() creates its own controller. This results in redundant timers/controllers and the abort in callTool() having no effect. Consider passing a single controller/signal through to fetchWithRetry(), or removing the unused controller/timeout in callTool().

Removed the unused AbortController and timeout from callTool() — fetchWithRetry() already has its own. Fixed in 41b317c.

Copilot · 2026-03-02T12:48:32Z

scripts/scb-client.ts

+          console.warn(`SCB MCP response for ${toolName} was not valid JSON; returning raw text`);
+          return text;


When JSON parsing fails, fetchWithRetry() returns raw text (a string). Callers like searchTables()/getTableData() then return that value as if it were SCBTableInfo[]/SCBDataPoint[], which can cause runtime errors downstream. Consider treating non-JSON responses as an error/null (or validating the parsed payload shape, e.g., Array.isArray) before returning it to typed callers.

Suggested change

console.warn(`SCB MCP response for ${toolName} was not valid JSON; returning raw text`);

return text;

console.warn(`SCB MCP response for ${toolName} was not valid JSON; treating as error`);

throw new Error(`SCB MCP response for ${toolName} was not valid JSON`);

Non-JSON responses now throw Error("SCB MCP response for ${toolName} was not valid JSON") instead of returning raw text. The error is caught by callTool() which returns null. Fixed in 41b317c.

Copilot · 2026-03-02T12:48:32Z

scripts/world-bank-client.ts

+  /** Tax revenue (% of GDP) — relevant to SkU taxation committee */
+  taxRevenue: 'GC.TAX.TOTL.GD.ZS',
+  /** Rule of law estimate — relevant to KU constitution committee */
+  rulOfLaw: 'RL.EST',


INDICATOR_IDS.rulOfLaw looks like a typo/inconsistent naming (missing 'e' in 'rule'). Since this key is exported and referenced elsewhere, it’s worth correcting to ruleOfLaw now to avoid locking in an awkward API surface.

Suggested change

rulOfLaw: 'RL.EST',

ruleOfLaw: 'RL.EST',

Renamed to ruleOfLaw and updated the reference in world-bank-context.ts. Fixed in 41b317c.

Copilot · 2026-03-02T12:48:32Z

scripts/statistical-claims-detector.ts

+    verificationSource: 'both',
+    worldBankIndicator: 'NY.GDP.MKTP.KD.ZG',


The gdp-share claim pattern is for statements like “X% of GDP”, but it maps to the GDP growth indicator (NY.GDP.MKTP.KD.ZG). GDP growth is not a “share of GDP” metric, so this cross-reference will produce misleading verifications. Consider either (a) removing the World Bank mapping for this generic pattern, or (b) mapping to a more appropriate indicator based on additional context (e.g., gov expenditure, military spending, etc.).

Suggested change

verificationSource: 'both',

worldBankIndicator: 'NY.GDP.MKTP.KD.ZG',

verificationSource: 'scb',

Changed gdp-share to verificationSource: 'scb' only (removed the misleading worldBankIndicator: 'NY.GDP.MKTP.KD.ZG' mapping). Fixed in 41b317c.

pethers · 2026-03-02T12:50:45Z

@copilot apply changes based on the comments in this thread and fix test generate-sitemap-html.test.ts (do not need to check for sitemap.html exist, not true anymore all sitemap generated at deploy time and not commited to repo anymore)

…er, non-JSON handling, typo, threshold separation, sitemap test Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>

Copilot · 2026-03-02T12:59:03Z

@copilot apply changes based on the comments in this thread and fix test generate-sitemap-html.test.ts (do not need t...

All 7 review comments addressed and sitemap test fixed in 41b317c:

scb-context.ts:46 — removed hard-coded count from comment
scb-context.ts:338 — brå now uses substring match (no Unicode \b issue)
scb-client.ts — removed unused AbortController in callTool(); non-JSON responses now throw instead of returning raw text
world-bank-client.ts — rulOfLaw → ruleOfLaw (+ reference update)
statistical-claims-detector.ts — gdp-share changed to SCB-only verification
article-quality-enhancer.ts — added separate recommendSCBContext threshold flag
generate-sitemap-html.test.ts — removed file existence check; all 2550 tests pass (0 failures)

github-actions · 2026-03-02T13:01:06Z

🔍 Lighthouse Performance Audit

Category	Score	Status
Performance	85/100	🟡
Accessibility	95/100	🟢
Best Practices	90/100	🟢
SEO	95/100	🟢

📥 Download full Lighthouse report

Budget Compliance: Performance budgets enforced via budget.json

Initial plan

b9b93a1

Copilot AI assigned Copilot and pethers Mar 2, 2026

Copilot started work on behalf of pethers March 2, 2026 11:43 View session

github-code-quality bot found potential problems Mar 2, 2026

View reviewed changes

tests/scb-client.test.ts

* @license Apache-2.0

*/

import { describe, it, expect, vi, beforeEach } from 'vitest';

Copilot AI and others added 2 commits March 2, 2026 12:09

Address code review: add logging in SCB client error paths

ad55dce

Co-authored-by: pethers <1726836+pethers@users.noreply.github.com> Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Analyse and improve news generation using data from World Bank and Swedish statistics~~ Add SCB client, statistical claims fact-checker, and complete World Bank/SCB→committee coverage Mar 2, 2026

Copilot finished work on behalf of pethers March 2, 2026 12:17

Copilot started work on behalf of pethers March 2, 2026 12:22 View session

Copilot AI changed the title ~~Add SCB client, statistical claims fact-checker, and complete World Bank/SCB→committee coverage~~ feat: Add SCB client/context modules and improve news generation test coverage Mar 2, 2026

Copilot finished work on behalf of pethers March 2, 2026 12:41

pethers requested a review from Copilot March 2, 2026 12:42

Copilot started reviewing on behalf of pethers March 2, 2026 12:43 View session

Copilot AI reviewed Mar 2, 2026

View reviewed changes

Copilot started work on behalf of pethers March 2, 2026 12:50 View session

fix: address PR review comments — Unicode regex, unused AbortControll…

41b317c

…er, non-JSON handling, typo, threshold separation, sitemap test Co-authored-by: pethers <1726836+pethers@users.noreply.github.com>

Copilot finished work on behalf of pethers March 2, 2026 13:00

pethers marked this pull request as ready for review March 2, 2026 13:07

pethers merged commit 5e34b93 into main Mar 2, 2026
17 checks passed

pethers deleted the copilot/analyse-agentic-workflows branch March 2, 2026 13:07

	if (options.recommendEconomicContext && !metrics.hasSCBContext) {
	if (metrics.hasStatisticalClaims && !metrics.hasSCBContext) {

	// SCB indicator mappings (12 domains → committees)
	// SCB indicator mappings (domains → committees)

	/\bbrå\b/i, // Crime statistics authority
	/brå/i, // Crime statistics authority (substring match to avoid Unicode \b issues)

		console.warn(`SCB MCP response for ${toolName} was not valid JSON; returning raw text`);
		return text;

		verificationSource: 'both',
		worldBankIndicator: 'NY.GDP.MKTP.KD.ZG',

	verificationSource: 'both',
	worldBankIndicator: 'NY.GDP.MKTP.KD.ZG',
	verificationSource: 'scb',

Conversation

Copilot AI commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New modules

Enhancements

Bug fixes

Tests (+163 new)

Usage

Uh oh!

github-actions bot commented Mar 2, 2026

🔍 Lighthouse Performance Audit

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

pethers commented Mar 2, 2026

Uh oh!

Copilot AI commented Mar 2, 2026

Uh oh!

github-actions bot commented Mar 2, 2026

🔍 Lighthouse Performance Audit

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Mar 2, 2026 •

edited

Loading