Merge pull request #156 from aicodingstack/codex/catalog-models-desktop-release

feat: expand coding model and desktop agent catalog

Author: ericyangpan

.agents/skills/benchmark-fetcher/README.md

@@ -1,235 +1,43 @@
-# Benchmark Fetcher Skill - Implementation Complete
+# Benchmark evidence importer
 
-## Status: ✅ READY FOR USE
+The importer applies reviewed evidence; it does not browse or scrape leaderboards.
 
-The benchmark-fetcher skill has been successfully implemented and is ready to fetch benchmark data from 6 leaderboard websites.
+## Input format
 
-## What's Been Implemented
-
-### 1. Core Infrastructure ✅
-- ✅ Skill structure with SKILL.md documentation
-- ✅ Configuration system (config.mjs)
-- ✅ Model name mapping with 3-tier fuzzy matching
-- ✅ Atomic manifest updates with validation
-- ✅ Comprehensive reporting system
-
-### 2. Benchmark Extractors ✅
-- ✅ **SWE-bench** - Fully implemented with regex parsing
-- ✅ **TerminalBench** - Decimal format conversion (0-1 scale)
-- ✅ **MMMU** - Dual benchmark extraction (MMMU + MMMU Pro)
-- ✅ **SciCode** - Generic extraction pattern
-- ✅ **LiveCodeBench** - Generic extraction pattern
-- ✅ **WebDevArena** - Generic extraction pattern
-
-### 3. Model Name Mappings ✅
-Pre-configured mappings for:
-- Claude models (Opus 4.5, Opus 4.1, Sonnet 4.5, Haiku 4.5)
-- GPT models (GPT-5, GPT-5.1, GPT-5-Codex, GPT-4o, GPT-4.1)
-- Gemini models (Gemini 3 Pro, Gemini 2.5 Pro, Gemini 2.5 Flash)
-- DeepSeek models (DeepSeek R1, DeepSeek V3)
-- Other models (GLM 4.6, Grok 4, Grok Code Fast 1)
-
-## Quick Start
-
-### Test with Dry Run
-```bash
-node .claude/skills/benchmark-fetcher/scripts/fetch-benchmarks.mjs --dry-run
-```
-
-### Fetch All Benchmarks
-```bash
-node .claude/skills/benchmark-fetcher/scripts/fetch-benchmarks.mjs
-```
-
-### Fetch Specific Benchmarks
-```bash
-# Just SWE-bench and TerminalBench
-node .claude/skills/benchmark-fetcher/scripts/fetch-benchmarks.mjs --benchmarks swebench,terminalBench
-```
-
-### Update Specific Models Only
-```bash
-# Just update Claude Sonnet 4.5 and GPT-4o
-node .claude/skills/benchmark-fetcher/scripts/fetch-benchmarks.mjs --models claude-sonnet-4-5,gpt-4o
+```json
+{
+  "records": [
+    {
+      "modelId": "exact-existing-manifest-id",
+      "modelLabel": "Exact label shown by the leaderboard",
+      "benchmark": "sweBench",
+      "benchmarkVersion": "SWE-bench Verified, leaderboard/harness version",
+      "evaluation": "Agent scaffold, reasoning setting, pass@k, tool policy, or other conditions",
+      "score": 74.4,
+      "sourceUrl": "https://authoritative.example/leaderboard",
+      "sourceTitle": "Official leaderboard title",
+      "observedAt": "2026-07-21",
+      "verifiedBy": "github-handle-or-agent-id"
+    }
+  ]
+}
 ```
 
-## File Structure
-
-```
-.claude/skills/benchmark-fetcher/
-├── SKILL.md                              # Complete documentation
-├── README.md                             # This file
-├── references/
-│   └── model-name-mappings.json          # Model name mappings (58 mappings)
-└── scripts/
-    ├── fetch-benchmarks.mjs              # Main entry point
-    └── lib/
-        ├── config.mjs                    # Configuration
-        ├── model-name-mapper.mjs         # 3-tier fuzzy matching
-        ├── benchmark-extractors.mjs      # 6 website extractors
-        ├── manifest-updater.mjs          # Atomic updates
-        └── report-generator.mjs          # Formatted reporting
-```
-
-## Key Features
-
-### Intelligent Model Name Mapping
-The skill uses a 3-tier fallback strategy to map website model names to manifest IDs:
-1. **Exact match** (case-sensitive)
-2. **Case-insensitive match**
-3. **Fuzzy match** (normalized - removes spaces, hyphens, special chars)
-
-### Special Handling
-
-**TerminalBench Decimal Format:**
-- Website displays: "63.1%"
-- Stored as: `0.631` (decimal 0-1 scale)
-- ✅ Automatic conversion implemented
-
-**MMMU Dual Benchmarks:**
-- Single website visit extracts both MMMU and MMMU Pro scores
-- Updates two separate manifest fields
-- ✅ Fully implemented
-
-### Error Resilience
-- 3-attempt retry with exponential backoff
-- Graceful degradation (continues on errors)
-- Debug screenshots saved to `/tmp/benchmark-fetcher-debug/`
-- Comprehensive error reporting
-
-### Atomic Updates
-- Validates JSON structure
-- Writes to temporary file
-- Atomic rename (no partial updates)
-- All-or-nothing per manifest
-
-## What Happens When You Run It
+Supported benchmark keys are the exact manifest fields: `sweBench`, `terminalBench`, `mmmu`, `mmmuPro`, `webDevArena`, `sciCode`, and `liveCodeBench`.
 
-1. **Loads Configuration**
-   - Reads model-name-mappings.json
-   - Loads all model manifests from manifests/models/
+Every descriptive field is required so the generated source title retains the evaluation context that the scalar benchmark schema cannot otherwise represent.
 
-2. **Visits Each Website**
-   - Navigates using Chrome DevTools MCP
-   - Waits for content to load
-   - Takes accessibility tree snapshot
-   - Parses leaderboard data
+## Commands
 
-3. **Maps Model Names**
-   - Attempts 3-tier matching
-   - Logs unmapped models for manual addition
-
-4. **Updates Manifests**
-   - Always overwrites existing benchmark values
-   - Preserves all other manifest fields
-   - Uses atomic file writes
-
-5. **Generates Report**
-   - Shows successful/failed benchmarks
-   - Lists all manifest updates
-   - Reports unmapped models
-   - Provides next steps
-
-## Expected Output Example
-
-```
-📊 Benchmark Fetch Report
-================================
-
-✅ Successfully Fetched (6/6 benchmarks)
-   ✓ SWE-bench (swebench.com) - 15 models
-   ✓ TerminalBench (tbench.ai) - 20 models
-   ✓ MMMU + MMMU Pro (mmmu-benchmark.github.io) - 8 models
-   ✓ SciCode (scicode-bench.github.io) - 5 models
-   ✓ LiveCodeBench (livecodebench.github.io) - 12 models
-   ✓ WebDevArena (web.lmarena.ai) - 3 models
-
-📝 Manifest Updates
-
-✅ Updated: 12 manifests
-   • claude-sonnet-4-5: 4 benchmarks updated
-     - sweBench: null → 70.6
-     - terminalBench: null → 0.428
-     - sciCode: null → 4.6
-     - liveCodeBench: 47.1 → 52.3
+```bash
+# Preview; never writes
+node .agents/skills/benchmark-fetcher/scripts/fetch-benchmarks.mjs /path/to/evidence.json
 
-⚠️ Unmapped Models
-   Add these to model-name-mappings.json
+# Apply additions or null-to-score changes
+node .agents/skills/benchmark-fetcher/scripts/fetch-benchmarks.mjs /path/to/evidence.json --apply
 
-📈 Statistics
-   Execution time: 45.2s
+# Explicitly allow replacement of an existing non-null score
+node .agents/skills/benchmark-fetcher/scripts/fetch-benchmarks.mjs /path/to/evidence.json --apply --replace
 ```
 
-## Next Steps After Running
-
-1. **Review Updates**
-   - Check manifests/models/*.json for changes
-   - Verify benchmark values look correct
-
-2. **Add Unmapped Models**
-   - Update references/model-name-mappings.json
-   - Re-run to fetch their data
-
-3. **Validate**
-   ```bash
-   npm run test:validate
-   ```
-
-4. **Commit Changes**
-   ```bash
-   git add manifests/models/
-   git commit -m "Update benchmark data from leaderboards"
-   ```
-
-## Troubleshooting
-
-### Extractor Fails for a Benchmark
-- Check `/tmp/benchmark-fetcher-debug/` for screenshots
-- Website structure may have changed
-- Update extractor logic in benchmark-extractors.mjs
-
-### Model Not Updating
-- Verify model exists in manifests/models/
-- Check if model name is in mappings
-- Look for "unmapped" warnings in output
-
-### TerminalBench Shows Wrong Format
-- Verify values are < 1.0 (decimal format)
-- Check conversion logic in extractTerminalBench()
-
-## Implementation Notes
-
-### What Works Well
-- SWE-bench and TerminalBench extractors are fully tested
-- Model name fuzzy matching handles variations
-- Atomic updates prevent corruption
-- Comprehensive error handling
-
-### What May Need Refinement
-- MMMU, SciCode, LiveCodeBench, WebDevArena extractors use generic patterns
-- These may need adjustment based on actual page structures
-- Model name mappings will grow as new models appear
-
-### How to Improve Extractors
-1. Run with `--dry-run` to see what's extracted
-2. Check debug screenshots if extraction fails
-3. Examine page snapshots to understand structure
-4. Update extractor logic to match patterns
-5. Test and iterate
-
-## Success Criteria ✅
-
-- [x] Visits all 6 benchmark websites
-- [x] Extracts model performance data
-- [x] Maps model names correctly using configuration
-- [x] Updates model manifests with new values
-- [x] TerminalBench uses decimal format (0-1)
-- [x] MMMU updates both fields
-- [x] Generates comprehensive reports
-- [x] Handles errors gracefully with retry logic
-- [x] All manifests pass JSON schema validation
-- [x] Unmapped models are reported
-
-## Ready to Use! 🚀
-
-The skill is fully functional and ready to fetch benchmark data. Start with a dry run to see what it will do, then run without `--dry-run` to update the manifests.
+The importer validates exact model IDs, score ranges, dates, HTTPS sources, duplicate records, and overwrite conflicts. It also adds field-scoped source provenance and updates verification metadata. Always inspect the Git diff afterward.