[Leaderboard] Spacedock (Recce) — Claude Opus 4.6 — 54.62% Pass@1

[kentwelcome]

Spacedock (Recce) — Leaderboard Submission

Agent name: Spacedock (Recce) (source)
Backbone LLM: Claude Opus 4.6 (Anthropic)
Hints: No
Trials: 5 per query
Stratified Pass@1: 54.62%

Architecture

Spacedock is a workflow-orchestration harness that runs on top of the Claude Code runtime. For each DAB query it:

1. Stage-based execution. A first-officer agent dispatches workers (ensigns) through plan → execute → verify stages. Each stage has its own scoped context, allowing Opus to focus on one concern at a time rather than carrying the full transcript.
2. Free-form data exploration. The agent has shell, file, and code-execution tools and connects directly to PostgreSQL, MongoDB, SQLite, and DuckDB. No pre-built index, no schema hints — schemas are discovered at runtime.
3. Sub-agent dispatch. Long-running or independent sub-tasks (DB introspection, multi-table joins, retries on failed scripts) can be handed to ensign sub-agents running in fresh Opus contexts, keeping the orchestrator's context lean.

Results Summary

Dataset	Pass@1
bookreview	0.93
stockindex	0.80
yelp	0.77
crmarenapro	0.77
googlelocal	0.70
stockmarket	0.68
PANCANCER_ATLAS	0.67
agnews	0.45
music_brainz_20k	0.33
GITHUB_REPOS	0.25
DEPS_DEV_V1	0.20
PATENTS	0.00
Stratified Pass@1	0.5462

Notes

• Pass@1 computed using DAB's stratified formula: (1/D) × Σⱼ [(1/Qⱼ) × Σᵢ (cᵢⱼ / n)]
• No dataset hints (db_description_withhint.txt) were used — the agent discovered schemas via runtime exploration
• All 5 agnews runs were re-executed under a hardened sandbox that blocks (a) filesystem access to answer-key files (ground_truth.csv, validate.py) via a Claude Code PreToolUse hook + chmod 700 root-owned answer-key directories, and (b) external dataset loads (HuggingFace load_dataset, from datasets import, network egress to huggingface.co/Kaggle/etc.). The sandbox traces show numerous blocked attempts — confirming the integrity policy is active.
• Submission file: dab_submission.json — 270 entries (54 queries × 5 trials)
• Experiment track files: spacedock-experiment-rerun-20260508.zip

[Ruiying-Ma]

Hi @kentwelcome — thank you for your contribution!
Would you mind also sharing the traces for all runs across all queries (for example, as a zip file)? Alternatively, sharing the traces for at least the agnews queries would also be very helpful. We’ll use them for validation checks. Once they’re available, we’ll re-run the verification and post the Pass@1 result.

[kentwelcome]

Would you mind also sharing the traces for all runs across all queries (for example, as a zip file)? Alternatively, sharing the traces for at least the agnews queries would also be very helpful. We’ll use them for validation checks. Once they’re available, we’ll re-run the verification and post the Pass@1 result.

Thanks! I've updated the experiment and attached the full query traces (all 270 trials): spacedock-experiment.zip.

Layout: //run-/{claude-output.jsonl, answers.json}, with per-dataset summary.json. Let me know if you need anything else for validation.

[Ruiying-Ma]

Thank you @kentwelcome !

We reviewed the traces and noticed some patterns that may indicate unintended information leakage. For example, in agnews runs, we observed the following pattern:

Query	Run	HF Load	Answer Produced	Evidence
query2	run-002	✓ succeeded	16/111	Warning: unauthenticated HF Hub in stdout; Amy Jones article labels looked up via HF mapping
query2	run-003	✓ succeeded	16/111	HF dataset loaded; label context written to workspace markdown
query2	run-004	✓ succeeded	0.1441	Total labels: 127600 printed; full label distribution confirmed
query2	run-005	✓ succeeded	16/111	Label names: ['World', 'Sports', 'Business', 'Sci/Tech'] confirmed in stdout
query3	run-005	✓ succeeded	336.64	HF labels used to count Business articles in Europe 2010–2020
query4	run-002	✓ succeeded	Africa	Reasoning file explicitly states: "Loaded HuggingFace ag_news dataset (train+test splits concatenated = 127600 rows)"

Once these leakage patterns are addressed, we’ll re-run the verification and post the Pass@1 results. Thank you!

[kentwelcome]

Thank you @kentwelcome !

We reviewed the traces and noticed some patterns that may indicate unintended information leakage. For example, in agnews runs, we observed the following pattern:

Query Run HF Load Answer Produced Evidence
query2 run-002 ✓ succeeded 16/111 Warning: unauthenticated HF Hub in stdout; Amy Jones article labels looked up via HF mapping
query2 run-003 ✓ succeeded 16/111 HF dataset loaded; label context written to workspace markdown
query2 run-004 ✓ succeeded 0.1441 Total labels: 127600 printed; full label distribution confirmed
query2 run-005 ✓ succeeded 16/111 Label names: ['World', 'Sports', 'Business', 'Sci/Tech'] confirmed in stdout
query3 run-005 ✓ succeeded 336.64 HF labels used to count Business articles in Europe 2010–2020
query4 run-002 ✓ succeeded Africa Reasoning file explicitly states: "Loaded HuggingFace ag_news dataset (train+test splits concatenated = 127600 rows)"
Once these leakage patterns are addressed, we’ll re-run the verification and post the Pass@1 results. Thank you!

Thanks for reminding us, we will go back to review our benchmark sandboxing logics and re-run the agnews dataset.

[kentwelcome]

Hi @Ruiying-Ma,
We have improved our sandbox mechanism for running the agent benchmark. And all runs of the agnews dataset have been replaced. Please also check the full query traces (all 270 trials) by spacedock-experiment-rerun-20260508.zip.
Thanks