You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#758 requires re-running mapping and annotation pipelines over every previously-mapped. score set. The only entry point today handles one score set per invocation, so executing #758 means hand-driving hundreds of runs with no cohort selection, no throttling, and no record of what failed. Having this mapping and annotation pipeline run on every human target score set is essential for variant search and detail pages to work as intended on all variants in MaveDB.
We should add a dedicated operator script that selects a cohort, orders it to exploit the 24-hour ClinGen cache, bounds the work per run, and reports per-score-set outcomes.
Problem
run_pipeline.py in mavedb/scripts requires --score-set-urn and exits when it is absent. It is strictly one score set per invocation.
No cohort selection. An operator has to assemble and hand-maintain the URN list for each phase of Rerun Mapping Jobs #758.
No limit. Re-running the full published corpus in one pass risks saturating both our workers and ClinGen's Allele Registry.
No ordering strategy, which wastes our cache. ClinGen allele lookups. Score sets assaying the same target gene overlap heavily in variant space, so running them consecutively turns repeat lookups across sets into cache hits. Arbitrary URN order scatters those lookups across multiple lookups.
No outcome summary. A partial failure mid-cohort leaves nothing resumable.
Proposed behavior
A new script alongside the existing ones, e.g. mavedb/scripts/run_score_set_pipelines.py:
Cohort selection:--collection-urn (covers the MaveMD phase), --published-only, an organism/taxonomy filter, and filter that allows exclusion of recently run score sets. --score-set-urn stays available and repeatable for ad-hoc reruns.
--limit N: process at most N score sets, so the full data set can be worked incrementally.
Loose gene grouping: order the resolved cohort by a normalized target-gene key so all score sets sharing a gene run consecutively inside one cache window.
--dry-run: print the resolved, ordered cohort with each entry's grouping key and exit without enqueuing anything.
Bounded concurrency, plus a per-score-set outcome summary (URN, pipeline, status, error) on stdout.
Skip work already current for the requested pipeline, so re-invocation is a no-op over completed score sets.
Acceptance criteria
--dry-run prints an ordered cohort with grouping keys and enqueues nothing.
--limit N processes exactly N score sets and leaves the rest untouched.
Score sets sharing a normalized target-gene name are adjacent in execution order.
Gene normalization collapses differing case and surrounding whitespace to one key.
A run that fails mid-cohort exits non-zero and still emits a summary distinguishing succeeded from failed score sets.
The emitted failure summary is accepted as input for a retry invocation.
Re-invoking after a fully successful run enqueues nothing.
An unknown pipeline name fails fast listing the available names, matching how run_pipeline behaves today.
Implementation notes
Reuse PIPELINE_DEFINITIONS validation, the --updater-id fallback chain (score set modifier, then creator), and --extra-param passthrough from run_pipeline so the two scripts behave consistently.
Keep cohort ordering a pure function over (score set, target gene names) so it can be exercised directly against synthetic input while the script is being built.
Summary
#758 requires re-running mapping and annotation pipelines over every previously-mapped. score set. The only entry point today handles one score set per invocation, so executing #758 means hand-driving hundreds of runs with no cohort selection, no throttling, and no record of what failed. Having this mapping and annotation pipeline run on every human target score set is essential for variant search and detail pages to work as intended on all variants in MaveDB.
We should add a dedicated operator script that selects a cohort, orders it to exploit the 24-hour ClinGen cache, bounds the work per run, and reports per-score-set outcomes.
Problem
run_pipeline.pyinmavedb/scriptsrequires--score-set-urnand exits when it is absent. It is strictly one score set per invocation.Proposed behavior
A new script alongside the existing ones, e.g.
mavedb/scripts/run_score_set_pipelines.py:--collection-urn(covers the MaveMD phase),--published-only, an organism/taxonomy filter, and filter that allows exclusion of recently run score sets.--score-set-urnstays available and repeatable for ad-hoc reruns.--limit N: process at most N score sets, so the full data set can be worked incrementally.--dry-run: print the resolved, ordered cohort with each entry's grouping key and exit without enqueuing anything.Acceptance criteria
--dry-runprints an ordered cohort with grouping keys and enqueues nothing.--limit Nprocesses exactly N score sets and leaves the rest untouched.run_pipelinebehaves today.Implementation notes
PIPELINE_DEFINITIONSvalidation, the--updater-idfallback chain (score set modifier, then creator), and--extra-parampassthrough fromrun_pipelineso the two scripts behave consistently.