Summary
CSV generation lives in a pair of long functions inside mavedb/lib/score_sets.py, interleaved with unrelated search, creation, and upload logic. It is hard to test and maintain due to the size and complexity of the function, but is too important to leave with weak test cases. Extract it into a dedicated module of tested helpers behind a thin composer, fix the bug, and leave a natural home for the variant-level composer the clinical download work will need (#799).
Problem
get_score_set_variants_as_csv runs roughly 220 lines and does everything inline: column planning per namespace, ClinVar versioned-namespace parsing, substrate query construction, pagination, and row assembly.
variant_to_csv_row runs roughly 143 more lines as a sequence of per-namespace loops.
- Both sit in a 1,289-line module beside score set search, variant creation, and upload parsing. There is no seam at which CSV behavior can be tested in isolation.
Consumers to preserve: the three score set CSV endpoints (variants, scores, counts), three call sites in the public data export script, and variants_to_csv_rows used during score set creation.
Proposed behavior
A dedicated module, e.g. mavedb/lib/score_set_csv.py, containing:
- Column planning — build the namespaced column map from a score set and requested namespaces, as a pure function.
- ClinVar versioned-namespace parsing — the
clinvar.YEAR_MONTH to MONTH_YEAR db_version translation, independently testable.
- Substrate query construction and pagination.
- Per-namespace value resolution that returns an explicit sentinel for unrecognized keys instead of falling through.
- Row assembly, and a thin composer that sequences the above and returns the CSV string.
csv_data_to_df stays where it is — it is the upload direction and a different concern.
Structure the module so a variant-level CSV composer can be added beside the score-set one idiomatically, since the clinical download discussion will land there. See #799.
Acceptance criteria
- Existing CSV endpoints produce byte-identical output to current
main for a fixture score set across every namespace combination they support (scores, counts, vep, gnomad, clingen, at least one ClinVar versioned namespace, and post-mapped HGVS on and off).
- An unrecognized column key yields the NA representation or raises explicitly. It can never emit an adjacent namespace's value. Covered by a direct test.
- Column planning, ClinVar namespace parsing, and row assembly each have unit tests that do not require a database session.
- No CSV generation logic remains in
mavedb/lib/score_sets.py apart from csv_data_to_df.
- The public data export script and score set creation path are updated to the new module and still pass their existing tests.
Summary
CSV generation lives in a pair of long functions inside
mavedb/lib/score_sets.py, interleaved with unrelated search, creation, and upload logic. It is hard to test and maintain due to the size and complexity of the function, but is too important to leave with weak test cases. Extract it into a dedicated module of tested helpers behind a thin composer, fix the bug, and leave a natural home for the variant-level composer the clinical download work will need (#799).Problem
get_score_set_variants_as_csvruns roughly 220 lines and does everything inline: column planning per namespace, ClinVar versioned-namespace parsing, substrate query construction, pagination, and row assembly.variant_to_csv_rowruns roughly 143 more lines as a sequence of per-namespace loops.Consumers to preserve: the three score set CSV endpoints (variants, scores, counts), three call sites in the public data export script, and
variants_to_csv_rowsused during score set creation.Proposed behavior
A dedicated module, e.g.
mavedb/lib/score_set_csv.py, containing:clinvar.YEAR_MONTHtoMONTH_YEARdb_version translation, independently testable.csv_data_to_dfstays where it is — it is the upload direction and a different concern.Structure the module so a variant-level CSV composer can be added beside the score-set one idiomatically, since the clinical download discussion will land there. See #799.
Acceptance criteria
mainfor a fixture score set across every namespace combination they support (scores, counts, vep, gnomad, clingen, at least one ClinVar versioned namespace, and post-mapped HGVS on and off).mavedb/lib/score_sets.pyapart fromcsv_data_to_df.