Skip to content

refactor: extract score set CSV generation into a dedicated, tested module #798

Description

@bencap

Summary

CSV generation lives in a pair of long functions inside mavedb/lib/score_sets.py, interleaved with unrelated search, creation, and upload logic. It is hard to test and maintain due to the size and complexity of the function, but is too important to leave with weak test cases. Extract it into a dedicated module of tested helpers behind a thin composer, fix the bug, and leave a natural home for the variant-level composer the clinical download work will need (#799).

Problem

  • get_score_set_variants_as_csv runs roughly 220 lines and does everything inline: column planning per namespace, ClinVar versioned-namespace parsing, substrate query construction, pagination, and row assembly.
  • variant_to_csv_row runs roughly 143 more lines as a sequence of per-namespace loops.
  • Both sit in a 1,289-line module beside score set search, variant creation, and upload parsing. There is no seam at which CSV behavior can be tested in isolation.

Consumers to preserve: the three score set CSV endpoints (variants, scores, counts), three call sites in the public data export script, and variants_to_csv_rows used during score set creation.

Proposed behavior

A dedicated module, e.g. mavedb/lib/score_set_csv.py, containing:

  • Column planning — build the namespaced column map from a score set and requested namespaces, as a pure function.
  • ClinVar versioned-namespace parsing — the clinvar.YEAR_MONTH to MONTH_YEAR db_version translation, independently testable.
  • Substrate query construction and pagination.
  • Per-namespace value resolution that returns an explicit sentinel for unrecognized keys instead of falling through.
  • Row assembly, and a thin composer that sequences the above and returns the CSV string.

csv_data_to_df stays where it is — it is the upload direction and a different concern.

Structure the module so a variant-level CSV composer can be added beside the score-set one idiomatically, since the clinical download discussion will land there. See #799.

Acceptance criteria

  • Existing CSV endpoints produce byte-identical output to current main for a fixture score set across every namespace combination they support (scores, counts, vep, gnomad, clingen, at least one ClinVar versioned namespace, and post-mapped HGVS on and off).
  • An unrecognized column key yields the NA representation or raises explicitly. It can never emit an adjacent namespace's value. Covered by a direct test.
  • Column planning, ClinVar namespace parsing, and row assembly each have unit tests that do not require a database session.
  • No CSV generation logic remains in mavedb/lib/score_sets.py apart from csv_data_to_df.
  • The public data export script and score set creation path are updated to the new module and still pass their existing tests.

Metadata

Metadata

Assignees

No one assigned

    Labels

    app: backendTask implementation touches the backendtype: maintenanceMaintaining this projecttype: testsTests for this project

    Type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions