Restructuring: top-level layout + workflow/papers/scratch division

[cailmdaley]

Restructuring: top-level layout + workflow / papers / scratch split

Reorganizes sp_validation around one principle — the things you run live at the top — with a clean split between analysis, paper figures, and personal scratch, plus a modular Snakemake workflow built for more than one person. Base branch: develop (untouched).

Layout

sp_validation/
├── src/sp_validation/   library code (incl. the glass_mock core)
├── cosmo_val/           shear / cosmology validation — code + config
├── cosmo_inference/     inference — code + config (cosmosis / cosmocov)
├── workflow/            ALL analysis — modular Snakemake, multi-person → results/
├── papers/              final-figure assembly only — bmodes, catalog, cosmo_val, harmonic
├── scripts/             reduction scripts (+ examples/, glass_mock/)
├── scratch/             per-person ad hoc work, tracked (cdaley/, guerrini/)
├── results/             analysis products + diagnostic plots (contents gitignored)
└── docs/  config/  + tests under src/sp_validation/tests/

Previously cosmo_val was buried inside notebooks/ while cosmo_inference/ sat at the top, so you had to hunt for where each piece lived. Now the things you actually run sit side by side, sharing src/ underneath.

Division of labor

The boundary is the inputs to a paper figure: everything up to that point is analysis, the figure itself is presentation.

• workflow/ — all analysis. Generic, reusable, modular. Produces analysis products and diagnostic plots into results/. The bulk of the work lives here.
• papers/{name}/ — final-figure assembly only. Figure PDFs, colour, layout. Tied to one paper; may never touch Snakemake.
• scratch/{person}/ — personal, ad hoc, tracked. One-off experiments and custom workflows; tracked because seeing each other's scratch is useful.

Modular workflow

Nothing here is computed once — the catalog changed ~20× in the first release, and every paper varies the data vector, covariance, and inference. So the workflow is parameterized: the rules are shared, the config changes per run. Snakemake's module directive imports the rules under your own config and an output prefix, with per-rule override:

module analysis:
    snakefile: "../../workflow/Snakefile"
    config:    config           # this run's catalog, cuts, blind
    prefix:    "results/bmodes"  # products land here — no clobbering
use rule * from analysis

Each run namespaces under results/{run}/; a --dry-run on every composition guards against silent breakage as the structure grows.

Notebooks

The analysis tree is now notebook-free — the top-level notebooks/ directory is gone, and every notebook was moved to a proper home, converted, or deleted:

• Reusable code → library / scripts. Reduction-notebook logic lives in src/sp_validation/; runnable workflows in scripts/examples/ (e.g. extract_info.py, calibrate_comprehensive_cat.py, leakage_minimal.py).
• Tutorial → docs. tutorial_UNIONS_SP_v1.0 is now the live Sphinx page "Using the weak-lensing catalogues".
• Paper-plot notebooks → papers/. The harmonic plot set and the catalog check_gaia plot moved under papers/{harmonic,catalog}/ (kept as notebooks — final-figure assembly).
• cosmo_val/ resolved. Generic helpers lifted into src/sp_validation/basic.py; the five working notebooks converted to jupytext percent-light scripts under scratch/guerrini/. cosmo_val/ now holds only code + config + README.
• Obsolete deleted. The exploratory reduction notebooks, glass_mock/validate_glass_mock, and defunct/ (quarantined since 2024) — all recoverable from develop history.
• Untouched: cosmo_inference/ keeps its own working notebooks in place.
• Discipline via tooling. nbstripout strips notebook outputs on commit, plus a large-file pre-commit hook (pre-commit install; see CONTRIBUTING).

Full per-notebook ledger (every .ipynb moved or deleted vs develop)

Moved → papers/harmonic/ (from cosmo_inference/notebooks/2D_harmonic_space_cosmic_shear_plots/, preserved as notebooks):
2025_09_26_plot_contours + its variants (_NL_modelling, _blind, _cl_vs_xi, _covariance, _glass_mock, _iNKA_vs_OneCov, _leakage, _scale_cut, _small_vs_large_scales, _weak_lensing), 2025_10_28_plot_whisker, S8_whisker

Moved → papers/catalog/ (from notebooks/cosmo_val/catalog_paper_plot/):
check_gaia

Promoted → docs (from notebooks/analyse_shear_cat/):
tutorial_UNIONS_SP_v1.0 → docs/source/using_the_catalogues.md

Converted → scripts in scratch/guerrini/ (from notebooks/cosmo_val/; reusable helpers chi2_and_pte, corr_from_cov, cov_from_one_covariance lifted to src/sp_validation/basic.py):
compute_pte_cell.py, one_covariance.py, plot_comparison.py, get_prior_leakage.py, exploration.py (+ its namaster_utils.py helper)

Deleted (recoverable from develop history):

• notebooks/: main_set_up, metacal_global, metacal_local, psf_leakage, maps, maps_local, match_stats, correlation, cosmology, write_cat, frac_error_local_calib, analyse_matched_stars_UNIONS_HSC, analyse_shear_cat/m2_SP_LF_alpha
• glass_mock/: validate_glass_mock
• defunct/: TD_WL_cycle2_2021, validation_local_cal

glass_mock

The generation core folded into src/sp_validation/glass_mock.py; the runner scripts moved to scripts/glass_mock/. The top-level glass_mock/ directory is gone.

Still open

Some scratch/ material may still deserve promotion into src/ or workflow/ and was intentionally left untouched in this PR — most notably scratch/guerrini/namaster_utils.py, a cohesive ~200-line NaMaster covariance toolkit that overlaps cosmo_val/harmonic_covariance_gaussian_sims.py. Consolidating the two (and adding tests) is a covariance-API decision best left to Sacha rather than forked here.

Safety net

A back-pressure guard suite holds structural invariants as files move — imports + standalone-scripts/ resolution, snakemake -n dry-runs, config-path existence, symlink integrity, and a dangling-reference / move-map guard. Sacha's develop foundation (paper plots, harmonic configs, library changes) is folded in, with the cosmology.py mnu KeyError fixed.

— Claude on behalf of Cail