Add Chrysalis E3SM v3 archive ingestion

[tomvothecoder]

Description

Adds a targeted E3SM v3 archive ingestor that runs on Chrysalis and uploads cases to SimBoard through the remote HPC endpoint. This is required because v3 performance data is stored on Chrysalis rather than a filesystem accessible from NERSC.

• Defines documented v3 simulation names and scans archive snapshots from 2024-01
• Reuses remote discovery, validation, deduplication, packaging, and upload logic
• Skips whole-snapshot checkpoints for safe filtered reconciliation
• Adds reconciliation summaries, regression tests, and operator documentation
• Closes [DevOps]: Add v3 ingestion script for Chrysalis #280

Checklist

• Code follows project style guidelines
• Self-reviewed code
• No new warnings
• Tests added or updated
• All tests pass locally
• Documentation/comments updated
• No breaking change

Deployment Notes (if any)

Run dry-run reconciliation on Chrysalis with an externally reachable SIMBOARD_API_BASE_URL and service-account token before enabling uploads.

[Copilot]

Pull request overview

Adds a Chrysalis-specific, targeted E3SM v3 archive ingestor that filters to a
static list of documented v3 simulations, scans archive snapshots from 2024-01,
packages each selected case, and uploads via the remote
/api/v1/ingestions/from-hpc-upload route (without using whole-snapshot
checkpoints).

Changes:

• Introduces chrysalis_v3_archive_ingestor.py and a v3 reconciliation summary
  that fails the run when expected simulations are missing/transient/incomplete.
• Extends the shared archive ingestion plumbing to support optional case-path
  filtering, optional archive checkpointing, and a run-report object for
  specialized runners.
• Adds operator documentation and dedicated regression tests for the targeted
  v3 backfill behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
docs/deploy/hpc-api-token-authentication.md	Documents Chrysalis v3 backfill workflow and upload endpoint usage.
docs/architecture/metadata-ingestion.md	Explains how the v3 backfill specializes remote automated upload mode and why checkpointing is disabled.
backend/tests/features/ingestion/test_chrysalis_v3_archive_ingestor.py	Adds regression coverage for v3 filtering, forced config, checkpoint bypass, and reconciliation outcomes.
backend/app/scripts/README.md	Documents the new Chrysalis v3 backfill script entry point and operational expectations.
backend/app/scripts/ingestion/nersc_archive_ingestor.py	Adds IngestorRunReport, configurable env overrides, and optional case filtering + report capture during archive scans/ingest.
backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py	Adds hooks for case filtering, disabling archive checkpointing, and emitting run-report details.
backend/app/scripts/ingestion/chrysalis_v3_archive_ingestor.py	Implements the targeted v3 archive scan/upload runner with reconciliation logging and non-checkpoint behavior.

[tomvothecoder]

Need clarification on what directories to process v3 data. The v3 simulations were ran on Chrysalis, which means the performance data is there too.

On the E3SM Data Docs, there are HPSS links to the output as well. We can either

1. Parse the original performance data on Chrysalis
2. See if the required metadata is available on NERSC

[tomvothecoder]

Need clarification on what directories to process v3 data. The v3 simulations were ran on Chrysalis, which means the performance data is there too.

On the E3SM Data Docs, there are HPSS links to the output as well. We can either

1. Parse the original performance data on Chrysalis

2. See if the required metadata is available on NERSC

Hey @chengzhuzhang, can you clarify where we should be ingesting v3 data?

[tomvothecoder]

LCRC E3SM V3 dry-run results

Summary: The V3 archive dry run completed successfully: all 30 expected simulations were found, 423 cases are ready for ingestion, and no missing, transient, or invalid executions were detected.

A dry run completed successfully against the Chrysalis archive.

Metric	Result
Exit code	0
Duration	248.046s
Expected simulations	30
Matched simulations	30
Missing simulations	0
Matching case directories	542
Accepted execution directories	632
Selected submission cases	423
Transient execution IDs	0
Invalid execution IDs	0
Incomplete execution IDs skipped	243
Ingestion failures	0

The runner found every expected V3 simulation, completed traversal, and found no transient or invalid execution metadata. The incomplete executions were excluded from submission as intended.

Reproduce on Chrysalis

Run from the repository's backend/ directory at the tested revision:

Run from the repository's backend/ directory at the tested revision:

git checkout devops/280-v3-ingestion
git rev-parse HEAD
uv sync

Create a local credentials file outside the repository:

mkdir -p ~/.config/simboard
cp app/scripts/ingestion/v3_data/lcrc-v3.env.example \
  ~/.config/simboard/lcrc-v3.env
chmod 600 ~/.config/simboard/lcrc-v3.env
Edit ~/.config/simboard/lcrc-v3.env and set:
SIMBOARD_API_BASE_URL=https://<simboard-api-host>
SIMBOARD_API_TOKEN=<service-account-token>
DRY_RUN=true

Optionally set OLD_PERF_ARCHIVE_ROOT when the archive is mounted somewhere other than the documented Chrysalis default.

Run the dry run:
LCRC_V3_ENV_FILE="$HOME/.config/simboard/lcrc-v3.env" \ ./app/scripts/ingestion/v3_data/lcrc_v3.sh

Confirm the final v3_ingestion_summary reports:

• missing_simulations=[]
• transient_execution_ids=0
• rejected_invalid_execution_ids=0
• exit_code=0

Live run
After reviewing the dry-run output, change DRY_RUN=false in the external env file and rerun the same wrapper:

LCRC_V3_ENV_FILE="$HOME/.config/simboard/lcrc-v3.env" \ ./app/scripts/ingestion/v3_data/lcrc_v3.sh

Then restore DRY_RUN=true and rerun the dry run. Successful uploads should leave selected_submission_cases=0 and increase the existing/processed execution counts.

[tomvothecoder]

Next steps: do a full run on Monday