[DevOps]: Add Chrysalis ingestion wrapper and deploy - #169
[DevOps]: Add Chrysalis ingestion wrapper and deploy #169tomvothecoder wants to merge 12 commits into
Conversation
|
@tomvothecoder Once I am clear on the boundaries to the term "NERSC ingestion wrapper", I should be able to comprehent "Chrysalis ingestion wrapper". The term "scheduler-agnostic" refers to Jenkins? (I always considered cron to be universal...). |
TonyB9000
left a comment
TonyB9000
left a comment
There was a problem hiding this comment.
Configures a call to the hpc_archive_ingestor. Understandable.
What process sets "SIMBOARD_API_BASE_URL" and "SIMBOARD_API_TOKEN"?
|
Hmmmm. The "Tom Requested your review" took me to the page with 7 files to examine, each with a "submit-review" option. As soon as I completed the first one, all 7 vanished... |
Accidentally tagged you for review. I meant to assign this PR you. It is fixed now. |
|
@tomvothecoder "Accidentally tagged you for review". OK, (I think colleges should offer a master's program in github). |
|
Chrysalis and other non-NERSC sites require upload-based ingestion rather than path-based ingestion, so follow-up work is tracked in #207 for a state-first HPC upload flow with DB-backed dedupe parity. |
|
Using the "upload-based' vs "path-based" terminology, my thought was that when the NERSC upload-receiving system was deliverd an upload from a non-NERSC system, it could open it in the existing NERSC PA-directory under (say) "From_crysalis/<new_exec_ids>" and then process it with the existing "path-based" codes - assuming PACE would not interfere with it (and vice-versa). But on second thought, to avoid PACE crossing, it would be best to open it in a separate "PACE-unaware" directory. |
|
@tomvothecoder I am preparing to exercise "hpc_upload_archive_ingestor.py" on chrysalis, to see the logs and flow (in dry-run) in action, discover parameter faults, etc. QUESTION: Although, on NERSC, the backend ingestion is "path-based" (returnsp paths for ingestion), it could in principle run the "https-transfer-based" codes just as easily. I might try a dryrun on NESRC/Perlmutter first, since that configuration is already a known item. Then, differences in behavior on chrysalis would stand out. Does that make sense? |
22a1a88 to
feae197
Compare
|
@tomvothecoder Apologies if I'm doing this wrong. I attempted to test the "hpc_upload" on NESRC, thinking "--help" might be helpful. To get started, I needed an environment where I could install things, so: |
|
Hey Tony, happy to help and no apologies needed.
SimBoard defines the Python backend dependencies in You can run
I'd checkout this branch now that I've rebased it on the latest The |
|
@tomvothecoder I get the latest stuff - but I have made progress. My latest run_script (NERSC dry_run test) says: The output indicates that I am missing "archive_root” and “has_api_token”. By examining the "nersc" "_build_config" function, I can see what variables exist to push into the environment. I'll checkout branch #169 on both NERSC and Chrysalis to do comparisons in outputs. |
|
@tomvothecoder git gets me again: You wrote: "I'd checkout this branch now that I've rebased it on the latest main commit." is "this branch" off of main, as you had advised? Or is it off of a fork?? ((test_simboard) ) (base) [ac.bartoletti1@chrlogin1 simboard]$ git branch -a
When I get too confused, I do a clean "git clone". Then I can do one of these: Which is appropriate in this case? |
|
@tomvothecoder If I pull down a branch off of someone's fork, am I in that fork, or can I pull that into a new branch of my local main? The persistence of branches and forks, between local and remote, is a bit of a mystery. |
This branch ( Something like this (I did not verify correctness): git remote add tomvothecoder https://github.com/tomvothecoder/simboard
git checkout tomvothecoder feature/154-ingestion-ites I usually work directly on upstream and not fork when possible, but in this case I use a fork for separate testing purposes. |
|
@tomvothecoder Sorry, I guess I must pull from your fork. Quick test: I cd to ``/home/ac.bartoletti1/gitrepo/simboard/backend" and issue
The result: |
|
@tomvothecoder The line in "chrysalis.sh" script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; echo $script_dir clearly wont work for defining "backend_root" as backend_root="$(cd "${script_dir}/../../../.." && pwd)" I will modify chrysalis.sh to provide a "backend_root" that does not depend upon the user location., at least for test purposes. |
|
@tomvothecoder Works better now that it can find "backentd/apps" When I use this for "chrysalis.sh": and issue these exports: I get: I guess, even "dry_run" requires real URLs and API_tokens. That is because we need "state" up front. |
|
Hi @tomvothecoder The document also says: One-case-per-request rule:
The term "alongside the archive" is a bit ambiguous. Would this be accurate?
Or am I misunderstanding the intent? |
Great to see the progress! Yes, the dry run needs to query the SimBoard database via the REST API. I will send the
Your info sounds more accurate, thanks for the suggestion. Can you point me to the source document with this info? I will update it. |
|
@tomvothecoder Running "chrysalis.sh" with the full (DRY_RUN) parameters yieded the following summary (folded for readability): Questions that arise:
Observation: The bulk of work getting to this point involved stuffing the right ENV VARS and having created an environment where misc modules like "dateutils" could be installed. On Chrysalis, I performed On NERSC/Perlmutter, I simply replaced "python3.12" with "python3.11". I intend to perform the same test on Perlmutter, just to exercise the mechanisms of networking. |
|
@tomvothecoder For comparison, running the equivalent commands on perlmutter (swapping our parameters where necessary), we obtain the summary: I suppose I should re-run the chrysalis test, using "OLD_PERF" as the root_PA directory. It is HUGE. |
Since this is a first-time dry-run on the Chrysalis
That exact counter is not in So yes: a test expecting that exact field is probably unrealistic or stale.
The ingestor script only asks SimBoard for enough existing ingestion state to decide which archive cases and their executions are new and may be candidates for ingestion. It does not fetch, return, or summarize the full database state. It also does not show the database query because the query is behind the SimBoard API, not inside the ingestor script. So this is not a Chrysalis-specific DB query in the ingestor. It is an API request filtered by the configured If more detail is needed, the API response or ingestor summary would need to be expanded to include counts like total known cases, known execution IDs, skipped known cases, and machine filter used. Happy for you to open a new GitHub issue to expand logging in https://github.com/E3SM-Project/simboard/blob/main/backend/app/scripts/ingestion/nersc_archive_ingestor.py and https://github.com/E3SM-Project/simboard/blob/main/backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py.
They are dry-run logging counters.
The point is to avoid massive logs when many candidate cases are found. It does not change which cases are candidates or which cases would be ingested. |
I don't think this is going to work yet as the directory structure of "OLD_PERF" is different from "performance_archive". We might also want to be targeted in what we ingest from "OLD_PERF". This will require guidance Rob/Jill. |
|
@tomvothecoder Certainly. It should not take long. I should grab the latest from main, then (not your fork, wherein I pushed the generic "simboard_pa_collection.sh" script the the .config files, I assume. |
|
@tomvothecoder After a fresh git-clone, it no longer finds "backend/.venv/bin/python". Must "make install" (why?) |
|
@tomvothecoder So far, so good: (Why does it say "machine name = perlmutter, when I am running it on chrysalis? . . .) |
|
@tomvothecoder FAILED to obtain state over API: |
You're using Check your
Could be due to the attempt in fetching state for |
|
@tomvothecoder It's ok now. I issued "export MACHINE_NAME=chrysalis" manually, then re-ran, and now it is advancing normally. Never had to do that before. Curiously, the hostname is actually "chrlogin1.lcrc.anl.gov", so how and why we would issue "chrysalis" is somewhat of a mystery. I was hoping some standard command would return "chrysalis" as the machine-name. But I will instead add
to the chrysalis.config file. |
|
Completed: |
The ingestion scripts don't have a
I think the issue is that I set the default |
fb76259 to
289a693
Compare
|
@tomvothecoder Side Note: What are the pros and cons of supplying a parameter via command line argument:
versus through environment variable
Which is more self-documenting? The method of ENV VARS has greater scope (any subsequent application in a chain can inherit the variables) but is this always desirable? |
I generally prefer supplying env variables through The reason why |
There was a problem hiding this comment.
@TonyB9000, I am reviewing commit 289a693 in this PR.
My understanding is:
- The site-specific
.shwrappers are being replaced by.configfiles. simboard_pa_collection.shbecomes the shared entry point for all sites.
This seems like a reasonable direction since it reduces duplicated workflow logic. Could you clarify how the site config is expected to be selected and deployed? The script currently always sources $HOME/.simboard.config, while the repository contains separate chrysalis.config and nersc.config files.
I understand that the cron job itself still needs to be user-owned because there is no sudo access for a system-level job. My related question is whether the shared SimBoard scripts, configs, logs, or other operational files should live in a project directory rather than that user’s home directory. The user-specific cron entry could then invoke the shared project-owned workflow while keeping only genuinely user-specific settings in the home directory.
| source $SIMBOARD_API_TOKEN_CMD | ||
| : "${SIMBOARD_API_TOKEN:?SIMBOARD_API_TOKEN failed to be set.}" |
There was a problem hiding this comment.
Where is this SIMBOARD_API_TOKEN_CMD at?
| export SIMBOARD_WORKDIR="${HOME}/Ops/simboard" | ||
| export SIMBOARD_REPODIR="${HOME}/gitrepo/simboard/backend" | ||
| export SIMBOARD_ENV_CMD="source $HOME/envs/test_simboard/bin/activate" |
There was a problem hiding this comment.
Would it make sense to keep SimBoard operational files in a shared project-owned directory rather than an individual user’s home directory? That may help avoid ownership, permission, continuity, and offboarding issues, while still leaving user home directories for personal overrides, credentials (E.g., SIMBOARD_API_TOKEN), debugging files, etc.
|
Note, I have two runs on Chrysalis with the latest commit of this branch using Staging Dir -- completed in 940 seconds 2026-08-05 12:18:50,791 [INFO]: archive_ingestor_core.py(_log_event:839) >> event=dry_run_summary_counts mode=dry-run discovered_cases=932 submission_qualified_cases=932 selected_submission_cases=932 execution_dirs_scanned=4040 execution_dirs_accepted=1910 skipped_incomplete=2130 skipped_invalid=0 skipped_transient=0
2026-08-05 12:18:50,791 [INFO]: archive_ingestor_core.py(_log_event:839) >> event=dry_run_summary_candidates accepted_execution_ids=1910 rejected_existing_execution_ids=0 rejected_incomplete_execution_ids=2130 rejected_invalid_execution_ids=0 transient_execution_ids=0 deferred_execution_ids=0 candidate_logs_emitted=20 candidate_logs_suppressed=912
2026-08-05 12:18:50,793 [INFO]: archive_ingestor_core.py(_log_event:839) >> event=run_finished mode=dry-run scan_mode=staging exit_code=0 duration_seconds=940.386Archive Dir -- In progress (currently at 671 seconds): 2026-08-05 12:19:49,967 [INFO]: archive_ingestor_core.py(_log_event:839) >> event=archive_scan_progress scan_mode=archive archive_root=/lcrc/group/e3sm/PERF_Chrysalis/OLD_PERF current_dir=/lcrc/group/e3sm/PERF_Chrysalis/OLD_PERF/2025-08/performance_archive_chrysalis_e3sm_2025_08_07_00_46_50/ac.dalei.hao/20250805_SSP245_ZATM_BGC_ne30pg2_f09_oEC60to30v3_FULL_FDBK_40years_restart_9_13months/832641.250806-004304/CaseDocs.832641.250806-004304 directories_visited=12250 discovered_cases=1211 execution_dirs_scanned=3105 execution_dirs_accepted=1758 skipped_transient=0 rejected_existing_execution_ids=0 duration_seconds=671.614 |
|
@TonyB9000 I pushed a few commits that improves the usability of dry-run, documents validation operations, and updates the bash wrapper script. Please review when you can and pull the latest commit in your repo (while addressing any merge conflicts).
|
|
@tomvothecoder I'll pull down the changes and review. |
|
@tomvothecoder I've at least confimed: When you simply cron a script to run (giving the full path), it ALWAYS treats the crontab user's home directory as the (initial) working directory. So, one must issue a "cd location" and "&& command" in order for the command to have "location" as the initial working directory. As I would like our nascent "simboard/operations" to be that base of operations, I will cron with More testing tomorrow. |
2 participants
Description
This adds a scheduler-agnostic HPC archive ingestor entrypoint and a thin Chrysalis site wrapper for existing Jenkins-driven metadata ingestion.
hpc_archive_ingestormodule that delegates to existing NERSC ingestorsites/chrysalis.shwrapper with Chrysalis archive and state defaults plus required API env varsTask
Finish turning this branch into a deployable ingestion path for Chrysalis first, then use the same pattern for other sites once access is available.
SIMBOARD_API_BASE_URLandSIMBOARD_API_TOKENshould be stored and injected in the Chrysalis Jenkins job.Checklist
Deployment Notes (if any)
No special deployment steps.
Local validation is currently blocked because PostgreSQL was unavailable at
127.0.0.1, somake backend-testand the targeted ingestion test file could not complete in this environment.