-
Notifications
You must be signed in to change notification settings - Fork 56
Updates for spack-stack: gsibec, fms, pfunit #1908
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
a39f9a0
Updates for spack-stack: gsibec, fms, pfunit
mathomp4 fe1f147
Add gsibec 1.4.1
mathomp4 63f8354
NAS updates
mathomp4 336eb89
Update gsibec in many files
mathomp4 95e9a3f
Updates for ifx stack
mathomp4 ecded67
Preliminary NAS TOSS5 setup
mathomp4 6f6f456
Fix for concretize oddity
mathomp4 decf0d4
Clean up packages.yamls
mathomp4 33df5cc
Merge remote-tracking branch 'jcsda/develop' into mathomp4-updates
mathomp4 00bbb6b
Fix fms
mathomp4 bcbc12a
Fix for Toss5
mathomp4 1e8ab3c
Update readme
mathomp4 a6e4e75
Fix for gnu
mathomp4 0c7883e
Fix gnu
mathomp4 24bf8aa
Fix for openmpi
mathomp4 f27d54b
Update to spack-packages@37965a7fc583e94c50cb0748199cf2d9b346be2d
mathomp4 e5f3bab
Move to CMake 3 for ecflow issue
mathomp4 5079134
Build p4est with -lm on gnu
mathomp4 0b58543
Fix bad commit
mathomp4 9a7319b
Updates for GCC 14 with hdf-eos2
mathomp4 05da1fe
More GNU fixes
mathomp4 f1ba211
Update readmes
mathomp4 bbc5959
Merge remote-tracking branch 'jcsda/develop' into mathomp4-updates
mathomp4 2f910c5
Update repos/builtin hash
mathomp4 c4f2c3a
Update to gsibec@1.4.2
mathomp4 bff28c1
Merge remote-tracking branch 'jcsda/develop' into mathomp4-updates
mathomp4 b69b426
Update repos/builtin hash to 340a3da5d2
mathomp4 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,246 @@ | ||
| # How to Build **spack-stack** at NAS on TOSS5 | ||
|
|
||
| This guide documents how to build **spack-stack** on NASA NAS TOSS5 systems. The previous install process used three stages across compute and login nodes, but recent testing shows installation can now be done directly on a compute node. | ||
|
|
||
| --- | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| - [Overview](#overview) | ||
| - [Machines Required](#machines-required) | ||
| - [Clone spack-stack](#clone-spack-stack) | ||
| - [Obtain an Interactive Compute Node](#obtain-an-interactive-compute-node) | ||
| - [Setup spack-stack](#setup-spack-stack) | ||
| - [Create Environments](#create-environments) | ||
| - [oneAPI - ifx Environment](#oneapi---ifx-environment) | ||
| - [oneAPI - ifort Environment](#oneapi---ifort-environment) | ||
| - [GCC Environment](#gcc-environment) | ||
| - [Activate the Environment](#activate-the-environment) | ||
| - [Concretize the Environment](#concretize-the-environment) | ||
| - [Create Source Cache (LOGIN NODE ONLY)](#create-source-cache-login-node-only) | ||
| - [Pre-Fetch Cargo Dependencies (LOGIN NODE ONLY)](#pre-fetch-cargo-dependencies-login-node-only) | ||
| - [Install Packages (COMPUTE NODE)](#install-packages-compute-node) | ||
| - [Update Module Files](#update-module-files) | ||
| - [Deactivate the Environment](#deactivate-the-environment) | ||
| - [Debugging Package Builds](#debugging-package-builds) | ||
| - [Deprecated: Legacy Three-Step Install](#deprecated-legacy-three-step-install) | ||
|
|
||
| --- | ||
|
|
||
| ## Overview | ||
|
|
||
| NAS login nodes and compute nodes have different constraints: | ||
|
|
||
| - **Login nodes**: | ||
| - Have internet access | ||
| - Limited to **2 processes** | ||
|
|
||
| - **Compute nodes** (Turin): | ||
| - No internet access | ||
| - Allow parallel builds | ||
|
|
||
| Use login nodes for setup steps that require internet access (e.g., concretization, mirroring sources, pre-fetching Cargo dependencies), then run installation on a compute node. | ||
|
|
||
| The install step itself is now a single compute-node command. | ||
|
|
||
| --- | ||
|
|
||
| ## Machines Required | ||
|
|
||
| You will need: | ||
|
|
||
| - **A login node** | ||
| Used for setup steps that require internet access. | ||
|
|
||
| - **A Turin compute node** | ||
| Used for package installation with higher parallelism. | ||
|
|
||
| --- | ||
|
|
||
| ## Clone spack-stack | ||
|
|
||
| Use the appropriate branch or tag: | ||
|
|
||
| ```bash | ||
| git clone --recurse-submodules https://github.com/JCSDA/spack-stack.git -b spack-stack-2.1.0 spack-stack-2.1.0 | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Obtain an Interactive Compute Node | ||
|
|
||
| NAS login nodes allow only **2 processes**, so use: | ||
|
|
||
| ```bash | ||
| qsub -I -V -X -l select=1:ncpus=128:mpiprocs=128:model=tur_ath -l walltime=12:00:00 -W group_list=s1873 -m b -N Interactive | ||
| ``` | ||
|
|
||
| This gives a **Turin** compute node for up to 12 hours. | ||
|
|
||
| --- | ||
|
|
||
| ## Setup spack-stack | ||
|
|
||
| Run on a **login node with internet**: | ||
|
|
||
| ```bash | ||
| cd spack-stack-2.1.0 | ||
| . setup.sh | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Create Environments | ||
|
|
||
| You only need to create each environment once. | ||
|
|
||
| ### oneAPI - ifx Environment | ||
|
|
||
| ```bash | ||
| spack stack create env --name ue-oneapi-2025.3.0 --template unified-dev --site nas-toss5 --compiler=oneapi-2025.3.0 | ||
| cd envs/ue-oneapi-2025.3.0 | ||
| ``` | ||
|
|
||
| ### oneAPI - ifort Environment | ||
|
|
||
| ```bash | ||
| spack stack create env --name ue-oneapi-2024.2.0 --template unified-dev --site nas-toss5 --compiler=oneapi-2024.2.0 | ||
| cd envs/ue-oneapi-2024.2.0 | ||
| ``` | ||
|
|
||
| ### GCC Environment | ||
|
|
||
| ```bash | ||
| spack stack create env --name ue-gcc-14.2.1 --template unified-dev --site nas-toss5 --compiler=gcc-14.2.1 | ||
| cd envs/ue-gcc-14.2.1 | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Activate the Environment | ||
|
|
||
| ```bash | ||
| spack env activate . | ||
| ``` | ||
|
|
||
| > **Important:** Run this in *every* terminal where you plan to run Spack commands. | ||
|
|
||
| --- | ||
|
|
||
| ## Concretize the Environment | ||
|
|
||
| Run on a **login node** (internet required for bootstrapping Clingo and other tools): | ||
|
|
||
| ```bash | ||
| spack concretize 2>&1 | tee log.concretize ; bell | ||
| ``` | ||
|
|
||
| ### Optional `bell` helper | ||
|
|
||
| ```bash | ||
| bell() { tput bel ; printf "\nFinished at: " ; date; } | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Create Source Cache (LOGIN NODE ONLY) | ||
|
|
||
| This downloads all source tarballs for your environment: | ||
|
|
||
| ```bash | ||
| spack mirror create -a -d /swbuild/gmao_SIteam/spack-stack/source-cache | ||
| ``` | ||
|
|
||
| > ⚠️ **Do not run this outside an activated environment.** | ||
| > Otherwise Spack will attempt to mirror **every** known package/version. | ||
|
|
||
| --- | ||
|
|
||
| ## Pre-Fetch Cargo Dependencies (LOGIN NODE ONLY) | ||
|
|
||
| Rust packages frequently require network access during build. Pre-fetch their dependencies: | ||
|
|
||
| ```bash | ||
| export CARGO_HOME=/swbuild/gmao_SIteam/spack-stack/cargo-cache | ||
| ../../util/fetch_cargo_deps.py | ||
| ``` | ||
|
|
||
| > ⚠️ **Set `CARGO_HOME` on compute nodes** before running `spack install`. | ||
|
|
||
| --- | ||
|
|
||
| ## Install Packages (COMPUTE NODE) | ||
|
|
||
| Run installation on a **compute node**: | ||
|
|
||
| ```bash | ||
| export CARGO_HOME=/swbuild/gmao_SIteam/spack-stack/cargo-cache | ||
| spack install -j 16 --verbose --fail-fast --show-log-on-error --no-check-signature 2>&1 | tee log.install ; bell | ||
| ``` | ||
|
|
||
| This replaces the former compute → login → compute install sequence. | ||
|
|
||
| > **Note:** You may need to re-run this command multiple times. Some builds fail intermittently but succeed on retry. | ||
|
|
||
| --- | ||
|
|
||
| ## Update Module Files | ||
|
|
||
| After installation completes, run: | ||
|
|
||
| ```bash | ||
| spack module tcl refresh -y --delete-tree ; bell | ||
| spack stack setup-meta-modules | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Deactivate the Environment | ||
|
|
||
| ```bash | ||
| spack env deactivate | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Debugging Package Builds | ||
|
|
||
| ```bash | ||
| spack clean | ||
| spack stage <package> | ||
| spack build-env <package> -- bash --norc --noprofile | ||
| ``` | ||
|
|
||
| This drops you into a clean build environment with the package’s full compiler/runtime environment loaded. | ||
|
|
||
| --- | ||
|
|
||
| ## Deprecated: Legacy Three-Step Install | ||
|
|
||
| > ⚠️ **Deprecated:** Keep this only as historical reference. Prefer the single compute-node install above. | ||
|
|
||
| Older workflows used three stages: | ||
|
|
||
| 1. Compute node: build dependencies for Rust-related Python packages and ecFlow. | ||
| 2. `athfe` login node: build `py-cryptography`, `py-maturin`, `py-rpds-py`, and `ecflow` with `-j 2`. | ||
| 3. Compute node: run full `spack install` to finish remaining packages. | ||
|
|
||
| Typical commands were: | ||
|
|
||
| ```bash | ||
| # Step 1 (compute node) | ||
| export CARGO_HOME=/swbuild/gmao_SIteam/spack-stack/cargo-cache | ||
| spack install -j 16 --verbose --fail-fast --show-log-on-error --no-check-signature \ | ||
| --only dependencies py-cryptography py-maturin py-rpds-py ecflow 2>&1 | tee log.install.deps-for-rust-and-ecflow ; bell | ||
|
|
||
| # Step 2 (athfe login node) | ||
| export CARGO_HOME=/swbuild/gmao_SIteam/spack-stack/cargo-cache | ||
| spack install -j 2 -p 1 --verbose --fail-fast --show-log-on-error --no-check-signature \ | ||
| py-cryptography py-maturin py-rpds-py ecflow 2>&1 | tee log.install.rust-and-ecflow ; bell | ||
|
|
||
| # Step 3 (compute node) | ||
| export CARGO_HOME=/swbuild/gmao_SIteam/spack-stack/cargo-cache | ||
| spack install -j 16 --verbose --fail-fast --show-log-on-error --no-check-signature 2>&1 | tee log.install.after-cargo ; bell | ||
| ``` | ||
|
|
||
| --- |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| config: | ||
| build_jobs: 6 | ||
|
|
||
| # Overrides for spack build and staging areas to speed up builds | ||
| # and avoid errors with Lustre file locking and xattr issues | ||
| build_stage: /swbuild/gmao_SIteam/spack-stack/cache/build_stage | ||
| test_stage: /swbuild/gmao_SIteam/spack-stack/cache/test_stage | ||
| source_cache: /swbuild/gmao_SIteam/spack-stack/cache/source_cache | ||
| misc_cache: /swbuild/gmao_SIteam/spack-stack/cache/misc_cache |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| mirrors: | ||
| local-source: | ||
| url: file:///swbuild/gmao_SIteam/spack-stack/source-cache | ||
| binary: false | ||
| local-binary: | ||
| url: file:///swbuild/gmao_SIteam/spack-stack/build-cache | ||
| binary: true | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| modules: | ||
| default: | ||
| enable:: | ||
| - tcl | ||
| tcl: | ||
| include: | ||
| # List of packages for which we need modules that are blacklisted by default | ||
| - python |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.