Notes on building the stack for Fujitsu A64FX

[trz42]

The stack is being builded since summer 2024. Recently (April 2025) we have taken a shared bot instance on Deucalion into production. We'll use this issue to coordinate the building of the stack.

---

Short summary of current status

2025-04-11: In total 271 modules have been built with the toolchains foss/2023a and foss/2023b.

Remember some TODOs

• Rebuild SitePackage.lua with changes done in add Lmod hook that prints warning for SciPy-bundle on NVIDIA Grace #1022 (in that PR building for a64fx failed)

System toolchain & miscellaneous improvements

• {2023.06}[system,a64fx] circumvent fusemount issue, improve module use, and add some packages #1025

[trz42]

Comment for installation of toolchain foss/2023a and all packages built with it.

To avoid re-building software within the 2023a stack, we have to be careful to use from-commitwhile building:

SciPy-bundle-2023.07-gfbf-2023a
R-bundle-CRAN-2023.12-foss-2023a
R-bundle-Bioconductor-3.18-foss-2023a-R-4.3.2

• if these apps are already ingested, we need to figure out if we have to rebuild them

Apps from EB 4.8.2 2023a easystack

• ALREADY in /cvmfs: add GCCcore/12.3.0 (grace PR: {2023.06}[2023a,grace] GCCcore 12.3.0 #979; sapphirerapids PR: {2023.06}[2023a,sapphirerapids] foss 2023a #925)
• ALREADY in /cvmfs: add foss/2023a (grace PR: {2023.06}[2023a,grace] foss 2023a, Rust 1.70.0 #983; sapphirerapids PR: {2023.06}[2023a,sapphirerapids] foss 2023a #925)
• MOST apps are in /cvmfs
  • remaining apps except SciPy-bundle rebuild
    • split up into several:
      • first Cython and TensorFlow dependencies: {2023.06}[2023a,a64fx] apps originally built with EB 4.8.2 #1028
      • all remaining apps except TensorFlow: {2023.06}[2023a,a64fx] second part of apps originally built with EB 4.8.2 #1033
      • TensorFlow: {2023.06}[2023a,a64fx] add TensorFlow 2.13 #1034 (par=2)
        • first build interactively to figure out what eb parameters let it build successfully
          • 27 extensions need to be built: grpcio (8/27) requires quite a bit of time (1h33 (par=2) - 1h36 (par=8) in different runs)
    • grace PRs
      • {2023.06}[2023a,grace] Apps from EB 4.8.2 2023a easystack #985
      • {2023.06}[2023a,grace] Remaining apps from EB 4.8.2 2023a easystack #986
    • sapphirerapids PR: {2023.06}[2023a,sapphirerapids] Remaining apps for 2023a with EB 4.8.2 #929
  • check if SciPy-bundle 2023.07 needs to be rebuilt -> separate PR
    • depends on Cython-3.0.8-GCCcore-12.3.0.eb
    • {2023.06}[2023a,a64fx] rebuild SciPy-bundle 2023.07 #1027
      • fakeroot is currently not configured for the build account => need to follow up later
  • Boost-1.82.0 is already in the stack, only the source URL had changed, so no need to do anything with it
  • What about Boost.Python? -> it doesn't seem to be built via EB 4.8.2

Apps from EB 4.9.0 2023a easystack

• {2023.06}[2023a,a64fx] first part of apps originally built with EB 4.9.0 #1038
  • grace PR: {2023.06}[2023a,grace] Apps from EB 4.9.0 2023a easystack #990
  • sapphirerapids PR: {2023.06}[2023a,sapphirerapids] Batch of EB 4.9.0 2023a apps #932
• {2023.06}[2023a,a64fx] add remaining apps originally built with EB 4.9.0 #1040
  • grace PR: {2023.06}[2023a,grace] Further apps from EB 4.9.0 2023a easystack #991 + {2023.06}[2023a,grace] Remaining apps from EB 4.9.0 2023a easystack #995
  • sapphirerapids PR: {2023.06}[2023a,sapphirerapids] Add remaining EB 490 2023a apps #934
    • added from-commit to libxc-6.2.2
    • added from-commit to ParMETIS-4.0.3
• {2023.06}[2023a,a64fx] PyTorch 2.1.2 originally built with EB 4.9.0 #1086

Apps from EB 4.9.1 2023a easystack

• done in several parts
  • part 1: {2023.06}[2023a,a64fx] apps originally built with EB 4.9.1 #1049
  • part 2: {2023.06}[2023a,a64fx] second batch of apps originally built with EB 4.9.1 #1060
  • grace PR: {2023.06}[2023a,grace] Apps from EB 4.9.1 2023a easystack #998, {2023.06}[2023a,grace] rebuild R-bundle-Bioconductor-3.18 in 2023a easystack #1007
  • sapphirerapids PR: {2023.06}[2023a,sapphirerapids] R-bundle-CRAN 2023.12 #941, {2023.06}[2023a,sapphirerapids] Add EB 4.9.1 easystack for 2023a #943, {2023.06}[2023a,sapphirerapids] R-bundle-Bioconductor 3.18 #944

Apps from EB 4.9.2 2023a easystack

• {2023.06}[2023a,a64fx] apps originally built with EB 4.9.2 #1091
  • grace PR: {2023.06}[2023a,grace] Apps from EB 4.9.2 2023a easystack #1002
  • sapphire rapids PRs: {2023.06}[2023a,sapphirerapids] OpenFOAM 10 and 11 #942, {2023.06}[2023a,sapphirerapids] Batch of apps for EB 4.9.2 / 2023a #945, {2023.06}[2023a,sapphirerapids] Remaining apps for EB 4.9.2 2023a easystack #954

[trz42]

Comment for installation of toolchain foss/2023b and all packages built with it.

Need to be careful when building the following packages because rebuilding doesn't work on Deucalion currently:

• scikit-build-core-0.9.3-GCCcore-13.2.0.eb (see {2023.06}[2023b] Rebuild scikit-build-core to include additional extension #957)

Apps from EB 4.9.0 2023b easystack

• add foss/2023b (grace PR: {2023.06}[2023b,grace] foss/2023b #982, sapphirerapids PR: {2023.06}[2023b,sapphirerapids] foss 2023b #926)
• build in two parts
  • part 1 includes most apps except Qt5: {2023.06}[2023b,a64fx] apps originally built with EB 4.9.0 #1050
  • part 2 includes Qt5: {2023.06}[2023b,a64fx] Qt5 5.15.13 #1061
  • grace PR: {2023.06}[2023b,grace] Remaining apps originally built with EB 4.9.0 #978
  • sapphirerapids PR: {2023.06}[2023b,sapphirerapids] Remaining EB 4.9.0 2023b apps #931

Apps from EB 4.9.1 2023b easystack

• {2023.06}[2023b,a64fx] apps originally built with EB 4.9.1 #1097
  • used from-commit for scikit-build-core-0.9.3-GCCcore-13.2.0.eb to avoid need for rebuild (see {2023.06}[2023b] Rebuild scikit-build-core to include additional extension #957)
  • grace PR: {2023.06}[2023b,grace] Apps originally built with EB 4.9.1 #987
  • sapphirerapids PR: {2023.06}[2023b,sapphirerapids] Add 2023b easystack for EB 4.9.1 #936
• GROMACS (skipped in the above because it failed in some tests)

Apps from EB 4.9.2 2023b easystack

• apps originally built with EB 4.9.2 (grace PR: {2023.06}[2023b,grace] apps originally built with EB 4.9.2 + changes to bot/build.sh and eessi_container.sh #989, sapphirerapids PR: {2023.06}[2023b,sapphirerapids] Add EB 4.9.2 easystack for 2023b #939)

Apps from EB 4.9.3 2023b easystack

• apps originally built with EB 4.9.3 (grace PR: {2023.06}[2023b,grace] apps originally built with EB 4.9.3 #992, sapphirerapids PR: {2023.06}[2023b,sapphirerapids] Add EB 4.9.3 easystack for 2023b #946)

Apps from EB 4.9.4 2023b easystack

• a couple of apps built with EB 4.9.4 (grace PR: {2023.06}[2023b,grace] a couple of apps built with EB 4.9.4 #993, sapphirerapids PR: {2023.06}[2023b,sapphirerapids] EB 4.9.4 easystack for 2023b #947)
• a few more apps built with EB 4.9.4 (grace PR: {2023.06}[system,2023b,grace] a few remaining apps for the system and 2023b toolchains #996, no sapphirerapids PR)

[trz42]

Comment for installation of toolchain foss/2022b and all packages built with it.

NVIDIA Grace // Sapphire Rapids PRs to consider (incl rebuilds):

• {2023.06}[2022b,grace] foss 2022b, HarfBuzz 5.3.1 #1003 // {2023.06}[2022b,sapphirerapids] foss 2022b #923
  • includes Python-3.10.8-GCCcore-12.2.0-bare: check if EB 4.9.4 includes updated easyblock from PR 3352
• {2023.06}[2022b,grace] foss 2022b, HarfBuzz 5.3.1 #1003, {2023.06}[2022b,grace] Qt5 5.15.7 #1005 // {2023.06}[2022b,sapphirerapids] HarfBuzz 5.3.1, Qt5 5.15.7, QuantumESPRESSO 7.2 #927
  • includes Python-3.10.8-GCCcore-12.2.0: check if EB 4.9.4 includes updated easyblock from PR 3352
• {2023.06}[2022b,grace] apps originally built with EB 4.9.0 #1013, add Lmod hook that prints warning for SciPy-bundle on NVIDIA Grace #1022 // {2023.06}[2022b,sapphirerapids] Easystack for 2022b with EB 4.9.0 #930
  • probably need to use from commit for 2 Boost easyconfigs
• {2023.06}[2022b,grace] apps originally built with EB 4.9.1 #1026 // {2023.06}[2022b,sapphirerapids] Add 2022b easystack for EB 4.9.1 #935
• Rebuild Python with ctypes fix for Sapphire Rapids #938
  • make sure that we don't need to rebuild

  - Python-3.10.8-GCCcore-12.2.0-bare:
      options:
        # See https://github.com/easybuilders/easybuild-easyblocks/pull/3352
        include-easyblocks-from-commit: 1ee17c0f7726c69e97442f53c65c5f041d65c94f
  - Python-3.10.8-GCCcore-12.2.0:
      options:
        # See https://github.com/easybuilders/easybuild-easyblocks/pull/3352
        include-easyblocks-from-commit: 1ee17c0f7726c69e97442f53c65c5f041d65c94f
  

• {2023.06}[2022b,grace] apps originally built with EB 4.9.2 #1029, {2023.06}[2022b,grace] remaining apps originally built with EB 4.9.2 #1031 // {2023.06}[2022b,sapphirerapids] Add EB 4.9.2 easystack for 2022b #940

Need to be careful when building the following packages because rebuilding doesn't work on Deucalion currently:

• Python-3.10.8-GCCcore-12.2.0-bare (see Rebuild Python with ctypes fix for Sapphire Rapids #938)
• Python-3.10.8-GCCcore-12.2.0 (see Rebuild Python with ctypes fix for Sapphire Rapids #938)

Apps originally built with EB 4.8.2

• {2023.06}[2022b,a64fx] foss/2022b #1051
  • grace PR: {2023.06}[2022b,grace] foss 2022b, HarfBuzz 5.3.1 #1003
  • sapphirerapids PRs: {2023.06}[2022b,sapphirerapids] foss 2022b #923, {2023.06}[2022b,sapphirerapids] HarfBuzz 5.3.1, Qt5 5.15.7, QuantumESPRESSO 7.2 #927
• {2023.06}[2022b,a64fx] HarfBuzz 5.3.1 (originally built with EB 4.8.2) #1098
  • grace PR: {2023.06}[2022b,grace] foss 2022b, HarfBuzz 5.3.1 #1003
  • sapphirerapids PRs: {2023.06}[2022b,sapphirerapids] HarfBuzz 5.3.1, Qt5 5.15.7, QuantumESPRESSO 7.2 #927

Apps originally built with EB 4.8.2, and also built here with EB 4.8.2

• Qt5 5.15.7
  • grace PR: {2023.06}[2022b,grace] Qt5 5.15.7 #1005
  • sapphirerapids PR: {2023.06}[2022b,sapphirerapids] HarfBuzz 5.3.1, Qt5 5.15.7, QuantumESPRESSO 7.2 #927
• QuantumESPRESSO 7.2
  • grace PR: {2023.06}[2022b,grace] QuantumESPRESSO 7.2 #1008
  • sapphirerapids PRs: {2023.06}[2022b,sapphirerapids] HarfBuzz 5.3.1, Qt5 5.15.7, QuantumESPRESSO 7.2 #927

Apps originally built with EB 4.9.0

• a bunch of apps (grace PR: {2023.06}[2022b,grace] apps originally built with EB 4.9.0 #1013, sapphirerapids PR: {2023.06}[2022b,sapphirerapids] Easystack for 2022b with EB 4.9.0 #930)
  • update for SitePackage.lua so module load SciPy-bundle/2023.02-gfbf-2022b (included in PR 1013) prints a message about a unusually high number of failing unit tests (general PR add Lmod hook that prints warning for SciPy-bundle on NVIDIA Grace #1022); the PR also fixes ReFrame test issues on Grace nodes

Apps originally built with EB 4.9.1

• another set of apps (grace PR: {2023.06}[2022b,grace] apps originally built with EB 4.9.1 #1026, sapphirerapids PR: {2023.06}[2022b,sapphirerapids] Add 2022b easystack for EB 4.9.1 #935)

Apps originally built with EB 4.9.2_

• part 1: 47 of 70 apps (grace PR: {2023.06}[2022b,grace] apps originally built with EB 4.9.2 #1029; sapphirerapids PR: {2023.06}[2022b,sapphirerapids] Add EB 4.9.2 easystack for 2022b #940)
• part 2: 23 of 70 apps (grace PR: {2023.06}[2022b,grace] remaining apps originally built with EB 4.9.2 #1031; sapphirerapids PR: {2023.06}[2022b,sapphirerapids] Add EB 4.9.2 easystack for 2022b #940)

[dagonzalezfo]

Adding BWA as part of initial test #1150

[dagonzalezfo]

About missing packages described here: #1024 (comment)

• EB 4.9.2: {2023.06}[2023b,a64fx] apps originally built with EB 4.9.2 #1151
• EB 4.9.2: {2023.06}[2022b,a64fx] remaining apps originally built with EB 4.9.2 #1176
• EB 4.9.3: {2023.06}[2023b,a64fx] apps originally built with EB 4.9.3 #1152
• EB 4.9.4: {2023.06}[2023b,a64fx] Lit and Astropy #1153
• EB 4.9.4: {2023.06}[2023b,a64fx] EB-4.9.4 additional packages #1154

[hvelab]

Only remaining:

• Batch of 4.9.0, numpy tests fail: {2023.06}[2022b,a64fx] apps originally built with EB 4.9.0 #1187
• GROMACS, tests failing: {2023.06}[2023b,a64fx] GROMACS built with EB 4.9.1 #1155
• Qt5 fails because of LLVM: {2023.06}[2022b,a64fx] Qt5 5.15.7 #1177
• old-ish one, fails for unknown reason: {2023.06}[2023a,a64fx] apps originally built with EB 4.9.2 #1091

[bedroge]

For that last one: I checked the logs of the last two builds jobs, and both just stopped somewhere in the middle. Slurm reports NODE_FAIL for both jobs though, so maybe they were taking up too many resources (disk space?). We might have to split that PR into smaller chunks.

[bedroge]

Just to log it here: I've just removed ParMETIS/4.0.3-gompi-2023b from the A64FX stack. This was installed, sort of, by accident in #1118: it was filtered for other CPU targets, but A64FX had an outdated EESSI-extend and because of that it was built as dependency of openCARP. The latter has been rebuilt in #1223 (also its dependencies PETSc and SuperLU_dist) to make sure they don't depend on ParMETIS anymore. Thus, it could be safely removed now, and this ensures that all stacks are similar again w.r.t. the ParMETIS situation.

[bedroge]

The a64fx stack is now on par with the others, except for the following three GROMACS versions:

• GROMACS/2024.3-foss-2023b
• GROMACS/2024.4-foss-2023b
• GROMACS/2024.1-foss-2023b

These are failing due to a failing test, an issue has been opened by @dagonzalezfo: https://gitlab.com/gromacs/gromacs/-/issues/5440.

[bedroge]

The last missing application (GROMACS 2024.1) was added in #1308. This allowed the CI that compares stacks to pass, see #1221 🎉

So, A64FX is fully on par with the other stacks now, and this has been reflected in the docs: EESSI/docs#625.