Triage of the 2026-08-17 full run: 263 failed tasks, kbet_pg and 5 methods produced nothing

[rcannood]

Triage of the full de.NBI run of 2026-08-17 (5w8BFLN1vMRRLW). Nextflow reported SUCCEEDED, but exitStrat() in labels_denbi.config ignores most non-zero exits after retrying, so 263 of 4369 tasks actually failed. One metric produced no scores at all, and five methods produced no results at all.

Where the data is

• Seqera run: https://cloud.seqera.io/orgs/openproblems-bio/workspaces/openproblems-bio/watch/5w8BFLN1vMRRLW
• Results: s3://openproblems-data/resources/task_batch_integration/results/run_2026-08-17_16-18-21/ with trace.txt, score_uns.yaml, report/ and logs/ (all .command.* and .exitcode, 4369/4369 tasks present, logs/_index.tsv maps work dir hash to task)
• Fetched off the cluster with common/scripts/fetch_denbi_run --logs --upload 5w8BFLN1vMRRLW. The full work dirs are not synced -- those only live on the cluster's /vol/scratch, which is deleted when the cluster is terminated, so grab anything you still need from there before teardown.
• Currently published results: https://openproblems.bio/benchmarks/batch_integration/v2.1.0-rc2/

Failures

exit	n	meaning
132	130	SIGILL, illegal instruction
137	65	OOM
140	30	SLURM walltime
1	28	application error
99	10	non-applicable, expected

19 of 26 scored methods have all six datasets. fadvi, geneformer, scanorama_correct, scgpt_finetuned and scprint produced nothing at all, and kbet_pg is absent from score_uns.yaml entirely.

Must fix: non-method components

• kbet_pg: 124/124 SIGILL, metric missing from the whole run. pegasuspy pulls hnswlib, which ships no PyPI wheels and whose setup.py compiles with -march=native. The image was built on an AVX-512 runner, so bare import hnswlib dies on the de.NBI workers (EPYC-Rome and Broadwell, AVX2 at most). docker run ghcr.io/openproblems-bio/task_batch_integration/metrics/kbet_pg:build_main python -c "import hnswlib" exits 132. The twin kbet_pg_label drew an AVX2-only runner and passed 248/248 -- same source, same version, different lottery ticket. The image on build_main still has 47 zmm references today, so a re-run fails again unchanged.
• bbknn: same landmine, not yet triggered. annoy has the same -march=native setup.py, and methods/bbknn:build_main currently carries 136 zmm references. It passed in this run; it will not pass in the next one.
• Pin the ISA at build time so this stops being a lottery. Both packages have an escape hatch, and viash emits the ENV lines correctly from a type: docker setup step placed before the pip step:

  setup:
    - type: docker
      env:
        - HNSWLIB_NO_NATIVE=1
        - CXXFLAGS=-march=x86-64-v3
        - ANNOY_COMPILER_ARGS=-D_CRT_SECURE_NO_WARNINGS,-fpermissive,-march=x86-64-v3,-O3,-ffast-math,-fno-associative-math,-DANNOYLIB_MULTITHREADED_BUILD,-std=c++14
    - type: python
      pypi: [ ... ]

  Verified against openproblems/base_python:1: no AVX-512 in either extension, AVX2 kernels retained, both import. Affects kbet_pg, kbet_pg_label, bbknn, scanorama_correct, scanorama_integrate, scalex.
• Move those env vars into the base images (openproblems/base_python, in openproblems-bio/openproblems) so no derived image anywhere can draw a bad ticket. The per-component fix above is the stopgap.
• kbet: 65 OOM + 4 walltime, all on hypomap and mouse_pancreas_atlas. Successful tasks peak at ~398 GB RSS against veryhighmem (200 GB x attempt, clamped to maxMemory = 480.GB), so this is the hardware ceiling and not a mislabel. Needs a decision rather than a patch: subsample, or drop kbet now that kbet_pg_label measures the same thing for a fraction of the cost.

Method bugs

• scanorama_correct: 6/6 SIGILL. Same -march=native problem as kbet_pg, via annoy. Confirmed by pulling the exact digest from the run, sha256:fc42f62d660a8804dc4469a4d489023f37076d348276b15aecc29278efbe3240: its annoylib.so has 136 AVX-512 references and reproduces the SIGILL on the real dkd input. The image has since been rebuilt and the current one runs clean, which is exactly the problem. Covered by the ISA pin above.
• scgpt_finetuned: 8/10, one line. script.py:176 uses adata.X.A; .A was removed from scipy sparse matrices in scipy 1.14. Use adata.X.toarray().
• geneformer: 8/10, HF 404. Duplicate of Method for geneformer has been blocked by new geneformer model! #96. Upstream ctheodoris/Geneformer renamed *_gc95M.pkl to *_gc104M.pkl and replaced gf-*-95M-i4096/ with Geneformer-V2-104M/ in the "add V2 models" commits of 2025-06-18, and every hf_hub_download tracks main, so it broke silently. Quickest unblock is to pin revision="42dbf0ae41244b93f1c3735fa393a966ffe64dfa" (2025-06-13, verified to still carry all four gc95M dictionaries and all four model directories); migrating to V2 is a separate call about what we are benchmarking.
• scprint: 12/12, triton vs CUDA. The image has torch 2.2.0+cu121 and triton 2.2.0, but openproblems/base_pytorch_nvidia:1 now ships CUDA 13.1 at /usr/local/cuda. Triton 2.2's ptx_get_version() only maps CUDA 10/11/12 and raises RuntimeError: Triton only support CUDA 10.0 or higher. Triton bundles its own CUDA 12.3 ptxas; setting TRITON_PTXAS_PATH=/usr/local/lib/python3.12/dist-packages/triton/third_party/cuda/bin/ptxas makes detection return 12.3 (verified in-image). Bumping scprint/torch is the real answer. Related: Inconsistent preprocessing for scPRINT #81.

Resource labels

• fadvi: 6/6 walltime, and it is a misconfiguration. config.vsh.yaml:100 is [midtime,midmem,midcpu] with no gpu, so SLURM sent it to -p cpu with no --gpus -- the log says GPU available: False, used: False -- and it hit the 4 h midtime ceiling on all six datasets. Add gpu and raise the time tier.
• scalex: 5/6 walltime, same shape. script.py:41 passes gpu=0, but the engine is openproblems/base_python:1 (no CUDA) and the label has no gpu, so it always runs on CPU. dkd took 28 min, everything larger blew past 8 h. Needs base_pytorch_nvidia:1 plus a gpu label.
• uce: 5/6 walltime, genuine. It did get its T4. dkd finished in 5 h 05, the rest hit hightime with ~40 h ETAs on a single card. veryhightime (24 h) is defined in _viash.yaml and used by no component; this is what it is for, though even 24 h may not cover hypomap and tabula_sapiens.
• mnnpy (5/6) and batchelor_mnn_correct (5/6): walltime, probably real limits. dkd in 23 min and 4 h 57 respectively, everything larger over 8 h. Worth one bump to veryhightime, and if they still time out we record it as a scaling limit of the methods rather than chasing it.

Not a bug

The 10 exit-99 tasks are correct: cellplm, scgpt_zeroshot, scimilarity and geneformer declaring themselves human-only on the two mouse datasets.

Suggested order

The two AVX-512 images first, since they are broken right now and one of them silently deletes a metric from the benchmark. Then the one-liners (scgpt_finetuned, geneformer, scprint, fadvi), then scalex, then the time tiers. kbet needs a decision, not a patch.

[rcannood]

Working through the list. PRs so far:

• Pin the ISA that source builds target in the base images core#50 -- pin the ISA in the base images. Covers kbet_pg, bbknn, scanorama_correct and friends; those components need no source change, only a rebuild against the fixed base. Verified on base_python: hnswlib goes from 47 AVX-512 instructions to 0 while keeping its AVX kernels, annoy likewise.
• Pin geneformer to the last upstream revision with the 95M assets #106 -- pin geneformer to 42dbf0ae. Verified on a de.NBI T4 by re-running the failing task's own work dir: all six downloads resolve and it reaches embedding extraction.
• Use toarray instead of the removed .A in scgpt_finetuned #107 -- adata.X.A to adata.X.toarray() in scgpt_finetuned. Verified the same way.
• Give fadvi a gpu label and bump scalex to midcpu #108 -- gpu label for fadvi, veryhightime for uce, mnnpy, batchelor_mnn_correct and scalex.
• Add the de.NBI launch script #109 -- commit the de.NBI launch script, with the --labels corrected from task_predict_modality.

Three things turned up while verifying that are not on the list above.

The GPU is currently unusable on de.NBI

openproblems/base_pytorch_nvidia:1 now resolves to nvcr pytorch 26.02 (torch 2.11, CUDA 13.1), but the de.NBI GPU workers run driver 565.57.01, which is CUDA 12.7. torch.cuda.is_available() returns False on the cluster:

UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 12070).
2.11.0a0+eb65b36914.nv26.02 13.1
False

This is new. On 2026-08-17 it worked -- scvi and scanvi both logged GPU available: True (cuda), used: True -- so it broke between that run and now, when the base image moved to 26.02.

Every method that keeps the base image's torch would now silently train on the CPU: uce, geneformer, drvi, scvi, scanvi, cellplm, condo, fadvi, scimilarity. scgpt_* and scprint are unaffected because they pip-install their own cu121 torch over the base.

The fix is on the cluster: nvidia-driver-580 is available in noble-updates on the workers. Until that lands, a re-run would produce a lot of CPU-speed timeouts, and the fadvi gpu label in #108 buys nothing.

scgpt_finetuned does not fit on a T4

With #107 applied it gets through preprocessing, splitting, tokenization and model loading, then:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 252.00 MiB.
GPU 0 has a total capacity of 14.57 GiB of which 64.75 MiB is free.

batch_size: 64 and epochs: 15 are hardcoded in the script rather than exposed as arguments, so there is no knob to turn. That is also contrary to the convention that a long-running method exposes whatever bounds its work. Worth its own issue: either expose the knobs, or record that scGPT fine-tuning needs a bigger card than de.NBI has.

Six methods have no functional test

uce, cellplm, scgpt_finetuned, scgpt_zeroshot, geneformer and scimilarity merge base_method.yaml, whose only test_resources entry is check_config.py. Everything else merges comp_method.yaml, which adds run_and_check_output.py.

Three of the five methods that produced nothing in this run are in that set, and CI could not have caught any of it -- viash test on geneformer passes today, with the 404 still in place.

@rcannood notes the reason is that these need a GPU and would fail on CPU-only CI runners, which is fair. Still worth a think about whether there is a cheaper signal available, since right now the first time we learn a foundation-model method is broken is a full benchmark run.

Still parked

@mumichae -- kbet (scib/R) and kbet_pg_label both compute kBET per cell type and average, so they measure the same thing; kbet peaked at ~398 GB RSS and lost 69 tasks, kbet_pg_label did all 248 without trouble. Is there a reason to keep the scib implementation for comparability with Luecken 2022, or can we drop it? Happy either way -- it just needs a decision before the next full run.

[rcannood]

The GPU driver is sorted, and the diagnosis was a bit more involved than "the driver is old".

Upgrading to 580 alone is not enough: the container runtime's CDI spec at /var/run/cdi/nvidia.yaml pins the exact .so versions it saw at boot, so after the swap every docker run --gpus all fails with

open /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.565.57.01: no such file or directory

The working sequence is install 580 → reload the nvidia kernel modules → nvidia-ctk cdi generate. That is now in the cluster's provisioning path (042-slurm-gpu-gres.patch in denbi-cluster-setup, on a gpu-driver-580 branch), so nodes get it on creation, and all four running T4 workers have been brought up to 580.173.02 by hand. Verified through SLURM with the stock image: torch.cuda.is_available() is True and tensors land on cuda:0.

Worth recording the dead end, since it looks like it should work: the nvcr image ships CUDA forward-compatibility libraries at /usr/local/cuda/compat/lib.real, and putting them on LD_LIBRARY_PATH really does make CUDA 13 run on driver 565 -- I measured it. They are never used automatically, though, for two reasons. Viash sets ENTRYPOINT [], which disables nvcr's entrypoint, and the container toolkit does not create the compat/lib symlink the image's own LD_LIBRARY_PATH already points at -- not in auto/CDI mode, and not in legacy mode with cuda-compat-mode = "ldconfig" either. Enabling them unconditionally would break on a host whose driver is newer than the compat libs, so it is not a fix worth baking into the base image.

That clears the blocker on the re-run. Remaining before it can go: #106, #107, #108, #109 and #112 merged, and openproblems-bio/core#50 merged and openproblems/base_python:1 republished first, so kbet_pg and bbknn rebuild against a base that does not re-roll the AVX-512 dice.