Found while verifying #107. With the .A fix applied, scgpt_finetuned gets through preprocessing, splitting, tokenization and model loading, then dies during the first training pass on a de.NBI T4:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 252.00 MiB.
GPU 0 has a total capacity of 14.57 GiB of which 64.75 MiB is free.
Including non-PyTorch memory, this process has 14.50 GiB memory in use.
on cellxgene_census/dkd, the smallest dataset, 39176 cells.
The component's label is biggpu, but on de.NBI that is still a single 16 GB T4 -- labels_denbi.config says as much: "Nothing here has more VRAM than a 16 GB T4, so biggpu falls back to the large node -- a method that really needs a bigger card will OOM rather than land on one."
The awkward part is that there is nothing to tune. src/methods/scgpt_finetuned/script.py hardcodes
"epochs": 15,
...
"batch_size": 64,
so neither the thing that drives memory nor the thing that drives runtime is reachable from the config. That also goes against the convention that a long-running method exposes whatever bounds its work, and caps it for CI via info.test_default.
Two things to settle:
- expose
--batch_size and --epochs as arguments, keeping the upstream defaults so nothing about the benchmark changes
- then decide whether a lower batch size on a T4 is an acceptable way to run scGPT fine-tuning, or whether this is simply a method that needs a bigger card than we have -- which is a legitimate, reportable outcome
Found while verifying #107. With the
.Afix applied,scgpt_finetunedgets through preprocessing, splitting, tokenization and model loading, then dies during the first training pass on a de.NBI T4:on
cellxgene_census/dkd, the smallest dataset, 39176 cells.The component's label is
biggpu, but on de.NBI that is still a single 16 GB T4 --labels_denbi.configsays as much: "Nothing here has more VRAM than a 16 GB T4, so biggpu falls back to the large node -- a method that really needs a bigger card will OOM rather than land on one."The awkward part is that there is nothing to tune.
src/methods/scgpt_finetuned/script.pyhardcodesso neither the thing that drives memory nor the thing that drives runtime is reachable from the config. That also goes against the convention that a long-running method exposes whatever bounds its work, and caps it for CI via
info.test_default.Two things to settle:
--batch_sizeand--epochsas arguments, keeping the upstream defaults so nothing about the benchmark changes