launch_slurm is broken for train_continue

[clessig]

What happened?

launch_slurm is currently broken since the stages param introduced to in run_train.py is not properly handled.

What are the steps to reproduce the bug?

../WeatherGenerator-private/hpc/launch-slurm.py --from-run-id ez94wl8b

Output:

3: usage: run_train.py [-h] {train,train_continue,inference} ...
3: run_train.py: error: unrecognized arguments: --from-run-id ez94wl8b --mini-epoch -1
2: usage: run_train.py [-h] {train,train_continue,inference} ...
2: run_train.py: error: unrecognized arguments: --from-run-id ez94wl8b --mini-epoch -1
1: usage: run_train.py [-h] {train,train_continue,inference} ...
0: usage: run_train.py [-h] {train,train_continue,inference} ...
1: run_train.py: error: unrecognized arguments: --from-run-id ez94wl8b --mini-epoch -1
0: run_train.py: error: unrecognized arguments: --from-run-id ez94wl8b --mini-epoch -1
srun: error: nid005263: tasks 0-3: Exited with exit code 2
srun: Terminating StepId=602194.0

Hedgedoc link to logs and more information. This ticket is public, do not attach files directly.

No response

[jehangirawan]

assign me

[github-actions]

Issue assigned to @jehangirawan.

[simone99n]

Could you try !124?

It should works, but I haven't try it (I have some other pending jobs since this morning..)

[jehangirawan]

I've opened !126 to fix this in the bash scripts directly.
https://gitlab.jsc.fz-juelich.de/esde/WeatherGenerator-private/-/merge_requests/126

Note on !124 — it should work currently via the _fix_argl fallback in
run_train.py, but that function is marked as temporary (TODO: remove after
grace period). Once removed, !124 would break.

!126 passes the subcommand explicitly in the bash scripts, making it
more future-proof.

Happy to discuss which approach is preferred!

[grassesi]

A combination of !124 and !126 is needed. Unfortunately the fix suggested in !124 is incomplete: $WEATHERGEN_STAGE is used in a pattern match statement in the batch scripts, "train_continue" needs to be added in all batch scripts, as this needs to be done anyway the fix in !126 can be implemented as well for every platform.

[tjhunter]

We should nearly never depend on environment variables. The only place where it is unfortunately necessary is to pass arguments to slurm.

!126 seems the right fix, however it is only implemented for juwels-booster in hpc/juwels_booster/weathergen_slurm.sh.

@shmh40 , I see you want to rely on _fix_argl in !124 which is already marked for removal. I prefer that we make the right fix already, at least for santis and hpc2020, in the same way as !126.

[tjhunter]

I see !127 addresses my comment. Thank you @shmh40 !

[simone99n]

Thanks @jehangirawan and @shmh40 for !126 and !127
I prepared !129 for the remaining HPCs.
After merge, we can close this issue

[grassesi]

Merged the PRs in the private repo