add: notebook smoke tests

[jbuchananr]

Changes

• Add CI workflow that runs every notebook end-to-end on Modal GPU function.
• We run standard model loading workflows on A10 gpus, we run modified version of training scripts with 0.01 epoch to verify it can run the training.

Included is a README.md in the util folder to understand how commands can be used. We introduced skipping commands to skip notebooks or skip specific cells in notebooks if required. In addition, in this util folder we have a file called modal_runner.py which contains the actual model function deployed to modal.

[jbuchananr]

@tuliren Should we create a separate workflow for these checks or is it sufficient to keep in check notebooks?

[tuliren]

This is fantastic!

[tuliren]

FYI, github workflow supports adding a markdown summary of the workflow. See details here.

This is an example of how the summary looks like: https://github.com/Liquid4All/labs/actions/runs/23210191532

Sometimes it is helpful to take a look at the output of the LLM response. If it is easy to do, can you ask coding agent to do that? I think the implementation may be to construct a paragraph of markdown text for each notebook that combines the notebook name, result["stdout"] and result["stderr"].

This is nice to have. Feel free to ignore this.