Add new component: quarto/partial

[fasterius]

This PR adds a new module: quarto/partial, which aims to add new functionality for Quarto within nf-core. The quartonotebook already exists, which can render Quarto/Jupyter/RMarkdown reports into HTML, but it only creates a single, monolithic report and can't be extended. Quarto has various ways of working with sub-reports or extending reports, which is what this new module is all about. The major use case is allowing for more granular reporting flow, dividing work up in arbitrary steps to reduce re-computation (e.g. when iterating over some analysis step or re-running a pipeline with a different set of parameters for only some of the reporting steps) while still allowing for a single output report.

There are two major ways of working with sub-reports or extended reports in Quarto: Quarto Projects, and Quarto Partials. Projects do not play well within a workflow manager context, as Quarto then needs to be the arbiter of what gets executed at what time, which depends on the working directory state (which will always be fresh in a Nextflow context). Partials, however, fits more nicely into the workflow manager context: a sub-report is first rendered to markdown (not HTML), which is then referred to in the final report using the {{< include partial.md >}} notation, and the final report is rendered only once.

While the quartonotebook module is already quite flexible, there are several reasons why incorporating partial rendering into it doesn't work well:

• Separation of concerns: the quartonotebook module renders a single Quarto document to HTML, where partials needs to render intermediate markdown documents.
• While some input/output is shared, there is some non-overlapping parts: partials do not use extensions, while the main output differs (an HTML versus a markdown + related files, e.g. images). Having a single module would mean that almost all output would be optional due to having to account for the different render modes, which feels brittle to me.
• Rendering a partial needs to explicitly use --to markdown, which overwrites whatever has been specified in the report. The quartonotebook module doesn't use --to at all, which allows a user to specify the format in the .qmd file themselves, which might include a custom format in e.g. an extension. Keeping the modules separate here makes it easier to work with both, circumventing having to use ext.args for every single module use (which would also be more opaque).
• Partials are more of a pure Quarto thing, and while it's probably possible to make it work with the other input types that quartonotebook can handle, I had some issues there. If somebody is working with partials, it is reasonable to assume a Quarto-centric system, thus removing support of the other formats (.ipynb and .rmd) and for the test suite.

A lot of the script: part of the modules remain similar, though, but I believe it's worth it due to the above mentioned things.

Should this module be accepted, the existing quartonotebook module would need to be renamed to e.g. quarto/notebook. I'm not sure what the procedure for renaming modules is, but I'm happy to make a PR both to the modules repository for this, as well as for all of the pipelines currently using the module (which is four, at time of writing) after its been updated.

PR checklist

• This comment contains a description of changes (with reason).
• If you've fixed a bug or added code that should be tested, add tests!
• If you've added a new tool - have you followed the module conventions in the contribution docs
• Remove all TODO statements.
• Broadcast software version numbers to topic: versions - See version_topics
• Follow the naming conventions.
• Follow the parameters requirements.
• Follow the input/output options guidelines.
• Add a resource label
• Use BioConda and BioContainers if possible to fulfil software requirements.
• Ensure that the test works with either Docker / Singularity. Conda CI tests can be quite flaky:
  • For modules:
    • nf-core modules test <MODULE> --profile docker
    • nf-core modules test <MODULE> --profile singularity
    • nf-core modules test <MODULE> --profile conda

[SPPearce]

When would we want to be able to change the artifact directory? Could it not just be hardcoded entirely?

[fasterius]

This is something I've thought about changing myself, and would love some more discussions around - I kept it for this PR to mirror how the original QUARTONOTEBOOK (and RMARKDOWNNOTEBOOK / JUPYTERNOTEBOOK) module works. In my experience, most of the time the default is used without changing anything (which is cheap in terms of added boilerplate code, as the input parameter can just be [] when using default values). However, I've been experimenting with changing the artifact dir to better mirror the various contents that different module invocations yield - for example, the way I run my projects I would divide up into separate steps, maybe one for quality controls and one for analyses. Being able to change the artifact directory means being able to publish outputs into e.g. results/qc and results/analysis instead of results/artifacts (with the possibility to add subdirectories for the different output types, of course).

[SPPearce]

Why?

[fasterius]

There were several iterations on the general module structure before I came to the conclusion that one run per sample was the best tradeoff overall. The problem is that while it's easy on the Nextflow side to name output partials along the lines of ${notebook}-${meta.id}, partials have the added complication of needing the be explicitly named in the final HTML rendering (which would be done with the QUARTONOTEBOOK module) in {{< include >}} blocks. This would mean that any pipeline would have to hard-code multiple of these partial names into the final notebook, which seems non-ideal to me from a workflow standpoint.

The way I currently use partials (and the envisioned use-case for this module) is sequential reports where each subsequent report uses the output of the previous one, handling all of the samples at the same time (which is usually at the end of a per-sample pipeline, where one wants to analyse all the samples in aggregate). Any pipeline wanting to use the one-partial-per-sample route could change prefix to e.g. ${notebook.baseName}-${meta.id} using ext.prefix though, but since the way I use it currently is one-partial-per-notebook I put this as the default.