Skip to content

workflows4: Fix workflow scripts and enhance prun options validation#86

Open
mightqxc wants to merge 16 commits into
masterfrom
workflows4
Open

workflows4: Fix workflow scripts and enhance prun options validation#86
mightqxc wants to merge 16 commits into
masterfrom
workflows4

Conversation

@mightqxc
Copy link
Copy Markdown
Contributor

No description provided.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends the native workflow tooling by adding an on-worker merge script, making the multistep_merge workflow template more self-contained (defaults + sandboxing the merge script), and expanding the workflow description schema to support secondary datasets, file-type filters, and containers.

Changes:

  • Add and package share/workflow_scripts/merge.sh and update the multistep_merge template to use it (including copying it into the sandbox).
  • Extend WorkflowDescription/WorkflowStep to serialize/validate new step attributes: inDsType, secondaryDSs, secondaryDsTypes, containerImage, and useAthenaPackages.
  • Adjust pchain_native startup to restore Rucio-related import paths from the original PYTHONPATH.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
share/workflow_scripts/merge.sh New worker-node merge helper that consumes --writeInputToTxt output and runs hadd.
setup.py Installs the new workflow script into etc/panda/share/workflow_scripts.
pandaclient/workflow_templates/multistep_merge.py Adds defaults, script discovery/copy-to-sandbox behavior, and configurable output/input filenames.
pandaclient/workflow_description.py Expands step schema + validation and updates documentation examples.
pandaclient/PChainScript_native.py Restores selected paths from the original environment to support importing rucio.
packages/full/pyproject.toml Ensures workflow scripts are included as shared wheel data.
examples/signal_background_combine_wfd.yaml Adds a multi-branch workflow example using secondary datasets and type filters.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pandaclient/workflow_templates/multistep_merge.py
Comment thread pandaclient/workflow_templates/multistep_merge.py
Comment thread pandaclient/workflow_description.py
Comment thread pandaclient/PChainScript_native.py
Comment thread share/workflow_scripts/merge.sh
mightqxc and others added 6 commits May 29, 2026 08:10
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@mightqxc mightqxc marked this pull request as ready for review May 29, 2026 06:20
@mightqxc mightqxc requested a review from tmaeno May 29, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants