Skip to content

Commit eb18859

Browse files
committed
Update WorkflowRunner.md
1 parent 1e1726c commit eb18859

File tree

1 file changed

+7
-9
lines changed

1 file changed

+7
-9
lines changed

MC/doc/WorkflowRunner.md

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,11 @@ not **what** is executed or how it is configured.
1313
The tool provides features of a typical data/task pipelining environment
1414
using a DAG approach. It allows to separate the concerns of workflow setup and workflow running - and as such
1515
allows to optimize workflow deployment during execution.
16-
It is inspired by similar ideas in ALICE alibuild and O2/DPL but differs in some aspects:
16+
It is inspired by similar ideas in ALICE alibuild and the ALILE Data Processing Layer (DPL) but differs in some aspects:
1717

1818
* allow to schedule tasks in parallel
19-
* be resource aware (e.g., don't schedule 2 tasks requires large memory at the same time)
20-
* allow to schedule any task, be at a simple executable, bash script, ROOT macro, or DPL workflow.
19+
* be resource aware (e.g., don't schedule 2 tasks requiring large memory at the same time)
20+
* allow to schedule any task, be it a simple executable, bash script, ROOT macro, or DPL workflow.
2121

2222
The goals of the tool are:
2323

@@ -27,7 +27,8 @@ The goals of the tool are:
2727
* skip-done features when run again with same input
2828
* if input changes, rerun only affected stages
2929
* automatic task skipping in case it is not relevant for the goal
30-
* dream: automatic task fusion (say DPL workflow for TPC digitization and TPC reco) when we don't need the intermediate files
30+
* file provenance tracking, cleanup of intermediate products
31+
* dream: automatic DPL fusion/pipelining (say DPL workflow for TPC digitization and TPC reco) when we don't need the intermediate files
3132
on disc
3233

3334
# Workflow specification
@@ -71,12 +72,12 @@ Naturally, task2 depends on task1 expressed via the `needs` list.
7172
Further keys in this format are:
7273
| field | description |
7374
| ----- | ----------- |
74-
| `resources` | estimated resource usage for average cpu load (250 = 2.5 CPUs) and maximal memory in MB. Used for scheduling. |
75+
| `resources` | estimated resource usage for average cpu load (250 = 2.5 CPUs) and maximal memory in MB. Used for scheduling. -1 is used for unknown or don't care. |
7576
| `timeframe` | timeframe index or -1 if not associated to any timeframe. May have influence on order of execution (prefer finish timeframe first) |
7677
| `cwd` | the workding directory where this is to be executed |
7778
| `label` | a list labels, describing this stage. Can be used to execute workfow in stages (such as 'do all digitization', 'run everthing for ITS'
7879

79-
A workflow is generated by different tools. A current example following the PWGHF embedding exercise can be found here ..
80+
A workflow is generated by different tools. A current example following the PWGHF embedding exercise can be found here https://github.com/AliceO2Group/O2DPG/blob/master/MC/run/PWGHF/create_embedding_workflow.py
8081

8182
## Example usage
8283

@@ -88,20 +89,17 @@ o2_dpg_workflow_runner.py -f workflow_sim.json
8889

8990
Show what you would run
9091
```
91-
alienv enter O2/latest O2DPG/latest
9292
o2_dpg_workflow_runner.py -f workflow_sim.json --dry-run
9393
```
9494

9595
## Future targeted features:
9696

9797
Run until everyting marked "RECO" is done
9898
```
99-
alienv enter O2/latest O2DPG/latest
10099
o2_dpg_workflow_runner.py -f workflow_sim.json --stages RECO
101100
```
102101

103102
Rerun worflow until AOD, skipping all tasks already done
104103
```
105-
alienv enter O2/latest O2DPG/latest
106104
o2_dpg_workflow_runner.py -f workflow_sim.json --stages AOD --skip-done
107105
```

0 commit comments

Comments
 (0)