You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: MC/doc/WorkflowRunner.md
+7-9Lines changed: 7 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,11 +13,11 @@ not **what** is executed or how it is configured.
13
13
The tool provides features of a typical data/task pipelining environment
14
14
using a DAG approach. It allows to separate the concerns of workflow setup and workflow running - and as such
15
15
allows to optimize workflow deployment during execution.
16
-
It is inspired by similar ideas in ALICE alibuild and O2/DPL but differs in some aspects:
16
+
It is inspired by similar ideas in ALICE alibuild and the ALILE Data Processing Layer (DPL) but differs in some aspects:
17
17
18
18
* allow to schedule tasks in parallel
19
-
* be resource aware (e.g., don't schedule 2 tasks requires large memory at the same time)
20
-
* allow to schedule any task, be at a simple executable, bash script, ROOT macro, or DPL workflow.
19
+
* be resource aware (e.g., don't schedule 2 tasks requiring large memory at the same time)
20
+
* allow to schedule any task, be it a simple executable, bash script, ROOT macro, or DPL workflow.
21
21
22
22
The goals of the tool are:
23
23
@@ -27,7 +27,8 @@ The goals of the tool are:
27
27
* skip-done features when run again with same input
28
28
* if input changes, rerun only affected stages
29
29
* automatic task skipping in case it is not relevant for the goal
30
-
* dream: automatic task fusion (say DPL workflow for TPC digitization and TPC reco) when we don't need the intermediate files
30
+
* file provenance tracking, cleanup of intermediate products
31
+
* dream: automatic DPL fusion/pipelining (say DPL workflow for TPC digitization and TPC reco) when we don't need the intermediate files
31
32
on disc
32
33
33
34
# Workflow specification
@@ -71,12 +72,12 @@ Naturally, task2 depends on task1 expressed via the `needs` list.
71
72
Further keys in this format are:
72
73
| field | description |
73
74
| ----- | ----------- |
74
-
|`resources`| estimated resource usage for average cpu load (250 = 2.5 CPUs) and maximal memory in MB. Used for scheduling. |
75
+
|`resources`| estimated resource usage for average cpu load (250 = 2.5 CPUs) and maximal memory in MB. Used for scheduling. -1 is used for unknown or don't care. |
75
76
|`timeframe`| timeframe index or -1 if not associated to any timeframe. May have influence on order of execution (prefer finish timeframe first) |
76
77
|`cwd`| the workding directory where this is to be executed |
77
78
| `label` | a list labels, describing this stage. Can be used to execute workfow in stages (such as 'do all digitization', 'run everthing for ITS'
78
79
79
-
A workflow is generated by different tools. A current example following the PWGHF embedding exercise can be found here ..
80
+
A workflow is generated by different tools. A current example following the PWGHF embedding exercise can be found here https://github.com/AliceO2Group/O2DPG/blob/master/MC/run/PWGHF/create_embedding_workflow.py
0 commit comments