[8.0] Interpret outputPath with LFN: prefix as an absolute one - #8603
[8.0] Interpret outputPath with LFN: prefix as an absolute one#8603atsareg wants to merge 9 commits into
Conversation
|
Thank you. All seems fine to me. |
fstagni
left a comment
fstagni
left a comment
There was a problem hiding this comment.
@aldbr has added a lot of tests for JobWrapper in https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/WorkloadManagementSystem/JobWrapper/test/Test_JobWrapper.py.
Can you see if (some of) those can be used here? I am always somewhat scared of changing the JobWrapper.
| # Check whether the list of LFNs has globbable patterns | ||
| globbedLfnList = [] | ||
| for lfn in lfnList: | ||
| lfnPath = os.path.dirname(lfn) | ||
| lfnLocal = os.path.basename(lfn) | ||
| globbedLfnList += [os.path.join(lfnPath, gLfn) for gLfn in List.uniqueElements(getGlobbedFiles(lfnLocal))] | ||
| if globbedLfnList != lfnList and globbedLfnList: | ||
| self.log.info("Found a pattern in the output data LFN list, LFNs to upload are:", ", ".join(globbedLfnList)) | ||
| lfnList = globbedLfnList | ||
|
|
||
| # Check whether the list of outputData has a globbable pattern |
There was a problem hiding this comment.
There is a code duplication between these inserted lines and those that follow, care to refactor for simplicity?
| # If output path is given with the LFN: prefix, take it as an absolute path | ||
| elif outputPath.startswith("LFN:"): | ||
| outputPath = outputPath[4:] | ||
| basePath = "" |
There was a problem hiding this comment.
Shouldn't this block be before the previous 2 lines? For the case when outputPath == "LFN:/some/where/some/thing.xyz"
Yes, there are many tests to JobWrapper added to the integration/9.0 . In 8.0 there are just few. Will see if I can add something here |
| - file names with wild cards; same after the file names expansion; | ||
| - file names in a form "LFN:/vo/full/destination/path/file.name"; in this case the file will be uploaded | ||
| to the specified LFN path without taking into account the OutputPath. Note that file.name here can be also | ||
| specified with wild cards. |
There was a problem hiding this comment.
Give an example of a wild card.
3 participants
This PR allows to define absolute outputPath by specifying it with the LFN: prefix.
It also allows using wild cards in outputData file names specified as LFNs
BEGINRELEASENOTES
*WorkloadManagement
NEW: JobWrapper - interpret outputPath with LFN: prefix as an absolute one
NEW: JobWrapper - allow wild cards in output LFNs
For examples look into release.notes
ENDRELEASENOTES