Skip to content

Commit f90c23b

Browse files
victorlinvictorlin
authored
Merge pull request #164: pathogen-repo-build: Support artifact-paths with AWS Batch
2 parents 0a9d8bb + 337b4a9 commit f90c23b

3 files changed

Lines changed: 141 additions & 34 deletions

File tree

‎.github/workflows/ci.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ jobs:
7171
with:
7272
repo: nextstrain/zika-tutorial
7373
runtime: aws-batch
74-
run: nextstrain build --env BOGUS_ENVVAR .
74+
run: nextstrain build --detach --no-download --env BOGUS_ENVVAR .
7575
artifact-name: build-output-aws-batch
7676
env: |
7777
BOGUS_ENVVAR: BOGUS_ENVVAR

‎.github/workflows/pathogen-repo-build.yaml‎

Lines changed: 105 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -104,15 +104,13 @@ on:
104104
105105
106106
build.log
107-
auspice/
108-
results/
109107
benchmarks/
110108
logs/
111109
.snakemake/log/
110+
auspice/ (not included for `aws-batch` runtime)
111+
results/ (not included for `aws-batch` runtime)
112112
113113
The "build.log" contains log messages from the `nextstrain build` command. The other paths are common output paths for Nextstrain builds. If a path does not exist in your build, then the action will still succeed and will print out a warning for the non-existent file(s). Use an exclude pattern for any of the default paths that you would like to exclude from the artifact (e.g. !build.log).
114-
115-
This is not supported for builds on AWS Batch because the workflow detaches from the build. Please use the `nextstrain build` command locally to reattach to AWS Batch builds to download outputs.
116114
type: string
117115
required: false
118116
outputs:
@@ -210,15 +208,13 @@ on:
210208
211209
212210
build.log
213-
auspice/
214-
results/
215211
benchmarks/
216212
logs/
217213
.snakemake/log/
214+
auspice/ (not included for `aws-batch` runtime)
215+
results/ (not included for `aws-batch` runtime)
218216
219217
The "build.log" contains log messages from the `nextstrain build` command. The other paths are common output paths for Nextstrain builds. If a path does not exist in your build, then the action will still succeed and will print out a warning for the non-existent file(s). Use an exclude pattern for any of the default paths that you would like to exclude from the artifact (e.g. !build.log).
220-
221-
This is not supported for builds on AWS Batch because the workflow detaches from the build. Please use the `nextstrain build` command locally to reattach to AWS Batch builds to download outputs.
222218
type: string
223219
required: false
224220
repo:
@@ -231,6 +227,13 @@ env:
231227
NEXTSTRAIN_GITHUB_DIR: .git/nextstrain/.github
232228
NEXTSTRAIN_BUILD_LOG: build.log
233229
NEXTSTRAIN_RUNTIME_ENVDIR: .git/nextstrain/env.d
230+
NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS: |
231+
build.log
232+
benchmarks/
233+
logs/
234+
.snakemake/log/
235+
${{ inputs.runtime != 'aws-batch' && 'auspice/' || '' }}
236+
${{ inputs.runtime != 'aws-batch' && 'results/' || '' }}
234237
permissions:
235238
id-token: write
236239
jobs:
@@ -348,18 +351,13 @@ jobs:
348351
run: |
349352
"$NEXTSTRAIN_GITHUB_DIR"/bin/interpolate-env < "$NEXTSTRAIN_GITHUB_DIR"/text-templates/attach-aws-batch.md \
350353
> "$GITHUB_STEP_SUMMARY"
351-
- if: always()
354+
- if: ${{ always() && !env.AWS_BATCH_JOB_ID }}
352355
uses: actions/upload-artifact@v7
353356
with:
354357
if-no-files-found: warn
355358
name: ${{ inputs.artifact-name }}
356359
path: |
357-
${{ env.NEXTSTRAIN_BUILD_LOG }}
358-
auspice/
359-
results/
360-
benchmarks/
361-
logs/
362-
.snakemake/log/
360+
${{ env.NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS }}
363361
${{ inputs.artifact-paths }}
364362
outputs:
365363
AWS_BATCH_JOB_ID: ${{ env.AWS_BATCH_JOB_ID }}
@@ -402,20 +400,34 @@ jobs:
402400
name: Attach to AWS Batch job
403401
env:
404402
AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }}
403+
ARTIFACT_PATHS: ${{ inputs.artifact-paths }}
405404
run: |
406405
# Using timeout to detach from build before we hit the job.timeout-minutes
407406
# to avoid cancelling the job in order to avoid a "Cancelled" status
408407
# for the whole workflow.¹ Using SIGINT to ensure that the GH Action
409408
# cleanly detaches from the build without cancelling it.²
410409
# ¹ <https://github.com/nextstrain/.github/issues/129#issuecomment-2956350306>
411410
# ² <https://github.com/nextstrain/.github/pull/134#discussion_r2325804695>
411+
412+
# Set download args for default and custom artifact paths, skipping
413+
# blank lines. Append directories with ** to match files within.
414+
download_args=()
415+
while IFS= read -r line; do
416+
[[ -z "$line" ]] && continue
417+
[[ "$line" == */ ]] && line+='**'
418+
download_args+=(--download "$line")
419+
done <<< "${NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS}${ARTIFACT_PATHS}"
420+
412421
exit_status=0
413422
timeout --signal INT 359m \
414423
nextstrain build \
415424
--aws-batch \
416425
--attach "$AWS_BATCH_JOB_ID" \
417426
--detach-on-interrupt \
418-
--no-download || exit_status=$?
427+
"${download_args[@]}" . \
428+
|& tee .git/"$NEXTSTRAIN_BUILD_LOG" || exit_status=$?
429+
430+
mv .git/"$NEXTSTRAIN_BUILD_LOG" "$NEXTSTRAIN_BUILD_LOG"
419431
420432
if [[ $exit_status == 0 ]]; then
421433
conclusion="success"
@@ -431,6 +443,14 @@ jobs:
431443
432444
echo "conclusion=$conclusion" | tee -a "$GITHUB_OUTPUT"
433445
exit $exit_status
446+
- if: ${{ always() && steps.attach.outputs.conclusion != 'timeout' }}
447+
uses: actions/upload-artifact@v7
448+
with:
449+
if-no-files-found: warn
450+
name: ${{ inputs.artifact-name }}
451+
path: |
452+
${{ env.NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS }}
453+
${{ inputs.artifact-paths }}
434454
# Emit a "conclusion" output for the job that's based on the conclusion
435455
# (success, timeout, failure) of the "attach" step above.
436456
# This is the conclusion we care about for the job.
@@ -470,20 +490,34 @@ jobs:
470490
name: Attach to AWS Batch job
471491
env:
472492
AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }}
493+
ARTIFACT_PATHS: ${{ inputs.artifact-paths }}
473494
run: |
474495
# Using timeout to detach from build before we hit the job.timeout-minutes
475496
# to avoid cancelling the job in order to avoid a "Cancelled" status
476497
# for the whole workflow.¹ Using SIGINT to ensure that the GH Action
477498
# cleanly detaches from the build without cancelling it.²
478499
# ¹ <https://github.com/nextstrain/.github/issues/129#issuecomment-2956350306>
479500
# ² <https://github.com/nextstrain/.github/pull/134#discussion_r2325804695>
501+
502+
# Set download args for default and custom artifact paths, skipping
503+
# blank lines. Append directories with ** to match files within.
504+
download_args=()
505+
while IFS= read -r line; do
506+
[[ -z "$line" ]] && continue
507+
[[ "$line" == */ ]] && line+='**'
508+
download_args+=(--download "$line")
509+
done <<< "${NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS}${ARTIFACT_PATHS}"
510+
480511
exit_status=0
481512
timeout --signal INT 359m \
482513
nextstrain build \
483514
--aws-batch \
484515
--attach "$AWS_BATCH_JOB_ID" \
485516
--detach-on-interrupt \
486-
--no-download || exit_status=$?
517+
"${download_args[@]}" . \
518+
|& tee .git/"$NEXTSTRAIN_BUILD_LOG" || exit_status=$?
519+
520+
mv .git/"$NEXTSTRAIN_BUILD_LOG" "$NEXTSTRAIN_BUILD_LOG"
487521
488522
if [[ $exit_status == 0 ]]; then
489523
conclusion="success"
@@ -499,6 +533,14 @@ jobs:
499533
500534
echo "conclusion=$conclusion" | tee -a "$GITHUB_OUTPUT"
501535
exit $exit_status
536+
- if: ${{ always() && steps.attach.outputs.conclusion != 'timeout' }}
537+
uses: actions/upload-artifact@v7
538+
with:
539+
if-no-files-found: warn
540+
name: ${{ inputs.artifact-name }}
541+
path: |
542+
${{ env.NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS }}
543+
${{ inputs.artifact-paths }}
502544
# Emit a "conclusion" output for the job that's based on the conclusion
503545
# (success, timeout, failure) of the "attach" step above.
504546
# This is the conclusion we care about for the job.
@@ -539,20 +581,34 @@ jobs:
539581
name: Attach to AWS Batch job
540582
env:
541583
AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }}
584+
ARTIFACT_PATHS: ${{ inputs.artifact-paths }}
542585
run: |
543586
# Using timeout to detach from build before we hit the job.timeout-minutes
544587
# to avoid cancelling the job in order to avoid a "Cancelled" status
545588
# for the whole workflow.¹ Using SIGINT to ensure that the GH Action
546589
# cleanly detaches from the build without cancelling it.²
547590
# ¹ <https://github.com/nextstrain/.github/issues/129#issuecomment-2956350306>
548591
# ² <https://github.com/nextstrain/.github/pull/134#discussion_r2325804695>
592+
593+
# Set download args for default and custom artifact paths, skipping
594+
# blank lines. Append directories with ** to match files within.
595+
download_args=()
596+
while IFS= read -r line; do
597+
[[ -z "$line" ]] && continue
598+
[[ "$line" == */ ]] && line+='**'
599+
download_args+=(--download "$line")
600+
done <<< "${NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS}${ARTIFACT_PATHS}"
601+
549602
exit_status=0
550603
timeout --signal INT 359m \
551604
nextstrain build \
552605
--aws-batch \
553606
--attach "$AWS_BATCH_JOB_ID" \
554607
--detach-on-interrupt \
555-
--no-download || exit_status=$?
608+
"${download_args[@]}" . \
609+
|& tee .git/"$NEXTSTRAIN_BUILD_LOG" || exit_status=$?
610+
611+
mv .git/"$NEXTSTRAIN_BUILD_LOG" "$NEXTSTRAIN_BUILD_LOG"
556612
557613
if [[ $exit_status == 0 ]]; then
558614
conclusion="success"
@@ -568,6 +624,14 @@ jobs:
568624
569625
echo "conclusion=$conclusion" | tee -a "$GITHUB_OUTPUT"
570626
exit $exit_status
627+
- if: ${{ always() && steps.attach.outputs.conclusion != 'timeout' }}
628+
uses: actions/upload-artifact@v7
629+
with:
630+
if-no-files-found: warn
631+
name: ${{ inputs.artifact-name }}
632+
path: |
633+
${{ env.NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS }}
634+
${{ inputs.artifact-paths }}
571635
# Emit a "conclusion" output for the job that's based on the conclusion
572636
# (success, timeout, failure) of the "attach" step above.
573637
# This is the conclusion we care about for the job.
@@ -608,20 +672,34 @@ jobs:
608672
name: Attach to AWS Batch job
609673
env:
610674
AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }}
675+
ARTIFACT_PATHS: ${{ inputs.artifact-paths }}
611676
run: |
612677
# Using timeout to detach from build before we hit the job.timeout-minutes
613678
# to avoid cancelling the job in order to avoid a "Cancelled" status
614679
# for the whole workflow.¹ Using SIGINT to ensure that the GH Action
615680
# cleanly detaches from the build without cancelling it.²
616681
# ¹ <https://github.com/nextstrain/.github/issues/129#issuecomment-2956350306>
617682
# ² <https://github.com/nextstrain/.github/pull/134#discussion_r2325804695>
683+
684+
# Set download args for default and custom artifact paths, skipping
685+
# blank lines. Append directories with ** to match files within.
686+
download_args=()
687+
while IFS= read -r line; do
688+
[[ -z "$line" ]] && continue
689+
[[ "$line" == */ ]] && line+='**'
690+
download_args+=(--download "$line")
691+
done <<< "${NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS}${ARTIFACT_PATHS}"
692+
618693
exit_status=0
619694
timeout --signal INT 359m \
620695
nextstrain build \
621696
--aws-batch \
622697
--attach "$AWS_BATCH_JOB_ID" \
623698
--detach-on-interrupt \
624-
--no-download || exit_status=$?
699+
"${download_args[@]}" . \
700+
|& tee .git/"$NEXTSTRAIN_BUILD_LOG" || exit_status=$?
701+
702+
mv .git/"$NEXTSTRAIN_BUILD_LOG" "$NEXTSTRAIN_BUILD_LOG"
625703
626704
if [[ $exit_status == 0 ]]; then
627705
conclusion="success"
@@ -637,6 +715,14 @@ jobs:
637715
638716
echo "conclusion=$conclusion" | tee -a "$GITHUB_OUTPUT"
639717
exit $exit_status
718+
- if: ${{ always() && steps.attach.outputs.conclusion != 'timeout' }}
719+
uses: actions/upload-artifact@v7
720+
with:
721+
if-no-files-found: warn
722+
name: ${{ inputs.artifact-name }}
723+
path: |
724+
${{ env.NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS }}
725+
${{ inputs.artifact-paths }}
640726
# Emit a "conclusion" output for the job that's based on the conclusion
641727
# (success, timeout, failure) of the "attach" step above.
642728
# This is the conclusion we care about for the job.

‎.github/workflows/pathogen-repo-build.yaml.in‎

Lines changed: 35 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -142,11 +142,11 @@ on:
142142
The default paths included in the artifact are:
143143

144144
build.log
145-
auspice/
146-
results/
147145
benchmarks/
148146
logs/
149147
.snakemake/log/
148+
auspice/ (not included for `aws-batch` runtime)
149+
results/ (not included for `aws-batch` runtime)
150150

151151
The "build.log" contains log messages from the
152152
`nextstrain build` command.
@@ -155,10 +155,6 @@ on:
155155
succeed and will print out a warning for the non-existent file(s).
156156
Use an exclude pattern for any of the default paths that you would like to
157157
exclude from the artifact (e.g. !build.log).
158-
159-
This is not supported for builds on AWS Batch because the workflow
160-
detaches from the build. Please use the `nextstrain build` command
161-
locally to reattach to AWS Batch builds to download outputs.
162158
type: string
163159
required: false
164160

@@ -189,6 +185,13 @@ env:
189185
NEXTSTRAIN_GITHUB_DIR: .git/nextstrain/.github
190186
NEXTSTRAIN_BUILD_LOG: build.log
191187
NEXTSTRAIN_RUNTIME_ENVDIR: .git/nextstrain/env.d
188+
NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS: |
189+
build.log
190+
benchmarks/
191+
logs/
192+
.snakemake/log/
193+
${{ inputs.runtime != 'aws-batch' && 'auspice/' || '' }}
194+
${{ inputs.runtime != 'aws-batch' && 'results/' || '' }}
192195

193196
permissions:
194197
id-token: write
@@ -329,18 +332,13 @@ jobs:
329332
"$NEXTSTRAIN_GITHUB_DIR"/bin/interpolate-env < "$NEXTSTRAIN_GITHUB_DIR"/text-templates/attach-aws-batch.md \
330333
> "$GITHUB_STEP_SUMMARY"
331334

332-
- if: always()
335+
- if: ${{ always() && !env.AWS_BATCH_JOB_ID }}
333336
uses: actions/upload-artifact@v7
334337
with:
335338
if-no-files-found: warn
336339
name: ${{ inputs.artifact-name }}
337340
path: |
338-
${{ env.NEXTSTRAIN_BUILD_LOG }}
339-
auspice/
340-
results/
341-
benchmarks/
342-
logs/
343-
.snakemake/log/
341+
${{ env.NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS }}
344342
${{ inputs.artifact-paths }}
345343

346344
outputs:
@@ -366,20 +364,34 @@ jobs:
366364
name: Attach to AWS Batch job
367365
env:
368366
AWS_BATCH_JOB_ID: ${{ needs.run-build.outputs.AWS_BATCH_JOB_ID }}
367+
ARTIFACT_PATHS: ${{ inputs.artifact-paths }}
369368
run: |
370369
# Using timeout to detach from build before we hit the job.timeout-minutes
371370
# to avoid cancelling the job in order to avoid a "Cancelled" status
372371
# for the whole workflow.¹ Using SIGINT to ensure that the GH Action
373372
# cleanly detaches from the build without cancelling it.²
374373
# ¹ <https://github.com/nextstrain/.github/issues/129#issuecomment-2956350306>
375374
# ² <https://github.com/nextstrain/.github/pull/134#discussion_r2325804695>
375+
376+
# Set download args for default and custom artifact paths, skipping
377+
# blank lines. Append directories with ** to match files within.
378+
download_args=()
379+
while IFS= read -r line; do
380+
[[ -z "$line" ]] && continue
381+
[[ "$line" == */ ]] && line+='**'
382+
download_args+=(--download "$line")
383+
done <<< "${NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS}${ARTIFACT_PATHS}"
384+
376385
exit_status=0
377386
timeout --signal INT 359m \
378387
nextstrain build \
379388
--aws-batch \
380389
--attach "$AWS_BATCH_JOB_ID" \
381390
--detach-on-interrupt \
382-
--no-download || exit_status=$?
391+
"${download_args[@]}" . \
392+
|& tee .git/"$NEXTSTRAIN_BUILD_LOG" || exit_status=$?
393+
394+
mv .git/"$NEXTSTRAIN_BUILD_LOG" "$NEXTSTRAIN_BUILD_LOG"
383395

384396
if [[ $exit_status == 0 ]]; then
385397
conclusion="success"
@@ -396,6 +408,15 @@ jobs:
396408
echo "conclusion=$conclusion" | tee -a "$GITHUB_OUTPUT"
397409
exit $exit_status
398410

411+
- if: ${{ always() && steps.attach.outputs.conclusion != 'timeout' }}
412+
uses: actions/upload-artifact@v7
413+
with:
414+
if-no-files-found: warn
415+
name: ${{ inputs.artifact-name }}
416+
path: |
417+
${{ env.NEXTSTRAIN_DEFAULT_ARTIFACT_PATHS }}
418+
${{ inputs.artifact-paths }}
419+
399420
# Emit a "conclusion" output for the job that's based on the conclusion
400421
# (success, timeout, failure) of the "attach" step above.
401422
# This is the conclusion we care about for the job.

0 commit comments

Comments
 (0)