Skip to content

ENT-13265: Retry rsync/ssh on proxy-target setup commands (3.24.x) #2265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
larsewi wants to merge 1 commit into
base: 3.24.x
Choose a base branch
from
+28 −5
Open

ENT-13265: Retry rsync/ssh on proxy-target setup commands (3.24.x) #2265

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 28 additions & 5 deletions ci/initialize-build-host.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,29 @@ command -v /sbin/ip 2>/dev/null && /sbin/ip addr || true
RSYNC="rsync --delete -zrlpt -T /tmp"
RSH="ssh -o BatchMode=yes"

# Retry a command up to 5 times with a 10 second pause between attempts.
# Used for the early proxy-target setup commands (rsync/ssh) where a brief
# network blip to an exotic build host should not fail the entire build.
try_run() {
max_tries=5
i=1
ret=0
while [ "$i" -le "$max_tries" ]; do
ret=0
"$@" || ret=$?
if [ "$ret" -eq 0 ]; then
return 0
fi
if [ "$i" -lt "$max_tries" ]; then
echo "try_run: attempt $i/$max_tries failed (exit $ret) for: $*; retrying in 10s..." >&2
sleep 10
fi
i=$((i + 1))
done
echo "try_run: command failed after $max_tries attempts: $*" >&2
return "$ret"
}

# Support launching scripts that were initially launched under bash.
if [ -n "$BASH_VERSION" ]
then
Expand Down Expand Up @@ -304,7 +327,7 @@ then
# --------------------------------------------------------------------------

# Put our currently executing script on the proxy target.
$RSYNC -e "$RSH" "$0" $login:commands-from-proxy.sh
try_run $RSYNC -e "$RSH" "$0" $login:commands-from-proxy.sh

# And the important parts of the environment.
for var in \
Expand Down Expand Up @@ -356,7 +379,7 @@ then
echo "PROXIED=1" >> env.sh
echo "export PROXIED" >> env.sh

$RSYNC -e "$RSH" env.sh $login:.
try_run $RSYNC -e "$RSH" env.sh $login:.

# And the helper tools, including this script.
# Note that only provisioned hosts will have this in HOME, since they use
Expand All @@ -365,7 +388,7 @@ then
# instead, synced separately below.
if [ -d $HOME/mender-qa ]
then
$RSYNC -e "$RSH" $HOME/mender-qa $login:.
try_run $RSYNC -e "$RSH" $HOME/mender-qa $login:.
fi

# Copy the workspace. If there is no workspace defined, we are not in the
Expand All @@ -379,8 +402,8 @@ then
echo "$WORKSPACE_REMOTE is not removed on build host."
exit 2
fi
$RSH $login mkdir -p "$WORKSPACE_REMOTE"
$RSYNC -e "$RSH" "$WORKSPACE"/ $login:"$WORKSPACE_REMOTE"/
try_run $RSH $login mkdir -p "$WORKSPACE_REMOTE"
try_run $RSYNC -e "$RSH" "$WORKSPACE"/ $login:"$WORKSPACE_REMOTE"/
fi

# --------------------------------------------------------------------------
Expand Down