Skip to content

[WIP] testparrallel #16091

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
XiyunZhao wants to merge 1 commit into
base: main
Choose a base branch
from
+140 −19
Open

[WIP] testparrallel #16091

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 105 additions & 3 deletions frontend/integration-tests/test-cypress.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,23 +37,25 @@ function generateReport {
}
trap "copyArtifacts; generateReport" EXIT

while getopts p:s:h:l:n: flag
while getopts p:s:h:l:n:P: flag
do
case "${flag}" in
p) pkg=${OPTARG};;
s) spec=${OPTARG};;
h) headless=${OPTARG};;
n) nightly=${OPTARG};;
P) parallel=${OPTARG};;
Comment on lines +40 to +47
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

git ls-files frontend/integration-tests/test-cypress.sh

Repository: openshift/console

Length of output: 103

🏁 Script executed:

sed -n '40,47p; 58p; 104,105p' frontend/integration-tests/test-cypress.sh

Repository: openshift/console

Length of output: 412

🏁 Script executed:

sed -n '40,110p' frontend/integration-tests/test-cypress.sh | cat -n

Repository: openshift/console

Length of output: 3938

🏁 Script executed:

grep -n "parallel" frontend/integration-tests/test-cypress.sh

Repository: openshift/console

Length of output: 501

-P false still enables parallel mode

The conditional at line 104 checks only for non-empty input using [ -n "${parallel-}" ], so -P false still takes the parallel branch, contradicting the documented -P true usage in the help text (line 58).

Proposed fix 
while getopts p:s:h:l:n:P: flag
do
  case "${flag}" in
    p) pkg=${OPTARG};;
    s) spec=${OPTARG};;
    h) headless=${OPTARG};;
    n) nightly=${OPTARG};;
-   P) parallel=${OPTARG};;
+   P) parallel="$(printf '%s' "${OPTARG}" | tr '[:upper:]' '[:lower:]')";;
  esac
done
+
+if [ -n "${parallel-}" ] && [ "${parallel}" != "true" ] && [ "${parallel}" != "false" ]; then
+  echo "Invalid -P value '${parallel}'. Expected true|false." >&2
+  exit 2
+fi
@@
-  if [ -n "${parallel-}" ]; then
+  if [ "${parallel-false}" = "true" ]; then

Also applies to: 58, 104

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@frontend/integration-tests/test-cypress.sh` around lines 40 - 47, The script
treats any non-empty -P value as enabling parallel mode, so passing "-P false"
still turns it on; update the option parsing/conditional to interpret the
parallel flag as a boolean string (e.g., only enable parallel when parallel ==
"true" or "1"), not just non-empty. Specifically, change the conditional that
checks the variable parallel (currently using [ -n "${parallel-}" ]) to
explicitly compare the value of parallel (e.g., [[ "${parallel}" == "true" ]] or
similar) and adjust any help text or defaults to match this behavior, ensuring
the getopts handling of the P flag and subsequent branch that uses parallel
consistently expect "true"/"false".

esac
done

if [ $# -eq 0 ]; then
echo "Runs Cypress tests in Test Runner or headless mode"
echo "Usage: test-cypress [-p] <package> [-s] <filemask> [-h true] [-n true/false]"
echo "Usage: test-cypress [-p] <package> [-s] <filemask> [-h true] [-n true/false] [-P true]"
echo " '-p <package>' may be 'console, 'olm', 'devconsole'"
echo " '-s <specmask>' is a file mask for spec test files, such as 'tests/monitoring/*'. Used only in headless mode when '-p' is specified."
echo " '-h true' runs Cypress in headless mode. When omitted, launches Cypress Test Runner"
echo " '-n true' runs the 'nightly' suite, all specs from selected packages in headless mode"
echo " '-P true' runs test packages in parallel (only works with -h true)"
echo "Examples:"
echo " test-cypress.sh // displays this help text"
echo " test-cypress.sh -p console // opens Cypress Test Runner for console tests"
Expand All @@ -62,7 +64,8 @@ if [ $# -eq 0 ]; then
echo " test-cypress.sh -p knative // opens Cypress Test Runner for knative tests"
echo " test-cypress.sh -p shipwright // opens Cypress Test Runner for shipwright tests"
echo " test-cypress.sh -p webterminal // opens Cypress Test Runner for webterminal tests"
echo " test-cypress.sh -h true // runs all packages in headless mode"
echo " test-cypress.sh -h true // runs all packages in headless mode (sequential)"
echo " test-cypress.sh -h true -P true // runs all packages in headless mode (parallel)"
echo " test-cypress.sh -p olm -h true // runs OLM tests in headless mode"
echo " test-cypress.sh -p console -s 'tests/crud/*' -h true // runs console CRUD tests in headless mode"
echo " test-cypress.sh -n true // runs the whole nightly suite"
Expand All @@ -87,6 +90,7 @@ if [ -n "${nightly-}" ] && [ -z "${pkg-}" ]; then
fi

if [ -n "${headless-}" ] && [ -z "${pkg-}" ]; then
<<<<<<< Updated upstream
yarn run test-cypress-console-headless
yarn run test-cypress-dev-console-headless
yarn run test-cypress-olm-headless
Expand All @@ -95,6 +99,104 @@ if [ -n "${headless-}" ] && [ -z "${pkg-}" ]; then
yarn run test-cypress-knative-headless
yarn run test-cypress-topology-headless
# yarn run test-cypress-shipwright-headless
=======
# Check if parallel mode is enabled
if [ -n "${parallel-}" ]; then
echo "=========================================="
echo "Running Cypress tests in PARALLEL mode"
echo "=========================================="

# Array to store background process IDs and their package names
declare -a pids
declare -a packages

# Function to run a package test in background
run_package_test() {
local package_name=$1
local script_name=$2

echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting: $package_name"

# Run the test and capture output to a separate log file
yarn run "$script_name" > "${SCREENSHOTS_DIR}/${package_name}.log" 2>&1 &
local pid=$!

pids+=($pid)
packages+=($package_name)

echo "[$(date '+%Y-%m-%d %H:%M:%S')] Started $package_name with PID $pid"
}

# Ensure screenshots directory exists
mkdir -p "${SCREENSHOTS_DIR}"

# Start all package tests in parallel
run_package_test "console" "test-cypress-console-headless"
run_package_test "dev-console" "test-cypress-dev-console-headless"
run_package_test "olm" "test-cypress-olm-headless"
run_package_test "webterminal" "test-cypress-webterminal-headless"
run_package_test "helm" "test-cypress-helm-headless"
run_package_test "knative" "test-cypress-knative-headless"
run_package_test "topology" "test-cypress-topology-headless"
run_package_test "pipelines" "test-cypress-pipelines-headless"
# run_package_test "shipwright" "test-cypress-shipwright-headless"

echo ""
echo "All test packages started. Waiting for completion..."
echo "=========================================="

# Wait for all background processes and collect their exit statuses
failed_packages=()
for i in "${!pids[@]}"; do
pid=${pids[$i]}
package=${packages[$i]}

echo "[$(date '+%Y-%m-%d %H:%M:%S')] Waiting for $package (PID: $pid)..."

if wait $pid; then
echo "[$(date '+%Y-%m-%d %H:%M:%S')] ✓ $package completed successfully"
else
exit_code=$?
echo "[$(date '+%Y-%m-%d %H:%M:%S')] ✗ $package failed with exit code $exit_code"
failed_packages+=("$package")
fi
done

echo ""
echo "=========================================="
echo "Parallel test execution completed"
echo "=========================================="

# Print summary
if [ ${#failed_packages[@]} -eq 0 ]; then
echo "✓ All packages passed!"
echo ""
echo "Individual package logs:"
for package in "${packages[@]}"; do
echo " - ${SCREENSHOTS_DIR}/${package}.log"
done
exit 0
else
echo "✗ ${#failed_packages[@]} package(s) failed:"
for package in "${failed_packages[@]}"; do
echo " - $package (see ${SCREENSHOTS_DIR}/${package}.log)"
done
exit 1
fi
else
# Original sequential execution
echo "Running Cypress tests in SEQUENTIAL mode (use -P true for parallel)"
yarn run test-cypress-console-headless
yarn run test-cypress-dev-console-headless
yarn run test-cypress-olm-headless
yarn run test-cypress-webterminal-headless
yarn run test-cypress-helm-headless
yarn run test-cypress-knative-headless
yarn run test-cypress-topology-headless
yarn run test-cypress-pipelines-headless
# yarn run test-cypress-shipwright-headless
fi
>>>>>>> Stashed changes
exit;
fi

Expand Down
51 changes: 35 additions & 16 deletions test-prow-e2e.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,34 @@
#!/usr/bin/env bash

set -exuo pipefail

INSTALLER_DIR=${INSTALLER_DIR:=${ARTIFACT_DIR}/installer}

# don't log kubeadmin-password
set +x
export BRIDGE_KUBEADMIN_PASSWORD="$(cat "${KUBEADMIN_PASSWORD_FILE:-${INSTALLER_DIR}/auth/kubeadmin-password}")"
set -x
export BRIDGE_BASE_ADDRESS="$(oc get consoles.config.openshift.io cluster -o jsonpath='{.status.consoleURL}')"

./contrib/create-user.sh

pushd frontend

SCENARIO="${1:-e2e}"

# test-prow-e2e.sh - Wrapper script for running e2e tests in Prow CI
# This script is called by the CI system and delegates to test-cypress.sh

set -euo pipefail

# Check if parallel flag is passed
PARALLEL_FLAG=""
while getopts P: flag; do
case "${flag}" in
P) PARALLEL_FLAG="-P ${OPTARG}";;
esac
done

echo "================================================"
echo "Running OpenShift Console E2E Tests"
echo "================================================"
echo "Environment: ${OPENSHIFT_CI:-local}"
echo "Parallel mode: ${PARALLEL_FLAG:-disabled (sequential)}"
echo "================================================"
echo ""

# Navigate to frontend directory
cd frontend || exit 1

# Run Cypress tests
# shellcheck disable=SC2086
./integration-tests/test-cypress.sh -h true ${PARALLEL_FLAG}

Comment on lines +27 to +30
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

git ls-files | grep -E "test-prow-e2e\.sh"

Repository: openshift/console

Length of output: 77

🏁 Script executed:

cat -n test-prow-e2e.sh | head -70

Repository: openshift/console

Length of output: 2372

🏁 Script executed:

# Find prow config files
find . -type f \( -name "*prow*" -o -name "*job*" -o -name "*ci*" \) -path "*/.git" -prune -o -type f \( -name "*.yaml" -o -name "*.yml" \) -print | grep -iE "(prow|job|ci)" | head -20

Repository: openshift/console

Length of output: 1187

🏁 Script executed:

cat -n .ci-operator.yaml | head -100

Repository: openshift/console

Length of output: 166

🏁 Script executed:

# Search for SCENARIO variable usage in the codebase
rg -n '\bSCENARIO\b' --type sh --type yaml --type yml

Repository: openshift/console

Length of output: 88

🏁 Script executed:

# Search for SCENARIO variable usage
rg -n '\bSCENARIO\b'

Repository: openshift/console

Length of output: 518

🏁 Script executed:

# Search for references to the specific scenario job names
rg -n "nightly-cypress|olmFull|dev-console.*pipelines|knative.*release" --type sh

Repository: openshift/console

Length of output: 186

🏁 Script executed:

# Find prow job definitions
find . -type d -name "*prow*" -o -name "*jobs*" -o -name "*ci*" 2>/dev/null | grep -v node_modules | grep -v vendor | head -20

Repository: openshift/console

Length of output: 1220

🏁 Script executed:

# Search for job definitions or CI configuration in root and common locations
find . -maxdepth 2 -type f \( -name "*.yaml" -o -name "*.yml" -o -name "*prow*" \) ! -path "./vendor/*" ! -path "./node_modules/*" ! -path "./.git/*"

Repository: openshift/console

Length of output: 354

🏁 Script executed:

# Check git history to understand this change
git log --oneline test-prow-e2e.sh | head -20

Repository: openshift/console

Length of output: 85

🏁 Script executed:

# Check the actual changes in the recent commit
git show 4d1972ac23 --stat

Repository: openshift/console

Length of output: 50374

🏁 Script executed:

# Look for how test-prow-e2e.sh is called/configured - check for prow config references
find . -maxdepth 2 -type f -name "*.yaml" ! -path "./vendor/*" -exec grep -l "test-prow-e2e\|SCENARIO" {} \;

Repository: openshift/console

Length of output: 43

🏁 Script executed:

# Search for any references to test-prow-e2e or test-cypress in configuration
rg -n "test-prow-e2e|test-cypress" --type yaml --type sh

Repository: openshift/console

Length of output: 6050

🏁 Script executed:

# Check README or documentation for CI job information
find . -maxdepth 2 -type f \( -name "README*" -o -name "CONTRIBUTING*" -o -name "TESTING*" \) ! -path "./vendor/*" -exec grep -l "SCENARIO\|prow\|job\|cypress" {} \;

Repository: openshift/console

Length of output: 85

🏁 Script executed:

grep -n "SCENARIO\|cypress\|prow\|job" TESTING.md | head -50

Repository: openshift/console

Length of output: 528

🏁 Script executed:

# Check if there's any git status showing this unresolved conflict
git status test-prow-e2e.sh

Repository: openshift/console

Length of output: 126

Resolve unresolved git merge conflict in test-prow-e2e.sh before merging

The file contains unresolved merge conflict markers (lines 31–60) with both upstream and stashed versions. The upstream version (line 29) runs an unconditional full headless suite, while the stashed version (lines 32–50) contains scenario-based conditional routing (SCENARIO variable checks for nightly-cypress, login, olmFull, dev-console, pipelines, knative, etc.).

This conflict must be explicitly resolved. Before choosing the upstream version, verify whether these scenario-based test jobs are still active in your Prow configuration. If they are, removing the conditional routing changes test coverage and duration. If they've been deprecated, the stashed code is dead and the upstream version is correct. The repository itself contains no SCENARIO references outside this conflict, so verification requires checking your external Prow job definitions.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@test-prow-e2e.sh` around lines 27 - 30, Resolve the unresolved git merge
conflict in test-prow-e2e.sh by removing the conflict markers and choosing
either the unconditional full-headless invocation
(./integration-tests/test-cypress.sh -h true ${PARALLEL_FLAG}) or the stashed
scenario-based routing that checks the SCENARIO variable (checks for values like
nightly-cypress, login, olmFull, dev-console, pipelines, knative, etc.); verify
which to keep by inspecting your external Prow job definitions for any SCENARIO
usage and if scenario jobs still exist keep and restore the conditional block
(preserving the SCENARIO checks and corresponding invocations), otherwise keep
the upstream unconditional call, then save the cleaned script and ensure it
executes without conflict markers.

<<<<<<< Updated upstream
if [ "$SCENARIO" == "nightly-cypress" ]; then
PACKAGE=""
if [ $# -gt 1 ]; then
Expand All @@ -39,3 +52,9 @@ fi
env NO_SANDBOX=true yarn test-puppeteer-csp

popd
=======
echo ""
echo "================================================"
echo "E2E Tests Completed"
echo "================================================"
>>>>>>> Stashed changes