Skip to content

K8SPG-943: check operator panic in e2e test #1441

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
egegunes merged 10 commits into from
Mar 4, 2026
Merged

K8SPG-943: check operator panic in e2e test #1441

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
c4cba6b
e2e tests should check for operator panics
mayankshah1607 Feb 13, 2026
6fa21f8
Merge branch 'main' into K8SPG-943-e2e-test
mayankshah1607 Feb 15, 2026
1d16eb1
Merge branch 'main' into K8SPG-943-e2e-test
mayankshah1607 Feb 16, 2026
1418e39
Merge branch 'main' into K8SPG-943-e2e-test
mayankshah1607 Feb 16, 2026
da35bb5
Merge branch 'main' into K8SPG-943-e2e-test
mayankshah1607 Mar 2, 2026
a2afa87
Merge branch 'main' into K8SPG-943-e2e-test
mayankshah1607 Mar 3, 2026
185e187
print next 100 log lines when panic detected
mayankshah1607 Mar 3, 2026
5f64d22
add check left out files
mayankshah1607 Mar 3, 2026
e5e159c
fix command pipeline
mayankshah1607 Mar 3, 2026
ecd71ad
Merge branch 'main' into K8SPG-943-e2e-test
hors Mar 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions e2e-tests/functions
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,14 @@ create_namespace() {
fi
}

check_operator_panic() {
local operator_pod=$(get_operator_pod)
if kubectl logs -n "${OPERATOR_NS:-$NAMESPACE}" "$operator_pod" -c operator | grep -q "Observed a panic"; then
Comment thread
mayankshah1607 marked this conversation as resolved.
Outdated
echo "Detected panic in operator"
Copy link
Copy Markdown
Contributor

@egegunes egegunes Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

detected panic but what is the panic? grep -q will suppress any output. we need to print the panic and the stacktrace if possible

exit 1
fi
}
Comment on lines +40 to +49
Copy link

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function doesn't verify that the operator pod exists before checking for panics. If get_operator_pod returns an empty string (no operator pod found), the kubectl logs command will fail, but the error will be masked by the pipeline to grep, causing the function to silently succeed without checking for panics.

Add validation that operator_pod is non-empty before attempting to retrieve logs. Consider also handling the case where kubectl logs fails due to the pod not existing or not being ready yet.

Copilot uses AI. Check for mistakes.
Comment on lines +40 to +49
Copy link

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Problem: Several test cleanup files in this repository were not updated with check_operator_panic while all other similar test directories were. The following cleanup files are missing the new check_operator_panic call that was added to 29 other test suites:
  • e2e-tests/tests/backup-enable-disable/99-remove-cluster-gracefully.yaml
  • e2e-tests/tests/cert-manager-tls/99-remove-cluster-gracefully.yaml
  • e2e-tests/tests/demand-backup-offline-snapshot/99-cleanup.yaml
  • e2e-tests/tests/dynamic-configuration/99-remove-cluster-gracefully.yaml
  • e2e-tests/tests/pg-tde/99-remove-cluster-gracefully.yaml
  • e2e-tests/tests/backup-enable-disable/98-remove-datasource-cluster-gracefully.yaml

  1. Why it matters: These test suites will not detect operator panics, creating inconsistent coverage.

  2. Fix: Add check_operator_panic before destroy_operator in each of these cleanup files, matching the pattern used in all other test suites in this PR.

Copilot uses AI. Check for mistakes.

deploy_operator() {
local cw_prefix=""

Expand Down
1 change: 1 addition & 0 deletions e2e-tests/tests/builtin-extensions/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/custom-envs/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/custom-extensions/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/custom-tls/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
destroy_cert_manager
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/database-init-sql/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/demand-backup/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/finalizers/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/huge-pages/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/init-deploy/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/major-upgrade/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/migration-backup-s3/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/migration-data-volumes/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/monitoring-pmm3/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/monitoring/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/one-pod/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/operator-self-healing/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/pitr/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/scaling/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/scheduled-backup/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/self-healing/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/sidecars/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60

1 change: 1 addition & 0 deletions e2e-tests/tests/standby-pgbackrest/99-cleanup.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/standby-streaming/99-cleanup.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,5 +24,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/start-from-backup/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/tablespaces/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/telemetry-transfer/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/upgrade-consistency/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/upgrade-minor/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
1 change: 1 addition & 0 deletions e2e-tests/tests/users/99-remove-cluster-gracefully.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@ commands:
source ../../functions

remove_all_finalizers
check_operator_panic
destroy_operator
timeout: 60
Loading