Bug 2027929 - release-openshift-origin-installer-e2e-aws-upgrade-4.6-to-4.7-to-4.8-to-4.9-ci fails due to missing admin ack
Summary: release-openshift-origin-installer-e2e-aws-upgrade-4.6-to-4.7-to-4.8-to-4.9-c...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Test Framework
Version: 4.8
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 4.9.z
Assignee: W. Trevor King
QA Contact:
URL:
Whiteboard:
Depends On: 2026806
Blocks: 2033426
TreeView+ depends on / blocked
 
Reported: 2021-12-01 04:50 UTC by W. Trevor King
Modified: 2021-12-16 19:24 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2026806
Environment:
job=release-openshift-origin-installer-e2e-aws-upgrade-4.6-to-4.7-to-4.8-to-4.9-ci=all
Last Closed: 2021-12-10 06:51:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 26656 0 None open Bug 2027929: test/e2e/upgrade/adminack: Poll gates for duration of update 2021-12-01 04:50:23 UTC

Description W. Trevor King 2021-12-01 04:50:01 UTC
+++ This bug was initially created as a clone of Bug #2026806 +++

release-openshift-origin-installer-e2e-aws-upgrade-4.6-to-4.7-to-4.8-to-4.9-ci

is failing frequently in CI, see:
https://testgrid.k8s.io/redhat-openshift-ocp-release-4.9-informing#release-openshift-origin-installer-e2e-aws-upgrade-4.6-to-4.7-to-4.8-to-4.9-ci

Job URI: https://prow.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.6-to-4.7-to-4.8-to-4.9-ci/1463537431096594432

Log:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.6-to-4.7-to-4.8-to-4.9-ci/1463537431096594432/artifacts/e2e-aws-upgrade/clusterversion.json | jq -r '.items[].status.conditions[] | .lastTransitionTime + " " + .type + "=" + .status + " " + (.reason // "-") + ": " + (.message // "-")'
2021-11-24T16:06:34Z RetrievedUpdates=False NoChannel: The update channel has not been configured.
2021-11-24T16:30:08Z Available=True -: Done applying 4.8.0-0.nightly-2021-11-24-020113
2021-11-24T19:32:46Z Failing=False -: -
2021-11-24T19:03:45Z Progressing=True ClusterOperatorUpdating: Working towards 4.9.0-0.ci-2021-11-24-092816: 205 of 738 done (27% complete), waiting on openshift-apiserver
2021-11-24T19:04:01Z Upgradeable=False AdminAckRequired: Kubernetes 1.22 and therefore OpenShift 4.9 remove several APIs which require admin consideration. Please see the knowledge article https://access.redhat.com/articles/6329921 for details and instructions.

Comment 2 W. Trevor King 2021-12-10 06:51:15 UTC
Today's run [1]:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.6-to-4.7-to-4.8-to-4.9-ci/1468974836503023616/artifacts/e2e-aws-upgrade/clusterversion.json | jq -r '.items[].status.conditions[] | .lastTransitionTime + " " + .type + "=" + .status + " " + .reason + ": " + .message'
2021-12-09T16:12:46Z RetrievedUpdates=False NoChannel: The update channel has not been configured.
2021-12-09T16:42:08Z Available=True : Done applying 4.8.0-0.nightly-2021-12-08-203416
2021-12-09T19:39:11Z Failing=False : 
2021-12-09T19:10:06Z Progressing=True ClusterOperatorUpdating: Working towards 4.9.0-0.ci-2021-12-09-065806: 205 of 738 done (27% complete), waiting on openshift-apiserver

so still stuck on opernshift-apiserver, but no longer setting Upgradeable=False AdminAckRequired.  And confirming the logs for the ack test-case:

$ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.6-to-4.7-to-4.8-to-4.9-ci/1468974836503023616/build-log.txt | grep -i 'Starting upgrade to\|admin.\?ack'
STEP: Building a namespace api object, basename check-for-admin-acks
STEP: Setting up admin ack test
Dec  9 16:42:50.653: INFO: Admin ack test setup complete
Dec  9 16:42:50.744: INFO: Skipping admin ack test. Admin ack is not in this baseline or contains no gates.
Dec  9 16:43:17.801: INFO: Starting upgrade to version= image=quay.io/openshift-release-dev/ocp-release:4.7.39-x86_64 attempt=58c4e669-7a3e-440f-a9be-316e9b0b83d2
Dec  9 16:52:50.830: INFO: Skipping admin ack test. Admin ack is not in this baseline or contains no gates.
Dec  9 17:02:50.832: INFO: Skipping admin ack test. Admin ack is not in this baseline or contains no gates.
Dec  9 17:12:50.831: INFO: Skipping admin ack test. Admin ack is not in this baseline or contains no gates.
Dec  9 17:22:50.832: INFO: Skipping admin ack test. Admin ack is not in this baseline or contains no gates.
Dec  9 17:32:50.833: INFO: Skipping admin ack test. Admin ack is not in this baseline or contains no gates.
Dec  9 17:42:50.829: INFO: Skipping admin ack test. Admin ack is not in this baseline or contains no gates.
Dec  9 17:52:50.834: INFO: Skipping admin ack test. Admin ack is not in this baseline or contains no gates.
Dec  9 17:55:49.412: INFO: Starting upgrade to version= image=registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-12-08-203416 attempt=bc6e46fd-f803-460a-aab6-884da931a4e6
Dec  9 18:02:51.006: INFO: Admin Ack verified
Dec  9 18:12:51.001: INFO: Admin Ack verified
Dec  9 18:22:51.001: INFO: Admin Ack verified
Dec  9 18:32:51.003: INFO: Admin Ack verified
Dec  9 18:42:51.001: INFO: Admin Ack verified
Dec  9 18:52:51.012: INFO: Admin Ack verified
Dec  9 19:02:51.011: INFO: Admin Ack verified
Dec  9 19:10:00.955: INFO: Starting upgrade to version= image=registry.ci.openshift.org/ocp/release:4.9.0-0.ci-2021-12-09-065806 attempt=2d00fb5e-69c2-4ae6-827e-140d6db90d30
Dec  9 19:12:51.005: INFO: Waiting for Upgradeable to be AdminAckRequired for "" ...
Dec  9 19:12:51.279: INFO: Waiting for Upgradeable to not be AdminAckRequired for "" ...
Dec  9 19:13:01.454: INFO: Admin Ack verified
Dec  9 19:22:51.009: INFO: Admin Ack verified
Dec  9 19:32:51.014: INFO: Admin Ack verified
Dec  9 19:42:51.004: INFO: Admin Ack verified
github.com/openshift/origin/test/extended/util.(*AdminAckTest).test(0xc00200de50, 0x8fc9d08, 0xc0029b1580, 0xc00200dd40)
...

And also:

$ curl -s https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.6-to-4.7-to-4.8-to-4.9-ci/1468974836503023616/build-log.txt | grep FAIL
Dec  9 19:53:20.747: FAIL: Error accessing configmap openshift-config-managed/admin-gates: Get "https://api.ci-op-w5q90zpi-9278e.origin-ci-int-aws.dev.rhcloud.com:6443/api/v1/namespaces/openshift-config-managed/configmaps/admin-gates": dial tcp 100.21.251.165:6443: i/o timeout

which is another hiccup like the one mentioned in [2].  Anyhow, the successful polling is good enough for me to mark this CURRENTRELEASE, and I've opened bug 2030972 to track that.

[1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.6-to-4.7-to-4.8-to-4.9-ci/1468974836503023616
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=2026806#c8


Note You need to log in before you can comment on or make changes to this bug.