In https://testgrid.k8s.io/redhat-openshift-ocp-release-4.5-blocking#release-openshift-origin-installer-e2e-gcp-4.5&sort-by-failures= the upper OLM test flakes a lot, timing out: fail [github.com/openshift/origin/test/extended/operators/olm.go:208]: Unexpected error: <*errors.errorString | 0xc0001d4970>: { s: "timed out waiting for the condition", } timed out waiting for the condition occurred Frequency: 50 times in 2d https://search.svc.ci.openshift.org/?search=github.com/openshift/origin/test/extended/operators/olm.go:208
*** Bug 1824199 has been marked as a duplicate of this bug. ***
Setting target release to current development version (4.5) for investigation. Where fixes (if any) are required/requested for prior versions, cloned BZs will be created when appropriate.
Job failure observed on e2e-aws-vpc-4.4: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-shared-vpc-4.4/96
Run the cases on 4.5 cluster. LGTM. Verified with PASS Here is the detailed: kuiwang@Kuis-MacBook-Pro origin % git log commit 7c3ca66a9dfce672a21172425856598e2d1a9916 (HEAD -> master, origin/master) Merge: 1ed88a07e7 deabbe06ae Author: OpenShift Merge Robot <openshift-merge-robot.github.com> Date: Sat May 9 07:00:48 2020 +0200 Merge pull request #24912 from gabemontero/reenable-template-test Bug 1731222: unskip template extended tests commit 1ed88a07e76c1d4433dd79de630fb0c065ad38be Merge: 4f9cdd5394 2b5eb08401 Author: OpenShift Merge Robot <openshift-merge-robot.github.com> Date: Sat May 9 01:32:40 2020 +0200 Merge pull request #24952 from jsafrane/gluster-ignore-notfound Bug 1829331: UPSTREAM: 90715: Ignore NotFound error when cleaning up Gluster tests commit 4f9cdd5394af68c56092d1b531d5a2dd17f4df40 Merge: dfd05bf95c 5ae335dd55 Author: OpenShift Merge Robot <openshift-merge-robot.github.com> Date: Fri May 8 20:10:05 2020 +0200 Merge pull request #24946 from exdx/fix/bug-1824200 Bug 1824200: use cockroachdb in OLM e2e test ... kuiwang@Kuis-MacBook-Pro origin % oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-0.nightly-2020-05-08-200452 True False 5h59m Cluster version is 4.5.0-0.nightly-2020-05-08-200452 I0511 13:38:20.019850 74557 test_context.go:423] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready I0511 13:38:23.311056 74559 test_context.go:423] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready started: (0/1/2) "[sig-operator] an end user use OLM can subscribe to the cockroachdb operator [Suite:openshift/conformance/parallel]" started: (0/2/2) "[sig-operator] an end user use OLM Report Upgradeable in OLM ClusterOperators status [Suite:openshift/conformance/parallel]" I0511 13:38:44.759686 74559 trace.go:116] Trace[1076855703]: "Reflector ListAndWatch" name:github.com/openshift/origin/pkg/monitor/pod.go:282 (started: 2020-05-11 13:38:23.340581 +0800 CST m=+3.431634841) (total time: 21.41750135s): Trace[1076855703]: [21.417121276s] [21.417121276s] Objects listed passed: (43.3s) 2020-05-11T05:39:06 "[sig-operator] an end user use OLM Report Upgradeable in OLM ClusterOperators status [Suite:openshift/conformance/parallel]" passed: (1m10s) 2020-05-11T05:39:33 "[sig-operator] an end user use OLM can subscribe to the cockroachdb operator [Suite:openshift/conformance/parallel]" Timeline: May 11 05:39:11.574 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/RequirementsUnknown requirements not yet checked May 11 05:39:11.913 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/RequirementsUnknown requirements not yet checked (2 times) May 11 05:39:11.914 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/RequirementsNotMet one or more requirements couldn't be found May 11 05:39:13.457 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/AllRequirementsMet all requirements found, attempting install May 11 05:39:13.494 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/InstallSucceeded waiting for install components to report healthy May 11 05:39:13.509 I ns/e2e-test-olm-23440-f59tp deployment/cockroachdb reason/ScalingReplicaSet Scaled up replica set cockroachdb-b8fcc6d78 to 1 May 11 05:39:14.063 I ns/e2e-test-olm-23440-f59tp replicaset/cockroachdb-b8fcc6d78 reason/SuccessfulCreate Created pod: cockroachdb-b8fcc6d78-vvc2x May 11 05:39:14.071 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ reason/Created May 11 05:39:14.341 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal reason/Scheduled May 11 05:39:14.341 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/InstallSucceeded waiting for install components to report healthy (2 times) May 11 05:39:15.736 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/InstallWaiting installing: waiting for deployment cockroachdb to become ready: Waiting for deployment spec update to be observed...\n May 11 05:39:16.388 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/InstallWaiting installing: waiting for deployment cockroachdb to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available...\n May 11 05:39:16.389 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/InstallWaiting installing: waiting for deployment cockroachdb to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available...\n (2 times) May 11 05:39:16.389 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x reason/AddedInterface Add eth0 [10.128.2.66/23] May 11 05:39:17.432 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/Pulling image/quay.io/helmoperators/cockroachdb:v2.1.11 May 11 05:39:17.557 I ns/openshift-config-operator deployment/openshift-config-operator reason/KubeCloudConfigController openshift-config-managed/kube-cloud-config ConfigMap was deleted as no longer required (563 times) May 11 05:39:18.659 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/Pulled image/quay.io/helmoperators/cockroachdb:v2.1.11 May 11 05:39:19.278 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/Created May 11 05:39:19.278 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/Started May 11 05:39:19.560 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/InstallSucceeded install strategy completed with no errors May 11 05:39:19.561 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/Ready May 11 05:39:27.479 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/Killing May 11 05:39:27.480 W ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal reason/GracefulDelete in 30s May 11 05:39:28.886 W ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal invariant violation (bug): pod should not transition Running->Pending even when terminated May 11 05:39:28.886 W ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/NotReady Writing JUnit report to junit_e2e_20200511-053933.xml 2 pass, 0 skip (1m10s)
This is failing 67% of MCO PRs recently: https://search.apps.build01.ci.devcluster.openshift.com/?search=an+end+user+use+OLM+can+subscribe+to+the+cockroachdb+operator+&maxAge=48h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job
we find the operator is not installed successfully in 150 seconds (expect Succeeded, but it is Installing after 150s) based on the log blob:null/7e43a637-9668-4138-ad98-c43669c6bbc8. do not know why it is slow to install operator sometimes. -- STEP: Cluster-admin user subscribe the operator resource May 28 21:01:34.916: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig process --ignore-unknown-parameters=true -f /tmp/fixture-testdata-dir169804195/test/extended/testdata/olm/operatorgroup.yaml -p NAME=test-operator NAMESPACE=e2e-test-olm-23440-fz2sk SOURCENAME=community-operators SOURCENAMESPACE=openshift-marketplace' May 28 21:01:35.273: INFO: Running 'oc --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig create -f /tmp/e2e-test-olm-23440-fz2sk-config.json' operatorgroup.operators.coreos.com/test-operator created May 28 21:01:45.639: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk operatorgroup test-operator -o=jsonpath={.status.namespaces}' May 28 21:01:45.992: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig process --ignore-unknown-parameters=true -f /tmp/fixture-testdata-dir169804195/test/extended/testdata/olm/cockroachdb-subscription.yaml -p NAME=test-operator NAMESPACE=e2e-test-olm-23440-fz2sk SOURCENAME=community-operators SOURCENAMESPACE=openshift-marketplace' May 28 21:01:46.339: INFO: Running 'oc --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig create -f /tmp/e2e-test-olm-23440-fz2sk-config.json' subscription.operators.coreos.com/test-operator created May 28 21:01:56.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:01:57.397: INFO: the output is Installing May 28 21:02:06.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:02:07.195: INFO: the output is Installing May 28 21:02:16.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:02:17.191: INFO: the output is Installing May 28 21:02:26.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:02:27.192: INFO: the output is Installing May 28 21:02:36.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:02:37.207: INFO: the output is Installing May 28 21:02:46.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:02:47.189: INFO: the output is Installing May 28 21:02:56.767: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:02:57.221: INFO: the output is Installing May 28 21:03:06.767: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:03:07.310: INFO: the output is Installing May 28 21:03:16.769: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:03:17.376: INFO: the output is Installing May 28 21:03:26.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:03:27.179: INFO: the output is Installing May 28 21:03:36.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:03:37.188: INFO: the output is Installing May 28 21:03:46.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:03:47.194: INFO: the output is Installing May 28 21:03:56.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:03:57.192: INFO: the output is Installing May 28 21:04:06.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:04:07.181: INFO: the output is Installing May 28 21:04:16.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:04:17.178: INFO: the output is Installing May 28 21:04:17.178: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}' May 28 21:04:17.600: INFO: the output is Installing --
The same test failes in 4.4: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.4-blocking#release-openshift-origin-installer-e2e-gcp-4.4&sort-by-flakiness= Daniel, is there a plan to backport the same fix to 4.4?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409