Bug 1824200 - release-openshift-origin-installer-e2e-gcp-4.5: [sig-operator] an end user use OLM can subscribe to the etcd operator [Suite:openshift/conformance/parallel]
Summary: release-openshift-origin-installer-e2e-gcp-4.5: [sig-operator] an end user us...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.5.0
Assignee: Daniel Sover
QA Contact: kuiwang
URL:
Whiteboard:
: 1824199 (view as bug list)
Depends On:
Blocks: 1847066
TreeView+ depends on / blocked
 
Reported: 2020-04-15 14:15 UTC by Stefan Schimanski
Modified: 2020-07-13 17:28 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1847066 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:27:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 24946 0 None closed Bug 1824200: use cockroachdb in OLM e2e test 2021-01-13 06:20:42 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:28:20 UTC

Description Stefan Schimanski 2020-04-15 14:15:57 UTC
In https://testgrid.k8s.io/redhat-openshift-ocp-release-4.5-blocking#release-openshift-origin-installer-e2e-gcp-4.5&sort-by-failures= the upper OLM test flakes a lot, timing out:

fail [github.com/openshift/origin/test/extended/operators/olm.go:208]: Unexpected error:
    <*errors.errorString | 0xc0001d4970>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
occurred

Frequency: 50 times in 2d https://search.svc.ci.openshift.org/?search=github.com/openshift/origin/test/extended/operators/olm.go:208

Comment 3 Stephen Cuppett 2020-04-15 17:41:46 UTC
*** Bug 1824199 has been marked as a duplicate of this bug. ***

Comment 5 Stephen Cuppett 2020-04-22 18:48:59 UTC
Setting target release to current development version (4.5) for investigation. Where fixes (if any) are required/requested for prior versions, cloned BZs will be created when appropriate.

Comment 6 Periklis Tsirakidis 2020-04-29 12:05:08 UTC
Job failure observed on e2e-aws-vpc-4.4: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-shared-vpc-4.4/96

Comment 11 kuiwang 2020-05-11 05:41:45 UTC
Run the cases on 4.5 cluster. LGTM. Verified with PASS


Here is the detailed:

kuiwang@Kuis-MacBook-Pro origin % git log
commit 7c3ca66a9dfce672a21172425856598e2d1a9916 (HEAD -> master, origin/master)
Merge: 1ed88a07e7 deabbe06ae
Author: OpenShift Merge Robot <openshift-merge-robot.github.com>
Date:   Sat May 9 07:00:48 2020 +0200

    Merge pull request #24912 from gabemontero/reenable-template-test
    
    Bug 1731222: unskip template extended tests

commit 1ed88a07e76c1d4433dd79de630fb0c065ad38be
Merge: 4f9cdd5394 2b5eb08401
Author: OpenShift Merge Robot <openshift-merge-robot.github.com>
Date:   Sat May 9 01:32:40 2020 +0200

    Merge pull request #24952 from jsafrane/gluster-ignore-notfound
    
    Bug 1829331: UPSTREAM: 90715: Ignore NotFound error when cleaning up Gluster tests

commit 4f9cdd5394af68c56092d1b531d5a2dd17f4df40
Merge: dfd05bf95c 5ae335dd55
Author: OpenShift Merge Robot <openshift-merge-robot.github.com>
Date:   Fri May 8 20:10:05 2020 +0200

    Merge pull request #24946 from exdx/fix/bug-1824200
    
    Bug 1824200: use cockroachdb in OLM e2e test

...

kuiwang@Kuis-MacBook-Pro origin % oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-05-08-200452   True        False         5h59m   Cluster version is 4.5.0-0.nightly-2020-05-08-200452


I0511 13:38:20.019850   74557 test_context.go:423] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready
I0511 13:38:23.311056   74559 test_context.go:423] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready
started: (0/1/2) "[sig-operator] an end user use OLM can subscribe to the cockroachdb operator [Suite:openshift/conformance/parallel]"

started: (0/2/2) "[sig-operator] an end user use OLM Report Upgradeable in OLM ClusterOperators status [Suite:openshift/conformance/parallel]"

I0511 13:38:44.759686   74559 trace.go:116] Trace[1076855703]: "Reflector ListAndWatch" name:github.com/openshift/origin/pkg/monitor/pod.go:282 (started: 2020-05-11 13:38:23.340581 +0800 CST m=+3.431634841) (total time: 21.41750135s):
Trace[1076855703]: [21.417121276s] [21.417121276s] Objects listed
passed: (43.3s) 2020-05-11T05:39:06 "[sig-operator] an end user use OLM Report Upgradeable in OLM ClusterOperators status [Suite:openshift/conformance/parallel]"

passed: (1m10s) 2020-05-11T05:39:33 "[sig-operator] an end user use OLM can subscribe to the cockroachdb operator [Suite:openshift/conformance/parallel]"


Timeline:

May 11 05:39:11.574 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/RequirementsUnknown requirements not yet checked
May 11 05:39:11.913 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/RequirementsUnknown requirements not yet checked (2 times)
May 11 05:39:11.914 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/RequirementsNotMet one or more requirements couldn't be found
May 11 05:39:13.457 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/AllRequirementsMet all requirements found, attempting install
May 11 05:39:13.494 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/InstallSucceeded waiting for install components to report healthy
May 11 05:39:13.509 I ns/e2e-test-olm-23440-f59tp deployment/cockroachdb reason/ScalingReplicaSet Scaled up replica set cockroachdb-b8fcc6d78 to 1
May 11 05:39:14.063 I ns/e2e-test-olm-23440-f59tp replicaset/cockroachdb-b8fcc6d78 reason/SuccessfulCreate Created pod: cockroachdb-b8fcc6d78-vvc2x
May 11 05:39:14.071 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ reason/Created
May 11 05:39:14.341 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal reason/Scheduled
May 11 05:39:14.341 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/InstallSucceeded waiting for install components to report healthy (2 times)
May 11 05:39:15.736 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/InstallWaiting installing: waiting for deployment cockroachdb to become ready: Waiting for deployment spec update to be observed...\n
May 11 05:39:16.388 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/InstallWaiting installing: waiting for deployment cockroachdb to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available...\n
May 11 05:39:16.389 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/InstallWaiting installing: waiting for deployment cockroachdb to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available...\n (2 times)
May 11 05:39:16.389 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x reason/AddedInterface Add eth0 [10.128.2.66/23]
May 11 05:39:17.432 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/Pulling image/quay.io/helmoperators/cockroachdb:v2.1.11
May 11 05:39:17.557 I ns/openshift-config-operator deployment/openshift-config-operator reason/KubeCloudConfigController openshift-config-managed/kube-cloud-config ConfigMap was deleted as no longer required (563 times)
May 11 05:39:18.659 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/Pulled image/quay.io/helmoperators/cockroachdb:v2.1.11
May 11 05:39:19.278 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/Created
May 11 05:39:19.278 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/Started
May 11 05:39:19.560 I ns/e2e-test-olm-23440-f59tp clusterserviceversion/cockroachdb.v2.1.11 reason/InstallSucceeded install strategy completed with no errors
May 11 05:39:19.561 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/Ready
May 11 05:39:27.479 I ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/Killing
May 11 05:39:27.480 W ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal reason/GracefulDelete in 30s
May 11 05:39:28.886 W ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal invariant violation (bug): pod should not transition Running->Pending even when terminated
May 11 05:39:28.886 W ns/e2e-test-olm-23440-f59tp pod/cockroachdb-b8fcc6d78-vvc2x node/ip-10-0-154-33.us-east-2.compute.internal container/cockroachdb reason/NotReady

Writing JUnit report to junit_e2e_20200511-053933.xml

2 pass, 0 skip (1m10s)

Comment 13 kuiwang 2020-05-29 00:23:59 UTC

we find the operator is not installed successfully in 150 seconds (expect Succeeded, but it is Installing after 150s) based on the log blob:null/7e43a637-9668-4138-ad98-c43669c6bbc8.
do not know why it is slow to install operator sometimes.
--
STEP: Cluster-admin user subscribe the operator resource
May 28 21:01:34.916: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig process --ignore-unknown-parameters=true -f /tmp/fixture-testdata-dir169804195/test/extended/testdata/olm/operatorgroup.yaml -p NAME=test-operator NAMESPACE=e2e-test-olm-23440-fz2sk SOURCENAME=community-operators SOURCENAMESPACE=openshift-marketplace'
May 28 21:01:35.273: INFO: Running 'oc --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig create -f /tmp/e2e-test-olm-23440-fz2sk-config.json'
operatorgroup.operators.coreos.com/test-operator created
May 28 21:01:45.639: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk operatorgroup test-operator -o=jsonpath={.status.namespaces}'
May 28 21:01:45.992: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig process --ignore-unknown-parameters=true -f /tmp/fixture-testdata-dir169804195/test/extended/testdata/olm/cockroachdb-subscription.yaml -p NAME=test-operator NAMESPACE=e2e-test-olm-23440-fz2sk SOURCENAME=community-operators SOURCENAMESPACE=openshift-marketplace'
May 28 21:01:46.339: INFO: Running 'oc --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig create -f /tmp/e2e-test-olm-23440-fz2sk-config.json'
subscription.operators.coreos.com/test-operator created
May 28 21:01:56.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:01:57.397: INFO: the output is Installing
May 28 21:02:06.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:02:07.195: INFO: the output is Installing
May 28 21:02:16.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:02:17.191: INFO: the output is Installing
May 28 21:02:26.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:02:27.192: INFO: the output is Installing
May 28 21:02:36.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:02:37.207: INFO: the output is Installing
May 28 21:02:46.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:02:47.189: INFO: the output is Installing
May 28 21:02:56.767: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:02:57.221: INFO: the output is Installing
May 28 21:03:06.767: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:03:07.310: INFO: the output is Installing
May 28 21:03:16.769: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:03:17.376: INFO: the output is Installing
May 28 21:03:26.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:03:27.179: INFO: the output is Installing
May 28 21:03:36.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:03:37.188: INFO: the output is Installing
May 28 21:03:46.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:03:47.194: INFO: the output is Installing
May 28 21:03:56.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:03:57.192: INFO: the output is Installing
May 28 21:04:06.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:04:07.181: INFO: the output is Installing
May 28 21:04:16.766: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:04:17.178: INFO: the output is Installing
May 28 21:04:17.178: INFO: Running 'oc --namespace=e2e-test-olm-23440-fz2sk --kubeconfig=/var/run/secrets/ci.openshift.io/multi-stage/kubeconfig get -n e2e-test-olm-23440-fz2sk csv cockroachdb.v2.1.11 -o=jsonpath={.status.phase}'
May 28 21:04:17.600: INFO: the output is Installing
--

Comment 14 Jan Chaloupka 2020-06-15 14:45:12 UTC
The same test failes in 4.4: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.4-blocking#release-openshift-origin-installer-e2e-gcp-4.4&sort-by-flakiness=

Daniel, is there a plan to backport the same fix to 4.4?

Comment 17 errata-xmlrpc 2020-07-13 17:27:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.