test: Managed cluster should have no crashlooping pods in core namespaces over four minutes
found that pods in the openshift-marketplace namespace sometimes are crashlooping.
*** This bug has been marked as a duplicate of bug 1949337 ***
Closing 1949337 as the duplicate and keeping re-opening this for tracking.
*** Bug 1949337 has been marked as a duplicate of this bug. ***
Looks like this could have been caused due to bad default CatalogSource images built in the pipeline. It appears to be transient however, and the images fixed since the last case of crashlooping pods. Running the marketplace-operator from latest master in a 4.8.0-0.ci-2021-04-20-092252 cluster did not show any crahlooping pods in the openshift-marketplace namespace:
$ oc get pods
NAME READY STATUS RESTARTS AGE
certified-operators-ttbgw 1/1 Running 0 121m
community-operators-htdwg 1/1 Running 0 121m
marketplace-operator-7fd8f5d9fb-tbzld 1/1 Running 0 17m
redhat-marketplace-6trz7 1/1 Running 0 121m
redhat-operators-p6n9q 1/1 Running 0 105m
The problem is still there, please don't close this BZ until the CI failure rate decreases:
An example of a failed job:
fail [github.com/openshift/origin/test/extended/operators/cluster.go:160]: Expected
<string | len:1, cap:1>: [
"Pod openshift-marketplace/community-operators-28lkj is not healthy: container registry-server exited with non-zero exit code",
to be empty
Either these pods should be fixed and shouldn't have non-zero exit codes, or an exception should be added for them . As this pod is not present after e2e tests have finished, most likely this pod was created (and deleted) by a test.
Moving the priority of this bz to urgent given that this is blocking CI
Since the pods are getting deleted after the tests, the logs from the tests have no indication regarding why the pods were getting killed in a loop apart from the log that states that the registry container exit with exit code 2. Opened https://bugzilla.redhat.com/show_bug.cgi?id=1952238 to report back logs from catalog pods to catalog operator on termination.
Once the fix for https://bugzilla.redhat.com/show_bug.cgi?id=1952238 goes in, we'll investigate the logs again.
*** Bug 1951617 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.