Bug 1981532 - Lots of events are firing in a loop in the openshift-marketplace namespace
Summary: Lots of events are firing in a loop in the openshift-marketplace namespace
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.8
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Per da Silva
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-07-12 18:30 UTC by Udi Kalifon
Modified: 2022-06-22 15:13 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-06-22 15:13:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Udi Kalifon 2021-07-12 18:30:59 UTC
Description of problem:
In the openshift-marketplace namespace there are events that fire all the time, without reconciling. It looks really bad especially if you log in to the UI, where you see them scrolling all the time in the Overview page, which looks really worrying. On the command line you can see them with:

oc get events -n openshift-marketplace

You can see that the registry-server is being started and stopped all the time, and you can see a lot of FailedScheduling events:

166m        Warning   FailedScheduling    pod/redhat-operators-49mbg                                            0/2 nodes are available: 2 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
126m        Normal    Started             pod/redhat-operators-mk46f                                            Started container registry-server
126m        Normal    Killing             pod/redhat-operators-mk46f                                            Stopping container registry-server


Version-Release number of selected component (if applicable):
4.8.0-rc.3


How reproducible:
100%


Steps to Reproduce:
1. Install a fresh cluster. I installed with the assisted installer, which also installs OLM, and I included the OCS operator. I installed 3 masters and 3 workers.
2. Log in to the UI to see the events, or check them with "oc get events -n openshift-marketplace"


Actual results:
Lots of events are constantly firing


Expected results:
There shouldn't be failures and restarts like that

Comment 7 Udi Kalifon 2021-08-04 08:27:41 UTC
The events that I saw can be found in the must-gather under quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-68bb88201fbf54c8c9709f224d2daaada8e7420f339e5c54d0628914c4401ebb/namespaces/openshift-marketplace/core/evets.yaml. There are a lot of events there firing in a loop.

When you say that the must gather "resembles a 4.6 cluster" - how can you see if it's a 4.6 or 4.8 cluster? All the clusters that I test on are 4.8, so this can't be a 4.6 cluster.

Comment 14 Jakub Hadvig 2022-06-13 11:59:40 UTC
The console is only rendering these alerts, there is nothing we can do with their firing.
Also since the severity is `medium` and the fact that 4.7 and 4.8 are in maintenance support Im putting it back to OLM.

Comment 15 tflannag 2022-06-22 15:13:12 UTC
Unfortunately, OLM isn't directly firing these events, but rather it's a side effect of the catalog polling implementation where kubelet is generating these events. Because these are normal-level events, and OLM cannot change this polling implementation given the current constraints, I'm going to close this out as NOTABUG.


Note You need to log in before you can comment on or make changes to this bug.