Bug 2230740

Summary:	KedaController failed to start - Back-off restarting failed container keda-admission and keda-operator
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Tiffany Nguyen <tunguyen>
Component:	Multi-Cloud Object Gateway	Assignee:	Naveen Paul <napaul>
Status:	ASSIGNED ---	QA Contact:	krishnaram Karthick <kramdoss>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.14	CC:	dzaken, odf-bz-bot
Target Milestone:	---	Keywords:	TestBlocker
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Tiffany Nguyen 2023-08-09 23:45:35 UTC

Description of problem (please be detailed as possible and provide log
snippests):

Keda controller is failling to create with back-off error

$ oc get pod -n openshift-keda
NAME                                                 READY   STATUS              RESTARTS         AGE
custom-metrics-autoscaler-operator-85d445c7d-qgktn   1/1     Running             13 (6h27m ago)   2d3h
keda-admission-5fcb586746-lm27g                      0/1     CrashLoopBackOff    5 (71s ago)      4m54s
keda-metrics-apiserver-598f6f6d4f-sq8j8              0/1     ContainerCreating   0                4m55s
keda-operator-6d84fcf49b-9vmtp                       0/1     CrashLoopBackOff    5 (101s ago)     4m55s

---------
<snipped>

Normal   Created         4m47s (x4 over 5m43s)  kubelet            Created container keda-operator
  Normal   Started         4m47s (x4 over 5m43s)  kubelet            Started container keda-operator
  Normal   Pulled          4m47s                  kubelet            Successfully pulled image "registry.redhat.io/custom-metrics-autoscaler/custom-metrics-autoscaler-rhel8@sha256:deb2fca967617fc31e95a4301f3aa7bb6f38a0a3cc180c0df35580b73665ad55" in 3.588043876s (3.588057519s including waiting)
  Warning  BackOff         39s (x27 over 5m39s)   kubelet            Back-off restarting failed container keda-operator in pod keda-operator-6d84fcf49b-9vmtp_openshift-keda(e9fd351f-e74c-44ef-87bb-7bf107380a63)

<snipped>
Normal   Created         5m41s (x4 over 6m52s)  kubelet            Created container keda-admission-webhooks
  Normal   Started         5m41s (x4 over 6m52s)  kubelet            Started container keda-admission-webhooks
  Warning  BackOff         2m3s (x25 over 6m48s)  kubelet            Back-off restarting failed container keda-admission-webhooks in pod keda-admission-5fcb586746-lm27g_openshift-keda(55dd9c69-97ff-44d6-b414-2c2baa1bf0bb)



Version of all relevant components (if applicable):
$ oc get clusterversion 
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.0-0.nightly-2023-07-31-181848   True        False         2d6h    Cluster version is 4.14.0-0.nightly-2023-07-31-181848

Build # "4.14.0-101"



Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?

None

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
None

Steps to Reproduce:
1. Install Custom Matrics Autoscaler from OperatorHub
2. Create KedaController
3. Check Keda running pods


Actual results:
KedaController failed to create

Expected results:
KedaController should create and run successfully.

Additional info:

Comment 2 Tiffany Nguyen 2023-08-10 04:06:11 UTC

Must gather logs -  http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/bz-2230740/

Comment 4 Naveen Paul 2023-08-14 11:27:48 UTC

@Tiffany Nguyen, Could you please install Keda before installing ODF and verify the issue? I wanted to make sure it's an issue with Noobaa or not, If keda issue still exists before installing ODF we can confirm that.
Because I am not sure it's a Noobaa issue. now in openshift we haven't enabled Keda based autoscaling.

Comment 5 Tiffany Nguyen 2023-08-14 20:06:22 UTC

I installed Keda on OCP cluster and it went well without any issue.  All keda pods are up and running.
$ oc get clusterversion 
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.14.0-0.nightly-2023-08-11-055332   True        False         84m     Cluster version is 4.14.0-0.nightly-2023-08-11-055332

$ oc get pod -n openshift-keda
NAME                                                  READY   STATUS    RESTARTS        AGE
custom-metrics-autoscaler-operator-57b66bb99b-jrgn7   1/1     Running   2 (19m ago)     65m
keda-admission-5fcb586746-6n662                       1/1     Running   0               5m14s
keda-metrics-apiserver-598f6f6d4f-84fzf               1/1     Running   0               5m15s
keda-operator-6d84fcf49b-vkrtq                        1/1     Running   1 (4m29s ago)   5m16s