Bug 1994652

Summary:	Getting clbo for multiples pods deploying latest 2.4 RHACM downstream snapshot
Product:	Red Hat Advanced Cluster Management for Kubernetes	Reporter:	Chad Crum <ccrum>
Component:	Cluster Lifecycle	Assignee:	Jian Qiu <jqiu>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	Derek Ho <dho>
Severity:	urgent	Docs Contact:	Christopher Dawson <cdawson>
Priority:	unspecified
Version:	rhacm-2.4	CC:	bjacot, ccrum, sasha, yuhe
Target Milestone:	---	Keywords:	TestBlocker
Target Release:	rhacm-2.4	Flags:	ccrum: rhacm-2.4?
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-08-17 16:13:07 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Chad Crum 2021-08-17 15:53:15 UTC

Description of the problem:

Deploying RHACM 2.4 downstream snapshot results in multiple pods crash looping.

submariner-addon-54db489c78-lrgz2                                 1/1     Running            6 (34s ago)      67m
managedcluster-import-controller-v2-c7676fd74-csfsd               0/1     CrashLoopBackOff   14 (3m12s ago)   65m
hive-operator-774646d57-dklrc                                     0/1     CrashLoopBackOff   16 (5m11s ago)   67m
management-ingress-c8c5f-5499494b95-ks2m9                         1/2     Running            22 (5m29s ago)   65m
management-ingress-c8c5f-5499494b95-nprns                         1/2     Running            22 (5m19s ago)   65m


Release version:

Operator snapshot version:
2.4.0-DOWNSTREAM-2021-08-17-07-07-34

OCP version:
4.9.0-0.nightly-2021-08-16-082143

Env:

Hub = ocp baremetal ipi, ipv4 connected (libvirt)

Browser Info:
N/A

Steps to reproduce:
1. Deploy ocp baremetal ipi, ipv4 connected hub
2. Created catalogsource from 2.4.0-DOWNSTREAM-2021-08-17-07-07-34
3. Deploy RHACM on hub via catalogsource
4. Create mch cr

Actual results:
(Hub was deployed ~70 minutes ago, so that would probably account for multicluster-operators-* pods)

multicluster-operators-channel-6554ffdf6b-tlxf4                   1/1     Running            1 (70m ago)      73m
multicluster-operators-application-59bbccd88c-7wdmn               4/4     Running            3 (70m ago)      73m
submariner-addon-54db489c78-lrgz2                                 1/1     Running            6 (6m21s ago)    73m
managedcluster-import-controller-v2-c7676fd74-csfsd               0/1     CrashLoopBackOff   15 (2m55s ago)   70m
hive-operator-774646d57-dklrc                                     0/1     CrashLoopBackOff   18 (9s ago)      73m
management-ingress-c8c5f-5499494b95-ks2m9                         1/2     CrashLoopBackOff   23 (4m36s ago)   70m
management-ingress-c8c5f-5499494b95-nprns                         1/2     CrashLoopBackOff   23 (4m16s ago)   70m


Expected results:
All pods eventually are running.

Additional info:

Testing with earlier RHACM 2.4 snapshots resulted in hive operator pod clbo, but other pods were running (Opened bz for this https://bugzilla.redhat.com/show_bug.cgi?id=1993366)

However in this case we have multiclusterhub cr applied and are getting more pods deployed.

Comment 2 Chad Crum 2021-08-17 16:13:07 UTC

After looking closer at the pods, these might not be related. Will open separate bugs on the individual pods. Closing this out.