Bug 1833038 - rook-ceph-osd pods in CrashLoopBackOff
Summary: rook-ceph-osd pods in CrashLoopBackOff
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenShift Container Storage
Classification: Red Hat Storage
Component: rook
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: OCS 4.4.0
Assignee: Travis Nielsen
QA Contact: Neha Berry
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-07 17:15 UTC by Coady LaCroix
Modified: 2020-06-04 12:54 UTC (History)
6 users (show)

Fixed In Version: 4.4.0-rc5
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-06-04 12:54:40 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift rook pull 52 0 None closed Bug 1833038: Remove invalid verbose params from lv activate 2021-02-12 13:47:30 UTC
Github rook rook pull 5438 0 None closed Remove invalid verbose params from lv activate 2021-02-12 13:47:30 UTC
Red Hat Product Errata RHBA-2020:2393 0 None None None 2020-06-04 12:54:46 UTC

Description Coady LaCroix 2020-05-07 17:15:23 UTC
Description of problem (please be detailed as possible and provide log
snippests):

During OCS 4.4 RC4 deployment the rook-ceph-osd pods end up in a CrashLoopBackOff state which causes the deployment to fail.

Jenkins job where we experienced this:
https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-deploy-ocs-cluster/7315//console

Mail thread with further details:
http://post-office.corp.redhat.com/archives/rhocs-eng/2020-May/msg00048.html


Version of all relevant components (if applicable):
ocs-olm-operator:4.4.0-418.ci
OCP 4.4


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Yes, we currently cannot deploy OCS 4.4 RC4.


Is there any workaround available to the best of your knowledge?

No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

1


Can this issue reproducible?

Yes

Can this issue reproduce from the UI?

Untested


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Deploy OCP 4.4
2. Deploy OCS 4.4 RC4 (4.4.0-418.ci)


Actual results:

OCS fails to deploy properly due to rook-ceph-osd pods being in CrashLoopBackOff state


Expected results:

rook-ceph-osd pods reach Running state and we continue on to a successful OCS deployment


Additional info:
must-gather logs: http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-au3r33-t1/jnk-au3r33-t1_20200506T131058/logs/failed_testcase_ocs_logs_1588771348/deployment_ocs_logs/

Comment 2 Travis Nielsen 2020-05-08 16:42:09 UTC
Reassigning while Seb is on holiday to get the RC builds unblocked...

Comment 5 Travis Nielsen 2020-05-08 17:27:42 UTC
Upstream PR is opened, still testing for downstream on an AWS cluster...

Comment 6 Travis Nielsen 2020-05-08 18:35:57 UTC
The change has been tested. I validated that I hit the issue on my aws cluster before applying the fix. After applying the fix, the OSDs started successfully.

The fix is merged downstream and ready for the next RC build.
https://github.com/openshift/rook/pull/52

Comment 10 errata-xmlrpc 2020-06-04 12:54:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2393


Note You need to log in before you can comment on or make changes to this bug.