Bug 1958373

Summary: OCS 4.8 deployment fails since 4.8.0-378.ci with storagecluster stuck in Progressing state
Product: [Red Hat Storage] Red Hat OpenShift Container Storage Reporter: Neha Berry <nberry>
Component: Multi-Cloud Object GatewayAssignee: Jacky Albo <jalbo>
Status: CLOSED ERRATA QA Contact: Neha Berry <nberry>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.8CC: etamir, muagarwa, nbecker, ocs-bugs
Target Milestone: ---Keywords: Automation, Regression
Target Release: OCS 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: v4.8.0-386.ci Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-03 18:16:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Neha Berry 2021-05-07 18:37:32 UTC
Description of problem (please be detailed as possible and provide log
snippests):
========================================================================
Note: We had a similar deployment failure in Bug 1952344 - OCS 4.8: v4.8.0-359 - storagecluster is in progressing state which was fixed with version 4.6.0-361.ci 

The last stable build of 4.8 where deployment succeeded is 4.8.0-374.ci. Since then, OCS deployment fails with CSV in Progressing state.

Creating a bug with details from the latest 4.8.0-382.ci build:
---------------------------------------------------------------------
2:50:07 - MainThread - ocs_ci.ocs.ocp - INFO - Resource ocs-operator.v4.8.0-382.ci is in phase: Succeeded!

13:36:38 - MainThread - ocs_ci.utility.utils - INFO - Executing command: oc -n openshift-storage get StorageCluster ocs-storagecluster -n openshift-storage -o yaml
13:36:39 - MainThread - ocs_ci.ocs.ocp - INFO - Resource ocs-storagecluster is in phase: Progressing!

# oc describe storagecluster
Last Heartbeat Time:   2021-05-06T17:38:27Z
    Last Transition Time:  2021-05-06T16:52:56Z
    Message:               Waiting on Nooba instance to finish initialization
    Reason:                NoobaaInitializing
    Status:                True
    Type:                  Progressing


Job links
---------------------

console logs =[1] = https://storage-jenkins-csb-ceph.cloud.paas.psi.redhat.com/job/ocs-
ci/381/console
[2] - https://storage-jenkins-csb-ceph.cloud.paas.psi.redhat.com/job/ocs-ci/378/console

Chat thread link - https://chat.google.com/room/AAAAREGEba8/yUKy4wDIDgY

Log location = https://storage-jenkins-csb-ceph.cloud.paas.psi.redhat.com/job/ocs-ci/381/artifact/logs/failed_testcase_ocs_logs_1620319669/test_deployment_ocs_logs/ocs_must_gather/




Version of all relevant components (if applicable):
=======================================================
openshift installer (4.8.0-0.nightly-2021-04-30-201824)
quay.io/rhceph-dev/ocs-registry:4.8.0-382.ci (4.8.0-378.ci)


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
=============================================================
Yes. Deployment fails

Is there any workaround available to the best of your knowledge?
====================================================================
Not sure

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
============================================================================
3

Can this issue reproducible?
==============================
Yes

Can this issue reproduce from the UI?
=========================================
Did not test

If this is a regression, please provide more details to justify this:
======================================================================
Yes

Steps to Reproduce:
=======================
1.Install OCS using ocs-ci
2.verify storage cluster is in ready state

Actual results
===================
storage cluster is in progressing state

Expected results:
====================
storage cluster should be in succeed phase

Additional info:
========================
oc describe storagecluster = https://storage-jenkins-csb-ceph.cloud.paas.psi.redhat.com/job/ocs-ci/381/artifact/logs/failed_testcase_ocs_logs_1620319669/test_deployment_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-1d34a430aee86cca33aa19a50d27bffa520a41d6490044563f601bb15d66e888/namespaces/openshift-storage/oc_output/storagecluster/*view*/



Initial analysis from the chat thread
==========================================


ocs-operator is waiting for NooBaa CR: Waiting on Nooba instance to finish initialization
NooBaa CR status: First endpoint is not ready yet
noobaa-endpoint Deployment is not able to create the noobaa-endpoint Pod:
'pods "noobaa-endpoint-65dc8bb596-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, spec.containers[0].securityContext.capabilities.add: Invalid value: "SETGID": capability may not be added, spec.containers[0].securityContext.capabilities.add: Invalid value: "SETUID": capability may not be added, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "noobaa-endpoint": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "rook-ceph": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount, provider "rook-ceph-csi": Forbidden: not usable by user or serviceaccount]'

Comment 9 errata-xmlrpc 2021-08-03 18:16:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Container Storage 4.8.0 container images bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3003