Bug 2083973 - Storage System stuck in Status "Condition: Progressing" while deploying from the CLI
Summary: Storage System stuck in Status "Condition: Progressing" while deploying from ...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: odf-operator
Version: 4.11
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Nitin Goyal
QA Contact: Martin Bukatovic
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-11 06:57 UTC by Keenon Lee
Modified: 2023-08-09 17:00 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-16 03:45:35 UTC
Embargoed:


Attachments (Terms of Use)

Description Keenon Lee 2022-05-11 06:57:38 UTC
Description of problem (please be detailed as possible and provide log
snippests):

Storage System stuck in Status "Condition: Progressing" while deploying from the CLI

Version of all relevant components (if applicable):


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes! Deployment is stuck.


Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?
Yes - saw it number of times on various deployments 


Can this issue reproduce from the UI?
No use web console works well 

If this is a regression, please provide more details to justify this:

redhat@jitli:~$ oc version
Client Version: 4.11.0-0.nightly-2022-04-01-172551
Server Version: 4.11.0-0.nightly-2022-05-09-161023
Kubernetes Version: v1.23.3+69213f8


Steps to Reproduce:
1. Create a cluster on vsphere(3 workers)

2. redhat@jitli:~$oc adm new-project openshift-storage

3. Install ODF Operator
redhat@jitli:~/work/src/data/OCP-45546$ oc apply -f og_odf_t1.yaml
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-storage-45546
  namespace: openshift-storage
spec:
  targetNamespaces:
  - openshift-storage

redhat@jitli:~/work/src/data/OCP-45546$ oc apply -f sub_odf_t1.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: odf-operator
  namespace: openshift-storage
spec:
  channel: stable-4.10
  installPlanApproval: Automatic
  name: odf-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace


wait...

redhat@jitli:~/work/src/data/OCP-45546$ oc get sub -n openshift-storage
NAME                                                              PACKAGE                   SOURCE             CHANNEL
mcg-operator-stable-4.10-redhat-operators-openshift-marketplace   mcg-operator              redhat-operators   stable-4.10
ocs-operator-stable-4.10-redhat-operators-openshift-marketplace   ocs-operator              redhat-operators   stable-4.10
odf-csi-addons-operator                                           odf-csi-addons-operator   redhat-operators   stable-4.10
odf-operator                                                      odf-operator              redhat-operators   stable-4.10


redhat@jitli:~/work/src/data/OCP-45546$ oc get csv -n openshift-storage
NAME                              DISPLAY                       VERSION   REPLACES   PHASE
mcg-operator.v4.10.0              NooBaa Operator               4.10.0               Succeeded
ocs-operator.v4.10.0              OpenShift Container Storage   4.10.0               Succeeded
odf-csi-addons-operator.v4.10.0   CSI Addons                    4.10.0               Succeeded
odf-operator.v4.10.0              OpenShift Data Foundation     4.10.0               Succeeded


then create StorageSystem:

redhat@jitli:~/work/src/data/OCP-45546$ cat storagesystem_t1.yaml 
apiVersion: odf.openshift.io/v1alpha1
kind: StorageSystem
metadata:
  name: ocs-storagecluster-storagesystem
  namespace: openshift-storage
spec:
  kind: storagecluster.ocs.openshift.io/v1
  name: ocs-storagecluster
  namespace: openshift-storage

redhat@jitli:~/work/src/data/OCP-45546$ oc apply -f storagesystem_t1.yaml


redhat@jitli:~/work/src/data/OCP-45546$ oc get storagesystem -n openshift-storage
NAME                               STORAGE-SYSTEM-KIND                  STORAGE-SYSTEM-NAME
ocs-storagecluster-storagesystem   storagecluster.ocs.openshift.io/v1   ocs-storagecluster

redhat@jitli:~/work/src/data/OCP-45546$ oc get storagesystem -n openshift-storage ocs-storagecluster-storagesystem -o yaml
apiVersion: odf.openshift.io/v1alpha1
kind: StorageSystem
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"odf.openshift.io/v1alpha1","kind":"StorageSystem","metadata":{"annotations":{},"name":"ocs-storagecluster-storagesystem","namespace":"openshift-storage"},"spec":{"kind":"storagecluster.ocs.openshift.io/v1","name":"ocs-storagecluster","namespace":"openshift-storage"}}
  creationTimestamp: "2022-05-11T04:58:10Z"
  finalizers:
  - storagesystem.odf.openshift.io
  generation: 1
  name: ocs-storagecluster-storagesystem
  namespace: openshift-storage
  resourceVersion: "345048"
  uid: 82cdfc6f-c6d6-4b67-b032-768234f3073d
spec:
  kind: storagecluster.ocs.openshift.io/v1
  name: ocs-storagecluster
  namespace: openshift-storage
status:
  conditions:
  - lastHeartbeatTime: "2022-05-11T06:26:41Z"
    lastTransitionTime: "2022-05-11T04:58:10Z"
    message: Reconcile is in progress
    reason: Reconciling
    status: "False"
    type: Available
  - lastHeartbeatTime: "2022-05-11T06:26:41Z"
    lastTransitionTime: "2022-05-11T04:58:10Z"
    message: Reconcile is in progress
    reason: Reconciling
    status: "True"
    type: Progressing
  - lastHeartbeatTime: "2022-05-11T06:26:41Z"
    lastTransitionTime: "2022-05-11T04:58:10Z"
    message: StorageSystem CR is valid
    reason: Valid
    status: "False"
    type: StorageSystemInvalid
  - lastHeartbeatTime: "2022-05-11T06:26:41Z"
    lastTransitionTime: "2022-05-11T04:58:10Z"
    reason: Ready
    status: "True"
    type: VendorCsvReady
  - lastHeartbeatTime: "2022-05-11T06:26:41Z"
    lastTransitionTime: "2022-05-11T04:58:10Z"
    message: StorageCluster.ocs.openshift.io "ocs-storagecluster" not found
    reason: NotFound
    status: "False"
    type: VendorSystemPresent



redhat@jitli:~/work/src/data/OCP-45546$ oc logs odf-operator-controller-manager-c5c97f895-q74pk -c manager -n openshift-storage

2022-05-11T05:20:01.575Z	ERROR	controller-runtime.manager.controller.storagesystem	Reconciler error	{"reconciler group": "odf.openshift.io", "reconciler kind": "StorageSystem", "name": "ocs-storagecluster-storagesystem", "namespace": "openshift-storage", "error": "StorageCluster.ocs.openshift.io \"ocs-storagecluster\" not found"}

FULL log: 
http://pastebin.test.redhat.com/1050948


So,I delete the storagesystem cr
redhat@jitli:~/work/src/data/OCP-45546$ oc delete -f storagesystem_t1.yaml
storagesystem.odf.openshift.io "ocs-storagecluster-storagesystem" deleted

I used web console to create storagesystem:
 1.Log into the OpenShift Web Console.
 2.On the left navigation bar, click Operators -> Install Operators -> 
StorageSystem.
 3.Click Create StorageSystem -> next next next (all default) -> Create

wait...

Conditions: Available, VendorCsvReady, VendorSystemPresent

redhat@jitli:~/work/src/data/OCP-45546$ oc get storagesystem -n openshift-storage ocs-storagecluster-storagesystem -o yaml
apiVersion: odf.openshift.io/v1alpha1
kind: StorageSystem
metadata:
  creationTimestamp: "2022-05-11T06:49:12Z"
  finalizers:
  - storagesystem.odf.openshift.io
  generation: 1
  name: ocs-storagecluster-storagesystem
  namespace: openshift-storage
  resourceVersion: "381788"
  uid: c0d60e24-f8fa-4a6b-89bf-3df82bec5104
spec:
  kind: storagecluster.ocs.openshift.io/v1
  name: ocs-storagecluster
  namespace: openshift-storage
status:
  conditions:
  - lastHeartbeatTime: "2022-05-11T06:49:12Z"
    lastTransitionTime: "2022-05-11T06:49:12Z"
    message: Reconcile is completed successfully
    reason: ReconcileCompleted
    status: "True"
    type: Available
  - lastHeartbeatTime: "2022-05-11T06:49:12Z"
    lastTransitionTime: "2022-05-11T06:49:12Z"
    message: Reconcile is completed successfully
    reason: ReconcileCompleted
    status: "False"
    type: Progressing
  - lastHeartbeatTime: "2022-05-11T06:49:12Z"
    lastTransitionTime: "2022-05-11T06:49:12Z"
    message: StorageSystem CR is valid
    reason: Valid
    status: "False"
    type: StorageSystemInvalid
  - lastHeartbeatTime: "2022-05-11T06:49:12Z"
    lastTransitionTime: "2022-05-11T06:49:12Z"
    reason: Ready
    status: "True"
    type: VendorCsvReady
  - lastHeartbeatTime: "2022-05-11T06:49:12Z"
    lastTransitionTime: "2022-05-11T06:49:12Z"
    reason: Found
    status: "True"
    type: VendorSystemPresent



Actual results:
    message: StorageCluster.ocs.openshift.io "ocs-storagecluster" not found


Expected results:
Storage System should reach status Ready. 


Additional info:

Did I forget which step? Or if there is any problem, please help point out
Thanks!

Comment 2 Keenon Lee 2022-05-11 07:03:49 UTC
https://github.com/red-hat-storage/odf-operator/blob/main/README.md
StorageSystem can be created from the console, using the StorageSystem creation wizard. From the CLI, a StorageSystem resource can be created using the example CR as follows,
oc create -f config/samples/ocs-storagecluster-storagesystem.yaml


I don't want to use web console to create StorageSystem

The purpose of listing the methods of using the web console is to show that the odf-operator is installed normally. Noly the StorageSystem cannot be created

Thanks a lot


Note You need to log in before you can comment on or make changes to this bug.