Bug 1968164

also test: test_ocs_failed_to_deploy_ocp_success
expect OCS to be failed, but in last test execution, OCS finished with success in AI, in actual OCS wont be able to install.
not sure it is the same bug, but might be same wrong status report (LMK if different bug is needed).

AI Operators:

    OpenShift Container Storage available
    OpenShift Local Storage available


From machine: 

[root@sealusa11 ~]# oc get StorageCluster -n openshift-storage
NAME                 AGE   PHASE         EXTERNAL   CREATED AT             VERSION
ocs-storagecluster   16m   Progressing              2021-06-06T15:46:36Z   4.7.0
[root@sealusa11 ~]# oc get po -n openshift-local-storage
NAME                                      READY   STATUS    RESTARTS   AGE
diskmaker-manager-4sflj                   1/1     Running   0          16m
diskmaker-manager-54xbv                   1/1     Running   0          16m
diskmaker-manager-8tmkk                   1/1     Running   0          16m
local-storage-operator-866667c768-lmxcs   1/1     Running   0          36m
[root@sealusa11 ~]# oc get pvc -n openshift-storage
NAME                          STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
ocs-deviceset-0-data-0qkv9q   Pending                                      localblock-sc   14m
ocs-deviceset-1-data-0mkgvl   Pending                                      localblock-sc   14m
ocs-deviceset-2-data-0bc69x   Pending                                      localblock-sc   14m

[root@sealusa11 ~]# oc get csv -n openshift-storage
NAME                  DISPLAY                       VERSION   REPLACES   PHASE
ocs-operator.v4.7.0   OpenShift Container Storage   4.7.0                Installing


Events:
  Type     Reason               Age                From                        Message
  ----     ------               ----               ----                        -------
  Normal   RequirementsUnknown  37m                operator-lifecycle-manager  requirements not yet checked
  Normal   RequirementsNotMet   37m (x2 over 37m)  operator-lifecycle-manager  one or more requirements couldn't be found
  Normal   InstallWaiting       36m (x2 over 36m)  operator-lifecycle-manager  installing: waiting for deployment rook-ceph-operator to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available...
  Normal   InstallSucceeded     35m                operator-lifecycle-manager  install strategy completed with no errors
  Warning  ComponentUnhealthy   16m (x2 over 16m)  operator-lifecycle-manager  installing: waiting for deployment ocs-operator to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available...
  Normal   NeedsReinstall       11m (x3 over 16m)  operator-lifecycle-manager  installing: waiting for deployment ocs-operator to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available...
  Normal   AllRequirementsMet   11m (x6 over 36m)  operator-lifecycle-manager  all requirements found, attempting install
  Normal   InstallSucceeded     11m (x5 over 36m)  operator-lifecycle-manager  waiting for install components to report healthy
  Normal   InstallWaiting       11m (x5 over 36m)  operator-lifecycle-manager  installing: waiting for deployment ocs-operator to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available...
  Warning  InstallCheckFailed   81s (x5 over 11m)  operator-lifecycle-manager  install timeout

Possible cause being investigated:
https://bugzilla.redhat.com/show_bug.cgi?id=1968606

We are currently investigating why this is happening.
In the meantime, how feasible is it for the AI code check for the StorageCluster status instead of the CSV for OCS operator?

Ondra how difficult it would be to change status monitoring to OCS operator?

It would be simple, but there are two problems:
1) After applying StorageCluster CR we need to wait before we start to wait for the Succeeded state. And don't check the state immediately.
2) As stated in bug 1968606 the OCS operator should not report Failed state in case it's still installing.

I would maybe suggest to fix this bug as adding a sleep before waiting to the proper CSV state after CR is applied. And ignore Failed state for a few times, before failing and marking the operator as it failed the installation.

(In reply to Ondra Machacek from comment #5)
> It would be simple, but there are two problems:
> 1) After applying StorageCluster CR we need to wait before we start to wait
> for the Succeeded state. And don't check the state immediately.
> 2) As stated in bug 1968606 the OCS operator should not report Failed state
> in case it's still installing.
> 
> I would maybe suggest to fix this bug as adding a sleep before waiting to
> the proper CSV state after CR is applied. And ignore Failed state for a few
> times, before failing and marking the operator as it failed the installation.

That sounds fine as well. 

Did anything change in the operator status check recently? I was wondering why this suddenly started happening/

Yes, this PR seems to cause it:

https://github.com/openshift/assisted-installer/pull/284

@omachace  @pkliczew, the bug needs to be fixed in Assisted-Installer. 


1) After applying StorageCluster CR we need to wait before we start to wait for the Succeeded state. And don't check the state immediately.
2) As stated in bug 1968606 the OCS operator should not report Failed state in case it's still installing.

I would maybe suggest to fix this bug as adding a sleep before waiting to the proper CSV state after CR is applied. And ignore Failed state for a few times, before failing and marking the operator as it failed the installation.


We could fix by any of the above approach.

I wonder how you will select timeout value. Will it work on all (fast and slow) environments?

Why not to check OCS operator for status instead?

Yes, we could check the StorageCluster CR status instead of checking csv status.

Adding a retry mechanism that ignores the failed state a few times.
PR: https://github.com/openshift/assisted-installer/pull/319

Issue still persist in Integration environment - In some cases, OCS become available in AI, while in actual when checking OCS storage class it is still Progressing rather than Ready (but become Ready after 10~ minutes).

CSV status of OCS is updated as:
1. It succeeds when OCS is deployed
2. It fails when the StorageCluster is being applied
3. It succeeds when the StorageCluster is applied

So, might be that it checks the csv status on phase 1 and see that it succeeded even if the StorageCluster CR is not applied, so we can check the status of the StorageCluster CR, till it's fixed in OCS.

Sure, it seems that checking the StorageCluster CR is the best workaround for now.

any updates on it?
keep reproduces https://auto-jenkins-csb-kniqe.apps.ocp4.prod.psi.redhat.com/job/ocp-assisted-installer-virt/2997/testReport/junit/test_ocs_validations/TestOCSValidations/Run_assisted_installer_Functional_API_Tests___test_ocs_multiple_insufficient_disks_three_hosts/

Rewant, Priyanka - does the introduction of OperatorCondition in OCS solve this? Changes needed to AI code too?

AI code will need to check the StorageCluster CR status instead of relying on CSV status only.With introduction of Operator Conditions, csv will not depend upon StorageCluster phases. 
@Rewant, how do we check the Operator status curently?

We check them from the CSV now(https://github.com/openshift/assisted-installer/blob/master/src/assisted_installer_controller/assisted_installer_controller.go#L481), We don't have anything to check the status of StorageCluster CR, we would need to implement it. Also, should we check only StorageCluster CR status or both StorageCluster and CSV status?

We need to check both: CSV status and StorageCluster status.

Then we would need to implement that in AI. I will check with Piotr about the changes we need to make and open tasks for same.

It's related to https://issues.redhat.com/browse/MGMT-10018, the StorageCluster is still not in a ready state but the operator is ready. There will be an email sent by @