Bug 1968181
| Summary: | [Assisted-4.7] [Integration][OCS] Operator deployment ends with failure, although OCS recovers and deploys successfully | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Lital Alon <lalon> | ||||||
| Component: | assisted-installer | Assignee: | Rewant <resoni> | ||||||
| assisted-installer sub component: | OCS Plugin | QA Contact: | Lital Alon <lalon> | ||||||
| Status: | CLOSED CURRENTRELEASE | Docs Contact: | |||||||
| Severity: | urgent | ||||||||
| Priority: | unspecified | CC: | cvultur, godas, lgamliel, nibalach, omachace, pkliczew, yobshans | ||||||
| Version: | 4.7 | ||||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | OCP-Metal-V1.0.23.1 | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2022-08-28 08:45:59 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Created attachment 1789121 [details]
cluster_logs
CSV events: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RequirementsUnknown 21m (x3 over 21m) operator-lifecycle-manager requirements not yet checked Normal RequirementsNotMet 21m (x2 over 21m) operator-lifecycle-manager one or more requirements couldn't be found Normal InstallWaiting 20m (x3 over 20m) operator-lifecycle-manager installing: waiting for deployment rook-ceph-operator to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available... Warning ComponentUnhealthy 5m34s (x2 over 5m35s) operator-lifecycle-manager installing: waiting for deployment ocs-operator to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available... Normal NeedsReinstall 5m32s (x2 over 5m33s) operator-lifecycle-manager installing: waiting for deployment ocs-operator to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available... Normal AllRequirementsMet 5m29s (x4 over 21m) operator-lifecycle-manager all requirements found, attempting install Normal InstallSucceeded 5m28s (x3 over 20m) operator-lifecycle-manager waiting for install components to report healthy Normal InstallWaiting 5m28s (x3 over 20m) operator-lifecycle-manager installing: waiting for deployment ocs-operator to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available... Normal InstallSucceeded 85s (x3 over 20m) operator-lifecycle-manager install strategy completed with no errors CSV events from another cluster that reproduced same behavior: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal RequirementsUnknown 135m operator-lifecycle-manager requirements not yet checked Normal RequirementsNotMet 135m operator-lifecycle-manager one or more requirements couldn't be found Normal InstallWaiting 134m operator-lifecycle-manager installing: waiting for deployment rook-ceph-operator to become ready: deployment "rook-ceph-operator" not available: Deployment does not have minimum availability. Normal InstallWaiting 116m operator-lifecycle-manager Webhooks not installed Normal InstallWaiting 116m (x4 over 134m) operator-lifecycle-manager installing: waiting for deployment ocs-operator to become ready: deployment "ocs-operator" not available: Deployment does not have minimum availability. Warning ComponentUnhealthy 115m (x3 over 118m) operator-lifecycle-manager installing: waiting for deployment ocs-operator to become ready: deployment "ocs-operator" not available: Deployment does not have minimum availability. Normal NeedsReinstall 115m (x4 over 118m) operator-lifecycle-manager installing: waiting for deployment ocs-operator to become ready: deployment "ocs-operator" not available: Deployment does not have minimum availability. Normal AllRequirementsMet 115m (x6 over 134m) operator-lifecycle-manager all requirements found, attempting install Normal InstallSucceeded 115m (x5 over 134m) operator-lifecycle-manager waiting for install components to report healthy Warning InstallCheckFailed 110m operator-lifecycle-manager install timeout Normal InstallSucceeded 109m (x3 over 133m) operator-lifecycle-manager install strategy completed with no errors Controller log: time="2021-06-07T12:04:45Z" level=info msg="Service acknowledged console operator is available for cluster b2d6ecdd-e77f-47a4-9882-2ee9cf3ec542" time="2021-06-07T12:05:15Z" level=info msg="Created temporary directory /tmp/controller-custom-manifests-499134409 to store custom manifest content." time="2021-06-07T12:05:15Z" level=info msg="Downloading file custom_manifests.yaml to /tmp/controller-custom-manifests-499134409/custom_manifests.yaml" time="2021-06-07T12:05:15Z" level=info msg="Downloading file kubeconfig-noingress to /tmp/controller-custom-manifests-499134409/kubeconfig-noingress" time="2021-06-07T12:05:16Z" level=info msg="Downloaded kubeconfig-noingress to /tmp/controller-custom-manifests-499134409/kubeconfig-noingress." time="2021-06-07T12:05:17Z" level=info msg="storagecluster.ocs.openshift.io/ocs-storagecluster created\n" time="2021-06-07T12:05:17Z" level=info msg="hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged created\n" time="2021-06-07T12:05:17Z" level=info msg="localvolumeset.local.storage.openshift.io/local-disks created\n" time="2021-06-07T12:05:17Z" level=info msg="Applying custom manifest file /tmp/controller-custom-manifests-499134409/custom_manifests.yaml succeed storagecluster.ocs.openshift.io/ocs-storagecluster created\nhyperconverged.hco.kubevirt.io/kubevirt-hyperconverged created\nlocalvolumeset.local.storage.openshift.io/local-disks created" time="2021-06-07T12:05:47Z" level=info msg="Checking OLM operators" time="2021-06-07T12:05:48Z" level=info msg="CSV ocs is in status Failed, message installing: waiting for deployment ocs-operator to become ready: deployment \"ocs-operator\" not available: Deployment does not have minimum availability.." time="2021-06-07T12:05:48Z" level=info msg="CSV cnv is in status Installing, message installing: waiting for deployment hco-operator to become ready: deployment \"hco-operator\" not available: Deployment does not have minimum availability.." time="2021-06-07T12:05:48Z" level=info msg="CSV lso is in status Succeeded, message install strategy completed with no errors." time="2021-06-07T12:06:17Z" level=info msg="Checking OLM operators" time="2021-06-07T12:06:18Z" level=info msg="CSV cnv is in status Installing, message installing: waiting for deployment hco-operator to become ready: deployment \"hco-operator\" not available: Deployment does not have minimum availability.." time="2021-06-07T12:06:47Z" level=info msg="Checking OLM operators" time="2021-06-07T12:06:48Z" level=info msg="CSV cnv is in status Installing, message installing: waiting for deployment hco-operator to become ready: deployment \"hco-operator\" not available: Deployment does not have minimum availability.." time="2021-06-07T12:07:17Z" level=info msg="Checking OLM operators" time="2021-06-07T12:07:18Z" level=info msg="CSV cnv is in status Installing, message installing: waiting for deployment hco-operator to become ready: deployment \"hco-operator\" not available: Deployment does not have minimum availability.." time="2021-06-07T12:07:43Z" level=info msg="Start uploading controller logs (intermediate snapshot)" time="2021-06-07T12:07:43Z" level=info msg="Uploading logs for assisted-installer-controller-gtkm6 in assisted-installer" time="2021-06-07T12:07:44Z" level=info msg="Done uploading controller logs to the service" time="2021-06-07T12:07:47Z" level=info msg="Checking OLM operators" time="2021-06-07T12:07:48Z" level=info msg="CSV cnv is in status Installing, message installing: waiting for deployment hco-operator to become ready: deployment \"hco-operator\" not available: Deployment does not have minimum availability.." time="2021-06-07T12:08:17Z" level=info msg="Checking OLM operators" time="2021-06-07T12:08:18Z" level=info msg="CSV cnv is in status Installing, message installing: waiting for deployment hco-operator to become ready: deployment \"hco-operator\" not available: Deployment does not have minimum availability.." time="2021-06-07T12:08:47Z" level=info msg="Checking OLM operators" time="2021-06-07T12:08:48Z" level=info msg="CSV cnv is in status Installing, message waiting for install components to report healthy." time="2021-06-07T12:09:17Z" level=info msg="Checking OLM operators" time="2021-06-07T12:09:18Z" level=info msg="CSV cnv is in status Succeeded, message install strategy completed with no errors." time="2021-06-07T12:09:47Z" level=info msg="Checking OLM operators" time="2021-06-07T12:09:47Z" level=info msg="Start complete installation step, with params success:true, error info " time="2021-06-07T12:10:49Z" level=info msg="Done complete installation step" time="2021-06-07T12:10:49Z" level=info msg="Finished PostInstallConfigs" time="2021-06-07T12:10:55Z" level=info msg="Cluster installation successfully finished." Possible cause being investigated: https://bugzilla.redhat.com/show_bug.cgi?id=1968606 Got the same problem with the OCS operator showing: status_info: "installing: waiting for deployment ocs-operator to become ready: deployment \"ocs-operator\" not available: Deployment does not have minimum availability." Still double checking the cluster seems fine Adding a retry mechanism that ignores the failed state a few times. PR: https://github.com/openshift/assisted-installer/pull/319 Please use Ondra's PR as workaround. Proper fix should be delivered in OCS and the workaround should be removed. @godas Should we use this BZ to track this issue in OCS or you prefer new one? Hi Piotr, We do have BZ https://bugzilla.redhat.com/show_bug.cgi?id=1968606 for OCS to track this. Do you mean BZ to track from AI side? I was not aware of BZ #1968606. Thanks! How can we make sure Ondra's fix will be removed once OCS bug is fixed? We will open one BZ from AI and make depends on to the ocs bz. So that we can track and do necessary changes when OCS side fix is available. This fix verified against Integration, and working well. No false failures so far. Verified on Staging OCP-Metal-V1.0.23.1 3 OCS clusters installed successfully |
Created attachment 1789120 [details] example Description of problem: Tried to deploy a cluster with OCS (minimal mode) The issue is that OCS deployment ended with a failure, although 5 minutes after that, OCS was Ready (like he was able to recover and deploy). I expect to have a failure only in cases where OCS is indeed fails to deploy. See attached example Issue reproduced few times, with CNV or without, and on all 3 modes on minimal, standard and compact. Version-Release number of selected component (if applicable): Integration How reproducible: Intermediate Additional info: From controller logs is seems like OCS is failing 1 sec after console operator became ready, and after this failur LSO got success (maybe a dependency issue? ) vtime="2021-06-06T14:19:15Z" level=info msg="CSV ocs is in status Failed, message installing: waiting for deployment ocs-operator to become ready: Waiting for rollout to finish: 0 of 1 updated replicas are available...\n." time="2021-06-06T14:19:15Z" level=info msg="CSV lso is in status Succeeded, message install strategy completed with no errors."