Description of problem (please be detailed as possible and provide log snippests): When cluster-wide proxy is configured, noobaa is unable to create first.bucket on Ceph RGW and fails to complete OCS installation. Version of all relevant components (if applicable): OCS 4.2.2 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? MCG stays unhealthy and cannot be used. Is there any workaround available to the best of your knowledge? Manually configure HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables in noobaa-core StatefulSet, as described in https://access.redhat.com/solutions/4776841 Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? Reproducible whenever cluster-wide proxy settings is used. Can this issue reproduce from the UI? Yes Steps to Reproduce: 1. Install OCP with cluster-wide proxy settings 2. Install OCS Operator 3. Create StorageCluster according to OCS documentation Actual results: All OCS components are installed, but MCG won't work. Expected results: MCG should work Additional info: https://bugzilla.redhat.com/show_bug.cgi?id=1793189 fixed only proxy settings for noobaa-operator. noobaa-core StatefulSet also needs proper NO_PROXY configuration.
Created attachment 1675284 [details] oc describe deploy noobaa-operator (to demonstrate proxy settings)
Created attachment 1675285 [details] oc describe sts noobaa-core Note that cluster-wide proxy settings aren't available.
Created attachment 1675286 [details] screenshot of error shown in OCP dashboard
Hi @vagner. the description is a bit unclear. do you mean that the default backing-store is not created? can you attach OCS must-gather diagnostics or noobaa operator+core logs?
@ashish thanks for the diags. In the must gather I don't see any problem. both noobaa and the default backingstore are healthy and in a ready state. if there are other logs that show the problem then I can debug it. I don't know how the cluster-wide proxy works exactly in OCP, or why noobaa-core sts did not get the proxy settings as noobaa-operator did. we can propagate any proxy settings from noobaa-operator to noobaa-core, but I don't think this is the way it should work
Created attachment 1675839 [details] ocs must-gather
(In reply to Danny from comment #5) > Hi @vagner. > the description is a bit unclear. do you mean that the default backing-store > is not created? Correct. > > can you attach OCS must-gather diagnostics or noobaa operator+core logs? OCS must-gather is now attached to this bz. How do I provider noobaa operator+core logs?
Created attachment 1675840 [details] diagnostics gathered from noobaa management interface
From the attached MG I can verify that not all deployments and statefulsets have the PROXY_* environment variables set. A closer examination reveal that only the deployments created by OLM (the various operators' deployments) have the PROXY_* environment variables set up properly. The following documentation https://access.redhat.com/documentation/en-us/openshift_container_platform/4.4/html/operators/olm-adding-operators-to-a-cluster suggests (section 4.2) that the OLM is the one responsible for managing these envs for the operators' deployments, specifically the paragraph: > If a cluster-wide egress proxy is configured, applications created from Operators using the Operator Lifecycle Manager (OLM) inherit the cluster-wide proxy settings > on their Deployments and Pods. Cluster administrators can also override these proxy settings by configuring the Operator’s Subscription. and > OLM handles these environment variables as a unit; if at least one of them is set, all three are considered overridden > and the cluster-wide defaults are not used for the subscribed Operator’s Deployments. From this, I can deduce that setting these env variables is not an automatic process and that the controller for each deployment/statefulset has the responsibility to set them. If this is indeed the case the resolution for this BZ is to set these environment vars for the noobaa-core deployment in the noobaa operator reconcile loop.
I fix was issued in the upstream project, see the links section
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenShift Container Storage 4.5.0 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3754