Created attachment 1710984 [details] openshift-cluster-csi-drivers namespace logs from must-gather Description of problem: OCP 4.6 installation cannot be completed on top of OSP with self signed certificates for public endpoints. Kuryr network type was used during this test. Version-Release number of selected component (if applicable): OCP: 4.6.0-0.nightly-2020-08-09-151434 OSP: RHOS-16.1-RHEL-8-20200804.n.0 (16.1.1 GA) Due to current issue https://bugzilla.redhat.com/show_bug.cgi?id=1866738, 46.82.202008080704-0 rhcos image has been used. How reproducible: always Steps to Reproduce: 1. Deploy OSP with self signed certs (the cacert is provided in clouds.yaml) 2. Install OCP 4.6 - "openshift-install create cluster" fails Actual results: ERROR Cluster operator storage Degraded is True with ManilaCSIDriverOperatorCR_ManilaController_SyncError: ManilaCSIDriverOperatorCRDegraded: ManilaControllerDegraded: Get "https://10.46.22.140:13000/": x509: certificate signed by unknown authority INFO Cluster operator storage Progressing is True with ManilaCSIDriverOperatorCR_WaitForOperator: ManilaCSIDriverOperatorCRProgressing: Waiting for Manila operator to report status INFO Cluster operator storage Available is False with ManilaCSIDriverOperatorCR_WaitForOperator: ManilaCSIDriverOperatorCRAvailable: Waiting for Manila operator to report status FATAL failed to initialize the cluster: Working towards 4.6.0-0.nightly-2020-08-09-151434: 99% complete Expected results: successful installation Error messages in failed manila pod: E0810 12:40:12.580733 1 base_controller.go:220] "ManilaController" controller failed to sync "key", err: Get "https://10.46.22.140:13000/": x509: certificate signed by unknown authority E0810 12:40:13.046397 1 reflector.go:127] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:opensh ift-cluster-csi-drivers:manila-csi-driver-operator" cannot list resource "configmaps" in API group "" in the namespace "openshift-manila-csi-driver" (shiftstack) [stack@undercloud-0 ~]$ oc get pv -A No resources found (shiftstack) [stack@undercloud-0 ~]$ oc get pvc -A No resources found Additional info: attached must-gather (shiftstack) [stack@undercloud-0 ~]$ oc get nodes NAME STATUS ROLES AGE VERSION ostest-ps9x4-master-0 Ready master 3h43m v1.19.0-rc.2+5241b27-dirty ostest-ps9x4-master-1 Ready master 3h43m v1.19.0-rc.2+5241b27-dirty ostest-ps9x4-master-2 Ready master 3h43m v1.19.0-rc.2+5241b27-dirty ostest-ps9x4-worker-cnzlt Ready worker 3h11m v1.19.0-rc.2+5241b27-dirty ostest-ps9x4-worker-drrp8 Ready worker 3h10m v1.19.0-rc.2+5241b27-dirty ostest-ps9x4-worker-njn5b Ready worker 3h11m v1.19.0-rc.2+5241b27-dirty (shiftstack) [stack@undercloud-0 ~]$ openstack server list +--------------------------------------+---------------------------+--------+-------------------------------------+----------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+---------------------------+--------+-------------------------------------+----------+--------+ | fadd89af-def9-41f6-a599-2a063641311d | ostest-ps9x4-worker-drrp8 | ACTIVE | ostest-ps9x4-openshift=10.196.3.225 | my-rhcos | | | 07e35039-0f77-43fa-a300-de98e75aeef3 | ostest-ps9x4-worker-njn5b | ACTIVE | ostest-ps9x4-openshift=10.196.2.118 | my-rhcos | | | efcaa3b2-d3b3-4e27-97ef-5689d1d7ac90 | ostest-ps9x4-worker-cnzlt | ACTIVE | ostest-ps9x4-openshift=10.196.2.18 | my-rhcos | | | 1da119e9-39b0-4167-9a9f-09784bd34e39 | ostest-ps9x4-master-2 | ACTIVE | ostest-ps9x4-openshift=10.196.1.226 | my-rhcos | | | f64465a7-55dc-4dfd-b113-6e92f69eb693 | ostest-ps9x4-master-0 | ACTIVE | ostest-ps9x4-openshift=10.196.0.152 | my-rhcos | | | d4cfd5fd-fed4-4334-b968-1a23499ca7b8 | ostest-ps9x4-master-1 | ACTIVE | ostest-ps9x4-openshift=10.196.2.69 | my-rhcos | | +--------------------------------------+---------------------------+--------+-------------------------------------+----------+--------+ (overcloud) [stack@undercloud-0 ~]$ openstack endpoint list +----------------------------------+-----------+--------------+----------------+---------+-----------+--------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+----------------+---------+-----------+--------------------------------------------------+ | 004971fbc9c24bc886b625c2957b8489 | regionOne | neutron | network | True | admin | http://172.17.1.96:9696 | | 0cbe76add0c6419e901187ceae2a688e | regionOne | neutron | network | True | internal | http://172.17.1.96:9696 | | 0d937a7898d94d1a8459441e4ae62e6c | regionOne | glance | image | True | internal | http://172.17.1.96:9292 | | 22249b6a10e24fbd878e993c4205757d | regionOne | glance | image | True | admin | http://172.17.1.96:9292 | | 23367b0bdebf4e8bbc7d22f1e91ee237 | regionOne | heat-cfn | cloudformation | True | internal | http://172.17.1.96:8000/v1 | | 30962be59db14e9abc6e1d145b534cec | regionOne | glance | image | True | public | https://10.46.22.140:13292 | | 41e7d30dc4c249da9ea72db9ed202bc3 | regionOne | cinderv2 | volumev2 | True | public | https://10.46.22.140:13776/v2/%(tenant_id)s | | 44ed760541204bb599bed00e2514c90f | regionOne | keystone | identity | True | internal | http://172.17.1.96:5000 | | 45e64391c45240118c73b5a307fdc5da | regionOne | octavia | load-balancer | True | public | https://10.46.22.140:13876 | | 4ec89098700948589d7cc58bbbdd8a88 | regionOne | heat | orchestration | True | admin | http://172.17.1.96:8004/v1/%(tenant_id)s | | 595a32702491491b88690f05a2e0002d | regionOne | cinderv2 | volumev2 | True | internal | http://172.17.1.96:8776/v2/%(tenant_id)s | | 665b6307f11045d8a3f12189077d6055 | regionOne | placement | placement | True | internal | http://172.17.1.96:8778/placement | | 67d338a5157d47b0aa447aade56a665e | regionOne | nova | compute | True | public | https://10.46.22.140:13774/v2.1 | | 6b04926f91544fecbdf3b2f0956baef0 | regionOne | cinderv3 | volumev3 | True | public | https://10.46.22.140:13776/v3/%(tenant_id)s | | 6e405308f35342e08c3b8ac3f85a1ab6 | regionOne | octavia | load-balancer | True | internal | http://172.17.1.96:9876 | | 6fabfa9c5b604e7d9319be6b668b7085 | regionOne | cinderv3 | volumev3 | True | internal | http://172.17.1.96:8776/v3/%(tenant_id)s | | 73e35f093cb2486c8b2eab52059f28c6 | regionOne | swift | object-store | True | internal | http://172.17.3.160:8080/v1/AUTH_%(tenant_id)s | | 771535b48ca948e19672099e37df7cef | regionOne | nova | compute | True | admin | http://172.17.1.96:8774/v2.1 | | 83025640c68c4709b96a8a7662226e75 | regionOne | placement | placement | True | public | https://10.46.22.140:13778/placement | | 94d3fb83cdde4c3d8b7e4189946896fd | regionOne | swift | object-store | True | public | https://10.46.22.140:13808/v1/AUTH_%(tenant_id)s | | 9ad3ab0201214c038ee94bd111d4517d | regionOne | cinderv2 | volumev2 | True | admin | http://172.17.1.96:8776/v2/%(tenant_id)s | | a4c0065ea55840d5bfab50acc868142f | regionOne | octavia | load-balancer | True | admin | http://172.17.1.96:9876 | | ae00af173d8f4cb29fe46b7d786626ea | regionOne | keystone | identity | True | admin | http://192.168.24.30:35357 | | b0141b3eeefe4ccb98f05601b6757c77 | regionOne | heat-cfn | cloudformation | True | public | https://10.46.22.140:13005/v1 | | b13ec5b4b89d429292b06b6b4583d6b2 | regionOne | swift | object-store | True | admin | http://172.17.3.160:8080 | | bec33172f6fd44acb041866b580c3115 | regionOne | keystone | identity | True | public | https://10.46.22.140:13000 | | bee42c17b40b4abab21369b9f87ab01a | regionOne | nova | compute | True | internal | http://172.17.1.96:8774/v2.1 | | bfce4e6719e7452287cdf97989205c8c | regionOne | heat | orchestration | True | internal | http://172.17.1.96:8004/v1/%(tenant_id)s | | c35fd7fa304948d38d1f6532a84c3342 | regionOne | cinderv3 | volumev3 | True | admin | http://172.17.1.96:8776/v3/%(tenant_id)s | | c667fa5ca14d4431a299c73e92f9b0e6 | regionOne | placement | placement | True | admin | http://172.17.1.96:8778/placement | | e64b5925221a45ebbd5315cad180f40b | regionOne | neutron | network | True | public | https://10.46.22.140:13696 | | eaee3a4b0cf0486b8bbbdf6a653d6665 | regionOne | heat-cfn | cloudformation | True | admin | http://172.17.1.96:8000/v1 | | eb473782621f4a3e87a7eee7a8391beb | regionOne | heat | orchestration | True | public | https://10.46.22.140:13004/v1/%(tenant_id)s | +----------------------------------+-----------+--------------+----------------+---------+-----------+--------------------------------------------------+
Created attachment 1711287 [details] manila pod logs
I've tried with 4.6.0-0.nightly-2020-08-12-155346 and found similar error. The installer fails: time="2020-08-13T04:32:27-04:00" level=debug msg="Still waiting for the cluster to initialize: Cluster operator storage is reporting a failure: ManilaCSIDriverOperatorCRDegraded: ManilaControllerDegraded: Get \"https://10.46.22.140:13000/\": x509: certificate signed by unknown authority" time="2020-08-13T04:38:39-04:00" level=info msg="Cluster operator insights Disabled is False with AsExpected: " time="2020-08-13T04:38:39-04:00" level=info msg="Cluster operator kube-apiserver Progressing is True with NodeInstaller: NodeInstallerProgressing: 1 nodes are at revision 6; 2 nodes are at revision 7" time="2020-08-13T04:38:39-04:00" level=error msg="Cluster operator storage Degraded is True with ManilaCSIDriverOperatorCR_ManilaController_SyncError: ManilaCSIDriverOperatorCRDegraded: ManilaControllerDegraded: Get \"https://10.46.22.140:13000/\": x509: certificate signed by unknown authority" time="2020-08-13T04:38:39-04:00" level=info msg="Cluster operator storage Progressing is True with ManilaCSIDriverOperatorCR_WaitForOperator: ManilaCSIDriverOperatorCRProgressing: Waiting for Manila operator to report status" time="2020-08-13T04:38:39-04:00" level=info msg="Cluster operator storage Available is False with ManilaCSIDriverOperatorCR_WaitForOperator: ManilaCSIDriverOperatorCRAvailable: Waiting for Manila operator to report status" time="2020-08-13T04:38:39-04:00" level=fatal msg="failed to initialize the cluster: Cluster operator storage is reporting a failure: ManilaCSIDriverOperatorCRDegraded: ManilaControllerDegraded: Get \"https://10.46.22.140:13000/\": x509: certificate signed by unknown authority" The manila pod log doesn't show the next message: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps is forbidden: User "system:serviceaccount:openshift-cluster-csi-drivers:manila-csi-driver-operator" cannot list resource "configmaps" in API group "" in the namespace "openshift-manila-csi-driver" but it keeps showing: E0813 08:24:40.987220 1 base_controller.go:220] "ManilaController" controller failed to sync "key", err: Get "https://10.46.22.140:13000/": x509: certificate signed by unknown authority Find attached manila pod logs.
Verified in 4.6.0-0.nightly-2020-08-18-085914 on top of OSP 16.1 RHOS-16.1-RHEL-8-20200804.n.0 compose. The cluster is installed successfully: DEBUG Time elapsed per stage: DEBUG Infrastructure: 1m52s DEBUG Bootstrap Complete: 19m51s DEBUG API: 4m56s DEBUG Bootstrap Destroy: 41s DEBUG Cluster Operators: 25m55s INFO Time elapsed: 49m5s manila csi driver pod is running ok: openshift-cluster-csi-drivers manila-csi-driver-operator-6b8678cc88-9t596 1/1 Running 0 72m
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196