Bug 1972294
| Summary: | OpenShift manila plugin fails to mount manila share | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | ksedmako |
| Component: | Storage | Assignee: | Mike Fedosin <mfedosin> |
| Storage sub component: | OpenStack CSI Drivers | QA Contact: | Jon Uriarte <juriarte> |
| Status: | CLOSED DUPLICATE | Docs Contact: | |
| Severity: | low | ||
| Priority: | unspecified | CC: | aos-bugs |
| Version: | 4.6 | ||
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-06-16 08:26:10 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** This bug has been marked as a duplicate of bug 1929136 *** Hello! I wrote a doc about manila share mounting: https://github.com/openshift/installer/pull/4803 I hope ShiftStack team will review and merge it soon. Just wanted to mentioned that the following modification worked as day 2 operation. The change was applied to the machineset and rolled out to the nodes. After that each node had an interface related to the manila network on OpenStack side and pod using manila storageclass were able to mount manila shares successfully:
networks:
- filter: {}
subnets:
- filter:
name: ocp4-njmk6-nodes
tags: openshiftClusterID=ocp4-njmk6
- filter: {}
noAllowedAddressPairs: true
uuid: 27671b90-c2bc-483f-b783-cc856f20ee5d #This is the id of the manila network as seen on OpenStack side.
|
Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Enable manila csi driver operator 2. Create PVC with storageclass csi-manila-ceph manila.csi.openstack.org 3. Crate a pod using above claim - the step that fails. Actual results: PVC is created successfully. Spawning a pod to use PVC fails getting stuck at ContainerCreating openshift-manila-csi-driver fails to: create manila port on OpenStack attach manila port to OpenShift node mount manila share on OpenShift node Expected results: Attach share to OpenShift node and achieve Running pod state. Setup validated to work manually using: openstack port create --network manila_net manila-test-port openstack server add port <instance_id/or_name> <port_id/or_name> manila access-allow <share_id/or_name> ip <ip_address_created_on_manila_net systemctl restart NetworkManager oc create -f <pod definition to use pvc> Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info: OpenShift plugin logs: oc logs openstack-manila-csi-nodeplugin-fbzt9 -c csi-driver I0603 12:29:33.953329 1 driver.go:124] Driver: manila.csi.openstack.org I0603 12:29:33.953447 1 driver.go:125] Driver version: 0.9.0@ I0603 12:29:33.953453 1 driver.go:126] CSI spec version: 1.2.0 I0603 12:29:33.953459 1 driver.go:129] Operating on NFS shares I0603 12:29:33.953469 1 driver.go:134] Topology awareness disabled I0603 12:29:33.953481 1 driver.go:197] Enabling controller service capability: CREATE_DELETE_VOLUME I0603 12:29:33.953487 1 driver.go:197] Enabling controller service capability: CREATE_DELETE_SNAPSHOT I0603 12:29:33.953493 1 driver.go:216] Enabling volume access mode: MULTI_NODE_MULTI_WRITER I0603 12:29:33.953498 1 driver.go:216] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER I0603 12:29:33.953501 1 driver.go:216] Enabling volume access mode: MULTI_NODE_READER_ONLY I0603 12:29:33.953505 1 driver.go:216] Enabling volume access mode: SINGLE_NODE_WRITER I0603 12:29:33.953509 1 driver.go:216] Enabling volume access mode: SINGLE_NODE_READER_ONLY I0603 12:29:33.954593 1 connection.go:261] Probing CSI driver for readiness I0603 12:29:33.956749 1 driver.go:262] proxying CSI driver nfs.csi.k8s.io version 2.0.0 I0603 12:29:33.957591 1 driver.go:227] Enabling node service capability: UNKNOWN I0603 12:29:33.957924 1 driver.go:326] listening for connections on &net.UnixAddr{Name:"/var/lib/kubelet/plugins/manila.csi.openstack.org/csi.sock", Net:"unix"} I0615 13:53:11.922241 1 builder.go:44] [ID:4] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 13:53:11.922334 1 driver.go:313] [ID:5] GRPC error: rpc error: code = Canceled desc = context canceled I0615 13:55:12.463593 1 builder.go:44] [ID:5] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 13:55:12.463730 1 driver.go:313] [ID:7] GRPC error: rpc error: code = Canceled desc = context canceled I0615 13:57:13.489533 1 builder.go:44] [ID:6] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 13:57:13.489621 1 driver.go:313] [ID:9] GRPC error: rpc error: code = Canceled desc = context canceled I0615 13:59:15.518814 1 builder.go:44] [ID:7] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 13:59:15.518893 1 driver.go:313] [ID:11] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:01:19.523361 1 builder.go:44] [ID:8] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:01:19.523423 1 driver.go:313] [ID:13] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:03:27.615845 1 builder.go:44] [ID:9] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:03:27.615926 1 driver.go:313] [ID:15] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:05:43.678115 1 builder.go:44] [ID:10] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:05:43.678232 1 driver.go:313] [ID:17] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:08:15.692745 1 builder.go:44] [ID:11] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:08:15.692822 1 driver.go:313] [ID:19] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:11:19.720613 1 builder.go:44] [ID:12] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:11:19.720725 1 driver.go:313] [ID:21] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:15:21.788699 1 builder.go:44] [ID:13] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:15:21.788778 1 driver.go:313] [ID:23] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:19:23.808683 1 builder.go:44] [ID:14] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:19:23.808770 1 driver.go:313] [ID:25] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:23:25.812357 1 builder.go:44] [ID:15] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:23:25.812463 1 driver.go:313] [ID:27] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:27:27.899661 1 builder.go:44] [ID:16] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:27:27.899769 1 driver.go:313] [ID:29] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:31:29.915406 1 builder.go:44] [ID:17] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:31:29.915480 1 driver.go:313] [ID:31] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:35:31.998713 1 builder.go:44] [ID:18] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:35:31.998768 1 driver.go:313] [ID:33] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:39:34.069656 1 builder.go:44] [ID:19] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:39:34.069755 1 driver.go:313] [ID:35] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:43:36.071951 1 builder.go:44] [ID:20] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:43:36.072094 1 driver.go:313] [ID:37] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:47:38.111509 1 builder.go:44] [ID:21] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:47:38.111566 1 driver.go:313] [ID:39] GRPC error: rpc error: code = Canceled desc = context canceled I0615 14:51:40.151235 1 builder.go:44] [ID:22] FWD GRPC error: rpc error: code = Canceled desc = context canceled E0615 14:51:40.151415 1 driver.go:313] [ID:41] GRPC error: rpc error: code = Canceled desc = context canceled [ksedmako@ksedmako base]$ oc logs openstack-manila-csi-nodeplugin-fbzt9 Error from server (BadRequest): a container name must be specified for pod openstack-manila-csi- oc logs openstack-manila-csi-nodeplugin-fbzt9 -c csi-node-driver-registrar I0603 12:29:36.035222 1 main.go:113] Version: v4.7.0-202103010125.p0-0-gde30f7a-dirty I0603 12:29:36.036303 1 main.go:137] Attempting to open a gRPC connection with: "/csi/csi.sock" I0603 12:29:36.036331 1 connection.go:153] Connecting to unix:///csi/csi.sock I0603 12:29:36.037475 1 main.go:144] Calling CSI driver to discover driver name I0603 12:29:36.038668 1 main.go:154] CSI driver name: "manila.csi.openstack.org" I0603 12:29:36.038716 1 node_register.go:52] Starting Registration Server at: /registration/manila.csi.openstack.org-reg.sock I0603 12:29:36.039080 1 node_register.go:61] Registration Server started at: /registration/manila.csi.openstack.org-reg.sock I0603 12:29:36.039157 1 node_register.go:83] Skipping healthz server because HTTP endpoint is set to: "" I0603 12:29:36.396780 1 main.go:80] Received GetInfo call: &InfoRequest{} I0603 12:29:36.422660 1 main.go:90] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,} E0615 14:48:39.967185 1 mount_linux.go:139] Mount failed: exit status 32 Mounting command: mount Mounting arguments: -t nfs 172.16.32.1:/volumes/_nogroup/0698a783-167d-44e0-82fe-0aa740a66292 /var/lib/kubelet/pods/fcc37d69-d2d0-4c2d-bcdc-a2ba5614d1d3/volumes/kubernetes.io~csi/pvc-957c8e39-dc1f-46ec-b2f3-908066d797eb/mount Output: mount.nfs: Connection timed out