1969345 – [manila-csi-driver-operator] openstack-manila-csi-nodeplugin pods stucked with ".. still connecting to unix:///var/lib/kubelet/plugins/csi-nfsplugin/csi.sock"

Bug 1969345 - [manila-csi-driver-operator] openstack-manila-csi-nodeplugin pods stucked with ".. still connecting to unix:///var/lib/kubelet/plugins/csi-nfsplugin/csi.sock"

Summary: [manila-csi-driver-operator] openstack-manila-csi-nodeplugin pods stucked wit...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Storage
Sub Component:
Version:	4.7
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	high
Target Milestone:	---
Target Release:	4.7.z
Assignee:	Eric Duen
QA Contact:	rlobillo
Docs Contact:
URL:
Whiteboard:
Depends On:	1944655
Blocks:
TreeView+	depends on / blocked

Reported:	2021-06-08 09:24 UTC by Matthew Booth
Modified:	2024-10-01 18:31 UTC (History)
CC List:	13 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1944655
Environment:
Last Closed:	2021-06-29 04:19:45 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	openshift cloud-provider-openstack pull 54	None	open	Bug 1969345: Backport ignore proxy connecting to CSI sockets	2021-06-17 01:00:46 UTC
Red Hat Knowledge Base (Article)	5983621	None	None	None	2021-06-18 12:22:05 UTC
Red Hat Product Errata	RHBA-2021:2502	None	None	None	2021-06-29 04:20:08 UTC

Comment 3 rlobillo 2021-06-18 10:51:31 UTC

Verified on 4.7.0-0.nightly-2021-06-18-030252 on OSP16.1 (RHOS-16.1-RHEL-8-20210506.n.1) with manila enabled.

Steps:

1 - Performing IPI installation and confirming that manila is working fine:

$ oc get pods,pvc
NAME                               READY   STATUS    RESTARTS   AGE
pod/demo-manila-5cc9769b8b-6r8h8   1/1     Running   0          58s

NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
persistentvolumeclaim/pvc-manila   Bound    pvc-a6e37be0-56c2-45a8-9562-3add351ef007   1Gi        RWO            csi-manila-default   58s

2 - Configuring the proxy:

$ oc get proxy cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Proxy
metadata:
  creationTimestamp: "2021-06-18T07:00:58Z"
  generation: 3
  name: cluster
  resourceVersion: "89477"
  selfLink: /apis/config.openshift.io/v1/proxies/cluster
  uid: ba203c45-a0cf-4e06-877a-cde8c2c3091a
spec:
  httpProxy: http://squid.corp.redhat.com:3128/
  noProxy: 10.46.43.140
  trustedCA:
    name: ""
status:
  httpProxy: http://squid.corp.redhat.com:3128/
  noProxy: .cluster.local,.svc,10.128.0.0/14,10.196.0.0/16,10.46.43.140,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.ostest.shiftstack.com,localhost

(where 10.46.43.140 is the OSP API).


3 - Wait until pods on openshift-manila-csi-driver are restarted.

$ oc get pods -n openshift-manila-csi-driver
NAME                                                     READY   STATUS    RESTARTS   AGE
csi-nodeplugin-nfsplugin-6qmnp                           1/1     Running   0          91m
csi-nodeplugin-nfsplugin-8ncgq                           1/1     Running   0          137m
csi-nodeplugin-nfsplugin-bxngm                           1/1     Running   0          137m
csi-nodeplugin-nfsplugin-lb2sv                           1/1     Running   0          137m
csi-nodeplugin-nfsplugin-q8dh9                           1/1     Running   0          93m
csi-nodeplugin-nfsplugin-vvk6w                           1/1     Running   0          96m
openstack-manila-csi-controllerplugin-59f84d78c9-rw944   4/4     Running   0          19s
openstack-manila-csi-nodeplugin-4f9bv                    2/2     Running   0          77s
openstack-manila-csi-nodeplugin-8rkxn                    2/2     Running   0          53s
openstack-manila-csi-nodeplugin-99cww                    2/2     Running   0          29s
openstack-manila-csi-nodeplugin-9pk2m                    2/2     Running   0          64s
openstack-manila-csi-nodeplugin-ln4mw                    2/2     Running   0          39s
openstack-manila-csi-nodeplugin-twfzn                    2/2     Running   0          58s


4 - Check nodeplugin logs:

$ for i in $(oc get pods -n openshift-manila-csi-driver -l component=nodeplugin -o NAME); do echo ***$i; oc logs -n openshift-manila-csi-driver $i -c csi-driver; echo; done
***pod/openstack-manila-csi-nodeplugin-4f9bv
I0618 10:28:49.847609       1 driver.go:124] Driver: manila.csi.openstack.org
I0618 10:28:49.847761       1 driver.go:125] Driver version: 0.9.0@
I0618 10:28:49.847771       1 driver.go:126] CSI spec version: 1.2.0
I0618 10:28:49.847790       1 driver.go:129] Operating on NFS shares
I0618 10:28:49.847805       1 driver.go:134] Topology awareness disabled
I0618 10:28:49.847839       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_VOLUME
I0618 10:28:49.847869       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0618 10:28:49.847936       1 driver.go:216] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0618 10:28:49.847946       1 driver.go:216] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0618 10:28:49.847953       1 driver.go:216] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0618 10:28:49.847961       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_WRITER
I0618 10:28:49.847968       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0618 10:28:49.849815       1 connection.go:261] Probing CSI driver for readiness
I0618 10:28:49.854943       1 driver.go:262] proxying CSI driver nfs.csi.k8s.io version 2.0.0
I0618 10:28:49.856372       1 driver.go:227] Enabling node service capability: UNKNOWN
I0618 10:28:49.857012       1 driver.go:326] listening for connections on &net.UnixAddr{Name:"/var/lib/kubelet/plugins/manila.csi.openstack.org/csi.sock", Net:"unix"}

***pod/openstack-manila-csi-nodeplugin-8rkxn
I0618 10:29:13.107566       1 driver.go:124] Driver: manila.csi.openstack.org
I0618 10:29:13.107684       1 driver.go:125] Driver version: 0.9.0@
I0618 10:29:13.107690       1 driver.go:126] CSI spec version: 1.2.0
I0618 10:29:13.107732       1 driver.go:129] Operating on NFS shares
I0618 10:29:13.107763       1 driver.go:134] Topology awareness disabled
I0618 10:29:13.107773       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_VOLUME
I0618 10:29:13.107778       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0618 10:29:13.107788       1 driver.go:216] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0618 10:29:13.107794       1 driver.go:216] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0618 10:29:13.107799       1 driver.go:216] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0618 10:29:13.107803       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_WRITER
I0618 10:29:13.107807       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0618 10:29:13.113584       1 connection.go:261] Probing CSI driver for readiness
I0618 10:29:13.116664       1 driver.go:262] proxying CSI driver nfs.csi.k8s.io version 2.0.0
I0618 10:29:13.117330       1 driver.go:227] Enabling node service capability: UNKNOWN
I0618 10:29:13.117636       1 driver.go:326] listening for connections on &net.UnixAddr{Name:"/var/lib/kubelet/plugins/manila.csi.openstack.org/csi.sock", Net:"unix"}

***pod/openstack-manila-csi-nodeplugin-99cww
I0618 10:29:37.966060       1 driver.go:124] Driver: manila.csi.openstack.org
I0618 10:29:37.966186       1 driver.go:125] Driver version: 0.9.0@
I0618 10:29:37.966192       1 driver.go:126] CSI spec version: 1.2.0
I0618 10:29:37.966204       1 driver.go:129] Operating on NFS shares
I0618 10:29:37.966227       1 driver.go:134] Topology awareness disabled
I0618 10:29:37.966292       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_VOLUME
I0618 10:29:37.966296       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0618 10:29:37.966301       1 driver.go:216] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0618 10:29:37.966305       1 driver.go:216] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0618 10:29:37.966308       1 driver.go:216] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0618 10:29:37.966312       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_WRITER
I0618 10:29:37.966315       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0618 10:29:37.967252       1 connection.go:261] Probing CSI driver for readiness
I0618 10:29:37.970302       1 driver.go:262] proxying CSI driver nfs.csi.k8s.io version 2.0.0
I0618 10:29:37.971723       1 driver.go:227] Enabling node service capability: UNKNOWN
I0618 10:29:37.972170       1 driver.go:326] listening for connections on &net.UnixAddr{Name:"/var/lib/kubelet/plugins/manila.csi.openstack.org/csi.sock", Net:"unix"}

***pod/openstack-manila-csi-nodeplugin-9pk2m
I0618 10:29:03.207566       1 driver.go:124] Driver: manila.csi.openstack.org
I0618 10:29:03.207886       1 driver.go:125] Driver version: 0.9.0@
I0618 10:29:03.207922       1 driver.go:126] CSI spec version: 1.2.0
I0618 10:29:03.207947       1 driver.go:129] Operating on NFS shares
I0618 10:29:03.207985       1 driver.go:134] Topology awareness disabled
I0618 10:29:03.208015       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_VOLUME
I0618 10:29:03.208040       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0618 10:29:03.208064       1 driver.go:216] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0618 10:29:03.208086       1 driver.go:216] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0618 10:29:03.208108       1 driver.go:216] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0618 10:29:03.208162       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_WRITER
I0618 10:29:03.208196       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0618 10:29:03.210441       1 connection.go:261] Probing CSI driver for readiness
I0618 10:29:03.218430       1 driver.go:262] proxying CSI driver nfs.csi.k8s.io version 2.0.0
I0618 10:29:03.220788       1 driver.go:227] Enabling node service capability: UNKNOWN
I0618 10:29:03.221196       1 driver.go:326] listening for connections on &net.UnixAddr{Name:"/var/lib/kubelet/plugins/manila.csi.openstack.org/csi.sock", Net:"unix"}

***pod/openstack-manila-csi-nodeplugin-ln4mw
I0618 10:29:27.672279       1 driver.go:124] Driver: manila.csi.openstack.org
I0618 10:29:27.672363       1 driver.go:125] Driver version: 0.9.0@
I0618 10:29:27.672368       1 driver.go:126] CSI spec version: 1.2.0
I0618 10:29:27.672373       1 driver.go:129] Operating on NFS shares
I0618 10:29:27.672379       1 driver.go:134] Topology awareness disabled
I0618 10:29:27.672390       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_VOLUME
I0618 10:29:27.672394       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0618 10:29:27.672398       1 driver.go:216] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0618 10:29:27.672401       1 driver.go:216] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0618 10:29:27.672405       1 driver.go:216] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0618 10:29:27.672408       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_WRITER
I0618 10:29:27.672411       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0618 10:29:27.673177       1 connection.go:261] Probing CSI driver for readiness
I0618 10:29:27.674803       1 driver.go:262] proxying CSI driver nfs.csi.k8s.io version 2.0.0
I0618 10:29:27.675282       1 driver.go:227] Enabling node service capability: UNKNOWN
I0618 10:29:27.675559       1 driver.go:326] listening for connections on &net.UnixAddr{Name:"/var/lib/kubelet/plugins/manila.csi.openstack.org/csi.sock", Net:"unix"}

***pod/openstack-manila-csi-nodeplugin-twfzn
I0618 10:29:09.035558       1 driver.go:124] Driver: manila.csi.openstack.org
I0618 10:29:09.036704       1 driver.go:125] Driver version: 0.9.0@
I0618 10:29:09.036737       1 driver.go:126] CSI spec version: 1.2.0
I0618 10:29:09.036755       1 driver.go:129] Operating on NFS shares
I0618 10:29:09.036770       1 driver.go:134] Topology awareness disabled
I0618 10:29:09.036807       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_VOLUME
I0618 10:29:09.036818       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0618 10:29:09.036832       1 driver.go:216] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0618 10:29:09.036841       1 driver.go:216] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0618 10:29:09.036848       1 driver.go:216] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0618 10:29:09.036856       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_WRITER
I0618 10:29:09.036863       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0618 10:29:09.041620       1 connection.go:261] Probing CSI driver for readiness
I0618 10:29:09.049713       1 driver.go:262] proxying CSI driver nfs.csi.k8s.io version 2.0.0
I0618 10:29:09.051546       1 driver.go:227] Enabling node service capability: UNKNOWN
I0618 10:29:09.052471       1 driver.go:326] listening for connections on &net.UnixAddr{Name:"/var/lib/kubelet/plugins/manila.csi.openstack.org/csi.sock", Net:"unix"}


5 - Create new pod + pvc using manila:

$ cat manila_1.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: "pvc-manila-1"
  namespace: "topologyaware-test"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-manila-default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-manila-1
  namespace: "topologyaware-test"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-manila-1
      cinder-az: nova
      nova-az: nova
  template:
    metadata:
      labels:
        app: demo-manila-1
        cinder-az: nova
        nova-az: nova
    spec:
      containers:
      - name: demo
        image: quay.io/kuryr/demo
        ports:
        - containerPort: 80
          protocol: TCP
        volumeMounts:
          - mountPath: /var/lib/www/data
            name: mydata
      nodeSelector:
        topology.cinder.csi.openstack.org/zone: nova
      volumes:
        - name: mydata
          persistentVolumeClaim:
            claimName: pvc-manila-1
            readOnly: false

$ oc apply -f manila_1.yaml 

$ oc get pods,pvc
NAME                                 READY   STATUS    RESTARTS   AGE
pod/demo-manila-1-5bc4fd9cb6-94rxg   1/1     Running   0          3m15s
pod/demo-manila-5cc9769b8b-wcfmv     1/1     Running   0          17m

NAME                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
persistentvolumeclaim/pvc-manila     Bound    pvc-a6e37be0-56c2-45a8-9562-3add351ef007   1Gi        RWO            csi-manila-default   23m
persistentvolumeclaim/pvc-manila-1   Bound    pvc-e6d3240c-92f5-43fa-891a-49a25f952277   1Gi        RWO            csi-manila-default   9m31s

Comment 7 errata-xmlrpc 2021-06-29 04:19:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.18 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2502

Note You need to log in before you can comment on or make changes to this bug.