Bug 1969345

Summary: [manila-csi-driver-operator] openstack-manila-csi-nodeplugin pods stucked with ".. still connecting to unix:///var/lib/kubelet/plugins/csi-nfsplugin/csi.sock"
Product: OpenShift Container Platform Reporter: Matthew Booth <mbooth>
Component: StorageAssignee: Eric Duen <eduen>
Storage sub component: OpenStack CSI Drivers QA Contact: rlobillo
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: medium CC: aos-bugs, asimonel, atn, eduen, gcharot, juriarte, mbooth, mfedosin, moddi, pprinett, rlobillo, tbarron, vimartin
Version: 4.7Keywords: Triaged
Target Milestone: ---   
Target Release: 4.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1944655 Environment:
Last Closed: 2021-06-29 04:19:45 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1944655    
Bug Blocks:    

Comment 3 rlobillo 2021-06-18 10:51:31 UTC 
Verified on 4.7.0-0.nightly-2021-06-18-030252 on OSP16.1 (RHOS-16.1-RHEL-8-20210506.n.1) with manila enabled.

Steps:

1 - Performing IPI installation and confirming that manila is working fine:

$ oc get pods,pvc
NAME                               READY   STATUS    RESTARTS   AGE
pod/demo-manila-5cc9769b8b-6r8h8   1/1     Running   0          58s

NAME                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
persistentvolumeclaim/pvc-manila   Bound    pvc-a6e37be0-56c2-45a8-9562-3add351ef007   1Gi        RWO            csi-manila-default   58s

2 - Configuring the proxy:

$ oc get proxy cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Proxy
metadata:
  creationTimestamp: "2021-06-18T07:00:58Z"
  generation: 3
  name: cluster
  resourceVersion: "89477"
  selfLink: /apis/config.openshift.io/v1/proxies/cluster
  uid: ba203c45-a0cf-4e06-877a-cde8c2c3091a
spec:
  httpProxy: http://squid.corp.redhat.com:3128/
  noProxy: 10.46.43.140
  trustedCA:
    name: ""
status:
  httpProxy: http://squid.corp.redhat.com:3128/
  noProxy: .cluster.local,.svc,10.128.0.0/14,10.196.0.0/16,10.46.43.140,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.ostest.shiftstack.com,localhost

(where 10.46.43.140 is the OSP API).


3 - Wait until pods on openshift-manila-csi-driver are restarted.

$ oc get pods -n openshift-manila-csi-driver
NAME                                                     READY   STATUS    RESTARTS   AGE
csi-nodeplugin-nfsplugin-6qmnp                           1/1     Running   0          91m
csi-nodeplugin-nfsplugin-8ncgq                           1/1     Running   0          137m
csi-nodeplugin-nfsplugin-bxngm                           1/1     Running   0          137m
csi-nodeplugin-nfsplugin-lb2sv                           1/1     Running   0          137m
csi-nodeplugin-nfsplugin-q8dh9                           1/1     Running   0          93m
csi-nodeplugin-nfsplugin-vvk6w                           1/1     Running   0          96m
openstack-manila-csi-controllerplugin-59f84d78c9-rw944   4/4     Running   0          19s
openstack-manila-csi-nodeplugin-4f9bv                    2/2     Running   0          77s
openstack-manila-csi-nodeplugin-8rkxn                    2/2     Running   0          53s
openstack-manila-csi-nodeplugin-99cww                    2/2     Running   0          29s
openstack-manila-csi-nodeplugin-9pk2m                    2/2     Running   0          64s
openstack-manila-csi-nodeplugin-ln4mw                    2/2     Running   0          39s
openstack-manila-csi-nodeplugin-twfzn                    2/2     Running   0          58s


4 - Check nodeplugin logs:

$ for i in $(oc get pods -n openshift-manila-csi-driver -l component=nodeplugin -o NAME); do echo ***$i; oc logs -n openshift-manila-csi-driver $i -c csi-driver; echo; done
***pod/openstack-manila-csi-nodeplugin-4f9bv
I0618 10:28:49.847609       1 driver.go:124] Driver: manila.csi.openstack.org
I0618 10:28:49.847761       1 driver.go:125] Driver version: 0.9.0@
I0618 10:28:49.847771       1 driver.go:126] CSI spec version: 1.2.0
I0618 10:28:49.847790       1 driver.go:129] Operating on NFS shares
I0618 10:28:49.847805       1 driver.go:134] Topology awareness disabled
I0618 10:28:49.847839       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_VOLUME
I0618 10:28:49.847869       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0618 10:28:49.847936       1 driver.go:216] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0618 10:28:49.847946       1 driver.go:216] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0618 10:28:49.847953       1 driver.go:216] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0618 10:28:49.847961       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_WRITER
I0618 10:28:49.847968       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0618 10:28:49.849815       1 connection.go:261] Probing CSI driver for readiness
I0618 10:28:49.854943       1 driver.go:262] proxying CSI driver nfs.csi.k8s.io version 2.0.0
I0618 10:28:49.856372       1 driver.go:227] Enabling node service capability: UNKNOWN
I0618 10:28:49.857012       1 driver.go:326] listening for connections on &net.UnixAddr{Name:"/var/lib/kubelet/plugins/manila.csi.openstack.org/csi.sock", Net:"unix"}

***pod/openstack-manila-csi-nodeplugin-8rkxn
I0618 10:29:13.107566       1 driver.go:124] Driver: manila.csi.openstack.org
I0618 10:29:13.107684       1 driver.go:125] Driver version: 0.9.0@
I0618 10:29:13.107690       1 driver.go:126] CSI spec version: 1.2.0
I0618 10:29:13.107732       1 driver.go:129] Operating on NFS shares
I0618 10:29:13.107763       1 driver.go:134] Topology awareness disabled
I0618 10:29:13.107773       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_VOLUME
I0618 10:29:13.107778       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0618 10:29:13.107788       1 driver.go:216] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0618 10:29:13.107794       1 driver.go:216] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0618 10:29:13.107799       1 driver.go:216] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0618 10:29:13.107803       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_WRITER
I0618 10:29:13.107807       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0618 10:29:13.113584       1 connection.go:261] Probing CSI driver for readiness
I0618 10:29:13.116664       1 driver.go:262] proxying CSI driver nfs.csi.k8s.io version 2.0.0
I0618 10:29:13.117330       1 driver.go:227] Enabling node service capability: UNKNOWN
I0618 10:29:13.117636       1 driver.go:326] listening for connections on &net.UnixAddr{Name:"/var/lib/kubelet/plugins/manila.csi.openstack.org/csi.sock", Net:"unix"}

***pod/openstack-manila-csi-nodeplugin-99cww
I0618 10:29:37.966060       1 driver.go:124] Driver: manila.csi.openstack.org
I0618 10:29:37.966186       1 driver.go:125] Driver version: 0.9.0@
I0618 10:29:37.966192       1 driver.go:126] CSI spec version: 1.2.0
I0618 10:29:37.966204       1 driver.go:129] Operating on NFS shares
I0618 10:29:37.966227       1 driver.go:134] Topology awareness disabled
I0618 10:29:37.966292       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_VOLUME
I0618 10:29:37.966296       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0618 10:29:37.966301       1 driver.go:216] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0618 10:29:37.966305       1 driver.go:216] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0618 10:29:37.966308       1 driver.go:216] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0618 10:29:37.966312       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_WRITER
I0618 10:29:37.966315       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0618 10:29:37.967252       1 connection.go:261] Probing CSI driver for readiness
I0618 10:29:37.970302       1 driver.go:262] proxying CSI driver nfs.csi.k8s.io version 2.0.0
I0618 10:29:37.971723       1 driver.go:227] Enabling node service capability: UNKNOWN
I0618 10:29:37.972170       1 driver.go:326] listening for connections on &net.UnixAddr{Name:"/var/lib/kubelet/plugins/manila.csi.openstack.org/csi.sock", Net:"unix"}

***pod/openstack-manila-csi-nodeplugin-9pk2m
I0618 10:29:03.207566       1 driver.go:124] Driver: manila.csi.openstack.org
I0618 10:29:03.207886       1 driver.go:125] Driver version: 0.9.0@
I0618 10:29:03.207922       1 driver.go:126] CSI spec version: 1.2.0
I0618 10:29:03.207947       1 driver.go:129] Operating on NFS shares
I0618 10:29:03.207985       1 driver.go:134] Topology awareness disabled
I0618 10:29:03.208015       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_VOLUME
I0618 10:29:03.208040       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0618 10:29:03.208064       1 driver.go:216] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0618 10:29:03.208086       1 driver.go:216] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0618 10:29:03.208108       1 driver.go:216] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0618 10:29:03.208162       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_WRITER
I0618 10:29:03.208196       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0618 10:29:03.210441       1 connection.go:261] Probing CSI driver for readiness
I0618 10:29:03.218430       1 driver.go:262] proxying CSI driver nfs.csi.k8s.io version 2.0.0
I0618 10:29:03.220788       1 driver.go:227] Enabling node service capability: UNKNOWN
I0618 10:29:03.221196       1 driver.go:326] listening for connections on &net.UnixAddr{Name:"/var/lib/kubelet/plugins/manila.csi.openstack.org/csi.sock", Net:"unix"}

***pod/openstack-manila-csi-nodeplugin-ln4mw
I0618 10:29:27.672279       1 driver.go:124] Driver: manila.csi.openstack.org
I0618 10:29:27.672363       1 driver.go:125] Driver version: 0.9.0@
I0618 10:29:27.672368       1 driver.go:126] CSI spec version: 1.2.0
I0618 10:29:27.672373       1 driver.go:129] Operating on NFS shares
I0618 10:29:27.672379       1 driver.go:134] Topology awareness disabled
I0618 10:29:27.672390       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_VOLUME
I0618 10:29:27.672394       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0618 10:29:27.672398       1 driver.go:216] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0618 10:29:27.672401       1 driver.go:216] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0618 10:29:27.672405       1 driver.go:216] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0618 10:29:27.672408       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_WRITER
I0618 10:29:27.672411       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0618 10:29:27.673177       1 connection.go:261] Probing CSI driver for readiness
I0618 10:29:27.674803       1 driver.go:262] proxying CSI driver nfs.csi.k8s.io version 2.0.0
I0618 10:29:27.675282       1 driver.go:227] Enabling node service capability: UNKNOWN
I0618 10:29:27.675559       1 driver.go:326] listening for connections on &net.UnixAddr{Name:"/var/lib/kubelet/plugins/manila.csi.openstack.org/csi.sock", Net:"unix"}

***pod/openstack-manila-csi-nodeplugin-twfzn
I0618 10:29:09.035558       1 driver.go:124] Driver: manila.csi.openstack.org
I0618 10:29:09.036704       1 driver.go:125] Driver version: 0.9.0@
I0618 10:29:09.036737       1 driver.go:126] CSI spec version: 1.2.0
I0618 10:29:09.036755       1 driver.go:129] Operating on NFS shares
I0618 10:29:09.036770       1 driver.go:134] Topology awareness disabled
I0618 10:29:09.036807       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_VOLUME
I0618 10:29:09.036818       1 driver.go:197] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0618 10:29:09.036832       1 driver.go:216] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0618 10:29:09.036841       1 driver.go:216] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I0618 10:29:09.036848       1 driver.go:216] Enabling volume access mode: MULTI_NODE_READER_ONLY
I0618 10:29:09.036856       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_WRITER
I0618 10:29:09.036863       1 driver.go:216] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I0618 10:29:09.041620       1 connection.go:261] Probing CSI driver for readiness
I0618 10:29:09.049713       1 driver.go:262] proxying CSI driver nfs.csi.k8s.io version 2.0.0
I0618 10:29:09.051546       1 driver.go:227] Enabling node service capability: UNKNOWN
I0618 10:29:09.052471       1 driver.go:326] listening for connections on &net.UnixAddr{Name:"/var/lib/kubelet/plugins/manila.csi.openstack.org/csi.sock", Net:"unix"}


5 - Create new pod + pvc using manila:

$ cat manila_1.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: "pvc-manila-1"
  namespace: "topologyaware-test"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-manila-default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-manila-1
  namespace: "topologyaware-test"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-manila-1
      cinder-az: nova
      nova-az: nova
  template:
    metadata:
      labels:
        app: demo-manila-1
        cinder-az: nova
        nova-az: nova
    spec:
      containers:
      - name: demo
        image: quay.io/kuryr/demo
        ports:
        - containerPort: 80
          protocol: TCP
        volumeMounts:
          - mountPath: /var/lib/www/data
            name: mydata
      nodeSelector:
        topology.cinder.csi.openstack.org/zone: nova
      volumes:
        - name: mydata
          persistentVolumeClaim:
            claimName: pvc-manila-1
            readOnly: false

$ oc apply -f manila_1.yaml 

$ oc get pods,pvc
NAME                                 READY   STATUS    RESTARTS   AGE
pod/demo-manila-1-5bc4fd9cb6-94rxg   1/1     Running   0          3m15s
pod/demo-manila-5cc9769b8b-wcfmv     1/1     Running   0          17m

NAME                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
persistentvolumeclaim/pvc-manila     Bound    pvc-a6e37be0-56c2-45a8-9562-3add351ef007   1Gi        RWO            csi-manila-default   23m
persistentvolumeclaim/pvc-manila-1   Bound    pvc-e6d3240c-92f5-43fa-891a-49a25f952277   1Gi        RWO            csi-manila-default   9m31s
Comment 7 errata-xmlrpc 2021-06-29 04:19:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.18 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2502