Bug 1658953 - Mount failed with hostpath driver deployed by csi-operator on next-gen env
Summary: Mount failed with hostpath driver deployed by csi-operator on next-gen env
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: ---
Assignee: Jan Safranek
QA Contact: Liang Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-13 09:02 UTC by Wenqi He
Modified: 2019-09-09 13:58 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-09 13:58:32 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Wenqi He 2018-12-13 09:02:02 UTC
Description of problem:
hostpath driver dose not work depolyed csi-operator on next-gen installation

Version-Release number of selected component (if applicable):
oc v4.0.0-0.82.0
kubernetes v1.11.0+6855010f70


How reproducible:
Always

Steps to Reproduce:
1. Install OCP by Next-gen on AWS https://github.com/openshift/installer/
2. Deoploy hostpath driver by csi-operator: oc apply -f deploy/samples/hostpath.yaml
3. Create pvc using hostpath sc
4. Create pod use that pvc

Actual results:
Failed to mount:
Events:
  Type     Reason                  Age                 From                                  Message
  ----     ------                  ----                ----                                  -------
  Normal   Scheduled               27m                 default-scheduler                     Successfully assigned default/hostpathpod to ip-10-0-42-127.ec2.internal
  Normal   SuccessfulAttachVolume  27m                 attachdetach-controller               AttachVolume.Attach succeeded for volume "pvc-456b764afeac11e8"
  Warning  FailedMount             4m (x10 over 25m)   kubelet, ip-10-0-42-127.ec2.internal  Unable to mount volumes for pod "hostpathpod_default(7861b8c9-feac-11e8-ad3b-0e3155c942d8)": timeout expired waiting for volumes to attach or mount for pod "default"/"hostpathpod". list of unmounted volumes=[hostpath]. list of unattached volumes=[hostpath default-token-k7nvs]
  Warning  FailedMount             39s (x21 over 27m)  kubelet, ip-10-0-42-127.ec2.internal  MountVolume.SetUp failed for volume "pvc-456b764afeac11e8" : rpc error: code = Unknown desc = mount failed: exit status 255
Mounting command: mount
Mounting arguments: -o bind /tmp/45775d51-feac-11e8-9516-0a580a800224 /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount
Output: mount: mounting /tmp/45775d51-feac-11e8-9516-0a580a800224 on /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount failed: No such file or directory



Expected results:
Pod is running

Master Log:
Did not find useful info
Node Log (of failed PODs):
# oc logs hostpath-node-lsz9j -c csi-driver -n openshift-csi-operator

I1213 08:05:58.631837       1 utils.go:97] GRPC request: volume_id:"45775d51-feac-11e8-9516-0a580a800224" target_path:"/var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount" volume_capability:<mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > > volume_attributes:<key:"storage.kubernetes.io/csiProvisionerIdentity" value:"1544685021365-8081-csi-hostpath" > 
I1213 08:05:58.632084       1 mount_linux.go:188] Detected OS without systemd
I1213 08:05:58.632237       1 nodeserver.go:79] target /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount
fstype ext4
device 
readonly false
attributes 45775d51-feac-11e8-9516-0a580a800224
 mountflags map[storage.kubernetes.io/csiProvisionerIdentity:1544685021365-8081-csi-hostpath]
%!(EXTRA []string=[])
I1213 08:05:58.632296       1 mount_linux.go:188] Detected OS without systemd
I1213 08:05:58.632310       1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-o bind /tmp/45775d51-feac-11e8-9516-0a580a800224 /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount])
E1213 08:05:58.633138       1 mount_linux.go:140] Mount failed: exit status 255
Mounting command: mount
Mounting arguments: -o bind /tmp/45775d51-feac-11e8-9516-0a580a800224 /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount
Output: mount: mounting /tmp/45775d51-feac-11e8-9516-0a580a800224 on /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount failed: No such file or directory

E1213 08:05:58.633166       1 utils.go:100] GRPC error: mount failed: exit status 255
Mounting command: mount
Mounting arguments: -o bind /tmp/45775d51-feac-11e8-9516-0a580a800224 /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount
Output: mount: mounting /tmp/45775d51-feac-11e8-9516-0a580a800224 on /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount failed: No such file or directory


PV Dump:
$ oc get pv pvc-456b764afeac11e8 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: csi-hostpath
  creationTimestamp: 2018-12-13T07:54:08Z
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-456b764afeac11e8
  resourceVersion: "110238"
  selfLink: /api/v1/persistentvolumes/pvc-456b764afeac11e8
  uid: 45774353-feac-11e8-a32f-0aaa93c407fa
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: pvc
    namespace: default
    resourceVersion: "110231"
    uid: 456b764a-feac-11e8-82b0-025c0624a272
  csi:
    driver: csi-hostpath
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1544685021365-8081-csi-hostpath
    volumeHandle: 45775d51-feac-11e8-9516-0a580a800224
  persistentVolumeReclaimPolicy: Delete
  storageClassName: sc1
status:
  phase: Bound


PVC Dump:
$ oc get pvc pvc -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"276380fe-fea6-11e8-b66d-0a580a800224","leaseDurationSeconds":15,"acquireTime":"2018-12-13T07:54:08Z","renewTime":"2018-12-13T07:54:10Z","leaderTransitions":0}'
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: csi-hostpath
  creationTimestamp: 2018-12-13T07:54:08Z
  finalizers:
  - kubernetes.io/pvc-protection
  name: pvc
  namespace: default
  resourceVersion: "110261"
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/pvc
  uid: 456b764a-feac-11e8-82b0-025c0624a272
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: sc1
  volumeName: pvc-456b764afeac11e8
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound


StorageClass Dump (if StorageClass used by PV/PVC):
$ oc get sc sc1 -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: 2018-12-13T07:10:13Z
  labels:
    csidriver.storage.openshift.io/owner-name: hostpath
    csidriver.storage.openshift.io/owner-namespace: openshift-csi-operator
  name: sc1
  resourceVersion: "81283"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/sc1
  uid: 227b108e-fea6-11e8-a32f-0aaa93c407fa
provisioner: csi-hostpath
reclaimPolicy: Delete
volumeBindingMode: Immediate


Additional info:
I tried both privileged and unprivileged pod, both failed with similar errors.

Comment 1 Jan Safranek 2019-01-02 11:29:47 UTC
HostPath is sort of debugging CSI driver, it basically expects that the cluster has single node. HostPath provisioner provisiones a volume as mkdir somewhere in /tmp on a node and all kubelets must see it (i.e. all kubelets share /tmp or there is only one kubelet). Maybe we could add a note about it somewhere.

How many nodes does your cluster have?

Comment 2 Jan Safranek 2019-01-02 11:49:58 UTC
Filled https://github.com/kubernetes-csi/drivers/pull/132 to document single-node requirement.

Comment 3 Wenqi He 2019-01-07 09:02:12 UTC
AFAIK, The next-gen installation can set node numbers, and actually I do not think a single node has its testing value for OCP... I am going to try install a single node to see whether it works or not.

Comment 4 Wenqi He 2019-01-10 04:04:12 UTC
I have tried to work with a single node Next-gen env, but blocked by bug #1646834
$ oc get nodes
NAME                           STATUS    ROLES     AGE       VERSION
ip-10-0-0-253.ec2.internal     Ready     master    2d        v1.11.0+4d56dbaf21
ip-10-0-142-236.ec2.internal   Ready     worker    2d        v1.11.0+4d56dbaf21
ip-10-0-22-101.ec2.internal    Ready     master    2d        v1.11.0+4d56dbaf21
ip-10-0-36-192.ec2.internal    Ready     master    2d        v1.11.0+4d56dbaf21

$ oc get pods
NAME                                   READY     STATUS             RESTARTS   AGE
hostpath-controller-6455875478-rsrd4   3/4       CrashLoopBackOff   8          15m
hostpath-node-rwczr                    3/3       Running            3          15m

  Warning  Unhealthy  34s (x4 over 4m)  kubelet, ip-10-0-142-236.ec2.internal  Liveness probe failed: Get http://10.131.0.28:9808/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Will try this one after bug #1646834 fixed. As Jan mentioned, hostpath is sort of debugging csi driver, I'd like to make this bug's priority a little low.

Comment 5 Jan Safranek 2019-03-06 08:49:24 UTC
Wenqi, bug #1646834 has been fixed, would you have time to re-test this?

Comment 6 Wenqi He 2019-03-08 09:21:44 UTC
Hi Jan,
I have tried, but failed. Detail steps are below:
Now Flexy is available on Next-gen installation, My flexy job id is 57616 for your reference.

$ oc get nodes
NAME                                           STATUS   ROLES    AGE   VERSION
ip-172-31-135-188.us-east-2.compute.internal   Ready    worker   48m   v1.12.4+c547bc3e86
ip-172-31-142-72.us-east-2.compute.internal    Ready    master   65m   v1.12.4+c547bc3e86
ip-172-31-159-154.us-east-2.compute.internal   Ready    master   65m   v1.12.4+c547bc3e86

$ oc apply -f ~/csi-operator/deploy/prerequisites
$ oc apply -f ~/csi-operator/deploy/operator.yaml
$ oc apply -f ~/csi-operator/deploy/samples/hostpath.yaml
$ oc get sc
NAME            PROVISIONER             AGE
gp2             kubernetes.io/aws-ebs   21m
sc1 (default)   csi-hostpath            2m26s
sc2             csi-hostpath            2m26s
$ oc get pvc
NAME   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc    Bound    pvc-c77cc8ca-417b-11e9-b724-06eba9b82fb0   1Gi        RWO            sc1            3s
$ oc get pv pvc-c77cc8ca-417b-11e9-b724-06eba9b82fb0 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: csi-hostpath
  creationTimestamp: 2019-03-08T08:25:49Z
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-c77cc8ca-417b-11e9-b724-06eba9b82fb0
  resourceVersion: "27119"
  selfLink: /api/v1/persistentvolumes/pvc-c77cc8ca-417b-11e9-b724-06eba9b82fb0
  uid: c784b82b-417b-11e9-ad72-025712d788ba
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: pvc
    namespace: default
    resourceVersion: "27115"
    uid: c77cc8ca-417b-11e9-b724-06eba9b82fb0
  csi:
    driver: csi-hostpath
    fsType: ext4
    volumeAttributes:
      storage.kubernetes.io/csiProvisionerIdentity: 1552033385985-8081-csi-hostpath
    volumeHandle: c783eaff-417b-11e9-8077-0a580a820018
  persistentVolumeReclaimPolicy: Delete
  storageClassName: sc1
status:
  phase: Bound

$ oc get pods
NAME                                   READY   STATUS              RESTARTS   AGE
hostpath-controller-74dd68c85b-hmqj6   4/4     Running             0          14m
hostpath-node-ddd22                    3/3     Running             0          14m
podtest                                0/1     ContainerCreating   0          18s
$ oc describe pods podtest
Name:               podtest
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               ip-172-31-135-188.us-east-2.compute.internal/172.31.135.188
Start Time:         Fri, 08 Mar 2019 16:37:28 +0800
Labels:             <none>
Annotations:        openshift.io/scc: anyuid
Status:             Pending
IP:                 
Containers:
  myfrontend:
    Container ID:   
    Image:          jhou/hello-openshift
    Image ID:       
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/hostpath from hostpath (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-sfjnj (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  hostpath:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pvc
    ReadOnly:   false
  default-token-sfjnj:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-sfjnj
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                From                                                   Message
  ----     ------                  ----               ----                                                   -------
  Normal   Scheduled               67s                default-scheduler                                      Successfully assigned default/podtest to ip-172-31-135-188.us-east-2.compute.internal
  Normal   SuccessfulAttachVolume  67s                attachdetach-controller                                AttachVolume.Attach succeeded for volume "pvc-c77cc8ca-417b-11e9-b724-06eba9b82fb0"
  Warning  FailedMount             27s (x7 over 59s)  kubelet, ip-172-31-135-188.us-east-2.compute.internal  MountVolume.MountDevice failed for volume "pvc-c77cc8ca-417b-11e9-b724-06eba9b82fb0" : driver name csi-hostpath not found in the list of registered CSI drivers

Comment 10 Jan Safranek 2019-09-09 13:58:32 UTC
We don't support CSI operator


Note You need to log in before you can comment on or make changes to this bug.