Hide Forgot
Description of problem: hostpath driver dose not work depolyed csi-operator on next-gen installation Version-Release number of selected component (if applicable): oc v4.0.0-0.82.0 kubernetes v1.11.0+6855010f70 How reproducible: Always Steps to Reproduce: 1. Install OCP by Next-gen on AWS https://github.com/openshift/installer/ 2. Deoploy hostpath driver by csi-operator: oc apply -f deploy/samples/hostpath.yaml 3. Create pvc using hostpath sc 4. Create pod use that pvc Actual results: Failed to mount: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 27m default-scheduler Successfully assigned default/hostpathpod to ip-10-0-42-127.ec2.internal Normal SuccessfulAttachVolume 27m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-456b764afeac11e8" Warning FailedMount 4m (x10 over 25m) kubelet, ip-10-0-42-127.ec2.internal Unable to mount volumes for pod "hostpathpod_default(7861b8c9-feac-11e8-ad3b-0e3155c942d8)": timeout expired waiting for volumes to attach or mount for pod "default"/"hostpathpod". list of unmounted volumes=[hostpath]. list of unattached volumes=[hostpath default-token-k7nvs] Warning FailedMount 39s (x21 over 27m) kubelet, ip-10-0-42-127.ec2.internal MountVolume.SetUp failed for volume "pvc-456b764afeac11e8" : rpc error: code = Unknown desc = mount failed: exit status 255 Mounting command: mount Mounting arguments: -o bind /tmp/45775d51-feac-11e8-9516-0a580a800224 /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount Output: mount: mounting /tmp/45775d51-feac-11e8-9516-0a580a800224 on /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount failed: No such file or directory Expected results: Pod is running Master Log: Did not find useful info Node Log (of failed PODs): # oc logs hostpath-node-lsz9j -c csi-driver -n openshift-csi-operator I1213 08:05:58.631837 1 utils.go:97] GRPC request: volume_id:"45775d51-feac-11e8-9516-0a580a800224" target_path:"/var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount" volume_capability:<mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER > > volume_attributes:<key:"storage.kubernetes.io/csiProvisionerIdentity" value:"1544685021365-8081-csi-hostpath" > I1213 08:05:58.632084 1 mount_linux.go:188] Detected OS without systemd I1213 08:05:58.632237 1 nodeserver.go:79] target /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount fstype ext4 device readonly false attributes 45775d51-feac-11e8-9516-0a580a800224 mountflags map[storage.kubernetes.io/csiProvisionerIdentity:1544685021365-8081-csi-hostpath] %!(EXTRA []string=[]) I1213 08:05:58.632296 1 mount_linux.go:188] Detected OS without systemd I1213 08:05:58.632310 1 mount_linux.go:135] Mounting cmd (mount) with arguments ([-o bind /tmp/45775d51-feac-11e8-9516-0a580a800224 /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount]) E1213 08:05:58.633138 1 mount_linux.go:140] Mount failed: exit status 255 Mounting command: mount Mounting arguments: -o bind /tmp/45775d51-feac-11e8-9516-0a580a800224 /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount Output: mount: mounting /tmp/45775d51-feac-11e8-9516-0a580a800224 on /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount failed: No such file or directory E1213 08:05:58.633166 1 utils.go:100] GRPC error: mount failed: exit status 255 Mounting command: mount Mounting arguments: -o bind /tmp/45775d51-feac-11e8-9516-0a580a800224 /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount Output: mount: mounting /tmp/45775d51-feac-11e8-9516-0a580a800224 on /var/lib/kubelet/pods/7861b8c9-feac-11e8-ad3b-0e3155c942d8/volumes/kubernetes.io~csi/pvc-456b764afeac11e8/mount failed: No such file or directory PV Dump: $ oc get pv pvc-456b764afeac11e8 -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: pv.kubernetes.io/provisioned-by: csi-hostpath creationTimestamp: 2018-12-13T07:54:08Z finalizers: - kubernetes.io/pv-protection name: pvc-456b764afeac11e8 resourceVersion: "110238" selfLink: /api/v1/persistentvolumes/pvc-456b764afeac11e8 uid: 45774353-feac-11e8-a32f-0aaa93c407fa spec: accessModes: - ReadWriteOnce capacity: storage: 1Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: pvc namespace: default resourceVersion: "110231" uid: 456b764a-feac-11e8-82b0-025c0624a272 csi: driver: csi-hostpath fsType: ext4 volumeAttributes: storage.kubernetes.io/csiProvisionerIdentity: 1544685021365-8081-csi-hostpath volumeHandle: 45775d51-feac-11e8-9516-0a580a800224 persistentVolumeReclaimPolicy: Delete storageClassName: sc1 status: phase: Bound PVC Dump: $ oc get pvc pvc -o yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"276380fe-fea6-11e8-b66d-0a580a800224","leaseDurationSeconds":15,"acquireTime":"2018-12-13T07:54:08Z","renewTime":"2018-12-13T07:54:10Z","leaderTransitions":0}' pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: csi-hostpath creationTimestamp: 2018-12-13T07:54:08Z finalizers: - kubernetes.io/pvc-protection name: pvc namespace: default resourceVersion: "110261" selfLink: /api/v1/namespaces/default/persistentvolumeclaims/pvc uid: 456b764a-feac-11e8-82b0-025c0624a272 spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: sc1 volumeName: pvc-456b764afeac11e8 status: accessModes: - ReadWriteOnce capacity: storage: 1Gi phase: Bound StorageClass Dump (if StorageClass used by PV/PVC): $ oc get sc sc1 -o yaml allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true" creationTimestamp: 2018-12-13T07:10:13Z labels: csidriver.storage.openshift.io/owner-name: hostpath csidriver.storage.openshift.io/owner-namespace: openshift-csi-operator name: sc1 resourceVersion: "81283" selfLink: /apis/storage.k8s.io/v1/storageclasses/sc1 uid: 227b108e-fea6-11e8-a32f-0aaa93c407fa provisioner: csi-hostpath reclaimPolicy: Delete volumeBindingMode: Immediate Additional info: I tried both privileged and unprivileged pod, both failed with similar errors.
HostPath is sort of debugging CSI driver, it basically expects that the cluster has single node. HostPath provisioner provisiones a volume as mkdir somewhere in /tmp on a node and all kubelets must see it (i.e. all kubelets share /tmp or there is only one kubelet). Maybe we could add a note about it somewhere. How many nodes does your cluster have?
Filled https://github.com/kubernetes-csi/drivers/pull/132 to document single-node requirement.
AFAIK, The next-gen installation can set node numbers, and actually I do not think a single node has its testing value for OCP... I am going to try install a single node to see whether it works or not.
I have tried to work with a single node Next-gen env, but blocked by bug #1646834 $ oc get nodes NAME STATUS ROLES AGE VERSION ip-10-0-0-253.ec2.internal Ready master 2d v1.11.0+4d56dbaf21 ip-10-0-142-236.ec2.internal Ready worker 2d v1.11.0+4d56dbaf21 ip-10-0-22-101.ec2.internal Ready master 2d v1.11.0+4d56dbaf21 ip-10-0-36-192.ec2.internal Ready master 2d v1.11.0+4d56dbaf21 $ oc get pods NAME READY STATUS RESTARTS AGE hostpath-controller-6455875478-rsrd4 3/4 CrashLoopBackOff 8 15m hostpath-node-rwczr 3/3 Running 3 15m Warning Unhealthy 34s (x4 over 4m) kubelet, ip-10-0-142-236.ec2.internal Liveness probe failed: Get http://10.131.0.28:9808/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers) Will try this one after bug #1646834 fixed. As Jan mentioned, hostpath is sort of debugging csi driver, I'd like to make this bug's priority a little low.
Wenqi, bug #1646834 has been fixed, would you have time to re-test this?
Hi Jan, I have tried, but failed. Detail steps are below: Now Flexy is available on Next-gen installation, My flexy job id is 57616 for your reference. $ oc get nodes NAME STATUS ROLES AGE VERSION ip-172-31-135-188.us-east-2.compute.internal Ready worker 48m v1.12.4+c547bc3e86 ip-172-31-142-72.us-east-2.compute.internal Ready master 65m v1.12.4+c547bc3e86 ip-172-31-159-154.us-east-2.compute.internal Ready master 65m v1.12.4+c547bc3e86 $ oc apply -f ~/csi-operator/deploy/prerequisites $ oc apply -f ~/csi-operator/deploy/operator.yaml $ oc apply -f ~/csi-operator/deploy/samples/hostpath.yaml $ oc get sc NAME PROVISIONER AGE gp2 kubernetes.io/aws-ebs 21m sc1 (default) csi-hostpath 2m26s sc2 csi-hostpath 2m26s $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc Bound pvc-c77cc8ca-417b-11e9-b724-06eba9b82fb0 1Gi RWO sc1 3s $ oc get pv pvc-c77cc8ca-417b-11e9-b724-06eba9b82fb0 -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: pv.kubernetes.io/provisioned-by: csi-hostpath creationTimestamp: 2019-03-08T08:25:49Z finalizers: - kubernetes.io/pv-protection name: pvc-c77cc8ca-417b-11e9-b724-06eba9b82fb0 resourceVersion: "27119" selfLink: /api/v1/persistentvolumes/pvc-c77cc8ca-417b-11e9-b724-06eba9b82fb0 uid: c784b82b-417b-11e9-ad72-025712d788ba spec: accessModes: - ReadWriteOnce capacity: storage: 1Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: pvc namespace: default resourceVersion: "27115" uid: c77cc8ca-417b-11e9-b724-06eba9b82fb0 csi: driver: csi-hostpath fsType: ext4 volumeAttributes: storage.kubernetes.io/csiProvisionerIdentity: 1552033385985-8081-csi-hostpath volumeHandle: c783eaff-417b-11e9-8077-0a580a820018 persistentVolumeReclaimPolicy: Delete storageClassName: sc1 status: phase: Bound $ oc get pods NAME READY STATUS RESTARTS AGE hostpath-controller-74dd68c85b-hmqj6 4/4 Running 0 14m hostpath-node-ddd22 3/3 Running 0 14m podtest 0/1 ContainerCreating 0 18s $ oc describe pods podtest Name: podtest Namespace: default Priority: 0 PriorityClassName: <none> Node: ip-172-31-135-188.us-east-2.compute.internal/172.31.135.188 Start Time: Fri, 08 Mar 2019 16:37:28 +0800 Labels: <none> Annotations: openshift.io/scc: anyuid Status: Pending IP: Containers: myfrontend: Container ID: Image: jhou/hello-openshift Image ID: Port: 80/TCP Host Port: 0/TCP State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <none> Mounts: /mnt/hostpath from hostpath (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-sfjnj (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: hostpath: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: pvc ReadOnly: false default-token-sfjnj: Type: Secret (a volume populated by a Secret) SecretName: default-token-sfjnj Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 67s default-scheduler Successfully assigned default/podtest to ip-172-31-135-188.us-east-2.compute.internal Normal SuccessfulAttachVolume 67s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-c77cc8ca-417b-11e9-b724-06eba9b82fb0" Warning FailedMount 27s (x7 over 59s) kubelet, ip-172-31-135-188.us-east-2.compute.internal MountVolume.MountDevice failed for volume "pvc-c77cc8ca-417b-11e9-b724-06eba9b82fb0" : driver name csi-hostpath not found in the list of registered CSI drivers
We don't support CSI operator