Hide Forgot
Description of problem: Mapping a PersistentVolume backed by an iSCSI device does not get mounted inside the container if the PersistentVolume is using VolumeMode=Block. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Create a PersistentVolume, PersistentVolumeClaim and Pod, --- apiVersion: v1 kind: PersistentVolume metadata: name: iscsi-disk-cirros-block labels: kubevirt.io: "" os: "cirros-block" spec: capacity: storage: 40Gi accessModes: - ReadWriteOnce volumeMode: Block iscsi: iqn: iqn.2017-01.io.kubevirt:sn.42 lun: 3 targetPortal: "127.0.0.1" --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: disk-cirros-block labels: kubevirt.io: "" spec: storageClassName: "" accessModes: - ReadWriteOnce volumeMode: Block resources: requests: storage: 40Gi selector: matchLabels: os: "cirros-block" --- apiVersion: v1 kind: Pod metadata: name: test-volume-mode namespace: default spec: containers: - command: - sleep - "360000" image: alpine imagePullPolicy: Always securityContext: privileged: true name: busybox volumeDevices: - devicePath: /dev/custom name: pvcvolume dnsPolicy: ClusterFirst hostNetwork: true restartPolicy: Always schedulerName: default-scheduler serviceAccount: default serviceAccountName: default volumes: - name: pvcvolume persistentVolumeClaim: claimName: disk-cirros-block 2. oc exec -it test-volume-mode -- /bin/sh 3. ls -lsa /dev Actual results: /dev/custom is not present Expected results: /dev/custom should be present. Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Target 1: iqn.2017-01.io.kubevirt:sn.42 System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No SWP: No Thin-provisioning: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 ... LUN: 3 Type: disk SCSI ID: IET 00010003 SCSI SN: beaf13 Size: 13 MB, Block size: 512 Online: Yes Removable media: No Prevent removal: No Readonly: No SWP: No Thin-provisioning: Yes Backing store type: rdwr Backing store path: /volumes/2-cirros.img Backing store flags: Account information: ACL information: ALL 2-cirros.img is a raw disk image. any image would expose this issue however. Additional info: BlockVolumes are alpha in Kubernetes 1.9/1.10. This issue occurs whether or not the pod is executed in privileged mode. (.securityContext.privileged=true/false) This upstream PR may be related: https://github.com/moby/moby/pull/36258
The impact to a user is that a user is only able to use file-system volumes for disk storage. This means that existing SANs can only be used by using file-system mode, which leads to worse performance characteristics.
I tested iscsi with latest upstream k8s on Fedora 25 by using local-up-cluster.sh. If I passed FEATURE_GATES=BlockVolume=true as environment variable, it worked as expected. And without passing it, I see the similar behaviour that there's no /dev/custom in the container. [w/ FEATURE_GATES=BlockVolume=true] # FEATURE_GATES=BlockVolume=true hack/local-up-cluster.sh % k create -f test-volume-mode-nonpriv.yaml # docker exec -it f3b67ffa723b ls -lsa /dev/custom 0 brw-rw---- 1 root disk 8, 16 Apr 16 19:51 /dev/custom [w/o FEATURE_GATES=BlockVolume=true] # hack/local-up-cluster.sh % k create -f test-volume-mode-nonpriv.yaml # docker exec -it 40624d66b480 ls -lsa /dev/custom ls: /dev/custom: No such file or directory I haven't tested it with OpenShift, but I guess that we need to test it with enabling BlockVolume by the similar way to the below URL. (By adding BlockVolume=true to feature-gates in /etc/origin/node/node-config.yaml and /etc/origin/master/master-config.yaml.) https://docs.openshift.com/container-platform/3.9/dev_guide/expanding_persistent_volumes.html
By "latest" do you mean latest release or head of git tree? https://github.com/kubernetes/kubernetes/pull/61549 This PR may have an effect on what you see, but I didn't think that made it into 1.10.
Also, to be clear, I was aware that I need to pass --feature-gates=BlockVolume=true on the kube-apiserver command line, and can confirm I did that, which is why I'm wondering about what version you tested against.
@sgott - I think you need to pass that feature-gate arg into the kubelet not the api server - as kubelet arg. Also this needs to be done on master and all nodes.
(In reply to Scott Creeley from comment #5) > @sgott - I think you need to pass that feature-gate arg into the kubelet not > the api server - as kubelet arg. Also this needs to be done on master and > all nodes. sorry, to be clear, you correctly passed in the feature gate to the master api server, but you also need to pass into the nodes kubelet args and restart each node as well.
I have added the feature-gates argument to kubelet, but no change. What version are you testing against?
Seems OCP 3.9 not support using iscsi volume with block volumetype.
hmmm...so it seems that is correct, if I look at origin 3.9 release, there is no block support in iSCSI. It's in master now but not in the 3.9 release and if I look in Kube, it's in the 1.10 release but not the 1.9 release.
Hi All, >>It's in master now but not in the 3.9 release and if I look in Kube, >>it's in the 1.10 release but not the 1.9 release. Yes, this is correct. We didn't have enough time to merge iSCSI block volume support before code freeze of Kubernetes v1.9. And then we got merged iSCSI(and also ceph RBD) at early phase of v1.10 release cycle. Thanks, Mitsuhiro
Let me share my test results to make current situation a bit clearer. In OpenShift3.9, if feature gate enabled, at least FC with non-privileged works, however, for iSCSI with privileged, we need to wait for both (*1) and (*2). +------------------------------------+---------+ | Conditions | Result | +-------------- -------- ------------+ | |Software/Ver. Plugin privileged | | +-------------- -------- ------------+---------+ k8s/1.10 FC false OK k8s/1.10 iSCSI false OK OpenShift/3.9 FC false OK OpenShift/3.9 FC true FAIL (*1) OpenShift/3.9 iSCSI false FAIL (*2) --------------- -------- ------------ --------- (*1) For privileged mode with volumeMode=Block, we may need to wait below PR to be merged in moby side. - devicePath for block volumes is not honored #62560 https://github.com/kubernetes/kubernetes/issues/62560 => fix issue: make --device works at privileged mode #36258 https://github.com/moby/moby/pull/36258 (*2) For iSCSI volumeMode=Block support, we may need to wait for 3.10. Because below PR is merged in k8s 1.10. - Block volumes Support: iSCSI plugin update #54752 https://github.com/kubernetes/kubernetes/pull/54752
The current documentation does not tell anything about supported storage backends Can this be added to the official documentation? https://docs.openshift.com/container-platform/3.9/architecture/additional_concepts/storage.html#block-volume-support
(In reply to Fabian Deutsch from comment #13) > The current documentation does not tell anything about supported storage > backends > > Can this be added to the official documentation? > > https://docs.openshift.com/container-platform/3.9/architecture/ > additional_concepts/storage.html#block-volume-support Hi Fabian, I'm EPM for Hitachi, and Masaki Kimura is working as Hitachi onsite engineer, so he won't have answer about official documentation. I believe we'd better to ask it to openshift-sme@redhat.com instead of asking Masaki on this CLOSED BZ. Since I'm writing this comment just because I got request from Masaki, so I may miss some background why you're asking Masaki about the documentation here. So If I missed your point/background, please let me know.
Hi Fabian, In openshift 3.9, we only support FC block volume. Now we are testing bolck volume for openshift 3.10, and then we'll update the documentation.