Bug 1568028 - iSCSI PersistentVolumes in VolumeMode=Block are not mapped to container
Summary: iSCSI PersistentVolumes in VolumeMode=Block are not mapped to container
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: hchen
QA Contact: Qin Ping
URL:
Whiteboard:
Depends On:
Blocks: 1581614
TreeView+ depends on / blocked
 
Reported: 2018-04-16 15:57 UTC by sgott
Modified: 2018-05-24 00:30 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1581614 (view as bug list)
Environment:
Last Closed: 2018-04-24 15:11:22 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description sgott 2018-04-16 15:57:14 UTC
Description of problem:
Mapping a PersistentVolume backed by an iSCSI device does not get mounted inside the container if the PersistentVolume is using VolumeMode=Block.

Version-Release number of selected component (if applicable):

How reproducible:
Always

Steps to Reproduce:
1.
Create a PersistentVolume, PersistentVolumeClaim and Pod,

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: iscsi-disk-cirros-block
  labels:
    kubevirt.io: ""
    os: "cirros-block"
spec:
  capacity:
    storage: 40Gi
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  iscsi:
    iqn: iqn.2017-01.io.kubevirt:sn.42
    lun: 3
    targetPortal: "127.0.0.1"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: disk-cirros-block
  labels:
    kubevirt.io: ""
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 40Gi
  selector:
    matchLabels:
      os: "cirros-block"
---
apiVersion: v1
kind: Pod
metadata:
  name: test-volume-mode
  namespace: default
spec:
  containers:
  - command:
    - sleep
    - "360000"
    image: alpine
    imagePullPolicy: Always
    securityContext:
      privileged: true
    name: busybox
    volumeDevices:
    - devicePath: /dev/custom
      name: pvcvolume
  dnsPolicy: ClusterFirst
  hostNetwork: true
  restartPolicy: Always
  schedulerName: default-scheduler
  serviceAccount: default
  serviceAccountName: default
  volumes:
    - name: pvcvolume
      persistentVolumeClaim:
        claimName: disk-cirros-block

2. oc exec -it test-volume-mode -- /bin/sh
3. ls -lsa /dev

Actual results:
/dev/custom is not present

Expected results:
/dev/custom should be present.

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):
Target 1: iqn.2017-01.io.kubevirt:sn.42        
    System information:                        
        Driver: iscsi                          
        State: ready                           
    I_T nexus information:                     
    LUN information:                           
        LUN: 0                                 
            Type: controller                                                                   
            SCSI ID: IET     00010000          
            SCSI SN: beaf10                    
            Size: 0 MB, Block size: 1          
            Online: Yes                        
            Removable media: No                
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
        ...
        LUN: 3                                 
            Type: disk                         
            SCSI ID: IET     00010003          
            SCSI SN: beaf13                    
            Size: 13 MB, Block size: 512       
            Online: Yes                        
            Removable media: No                
            Prevent removal: No                
            Readonly: No                       
            SWP: No                            
            Thin-provisioning: Yes             
            Backing store type: rdwr           
            Backing store path: /volumes/2-cirros.img                                          
            Backing store flags:               
    Account information:                       
    ACL information:                           
        ALL 

2-cirros.img is a raw disk image. any image would expose this issue however.

Additional info:
BlockVolumes are alpha in Kubernetes 1.9/1.10.

This issue occurs whether or not the pod is executed in privileged mode. (.securityContext.privileged=true/false)

This upstream PR may be related:
https://github.com/moby/moby/pull/36258

Comment 1 Fabian Deutsch 2018-04-16 18:15:33 UTC
The impact to a user is that a user is only able to use file-system volumes for disk storage.
This means that existing SANs can only be used by using file-system mode, which leads to worse performance characteristics.

Comment 2 Masaki Kimura (Hitachi) 2018-04-16 20:51:41 UTC
I tested iscsi with latest upstream k8s on Fedora 25 by using local-up-cluster.sh. 
If I passed FEATURE_GATES=BlockVolume=true as environment variable, it worked as expected.
And without passing it, I see the similar behaviour that there's no /dev/custom in the container.

[w/ FEATURE_GATES=BlockVolume=true]
# FEATURE_GATES=BlockVolume=true hack/local-up-cluster.sh  
% k create -f test-volume-mode-nonpriv.yaml 
# docker exec -it f3b67ffa723b ls -lsa /dev/custom
     0 brw-rw----    1 root     disk        8,  16 Apr 16 19:51 /dev/custom

[w/o FEATURE_GATES=BlockVolume=true]
# hack/local-up-cluster.sh
% k create -f test-volume-mode-nonpriv.yaml 
# docker exec -it 40624d66b480 ls -lsa /dev/custom
ls: /dev/custom: No such file or directory


I haven't tested it with OpenShift, but I guess that we need to 
test it with enabling BlockVolume by the similar way to the below URL.
(By adding BlockVolume=true to feature-gates in /etc/origin/node/node-config.yaml and /etc/origin/master/master-config.yaml.)

https://docs.openshift.com/container-platform/3.9/dev_guide/expanding_persistent_volumes.html

Comment 3 sgott 2018-04-16 21:51:11 UTC
By "latest" do you mean latest release or head of git tree?

https://github.com/kubernetes/kubernetes/pull/61549

This PR may have an effect on what you see, but I didn't think that made it into 1.10.

Comment 4 sgott 2018-04-17 15:16:53 UTC
Also, to be clear, I was aware that I need to pass --feature-gates=BlockVolume=true on the kube-apiserver command line, and can confirm I did that, which is why I'm wondering about what version you tested against.

Comment 5 Scott Creeley 2018-04-17 20:50:03 UTC
@sgott - I think you need to pass that feature-gate arg into the kubelet not the api server - as kubelet arg.  Also this needs to be done on master and all nodes.

Comment 6 Scott Creeley 2018-04-17 20:54:19 UTC
(In reply to Scott Creeley from comment #5)
> @sgott - I think you need to pass that feature-gate arg into the kubelet not
> the api server - as kubelet arg.  Also this needs to be done on master and
> all nodes.

sorry, to be clear, you correctly passed in the feature gate to the master api server, but you also need to pass into the nodes kubelet args and restart each node as well.

Comment 7 sgott 2018-04-17 22:08:14 UTC
I have added the feature-gates argument to kubelet, but no change.

What version are you testing against?

Comment 8 Qin Ping 2018-04-18 01:35:30 UTC
Seems OCP 3.9 not support using iscsi volume with block volumetype.

Comment 9 Scott Creeley 2018-04-18 04:03:40 UTC
hmmm...so it seems that is correct, if I look at origin 3.9 release, there is no block support in iSCSI.  It's in master now but not in the 3.9 release and if I look in Kube, it's in the 1.10 release but not the 1.9 release.

Comment 10 Mitsuhiro Tanino 2018-04-19 00:05:02 UTC
Hi All,

>>It's in master now but not in the 3.9 release and if I look in Kube,
>>it's in the 1.10 release but not the 1.9 release.

Yes, this is correct. We didn't have enough time to merge iSCSI block volume support before code freeze of Kubernetes v1.9. And then we got merged iSCSI(and also ceph RBD) at early phase of v1.10 release cycle.

Thanks,
Mitsuhiro

Comment 11 Masaki Kimura (Hitachi) 2018-04-19 20:35:50 UTC
Let me share my test results to make current situation a bit clearer.
In OpenShift3.9, if feature gate enabled, at least FC with non-privileged works,
however, for iSCSI with privileged, we need to wait for both (*1) and (*2).

+------------------------------------+---------+
|             Conditions             | Result  |
+-------------- -------- ------------+         |
|Software/Ver.   Plugin   privileged |         |
+-------------- -------- ------------+---------+
 k8s/1.10        FC        false       OK
 k8s/1.10        iSCSI     false       OK
 OpenShift/3.9   FC        false       OK
 OpenShift/3.9   FC        true        FAIL (*1)
 OpenShift/3.9   iSCSI     false       FAIL (*2)
--------------- -------- ------------ ---------

(*1) For privileged mode with volumeMode=Block, we may need to wait
     below PR to be merged in moby side. 
     - devicePath for block volumes is not honored #62560
       https://github.com/kubernetes/kubernetes/issues/62560
       => fix issue: make --device works at privileged mode #36258
          https://github.com/moby/moby/pull/36258

(*2) For iSCSI volumeMode=Block support, we may need to wait for 3.10. 
     Because below PR is merged in k8s 1.10.
     - Block volumes Support: iSCSI plugin update #54752
       https://github.com/kubernetes/kubernetes/pull/54752

Comment 13 Fabian Deutsch 2018-05-23 10:15:53 UTC
The current documentation does not tell anything about supported storage backends

Can this be added to the official documentation?

https://docs.openshift.com/container-platform/3.9/architecture/additional_concepts/storage.html#block-volume-support

Comment 14 Yoshiki Ohmura 2018-05-23 16:28:08 UTC
(In reply to Fabian Deutsch from comment #13)
> The current documentation does not tell anything about supported storage
> backends
> 
> Can this be added to the official documentation?
> 
> https://docs.openshift.com/container-platform/3.9/architecture/
> additional_concepts/storage.html#block-volume-support

Hi Fabian,

I'm EPM for Hitachi, and Masaki Kimura is working as Hitachi onsite engineer, so he won't have answer about official documentation.

I believe we'd better to ask it to openshift-sme instead of asking Masaki on this CLOSED BZ.

Since I'm writing this comment just because I got request from Masaki, so I may miss some background why you're asking Masaki about the documentation here.
So If I missed your point/background, please let me know.

Comment 15 Qin Ping 2018-05-24 00:30:13 UTC
Hi Fabian,

In openshift 3.9, we only support FC block volume. Now we are testing bolck volume for openshift 3.10, and then we'll update the documentation.


Note You need to log in before you can comment on or make changes to this bug.