Description of problem: When creating 2 inline volume pods using the same iscs volume withe readOnly=false, the second pod can not run for "MountVolume.WaitForAttach failed for volume "iscsi" : iscsi: create /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-172.30.63.54:3260:iscsi/172.30.63.54:3260-iqn.2016-04.test.com:storage.target00-lun-0/iscsi.json err open /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-172.30.63.54:3260:iscsi/172.30.63.54:3260-iqn.2016-04.test.com:storage.target00-lun-0/iscsi.json: read-only file system" Version-Release number of selected component (if applicable): 4.5.0-0.nightly-2020-05-19-041951 How reproducible: Always Steps to Reproduce: 1. Create a inline iscsi volume pod with readOnly=false. 2. Delete this pod 3. Create 2 inline volume pods using the same iscsi volume with readOnly=true. Actual results: The second readonly pod did not run successfully, and report err msg: "MountVolume.WaitForAttach failed for volume "iscsi" : iscsi: create /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-172.30.63.54:3260:iscsi/172.30.63.54:3260-iqn.2016-04.test.com:storage.target00-lun-0/iscsi.json err open /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-172.30.63.54:3260:iscsi/172.30.63.54:3260-iqn.2016-04.test.com:storage.target00-lun-0/iscsi.json: read-only file system" Expected results: The second readonly pod runs successfully. Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info: Inline volume pod we used: { "apiVersion": "v1", "kind": "Pod", "metadata": { "name": "iscsi" }, "spec": { "containers": [ { "name": "iscsi", "image": "quay.io/openshifttest/storage@sha256:a05b96d373be86f46e76817487027a7f5b8b5f87c0ac18a246b018df11529b40", "securityContext": { "privileged": true }, "imagePullPolicy": "IfNotPresent", "volumeMounts": [ { "mountPath": "/mnt/iscsi", "name": "iscsi" } ] } ], "volumes": [ { "name": "iscsi", "iscsi": { "targetPortal": "#POD_IP#:3260", "iqn": "iqn.2016-04.test.com:storage.target00", "lun": 0, "iface": "default", "fsType": "ext4", "readOnly": true } } ] } }
It looks like that kubelet actually stores its iscsi.json file into the mounted volume! And in case the volume is read-only, it can't store the data there at all. This is wrong, kubelet should not touch data on the volume at all. Upstream fix: https://github.com/kubernetes/kubernetes/pull/91738
The upstream fix needs some updates.
Waiting for 1.19 rebase to land.
Rebase has landed. In this case, please check that you have kubelet 1.19 (release candidate is OK).
verified with: 4.6.0-0.nightly-2020-08-04-002217
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196