Bug 1837971
| Summary: | iscsi doesn't support ROX well | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Qin Ping <piqin> |
| Component: | Storage | Assignee: | Jan Safranek <jsafrane> |
| Storage sub component: | Kubernetes | QA Contact: | Qin Ping <piqin> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | low | ||
| Priority: | unspecified | CC: | aos-bugs, jsafrane |
| Version: | 4.5 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.6.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-10-27 16:00:21 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
It looks like that kubelet actually stores its iscsi.json file into the mounted volume! And in case the volume is read-only, it can't store the data there at all. This is wrong, kubelet should not touch data on the volume at all. Upstream fix: https://github.com/kubernetes/kubernetes/pull/91738 The upstream fix needs some updates. Waiting for 1.19 rebase to land. Rebase has landed. In this case, please check that you have kubelet 1.19 (release candidate is OK). verified with: 4.6.0-0.nightly-2020-08-04-002217 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 |
Description of problem: When creating 2 inline volume pods using the same iscs volume withe readOnly=false, the second pod can not run for "MountVolume.WaitForAttach failed for volume "iscsi" : iscsi: create /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-172.30.63.54:3260:iscsi/172.30.63.54:3260-iqn.2016-04.test.com:storage.target00-lun-0/iscsi.json err open /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-172.30.63.54:3260:iscsi/172.30.63.54:3260-iqn.2016-04.test.com:storage.target00-lun-0/iscsi.json: read-only file system" Version-Release number of selected component (if applicable): 4.5.0-0.nightly-2020-05-19-041951 How reproducible: Always Steps to Reproduce: 1. Create a inline iscsi volume pod with readOnly=false. 2. Delete this pod 3. Create 2 inline volume pods using the same iscsi volume with readOnly=true. Actual results: The second readonly pod did not run successfully, and report err msg: "MountVolume.WaitForAttach failed for volume "iscsi" : iscsi: create /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-172.30.63.54:3260:iscsi/172.30.63.54:3260-iqn.2016-04.test.com:storage.target00-lun-0/iscsi.json err open /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-172.30.63.54:3260:iscsi/172.30.63.54:3260-iqn.2016-04.test.com:storage.target00-lun-0/iscsi.json: read-only file system" Expected results: The second readonly pod runs successfully. Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info: Inline volume pod we used: { "apiVersion": "v1", "kind": "Pod", "metadata": { "name": "iscsi" }, "spec": { "containers": [ { "name": "iscsi", "image": "quay.io/openshifttest/storage@sha256:a05b96d373be86f46e76817487027a7f5b8b5f87c0ac18a246b018df11529b40", "securityContext": { "privileged": true }, "imagePullPolicy": "IfNotPresent", "volumeMounts": [ { "mountPath": "/mnt/iscsi", "name": "iscsi" } ] } ], "volumes": [ { "name": "iscsi", "iscsi": { "targetPortal": "#POD_IP#:3260", "iqn": "iqn.2016-04.test.com:storage.target00", "lun": 0, "iface": "default", "fsType": "ext4", "readOnly": true } } ] } }