Bug 1837971 - iscsi doesn't support ROX well
Summary: iscsi doesn't support ROX well
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.5
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 4.6.0
Assignee: Jan Safranek
QA Contact: Qin Ping
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-20 10:14 UTC by Qin Ping
Modified: 2020-10-27 16:00 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:00:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:00:50 UTC

Description Qin Ping 2020-05-20 10:14:15 UTC
Description of problem:
When creating 2 inline volume pods using the same iscs volume withe readOnly=false, the second pod can not run for "MountVolume.WaitForAttach failed for volume "iscsi" : iscsi: create /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-172.30.63.54:3260:iscsi/172.30.63.54:3260-iqn.2016-04.test.com:storage.target00-lun-0/iscsi.json err open /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-172.30.63.54:3260:iscsi/172.30.63.54:3260-iqn.2016-04.test.com:storage.target00-lun-0/iscsi.json: read-only file system"

Version-Release number of selected component (if applicable):
4.5.0-0.nightly-2020-05-19-041951

How reproducible:
Always

Steps to Reproduce:
1. Create a inline iscsi volume pod with readOnly=false.
2. Delete this pod
3. Create 2 inline volume pods using the same iscsi volume with readOnly=true.


Actual results:
The second readonly pod did not run successfully, and report err msg:
"MountVolume.WaitForAttach failed for volume "iscsi" : iscsi: create /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-172.30.63.54:3260:iscsi/172.30.63.54:3260-iqn.2016-04.test.com:storage.target00-lun-0/iscsi.json err open /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-172.30.63.54:3260:iscsi/172.30.63.54:3260-iqn.2016-04.test.com:storage.target00-lun-0/iscsi.json: read-only file system"

Expected results:
The second readonly pod runs successfully.

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
Inline volume pod we used:
{
    "apiVersion": "v1",
    "kind": "Pod",
    "metadata": {
        "name": "iscsi"
    },
    "spec": {
        "containers": [
            {
                "name": "iscsi",
                "image": "quay.io/openshifttest/storage@sha256:a05b96d373be86f46e76817487027a7f5b8b5f87c0ac18a246b018df11529b40",
                "securityContext": {
                    "privileged": true
                },
                "imagePullPolicy": "IfNotPresent",
                "volumeMounts": [
                    {
                        "mountPath": "/mnt/iscsi",
                        "name": "iscsi"
                    }
                ]
            }
        ],
        "volumes": [
            {
                "name": "iscsi",
                "iscsi": {
                    "targetPortal": "#POD_IP#:3260",
                    "iqn": "iqn.2016-04.test.com:storage.target00",
                    "lun": 0,
                    "iface": "default",
                    "fsType": "ext4",
                    "readOnly": true
                }
            }
        ]
    }
}

Comment 1 Jan Safranek 2020-06-08 07:41:15 UTC
It looks like that kubelet actually stores its iscsi.json file into the mounted volume! And in case the volume is read-only, it can't store the data there at all. This is wrong, kubelet should not touch data on the volume at all.


Upstream fix: https://github.com/kubernetes/kubernetes/pull/91738

Comment 2 Jan Safranek 2020-06-17 09:33:17 UTC
The upstream fix needs some updates.

Comment 3 Jan Safranek 2020-07-08 08:31:48 UTC
Waiting for 1.19 rebase to land.

Comment 4 Jan Safranek 2020-07-29 16:58:49 UTC
Rebase has landed. In this case, please check that you have kubelet 1.19 (release candidate is OK).

Comment 7 Qin Ping 2020-08-04 06:44:48 UTC
verified with: 4.6.0-0.nightly-2020-08-04-002217

Comment 9 errata-xmlrpc 2020-10-27 16:00:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.