Bug 1632418

Summary: iSCSI doesn't give enough time multipathd to create multipath device
Product: OpenShift Container Platform Reporter: Ryan Howe <rhowe>
Component: StorageAssignee: Jan Safranek <jsafrane>
Status: CLOSED ERRATA QA Contact: Liang Xia <lxia>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.10.0CC: aos-bugs, aos-storage-staff, bbennett, bchilds, jhou, lxia, per.carlson, vlaad, wsun
Target Milestone: ---   
Target Release: 3.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-11 16:39:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ryan Howe 2018-09-24 18:17:37 UTC
Description of problem:

When mounting iSCSI PV kubelet doesn't give multipathd enough time to create multipath device and fallbacks to mounting single SCSI device. 

Version-Release number of selected component (if applicable):
3.9
3.10

How reproducible:
Intermittent 

Steps to Reproduce:
1. Create a PV with 2 or more portals
2. Schedule to node. 

Actual results:
Node used  individual path created for device as it took to long for multipathd to construct a multipath device.

Expected results:
For OpenShift to allow enough time so that the multipath device to be created and used. If fail to use multipath log accordingly 

Additional info:
 Bug/Issue has been resolved upstream 1.12+ 

  Upstream Issue: https://github.com/kubernetes/kubernetes/issues/60894
  PR: https://github.com/kubernetes/kubernetes/pull/67140

Comment 3 Jan Safranek 2018-10-09 07:55:54 UTC
3.10.z PR: https://github.com/openshift/ose/pull/1431

Comment 7 Jianwei Hou 2018-10-22 16:05:47 UTC
Verified on v3.10.57

With PV having 2+ portals, the multipath device is used.

 mount|grep iscsi
/dev/mapper/mpatha on /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.73.131.159:3260-iqn.2016-12.org.gluster-block:f6cc4639-4efc-4ef4-9fa2-09bdae6e8be8-lun-0 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/mpatha on /var/lib/origin/openshift.local.volumes/pods/f9b52515-d611-11e8-a30f-fa163e85866a/volumes/kubernetes.io~iscsi/pvc-7f7ee170-d60f-11e8-8de7-fa163e85866a type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/mpathb on /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.73.131.159:3260-iqn.2016-12.org.gluster-block:413cd00f-8bf8-4858-bc93-3843c7717b27-lun-0 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/mpathb on /var/lib/origin/openshift.local.volumes/pods/5fedde0b-d612-11e8-a30f-fa163e85866a/volumes/kubernetes.io~iscsi/pvc-54c95b8c-d612-11e8-a30f-fa163e85866a type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
[root@qe-lxia-310-node-1 ~]# mount|grep iscsi
/dev/mapper/mpatha on /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.73.131.159:3260-iqn.2016-12.org.gluster-block:f6cc4639-4efc-4ef4-9fa2-09bdae6e8be8-lun-0 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/mpatha on /var/lib/origin/openshift.local.volumes/pods/f9b52515-d611-11e8-a30f-fa163e85866a/volumes/kubernetes.io~iscsi/pvc-7f7ee170-d60f-11e8-8de7-fa163e85866a type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/mpathb on /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.73.131.159:3260-iqn.2016-12.org.gluster-block:413cd00f-8bf8-4858-bc93-3843c7717b27-lun-0 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/mpathb on /var/lib/origin/openshift.local.volumes/pods/5fedde0b-d612-11e8-a30f-fa163e85866a/volumes/kubernetes.io~iscsi/pvc-54c95b8c-d612-11e8-a30f-fa163e85866a type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/mpathc on /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.73.131.159:3260-iqn.2016-12.org.gluster-block:177817b4-f1e4-4b71-b261-f0ae67d6b87a-lun-0 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/mpathc on /var/lib/origin/openshift.local.volumes/pods/0598a069-d613-11e8-a30f-fa163e85866a/volumes/kubernetes.io~iscsi/pvc-fc52dc4b-d612-11e8-a30f-fa163e85866a type xfs (rw,relatime,seclabel,attr2,inode64,noquota)



multipath -ll
Oct 22 12:04:03 | sdh: alua not supported
Oct 22 12:04:04 | sdi: alua not supported
mpathc (36001405177817b4f1e44b71b261f0ae6) dm-3 LIO-ORG ,TCMU device
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 12:0:0:0 sdg 8:96  active ready running
|-+- policy='round-robin 0' prio=-1 status=enabled
| `- 13:0:0:0 sdh 8:112 active ready running
`-+- policy='round-robin 0' prio=-1 status=enabled
  `- 14:0:0:0 sdi 8:128 active ready running
Oct 22 12:04:06 | sde: alua not supported
Oct 22 12:04:07 | sdf: alua not supported
mpathb (36001405413cd00f8bf84858bc933843c) dm-2 LIO-ORG ,TCMU device
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 9:0:0:0  sdd 8:48  active ready running
|-+- policy='round-robin 0' prio=-1 status=enabled
| `- 10:0:0:0 sde 8:64  active ready running
`-+- policy='round-robin 0' prio=-1 status=enabled
  `- 11:0:0:0 sdf 8:80  active ready running
Oct 22 12:04:09 | sdb: alua not supported
Oct 22 12:04:10 | sdc: alua not supported
mpatha (36001405f6cc46394efc4ef49fa209bda) dm-1 LIO-ORG ,TCMU device
size=1.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| `- 6:0:0:0  sda 8:0   active ready running
|-+- policy='round-robin 0' prio=-1 status=enabled
| `- 7:0:0:0  sdb 8:16  active ready running
`-+- policy='round-robin 0' prio=-1 status=enabled
  `- 8:0:0:0  sdc 8:32  active ready running

Comment 9 errata-xmlrpc 2018-11-11 16:39:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2709