Bug 1609703

Description of problem:
APP pod unable to start after target port failure in cases where single paths are mounted on APP pods(BZ#1599742)

This issue is related and a side-effect of  https://bugzilla.redhat.com/show_bug.cgi?id=1599742. When app pods get bind-mounted with single devices instead of mpath, after a path failure, APP pod fails to get restored. This issue wont be seen if we have mpath devices mounted instead of single paths. 

Description
==============

In some cases, it is seen that while creating app pods with bind-mounted block devices, no mpath device name is created against the device on the initiator side(https://bugzilla.redhat.com/show_bug.cgi?id=1599742).
Due to this, instead of a mpath device(/dev/mapper/mpatha, etc) getting mounted , single devices (/dev/sdk, etc) get mounted on the app pod. Since single paths are mounted, if somehow the said path goes down, the app pod goes into CrashLoopBackOff state(which is expected). But once the path is restored and the app pod is re-created, the mount path on initiator is still showing input/output error.

Note: multipath service was up and running on all the initiator nodes

We see following 2 issues based on 2 different scenarios :
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Scenario- 1. Once path is restored, the app pod tries to re-spin on the same initiator node
*********************************************************************************************
In this case, when we delete the app pod so as to make it use the restored path again, the app pod creation is stuck with following error message

Sequence of events
====================
1. The app pod is in Running state on Node-A and has bind-mounted path - /dev/sdj as no mpath device is created for /dev/sdj(belonging to target-X)
2. Failed path sdj by blocking the 3260 port in its target portal IP-X
3. The app pod goes into CrashLoopBackState
4. The path is restored to target-X. All other devices which had mpath, show the transition from Failed->Active state. /dev/sdj doesnt have mpath, hence couldn't validate the same using multipath -ll
5. TO make the pod use the restored path again, deleted the app pod. With replica=1, the app pod tried to re-spin, this time on the same initiator node.
6. The app pod stays in ContainerCreating state and oc describe shows following error message:

------------------------------------------------------ 
Warning  FailedMount  31s (x7 over 1m)  kubelet, dhcp46-177.lab.eng.blr.redhat.com  MountVolume.WaitForAttach failed for volume "pvc-0cb7c180-92e9-11e8-8e5c-005056a5f3b5" : Heuristic determination of mount point failed:stat /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/iscsi/iface-default/10.70.46.172:3260-iqn.2016-12.org.gluster-block:4b9d7ead-83fc-41a9-9c94-83e2244fc1b4-lun-0: input/output error
------------------------------------------------------ 
7. The app pod is still in the same state, and has had x340 re-tries over 12h. The iscsiadm login to the iqn is also intact. The mount point is input/output error


Scenario- 2. Once path is restored, the app pod tries to re-spin on a different initiator node
***********************************************************************************************

In this case, the app pod tries to re-spin on another node. 

Sequence of events
====================
1. The app pod is running on Node-A and has bind-mounted path - /dev/sdm (no mpath device created, BZ#1599742)
2. Failed path /dev/sdm by blocking the 3260 port in its target portal IP-X
3. The app pod goes into CrashLoopBackState as expected. The mount path in initiator becomes input/output error.
4. The path is restored to target-X. All other devices which had mpath, show the transition from Failed->Active state. /dev/sdm doesnt have mpath, hence couldn't validate the same using multipath -ll
5. TO make the pod use the restored path again, deleted the app pod. With replica=1, the app pod tried to re-spin, this time on a different initiator node-B.
6. The app pod stays in ContainerCreating state for some time - issue reported:

---------------------------------------

  Warning  FailedAttachVolume  3m    attachdetach-controller                    Multi-Attach error for volume "pvc-857a1bd1-93b5-11e8-8e5c-005056a5f3b5" Volume is already exclusively attached to one node and can't be attached to another
  Warning  FailedMount         1m    kubelet, dhcp47-62.lab.eng.blr.redhat.com  Unable to mount volumes for pod "mongo-3-1-9d77w_glusterfs(16db0ded-93bd-11e8-8e5c-005056a5f3b5)": timeout expired waiting for volumes to attach or mount for pod "glusterfs"/"mongo-3-1-9d77w". list of unmounted volumes=[mongo-3-data]. list of unattached volumes=[mongo-3-data default-token-dhnj5]

---------------------------------------

7. After some time, the app pod does get successfully created on the other node.

8. But it is seen that the iqn is still logged in from Node-A. Maybe the logout would have failed due to the mount path being input/output error.

9. Thus the same IQN is logged in from 2 initiator nodes which is not a desirable case.



More details are attached in subsequent comments.



Version-Release number of selected component (if applicable):
# oc version
oc v3.10.14

heketi-7.0.0-5.el7rhgs.x86_64
Glusterfs= 3.8.4-54.15
gluster-block-0.2.1-23.el7rhgs.x86_64

How reproducible:
++++++++++++++++++++++
2x2

Steps to Reproduce:
++++++++++++++++++
1. Create APP pods with bind-mounted block pvcs(HA=2/HA=3) 
2. Due to BZ#1599742 , it may happen that for some APP pods, the mount path is a single device instead of device mapper name
E.g. 
# oc rsh mongo-2-1-5hvxq df -kh|grep mongo
/dev/sdj                                                                                            1.0G  236M  779M    24% /var/lib/mongodb/data

3. Now bring down the target which services path, say above /dev/sdj path - 10.70.46.172 by rejecting 3260 in iptables.
lrwxrwxrwx. 1 root root 9 Jul 29 10:07 /dev/disk/by-path/ip-10.70.46.172:3260-iscsi-iqn.2016-12.org.gluster-block:4b9d7ead-83fc-41a9-9c94-83e2244fc1b4-lun-0 -> ../../sdj

4. The app pod should crash as expected , since /dev/sdj was mounted as a single path

5. Bring up the path and deleted the pod.

6. Check if the pod comes up on the same initiator or a different one and validate scenarios #1 and #2 above.


Actual results:
+++++++++++++++++++
1. Single path is getting mounted which is unacceptable as no HA is involved.
2. When the app pod comes up on the same initiator, the app pod fails to get re-created. The mount path remains showing i/o error. The umount seems to be unsuccessful.
3. When the app pod comes up on a different initiator, the app pod does get re-created. But the mount path still stays behind in initial node(with  i/o error) and not unmounted. Hence, for the same iqn, we still see logins from 2 nodes - new and old initiator.

Expected results:
++++++++++++++++++++
1. Always, mpath device should get mounted on the app pods to get the benefits of multipath.
2. Moreover , if single device is mounted, it should get unmounted successfully from old node,  when APP pod gets re-created on a new initiator.