Bug 1701826

Summary: OCS uninstall playook fails to remove multipath entries from one of the nodes.
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Ashmitha Ambastha <asambast>
Component: doc-Container_Native_Storage_with_OpenShiftAssignee: Disha Walvekar <dwalveka>
Status: CLOSED CURRENTRELEASE QA Contact: Prasanth <pprakash>
Severity: high Docs Contact:
Priority: unspecified    
Version: cns-3.10CC: asriram, dwalveka, hchiramm, jmulligan, kramdoss, madam, pprakash, prasanna.kalever, puebele, rcyriac, rhs-bugs, rtalur, sarumuga, storage-doc
Target Milestone: ---Keywords: ZStream
Target Release: OCS 3.11.z Async   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-01-11 17:33:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1845059    

Description Ashmitha Ambastha 2019-04-22 06:40:33 UTC 
Description of problem:
On a set up of OCP 3.10 + OCS 3.10, while uninstalling using uninstall playbook at  /usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/uninstall.yml
with wipefs set to true while running the uninstall playbook, the playbook passes but on one of the nodes, mpath entries can still be seen. 


# lsblk
NAME                              MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
fd0                                 2:0    1    4K  0 disk  
sda                                 8:0    0  100G  0 disk  
├─sda1                              8:1    0    1G  0 part  /boot
└─sda2                              8:2    0   95G  0 part  
  ├─rhel_dhcp47--20-root          253:0    0   50G  0 lvm   /
  ├─rhel_dhcp47--20-swap          253:1    0   15G  0 lvm   
  └─rhel_dhcp47--20-home          253:20   0   30G  0 lvm   /home
sdb                                 8:16   0  100G  0 disk  
└─sdb1                              8:17   0  100G  0 part  /var
sdc                                 8:32   0  100G  0 disk  
└─sdc1                              8:33   0  100G  0 part  
  ├─docker--vg-docker--pool_tmeta 253:17   0  104M  0 lvm   
  │ └─docker--vg-docker--pool     253:19   0 39.8G  0 lvm   
  │   ├─docker-8:17-398323-f97364f295e18c12b1af9b19d12622a8b3312471ed2ceb0e19a97b6f85c749ea
                                  253:21   0   10G  0 dm    /var/lib/docker/devicemap
  │   ├─docker-8:17-398323-bdc105597034afd284c8a8eaf399ac812cafa1339d688f49471371ffc1832fb9
                                  253:22   0   10G  0 dm    /var/lib/docker/devicemap
  │   ├─docker-8:17-398323-be1e70ac568690ba0f6972447136361e249994ed1c6610ae1ded67f01ecd2334
                                  253:23   0   10G  0 dm    /var/lib/docker/devicemap
  │   ├─docker-8:17-398323-2bf827b78b0de038894484026e1cb7078cc6487c15c204dbb8fdf9bd7359e3cc
                                  253:24   0   10G  0 dm    /var/lib/docker/devicemap
  │   ├─docker-8:17-398323-3c7c8dfd2f41678adb1d9848f9c4ea9765eeebdefe1e3639b5d23e0bb4be801c
                                  253:25   0   10G  0 dm    /var/lib/docker/devicemap
  │   ├─docker-8:17-398323-e48660c869aba3f7ac1a0324e978911dcf89789cd0ebb8e33a81c693f7c6421c
                                  253:26   0   10G  0 dm    /var/lib/docker/devicemap
  │   ├─docker-8:17-398323-e3cc4bbbd577685034f534a844106baddec901a49fbb2904a166fe366939503f
                                  253:28   0   10G  0 dm    /var/lib/docker/devicemap
  │   └─docker-8:17-398323-097b13b53a1043b165c5d45de4c2d647457c202066bcddb587eb9681b895be94
                                  253:29   0   10G  0 dm    /var/lib/docker/devicemap
  └─docker--vg-docker--pool_tdata 253:18   0 39.8G  0 lvm   
    └─docker--vg-docker--pool     253:19   0 39.8G  0 lvm   
      ├─docker-8:17-398323-f97364f295e18c12b1af9b19d12622a8b3312471ed2ceb0e19a97b6f85c749ea
                                  253:21   0   10G  0 dm    /var/lib/docker/devicemap
      ├─docker-8:17-398323-bdc105597034afd284c8a8eaf399ac812cafa1339d688f49471371ffc1832fb9
                                  253:22   0   10G  0 dm    /var/lib/docker/devicemap
      ├─docker-8:17-398323-be1e70ac568690ba0f6972447136361e249994ed1c6610ae1ded67f01ecd2334
                                  253:23   0   10G  0 dm    /var/lib/docker/devicemap
      ├─docker-8:17-398323-2bf827b78b0de038894484026e1cb7078cc6487c15c204dbb8fdf9bd7359e3cc
                                  253:24   0   10G  0 dm    /var/lib/docker/devicemap
      ├─docker-8:17-398323-3c7c8dfd2f41678adb1d9848f9c4ea9765eeebdefe1e3639b5d23e0bb4be801c
                                  253:25   0   10G  0 dm    /var/lib/docker/devicemap
      ├─docker-8:17-398323-e48660c869aba3f7ac1a0324e978911dcf89789cd0ebb8e33a81c693f7c6421c
                                  253:26   0   10G  0 dm    /var/lib/docker/devicemap
      ├─docker-8:17-398323-e3cc4bbbd577685034f534a844106baddec901a49fbb2904a166fe366939503f
                                  253:28   0   10G  0 dm    /var/lib/docker/devicemap
      └─docker-8:17-398323-097b13b53a1043b165c5d45de4c2d647457c202066bcddb587eb9681b895be94
                                  253:29   0   10G  0 dm    /var/lib/docker/devicemap
sdd                                 8:48   0 1001G  0 disk  
sde                                 8:64   0 1001G  0 disk  
sdf                                 8:80   0   20G  0 disk  
└─mpatha                          253:38   0   20G  0 mpath 
sdg                                 8:96   0   20G  0 disk  
└─mpatha                          253:38   0   20G  0 mpath 
sdh                                 8:112  0   20G  0 disk  
└─mpatha                          253:38   0   20G  0 mpath 
sr0                                11:0    1 1024M  0 rom   



# iscsiadm -m session
tcp: [1] 10.70.46.114:3260,3 iqn.2016-12.org.gluster-block:b0c40ba9-9672-484e-aa45-5024d4436824 (non-flash)
tcp: [2] 10.70.47.62:3260,1 iqn.2016-12.org.gluster-block:b0c40ba9-9672-484e-aa45-5024d4436824 (non-flash)
tcp: [3] 10.70.46.176:3260,2 iqn.2016-12.org.gluster-block:b0c40ba9-9672-484e-aa45-5024d4436824 (non-flash)

# multipath -ll
mpatha (36001405b0c40ba99672484eaa455024d) dm-38 LIO-ORG ,TCMU device     
size=20G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=0 status=enabled
| `- 34:0:0:0 sdg 8:96  failed faulty running
|-+- policy='round-robin 0' prio=0 status=enabled
| `- 35:0:0:0 sdh 8:112 failed faulty running
`-+- policy='round-robin 0' prio=0 status=enabled
  `- 33:0:0:0 sdf 8:80  failed faulty running

# oc version
oc v3.10.127
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://dhcp47-17.lab.eng.blr.redhat.com:8443
openshift v3.10.127
kubernetes v1.10.0+b81c8f8

# rpm -qa | grep openshift-ansible
openshift-ansible-roles-3.10.127-1.git.0.131da09.el7.noarch
openshift-ansible-docs-3.10.127-1.git.0.131da09.el7.noarch
openshift-ansible-playbooks-3.10.127-1.git.0.131da09.el7.noarch
openshift-ansible-3.10.127-1.git.0.131da09.el7.noarch

Version-Release number of selected component (if applicable): OCP 3.10 + OCS 3.10

How reproducible: 2/3 times

Steps to Reproduce:
1. On a set up of OCP 3.10 and OCS 3.10, uninstall  OCS using following command, 
ansible-playbook -i <path_to_inventory_file> -e "openshift_storage_glusterfs_wipe=true" /usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/uninstall.yml
2. Check all the nodes, lsblk output and mpatha entries were listed.
3. Check multipath -ll and iscsiadm -m session results

Actual results: All multipath entries were not removed. 

Expected results:  All multipath entries should be removed. 

Additional info: re-running the playbook worked on cleaning the setup once. Attaching logs for the 1st run of the playbook.