1701826 – OCS uninstall playook fails to remove multipath entries from one of the nodes.

Bug 1701826 - OCS uninstall playook fails to remove multipath entries from one of the nodes.

Summary: OCS uninstall playook fails to remove multipath entries from one of the nodes.

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	doc-Container_Native_Storage_with_OpenShift
Sub Component:
Version:	cns-3.10
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	OCS 3.11.z Async
Assignee:	Disha Walvekar
QA Contact:	Prasanth
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1845059
TreeView+	depends on / blocked

Reported:	2019-04-22 06:40 UTC by Ashmitha Ambastha
Modified:	2021-01-11 17:33 UTC (History)
CC List:	14 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-01-11 17:33:50 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Ashmitha Ambastha 2019-04-22 06:40:33 UTC

Description of problem:
On a set up of OCP 3.10 + OCS 3.10, while uninstalling using uninstall playbook at  /usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/uninstall.yml
with wipefs set to true while running the uninstall playbook, the playbook passes but on one of the nodes, mpath entries can still be seen. 


# lsblk
NAME                              MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
fd0                                 2:0    1    4K  0 disk  
sda                                 8:0    0  100G  0 disk  
├─sda1                              8:1    0    1G  0 part  /boot
└─sda2                              8:2    0   95G  0 part  
  ├─rhel_dhcp47--20-root          253:0    0   50G  0 lvm   /
  ├─rhel_dhcp47--20-swap          253:1    0   15G  0 lvm   
  └─rhel_dhcp47--20-home          253:20   0   30G  0 lvm   /home
sdb                                 8:16   0  100G  0 disk  
└─sdb1                              8:17   0  100G  0 part  /var
sdc                                 8:32   0  100G  0 disk  
└─sdc1                              8:33   0  100G  0 part  
  ├─docker--vg-docker--pool_tmeta 253:17   0  104M  0 lvm   
  │ └─docker--vg-docker--pool     253:19   0 39.8G  0 lvm   
  │   ├─docker-8:17-398323-f97364f295e18c12b1af9b19d12622a8b3312471ed2ceb0e19a97b6f85c749ea
                                  253:21   0   10G  0 dm    /var/lib/docker/devicemap
  │   ├─docker-8:17-398323-bdc105597034afd284c8a8eaf399ac812cafa1339d688f49471371ffc1832fb9
                                  253:22   0   10G  0 dm    /var/lib/docker/devicemap
  │   ├─docker-8:17-398323-be1e70ac568690ba0f6972447136361e249994ed1c6610ae1ded67f01ecd2334
                                  253:23   0   10G  0 dm    /var/lib/docker/devicemap
  │   ├─docker-8:17-398323-2bf827b78b0de038894484026e1cb7078cc6487c15c204dbb8fdf9bd7359e3cc
                                  253:24   0   10G  0 dm    /var/lib/docker/devicemap
  │   ├─docker-8:17-398323-3c7c8dfd2f41678adb1d9848f9c4ea9765eeebdefe1e3639b5d23e0bb4be801c
                                  253:25   0   10G  0 dm    /var/lib/docker/devicemap
  │   ├─docker-8:17-398323-e48660c869aba3f7ac1a0324e978911dcf89789cd0ebb8e33a81c693f7c6421c
                                  253:26   0   10G  0 dm    /var/lib/docker/devicemap
  │   ├─docker-8:17-398323-e3cc4bbbd577685034f534a844106baddec901a49fbb2904a166fe366939503f
                                  253:28   0   10G  0 dm    /var/lib/docker/devicemap
  │   └─docker-8:17-398323-097b13b53a1043b165c5d45de4c2d647457c202066bcddb587eb9681b895be94
                                  253:29   0   10G  0 dm    /var/lib/docker/devicemap
  └─docker--vg-docker--pool_tdata 253:18   0 39.8G  0 lvm   
    └─docker--vg-docker--pool     253:19   0 39.8G  0 lvm   
      ├─docker-8:17-398323-f97364f295e18c12b1af9b19d12622a8b3312471ed2ceb0e19a97b6f85c749ea
                                  253:21   0   10G  0 dm    /var/lib/docker/devicemap
      ├─docker-8:17-398323-bdc105597034afd284c8a8eaf399ac812cafa1339d688f49471371ffc1832fb9
                                  253:22   0   10G  0 dm    /var/lib/docker/devicemap
      ├─docker-8:17-398323-be1e70ac568690ba0f6972447136361e249994ed1c6610ae1ded67f01ecd2334
                                  253:23   0   10G  0 dm    /var/lib/docker/devicemap
      ├─docker-8:17-398323-2bf827b78b0de038894484026e1cb7078cc6487c15c204dbb8fdf9bd7359e3cc
                                  253:24   0   10G  0 dm    /var/lib/docker/devicemap
      ├─docker-8:17-398323-3c7c8dfd2f41678adb1d9848f9c4ea9765eeebdefe1e3639b5d23e0bb4be801c
                                  253:25   0   10G  0 dm    /var/lib/docker/devicemap
      ├─docker-8:17-398323-e48660c869aba3f7ac1a0324e978911dcf89789cd0ebb8e33a81c693f7c6421c
                                  253:26   0   10G  0 dm    /var/lib/docker/devicemap
      ├─docker-8:17-398323-e3cc4bbbd577685034f534a844106baddec901a49fbb2904a166fe366939503f
                                  253:28   0   10G  0 dm    /var/lib/docker/devicemap
      └─docker-8:17-398323-097b13b53a1043b165c5d45de4c2d647457c202066bcddb587eb9681b895be94
                                  253:29   0   10G  0 dm    /var/lib/docker/devicemap
sdd                                 8:48   0 1001G  0 disk  
sde                                 8:64   0 1001G  0 disk  
sdf                                 8:80   0   20G  0 disk  
└─mpatha                          253:38   0   20G  0 mpath 
sdg                                 8:96   0   20G  0 disk  
└─mpatha                          253:38   0   20G  0 mpath 
sdh                                 8:112  0   20G  0 disk  
└─mpatha                          253:38   0   20G  0 mpath 
sr0                                11:0    1 1024M  0 rom   



# iscsiadm -m session
tcp: [1] 10.70.46.114:3260,3 iqn.2016-12.org.gluster-block:b0c40ba9-9672-484e-aa45-5024d4436824 (non-flash)
tcp: [2] 10.70.47.62:3260,1 iqn.2016-12.org.gluster-block:b0c40ba9-9672-484e-aa45-5024d4436824 (non-flash)
tcp: [3] 10.70.46.176:3260,2 iqn.2016-12.org.gluster-block:b0c40ba9-9672-484e-aa45-5024d4436824 (non-flash)

# multipath -ll
mpatha (36001405b0c40ba99672484eaa455024d) dm-38 LIO-ORG ,TCMU device     
size=20G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=0 status=enabled
| `- 34:0:0:0 sdg 8:96  failed faulty running
|-+- policy='round-robin 0' prio=0 status=enabled
| `- 35:0:0:0 sdh 8:112 failed faulty running
`-+- policy='round-robin 0' prio=0 status=enabled
  `- 33:0:0:0 sdf 8:80  failed faulty running

# oc version
oc v3.10.127
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://dhcp47-17.lab.eng.blr.redhat.com:8443
openshift v3.10.127
kubernetes v1.10.0+b81c8f8

# rpm -qa | grep openshift-ansible
openshift-ansible-roles-3.10.127-1.git.0.131da09.el7.noarch
openshift-ansible-docs-3.10.127-1.git.0.131da09.el7.noarch
openshift-ansible-playbooks-3.10.127-1.git.0.131da09.el7.noarch
openshift-ansible-3.10.127-1.git.0.131da09.el7.noarch

Version-Release number of selected component (if applicable): OCP 3.10 + OCS 3.10

How reproducible: 2/3 times

Steps to Reproduce:
1. On a set up of OCP 3.10 and OCS 3.10, uninstall  OCS using following command, 
ansible-playbook -i <path_to_inventory_file> -e "openshift_storage_glusterfs_wipe=true" /usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/uninstall.yml
2. Check all the nodes, lsblk output and mpatha entries were listed.
3. Check multipath -ll and iscsiadm -m session results

Actual results: All multipath entries were not removed. 

Expected results:  All multipath entries should be removed. 

Additional info: re-running the playbook worked on cleaning the setup once. Attaching logs for the 1st run of the playbook.

Note You need to log in before you can comment on or make changes to this bug.