Bug 1591676

Summary: Uninstall playbook fails to remove docker-storage due to failure of vgremove
Product: OpenShift Container Platform Reporter: Kenjiro Nakayama <knakayam>
Component: InstallerAssignee: Vadim Rutkovsky <vrutkovs>
Status: CLOSED DUPLICATE QA Contact: sheng.lao <shlao>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.9.0CC: aos-bugs, gmarcote, jkaur, jokerman, mmccomas, rbost
Target Milestone: ---   
Target Release: 3.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-10 11:05:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kenjiro Nakayama 2018-06-15 09:18:22 UTC
Description of problem:
---
- When we run uninstall playbook, it fails sometimes due to following error.

  2018-06-14 14:57:10,681 p=40718 u=root |  TASK [Wipe out Docker storage contents] *************************************************************************************************************************************
  2018-06-14 14:57:11,244 p=40718 u=root |  failed: [xx.xx.xx.xx] (item=  dockervg) => {"changed": true, "cmd": ["vgremove", "-f", "dockervg"], "delta": "0:00:00.016968", "end": "2018-06-14 14:57:11.222545", "failed": true, "item": "  dockervg", "msg": "non-zero return code", "rc": 5, "start": "2018-06-14 14:57:11.205577", "stderr": "  Logical volume dockervg/dockerlv contains a filesystem in use.", "stderr_lines": ["  Logical volume dockervg/dockerlv contains a filesystem in use."], "stdout": "", "stdout_lines": []}

Version
----
  ansible-2.4.4.0-1.el7ae.noarch                              Tue Jun 12 09:38:29 2018
  openshift-ansible-3.9.30-1.git.7.46f8678.el7.noarch         Tue Jun 12 09:38:36 2018
  openshift-ansible-docs-3.9.30-1.git.7.46f8678.el7.noarch    Tue Jun 12 09:38:34 2018
  openshift-ansible-playbooks-3.9.30-1.git.7.46f8678.el7.noarch Tue Jun 12 09:38:34 2018
  openshift-ansible-roles-3.9.30-1.git.7.46f8678.el7.noarch   Tue Jun 12 09:38:35 2018

How reproducible: not 100% (but 100% on customer's env)

Steps to Reproduce:
---
1. Install OCP 3.9 (though the cu installed standalone registry, it doesn't matter I think.)
2. Run uninstall playbook

Actual results:
---
- Got error 

Expected results:
---
- No error

Additional info:
---
- Proposed patch https://github.com/openshift/openshift-ansible/pull/8777
- Not only this issue, but also current playbook remove VG which contains 'docker' anyway.

Comment 2 Kenjiro Nakayama 2018-06-18 04:57:18 UTC
> How reproducible: not 100% (but 100% on customer's env)

This issue could be reproducible 100% when CONTAINER_ROOT_LV_MOUNT_PATH in /etc/sysconfig/docker-storage-setup was specified like below:

  # cat /etc/sysconfig/docker-storage-setup
  STORAGE_DRIVER=overlay2
  DEVS=/dev/vdb
  CONTAINER_ROOT_LV_NAME=dockerlv
  CONTAINER_ROOT_LV_SIZE=100%FREE
  CONTAINER_ROOT_LV_MOUNT_PATH=/var/lib/docker
  VG=dockervg
  WIPE_SIGNATURES=true
  
  # systemctl stop docker

  # vgremove -f dockervg
    Logical volume dockervg/dockerlv contains a filesystem in use.

Comment 3 Robert Bost 2018-07-14 19:39:40 UTC
This workaround seems to help:

  https://bugzilla.redhat.com/show_bug.cgi?id=1572266#c38

Comment 5 Scott Dodson 2018-07-16 14:32:21 UTC
*** Bug 1599990 has been marked as a duplicate of this bug. ***

Comment 6 Scott Dodson 2018-07-16 14:33:38 UTC
https://github.com/openshift/openshift-ansible/pull/8823 release-3.10 backport

Comment 7 Vadim Rutkovsky 2018-07-20 13:19:06 UTC
Fix is available in openshift-ansible-3.10.21-1

Comment 10 Vadim Rutkovsky 2018-08-10 11:05:04 UTC
Marking this as a duplicate of #1569553

Docker uninstall should be controlled by a boolean and leave `/var/lib/docker` untouched (as it might be stored on a different path). Mike's PR has been merged to master and not yet backported to 3.10

*** This bug has been marked as a duplicate of bug 1569553 ***

Comment 11 Scott Dodson 2019-04-18 15:16:09 UTC
*** Bug 1620909 has been marked as a duplicate of this bug. ***