Bug 1257468 - gdeploy: creation of thin pool stuck after brick cleanup
gdeploy: creation of thin pool stuck after brick cleanup
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: gdeploy (Show other bugs)
3.1
Unspecified Unspecified
unspecified Severity unspecified
: ---
: RHGS 3.1.1
Assigned To: Nandaja Varma
Anush Shetty
: ZStream
Depends On:
Blocks: 1251815
  Show dependency treegraph
 
Reported: 2015-08-27 03:00 EDT by Anush Shetty
Modified: 2016-07-03 21:12 EDT (History)
4 users (show)

See Also:
Fixed In Version: gdeploy-1.0-8
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-05 03:24:50 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Anush Shetty 2015-08-27 03:00:02 EDT
Description of problem: After a successful run of gdeploy backend setup, the backend bricks were clean up by umounting the brick mount and deleting the logical volumes, volume groups and physical volumes on the devices. 

Running gdeploy to configure bricks again resulted in the process being stuck while creating logical volumes on the hosts. 


Version-Release number of selected component (if applicable): gdeploy-1.0-6.el7rhgs.noarch.rpm


How reproducible: Always


Steps to Reproduce:
1. Run gdeploy to configure backend bricks
2. After a successful run, clean up by the brick mounts: umount, remove logical and physical volumes.
3. Run gdeploy again with the same configuration file.

Actual results:

gdeploy stuck while creating logical volumes.

gdeploy in verbose mode:

changed: [rhshdp05.lab.eng.blr.redhat.com] => (item=GLUSTER_vg2) => {"changed": 1, "item": "GLUSTER_vg2", "msg": "  Rounding up size to full physical extent 1.75 GiB\n  Logical volume \"metadata\" created.\n"}
changed: [rhshdp06.lab.eng.blr.redhat.com] => (item=GLUSTER_vg2) => {"changed": 1, "item": "GLUSTER_vg2", "msg": "  Rounding up size to full physical extent 1.75 GiB\n  Logical volume \"metadata\" created.\n"}
changed: [rhshdp03.lab.eng.blr.redhat.com] => (item=GLUSTER_vg2) => {"changed": 1, "item": "GLUSTER_vg2", "msg": "  Rounding up size to full physical extent 512.00 MiB\n  Logical volume \"metadata\" created.\n"}

TASK: [Create a thin pool] **************************************************** 
<rhshdp03.lab.eng.blr.redhat.com> ESTABLISH CONNECTION FOR USER: root
<rhshdp03.lab.eng.blr.redhat.com> REMOTE_MODULE lv action=create lvname=GLUSTER_pool1 lvtype='thin' compute=rhs disktype="jbod" vgname=GLUSTER_vg1
<rhshdp03.lab.eng.blr.redhat.com> EXEC ssh -C -tt -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/root/.ansible/cp/ansible-ssh-%h-%p-%r" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 rhshdp03.lab.eng.blr.redhat.com /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1440658478.89-25346136994452 && echo $HOME/.ansible/tmp/ansible-tmp-1440658478.89-25346136994452'
<rhshdp05.lab.eng.blr.redhat.com> ESTABLISH CONNECTION FOR USER: root
<rhshdp05.lab.eng.blr.redhat.com> REMOTE_MODULE lv action=create lvname=GLUSTER_pool1 lvtype='thin' compute=rhs disktype="jbod" vgname=GLUSTER_vg1
<rhshdp06.lab.eng.blr.redhat.com> ESTABLISH CONNECTION FOR USER: root
<rhshdp05.lab.eng.blr.redhat.com> EXEC ssh -C -tt -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/root/.ansible/cp/ansible-ssh-%h-%p-%r" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 rhshdp05.lab.eng.blr.redhat.com /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1440658478.91-59915647038493 && echo $HOME/.ansible/tmp/ansible-tmp-1440658478.91-59915647038493'
<rhshdp06.lab.eng.blr.redhat.com> REMOTE_MODULE lv action=create lvname=GLUSTER_pool1 lvtype='thin' compute=rhs disktype="jbod" vgname=GLUSTER_vg1
<rhshdp06.lab.eng.blr.redhat.com> EXEC ssh -C -tt -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/root/.ansible/cp/ansible-ssh-%h-%p-%r" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 rhshdp06.lab.eng.blr.redhat.com /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1440658478.93-259397830188625 && echo $HOME/.ansible/tmp/ansible-tmp-1440658478.93-259397830188625'
<rhshdp03.lab.eng.blr.redhat.com> PUT /tmp/tmp_rmobG TO /root/.ansible/tmp/ansible-tmp-1440658478.89-25346136994452/lv
<rhshdp05.lab.eng.blr.redhat.com> PUT /tmp/tmpNfYBjg TO /root/.ansible/tmp/ansible-tmp-1440658478.91-59915647038493/lv
<rhshdp06.lab.eng.blr.redhat.com> PUT /tmp/tmpfnOCHC TO /root/.ansible/tmp/ansible-tmp-1440658478.93-259397830188625/lv
<rhshdp03.lab.eng.blr.redhat.com> EXEC ssh -C -tt -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/root/.ansible/cp/ansible-ssh-%h-%p-%r" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 rhshdp03.lab.eng.blr.redhat.com /bin/sh -c 'LANG=C LC_CTYPE=C /usr/bin/python /root/.ansible/tmp/ansible-tmp-1440658478.89-25346136994452/lv; rm -rf /root/.ansible/tmp/ansible-tmp-1440658478.89-25346136994452/ >/dev/null 2>&1'
<rhshdp05.lab.eng.blr.redhat.com> EXEC ssh -C -tt -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/root/.ansible/cp/ansible-ssh-%h-%p-%r" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 rhshdp05.lab.eng.blr.redhat.com /bin/sh -c 'LANG=C LC_CTYPE=C /usr/bin/python /root/.ansible/tmp/ansible-tmp-1440658478.91-59915647038493/lv; rm -rf /root/.ansible/tmp/ansible-tmp-1440658478.91-59915647038493/ >/dev/null 2>&1'
<rhshdp06.lab.eng.blr.redhat.com> EXEC ssh -C -tt -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/root/.ansible/cp/ansible-ssh-%h-%p-%r" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 rhshdp06.lab.eng.blr.redhat.com /bin/sh -c 


On the hosts:

# lvs
^C  Interrupted...
  Giving up waiting for lock.
  /run/lock/lvm/V_GLUSTER_vg1: flock failed: Interrupted system call
  Can't get lock for GLUSTER_vg1
  Cannot process volume group GLUSTER_vg1
  LV       VG          Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  metadata GLUSTER_vg2 -wi-a----- 1.75g                                                    

# ps ax | grep lv
  386 ?        Ssl    0:00 /usr/sbin/lvmetad -f
32031 pts/1    Ss+    0:00 /bin/sh -c LANG=C LC_CTYPE=C /usr/bin/python /root/.ansible/tmp/ansible-tmp-1440658478.91-59915647038493/lv; rm -rf /root/.ansible/tmp/ansible-tmp-1440658478.91-59915647038493/ >/dev/null 2>&1
32037 pts/1    S+     0:00 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1440658478.91-59915647038493/lv
32046 pts/1    S+     0:00 /usr/sbin/lvcreate -L 104326144.0K -n GLUSTER_pool1 GLUSTER_vg1
32257 pts/0    S+     0:00 grep --color=auto lv


Additional info:

The lvs, vgs and pvs had to cleaned up again to get the gdepoy run successfully.
Comment 2 Nandaja Varma 2015-08-27 04:02:15 EDT
Anush,

Can you provide the configuration file used? (Just to clarify some doubts)
Comment 4 Nandaja Varma 2015-09-01 10:23:12 EDT
The root cause of the issue was, lvcreate was prompting if it should wipe the xfs signature. This prompt was not shown by ansible. The prompt was as follows:
 Rounding up size to full physical extent 99.50 GiB
WARNING: xfs signature detected on /dev/GLUSTER_vg1/GLUSTER_pool1 at offset 0. Wipe it? [y/n]: 

The lvcreate command has been modified to not wipe any signatures left behind. It will not prompt for this anymore.
Comment 5 Anush Shetty 2015-09-08 05:17:16 EDT
Verified with gdeploy-1.0-9.el7rhgs.noarch
Comment 8 errata-xmlrpc 2015-10-05 03:24:50 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1845.html

Note You need to log in before you can comment on or make changes to this bug.