Bug 1534953

Summary: Heketi device deletion fails
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Rachael <rgeorge>
Component: heketiAssignee: Michael Adam <madam>
Status: CLOSED ERRATA QA Contact: vinutha <vinug>
Severity: high Docs Contact:
Priority: unspecified    
Version: cns-3.6CC: hchiramm, kramdoss, ndevos, pprakash, rcyriac, rgeorge, rhs-bugs, rtalur, storage-qa-internal
Target Milestone: ---   
Target Release: CNS 3.10   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: heketi-6.0.0-14.el7rhgs Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-12 09:22:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1571620, 1584639    
Bug Blocks: 1568861    

Description Rachael 2018-01-16 10:51:33 UTC
Description of problem:

On running device delete after performing a successful device removal, the device deletion hangs.

[root@dhcp46-180 ~]# heketi-cli device info 063a2e602443dba541f56e14cb505e4f
Device Id: 063a2e602443dba541f56e14cb505e4f
Name: /dev/sdd
State: failed
Size (GiB): 19
Used (GiB): 0
Free (GiB): 19
Bricks:

[root@dhcp46-180 ~]# heketi-cli device delete 063a2e602443dba541f56e14cb505e4f


Version-Release number of selected component (if applicable):

cns-deploy-5.0.0-57.el7rhgs.x86_64
heketi-5.0.0-19.el7rhgs.x86_64
heketi-client-5.0.0-19.el7rhgs.x86_64


How reproducible:


Steps to Reproduce:
1. 
2.
3.

Actual results:
Device deletion hangs

Expected results:
Device Deletion should be successful

Additional info:

Comment 2 Rachael 2018-01-16 11:01:08 UTC
(In reply to Rachael from comment #0)
> Description of problem:
> 
> On running device delete after performing a successful device removal, the
> device deletion hangs.
> 
> [root@dhcp46-180 ~]# heketi-cli device info 063a2e602443dba541f56e14cb505e4f
> Device Id: 063a2e602443dba541f56e14cb505e4f
> Name: /dev/sdd
> State: failed
> Size (GiB): 19
> Used (GiB): 0
> Free (GiB): 19
> Bricks:
> 
> [root@dhcp46-180 ~]# heketi-cli device delete
> 063a2e602443dba541f56e14cb505e4f
> 
> 
> Version-Release number of selected component (if applicable):
> 
> cns-deploy-5.0.0-57.el7rhgs.x86_64
> heketi-5.0.0-19.el7rhgs.x86_64
> heketi-client-5.0.0-19.el7rhgs.x86_64
> 
> 
> How reproducible:
> 
> 
> Steps to Reproduce:
> 1. 
> 2.
> 3.
> 
> Actual results:
> Device deletion hangs
> 
> Expected results:
> Device Deletion should be successful
> 
> Additional info:

Heketi logs: http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1534953/

Comment 3 Niels de Vos 2018-05-14 12:14:09 UTC
Hi Rachael, did this happen once, or is it easy reproducible?

The logs do not contain much detail, it shows the job being stuck, but no reason why. If it is possible to reproduce, we may be able to debug a little more. I do not expect we have time to troubleshoot this soon, moving it to cns-3.11 for now. If you expect this problem to (almost) always to happen, we might be able to increase the priority.

Comment 4 Rachael 2018-05-15 04:19:47 UTC
Hi Niels, this issue was hit more than once, in this version: heketi-5.0.0-19.el7rhgs.x86_64, but it is not easily reproducible. However, I have not seen this behaviour in the latest version of Heketi.

Comment 5 Niels de Vos 2018-05-15 06:34:47 UTC
Thanks Rachael! In that case it is good possible that the improved database operation framework within Heketi prevents this problem from happening.

Re-adding to cns-3.10.0 as the fix is most likely included already. This will be a QE verification effort.

Comment 9 Raghavendra Talur 2018-06-04 17:05:32 UTC
As Niels pointed out in comment 5, improved database operation in heketi 6 and above prevents this from happening. Moving it to ON_QA

Comment 10 Rachael 2018-06-12 06:53:51 UTC
Verification of this bug is blocked due to https://bugzilla.redhat.com/show_bug.cgi?id=1584639

Comment 15 errata-xmlrpc 2018-09-12 09:22:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2686