Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1565977 - [stability] error in creating 100gb of arbiter volume from heketi-cli
Summary: [stability] error in creating 100gb of arbiter volume from heketi-cli
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: heketi
Version: rhgs-3.3
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: CNS 3.10
Assignee: John Mulligan
QA Contact: vinutha
URL:
Whiteboard:
Depends On:
Blocks: 1568862
TreeView+ depends on / blocked
 
Reported: 2018-04-11 08:40 UTC by Nitin Goyal
Modified: 2018-09-12 09:23 UTC (History)
9 users (show)

Fixed In Version: heketi-7.0.0-2.el7rhgs
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-09-12 09:22:12 UTC
Target Upstream Version:


Attachments (Terms of Use)
attachment contains logs of heketi-cli and topolgy info (160.00 KB, application/x-tar)
2018-04-11 08:40 UTC, Nitin Goyal
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github https://github.com/heketi heketi issues 1220 0 None None None 2018-06-13 15:07:51 UTC
Red Hat Product Errata RHEA-2018:2686 0 None None None 2018-09-12 09:23:14 UTC

Description Nitin Goyal 2018-04-11 08:40:07 UTC
Created attachment 1420232 [details]
attachment contains logs of heketi-cli and topolgy info

Description of problem: heketi-cli is not able to create 100gb arbiter volume.


Version-Release number of selected component (if applicable): 6.0.0


How reproducible: 


Steps to Reproduce:
1. first create 100 gb of volume.
2. mount it on client
3. create some files.
4. check is files are created on backend.
5. delete that volume.
6. if see at backend bricks are not umounted.
7. create new 100 gb volume.
8. it will give you error.

Actual results: Error: Unable to execute command on glusterfs-storage-wxtw7:   Volume group "vg_334d74d7f871f4bf1a8530d64b20a32e" has insufficient free space (25311 extents): 25600 required.


Expected results: it should create 100 gb volume


Additional info:

Comment 2 Nitin Goyal 2018-04-11 08:59:05 UTC
I want to add more info : 
1. The bricks for the deleted volume is not unmounted. We can see there is 100 gb of brick which is mounted. 

sh-4.2# df -Th
Filesystem                                                                             Type      Size  Used Avail Use% Mounted on
/dev/dm-9                                                                              xfs        10G  1.1G  9.0G  11% /
/dev/mapper/rhel_dhcp46--93-root                                                       xfs        50G  2.8G   48G   6% /run
devtmpfs                                                                               devtmpfs   24G     0   24G   0% /dev
shm                                                                                    tmpfs      64M     0   64M   0% /dev/shm
tmpfs                                                                                  tmpfs      24G  1.7M   24G   1% /run/lvm
tmpfs                                                                                  tmpfs      24G     0   24G   0% /sys/fs/cgroup
tmpfs                                                                                  tmpfs      24G   16K   24G   1% /run/secrets/kubernetes.io/serviceaccount
/dev/mapper/vg_8b324cb5b1abdfe86678249dcd5c8a17-brick_a5708eddf8de3e8431903b8ed15be5f2 xfs       2.0G   33M  2.0G   2% /var/lib/heketi/mounts/vg_8b324cb5b1abdfe86678249dcd5c8a17/brick_a5708eddf8de3e8431903b8ed15be5f2
/dev/mapper/vg_8b324cb5b1abdfe86678249dcd5c8a17-brick_b04bd82fd063eaec575e337075a1e5ff xfs       100G   34M  100G   1% /var/lib/heketi/mounts/vg_8b324cb5b1abdfe86678249dcd5c8a17/brick_b04bd82fd063eaec575e337075a1e5ff

2. There is sufficient space for creating new volume so it should not give that error.

Comment 3 Nitin Goyal 2018-04-11 09:56:37 UTC
After that it allows me to create volume even of bigger size. It shows very unexpected behaviour.

sh-4.2# heketi-cli volume create --size=100 --gluster-volume-options='user.heketi.arbiter true'
Error: Unable to execute command on glusterfs-storage-wxtw7:   Volume group "vg_334d74d7f871f4bf1a8530d64b20a32e" has insufficient free space (25311 extents): 25600 required.
sh-4.2# heketi-cli volume create --size=150 --gluster-volume-options='user.heketi.arbiter true'
Error: Unable to execute command on glusterfs-storage-wxtw7:   Volume group "vg_334d74d7f871f4bf1a8530d64b20a32e" has insufficient free space (25247 extents): 38400 required.
sh-4.2# heketi-cli volume create --size=200 --gluster-volume-options='user.heketi.arbiter true'
Name: vol_41259ec4c7780169213a7d5dc9f61ae0
Size: 200
Volume Id: 41259ec4c7780169213a7d5dc9f61ae0
Cluster Id: f63905e6bcbbca9dda0bfdc88a1ffa12
Mount: 10.70.47.89:vol_41259ec4c7780169213a7d5dc9f61ae0
Mount Options: backup-volfile-servers=10.70.46.72,10.70.46.198,10.70.46.46
Block: false
Free Size: 0
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

Comment 4 Nitin Goyal 2018-04-11 10:49:39 UTC
The bricks are not getting unmounted after deleting the volume when we are in the directory where the volume is mounted. If we are outside the directory then everything works fine and the bricks are unmounted successfully.

Comment 5 Nitin Goyal 2018-04-12 09:00:56 UTC
This might be because of previous volume bricks are not deleted.

Comment 6 Michael Adam 2018-05-08 06:40:05 UTC
removing in-flight bz tracker:
this is only applicable for the bugs raised after the release has exited planning

Comment 7 Michael Adam 2018-05-08 07:14:41 UTC
This actually looks very similar in some respect to Bug #1534146.

Here there's the special case of the unmount failing.

Might be the case again of the placer not acting on the space really required but on the space from the request which can be slightly different. Only that the delta in this case and the other is just slightly bigger than the overallocation we're actually doing.

John could you look into these reports?

Comment 8 Michael Adam 2018-05-15 19:42:16 UTC
After looking into this with John:

The delete operation failed because the umount and subsequent operations failed on the brick. But curiously, the volume delete heketi operation succeeded:

[heketi] INFO 2018/04/11 10:51:08 Delete Volume succeeded

Hence heketi believes that the space has been freed up and hence takes the device into account again for future volume creates. In reality 100GB are still in use, so only 99GB are free.

This is a bug in the volume delete code (error handling) that we need to fix.

The fact that different volume sizes (100, 150, 200 GB) had different results, seems to be coincidence. subsequent tries for 100GB should have a similar pattern.

Comment 12 John Mulligan 2018-05-31 20:16:32 UTC
Note that the fix I have posted upstream (@ https://github.com/heketi/heketi/pull/1200 ) does not fix the failure to unmount as that's a gluster issue. However, it will prevent heketi from thinking the brick was successfully removed when it was not.

Comment 13 Michael Adam 2018-06-01 06:29:31 UTC
patch merged upstream.

Comment 14 Raghavendra Talur 2018-06-05 12:56:54 UTC
As mentioned in comment 12 , heketi won't attempt and fail a volume creation. It will fail the creation because it knows that the bricks have not been cleaned up.

Comment 15 Nitin Goyal 2018-06-11 09:46:21 UTC
volume is deleted in gluster but it is still there in heketi-cli and bricks are also there at backend.

[root@dhcp46-81 home]# heketi-cli volume list
Id:4f9c0249834919dd372e8fb3344cd7bd    Cluster:dc1c772ff615f0b094c13f822ebffd71    Name:vol2
Id:ab65b7e5834e8ddf856afb7bfe1c588f    Cluster:dc1c772ff615f0b094c13f822ebffd71    Name:vol1
Id:be3468334af0dfad99539bb30db00c5f    Cluster:dc1c772ff615f0b094c13f822ebffd71    Name:heketidbstorage
[root@dhcp46-81 home]# heketi-cli volume delete ab65b7e5834e8ddf856afb7bfe1c588f
Error: Unable to execute command on glusterfs-storage-2ddzp: umount: /var/lib/heketi/mounts/vg_7bd0718ec9b17e44856735b60d662d17/brick_e706f784a6791cb397d02064827a5217: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

Comment 16 Nitin Goyal 2018-06-11 09:52:03 UTC
Logs and sosreports.

http://rhsqe-repo.lab.eng.blr.redhat.com/cns/sosreports/BZ-1565977/

Comment 17 Niels de Vos 2018-06-11 11:23:35 UTC
From the log of the brick process:

- glusterfs-storage-2ddzp
- /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_7bd0718ec9b17e44856735b60d662d17-brick_e706f784a6791cb397d02064827a5217-brick.log

[glusterfsd-mgmt.c:262:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /var/lib/heketi/mounts/vg_7bd0718ec9b17e44856735b60d662d17/brick_e706f784a6791cb397d02064827a5217/brick
[glusterfsd-mgmt.c:234:glusterfs_handle_terminate] 0-glusterfs: can't terminate /var/lib/heketi/mounts/vg_7bd0718ec9b17e44856735b60d662d17/brick_e706f784a6791cb397d02064827a5217/brick - not found

Failing to unmount a brick might be the problem from bz#1524336:
- glusterfs-server-3.8.4-54.8.el7rhgs.x86_64 while testing
- glusterfs-3.12.2-8 should have the fix (RHGS-3.4)

The heketi database still contains the bricks (all 3) that are part of the volume. I think this is the expected behaviour for 'failing to delete a volume'. The space should not be free'd in the database, as the bricks are still in use too.

I guess that deleting the volume a 2nd time will fail (the volume does not exist in Gluster anymore). The database is consistent with the space accounting, but not with the volumes.


John, what is the intended behaviour for this?

Nitin, this problem is quite different from the description in comment #0 and summary. Does it really block verification of the fix for the initial issue?

Comment 18 Nitin Goyal 2018-06-11 11:47:33 UTC
Niels de Vos,

volume creation if failed because of unmounted bricks that time. Now again bricks are not unmounted successfully that's why i think this is a same issue.

Comment 19 John Mulligan 2018-06-11 15:18:23 UTC
Niels, your summary appears correct to me.

The heketi database needs to retain the bricks in order to try the delete again in the future. If we "forgot" those brick entries we would not know what needed to be deleted on a future retry of the delete.

The inability to unmount the brick is indeed a known gluster (rather than heketi) issue that is unlikely to be fixed for CNS 3.10. What we decided to do in Heketi was not to "lie" and say that the volume was successfully deleted when it was not.

---

(In reply to Nitin Goyal from comment #18)
> Niels de Vos,
> 
> volume creation if failed because of unmounted bricks that time. Now again
> bricks are not unmounted successfully that's why i think this is a same
> issue.

Nitin, I don't understand what you mean by this.

If you try to delete the volume again am I correct that it also fails?

---

Michael,
As noted above the Heketi DB retains the information we need to clean up but does not reflect the state in gluster. I think this was OK when we last discussed it but it would lead to needing manual cleanup. We can try to squeeze making delete more "idempotent" or we can just declare that failure is an option and do that work as a future enhancement. Any opinions?

Comment 20 Nitin Goyal 2018-06-12 05:51:26 UTC
> > volume creation if failed because of unmounted bricks that time. Now again
> > bricks are not unmounted successfully that's why i think this is a same
> > issue.
> 
> Nitin, I don't understand what you mean by this.
This is a reply to Niels that this is a same issue

> If you try to delete the volume again am I correct that it also fails?

John,

Yes it is failing again. Because of fail we can not create new volume.

[root@dhcp46-81 home]# heketi-cli volume list
Id:4f9c0249834919dd372e8fb3344cd7bd    Cluster:dc1c772ff615f0b094c13f822ebffd71    Name:vol2
Id:63d905ff4bf00223215f44ccdc429121    Cluster:dc1c772ff615f0b094c13f822ebffd71    Name:vol_63d905ff4bf00223215f44ccdc429121
Id:ab65b7e5834e8ddf856afb7bfe1c588f    Cluster:dc1c772ff615f0b094c13f822ebffd71    Name:vol1
Id:be3468334af0dfad99539bb30db00c5f    Cluster:dc1c772ff615f0b094c13f822ebffd71    Name:heketidbstorage
[root@dhcp46-81 home]# heketi-cli volume delete ab65b7e5834e8ddf856afb7bfe1c588f
Error: Unable to delete volume vol1: Unable to execute command on glusterfs-storage-nhmmx: volume delete: vol1: failed: Volume vol1 does not exist

Comment 21 John Mulligan 2018-06-21 14:34:56 UTC
https://github.com/heketi/heketi/pull/1222

Comment 22 Nitin Goyal 2018-08-27 08:14:29 UTC
I tried to reproduce this issue in heketi-7.0.0-7.el7rhgs.x86_64, But not able to reproduce the issue. So marking this as verified.

Comment 24 errata-xmlrpc 2018-09-12 09:22:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2686


Note You need to log in before you can comment on or make changes to this bug.