Created attachment 1420232 [details] attachment contains logs of heketi-cli and topolgy info Description of problem: heketi-cli is not able to create 100gb arbiter volume. Version-Release number of selected component (if applicable): 6.0.0 How reproducible: Steps to Reproduce: 1. first create 100 gb of volume. 2. mount it on client 3. create some files. 4. check is files are created on backend. 5. delete that volume. 6. if see at backend bricks are not umounted. 7. create new 100 gb volume. 8. it will give you error. Actual results: Error: Unable to execute command on glusterfs-storage-wxtw7: Volume group "vg_334d74d7f871f4bf1a8530d64b20a32e" has insufficient free space (25311 extents): 25600 required. Expected results: it should create 100 gb volume Additional info:
I want to add more info : 1. The bricks for the deleted volume is not unmounted. We can see there is 100 gb of brick which is mounted. sh-4.2# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/dm-9 xfs 10G 1.1G 9.0G 11% / /dev/mapper/rhel_dhcp46--93-root xfs 50G 2.8G 48G 6% /run devtmpfs devtmpfs 24G 0 24G 0% /dev shm tmpfs 64M 0 64M 0% /dev/shm tmpfs tmpfs 24G 1.7M 24G 1% /run/lvm tmpfs tmpfs 24G 0 24G 0% /sys/fs/cgroup tmpfs tmpfs 24G 16K 24G 1% /run/secrets/kubernetes.io/serviceaccount /dev/mapper/vg_8b324cb5b1abdfe86678249dcd5c8a17-brick_a5708eddf8de3e8431903b8ed15be5f2 xfs 2.0G 33M 2.0G 2% /var/lib/heketi/mounts/vg_8b324cb5b1abdfe86678249dcd5c8a17/brick_a5708eddf8de3e8431903b8ed15be5f2 /dev/mapper/vg_8b324cb5b1abdfe86678249dcd5c8a17-brick_b04bd82fd063eaec575e337075a1e5ff xfs 100G 34M 100G 1% /var/lib/heketi/mounts/vg_8b324cb5b1abdfe86678249dcd5c8a17/brick_b04bd82fd063eaec575e337075a1e5ff 2. There is sufficient space for creating new volume so it should not give that error.
After that it allows me to create volume even of bigger size. It shows very unexpected behaviour. sh-4.2# heketi-cli volume create --size=100 --gluster-volume-options='user.heketi.arbiter true' Error: Unable to execute command on glusterfs-storage-wxtw7: Volume group "vg_334d74d7f871f4bf1a8530d64b20a32e" has insufficient free space (25311 extents): 25600 required. sh-4.2# heketi-cli volume create --size=150 --gluster-volume-options='user.heketi.arbiter true' Error: Unable to execute command on glusterfs-storage-wxtw7: Volume group "vg_334d74d7f871f4bf1a8530d64b20a32e" has insufficient free space (25247 extents): 38400 required. sh-4.2# heketi-cli volume create --size=200 --gluster-volume-options='user.heketi.arbiter true' Name: vol_41259ec4c7780169213a7d5dc9f61ae0 Size: 200 Volume Id: 41259ec4c7780169213a7d5dc9f61ae0 Cluster Id: f63905e6bcbbca9dda0bfdc88a1ffa12 Mount: 10.70.47.89:vol_41259ec4c7780169213a7d5dc9f61ae0 Mount Options: backup-volfile-servers=10.70.46.72,10.70.46.198,10.70.46.46 Block: false Free Size: 0 Block Volumes: [] Durability Type: replicate Distributed+Replica: 3
The bricks are not getting unmounted after deleting the volume when we are in the directory where the volume is mounted. If we are outside the directory then everything works fine and the bricks are unmounted successfully.
This might be because of previous volume bricks are not deleted.
removing in-flight bz tracker: this is only applicable for the bugs raised after the release has exited planning
This actually looks very similar in some respect to Bug #1534146. Here there's the special case of the unmount failing. Might be the case again of the placer not acting on the space really required but on the space from the request which can be slightly different. Only that the delta in this case and the other is just slightly bigger than the overallocation we're actually doing. John could you look into these reports?
After looking into this with John: The delete operation failed because the umount and subsequent operations failed on the brick. But curiously, the volume delete heketi operation succeeded: [heketi] INFO 2018/04/11 10:51:08 Delete Volume succeeded Hence heketi believes that the space has been freed up and hence takes the device into account again for future volume creates. In reality 100GB are still in use, so only 99GB are free. This is a bug in the volume delete code (error handling) that we need to fix. The fact that different volume sizes (100, 150, 200 GB) had different results, seems to be coincidence. subsequent tries for 100GB should have a similar pattern.
Note that the fix I have posted upstream (@ https://github.com/heketi/heketi/pull/1200 ) does not fix the failure to unmount as that's a gluster issue. However, it will prevent heketi from thinking the brick was successfully removed when it was not.
patch merged upstream.
As mentioned in comment 12 , heketi won't attempt and fail a volume creation. It will fail the creation because it knows that the bricks have not been cleaned up.
volume is deleted in gluster but it is still there in heketi-cli and bricks are also there at backend. [root@dhcp46-81 home]# heketi-cli volume list Id:4f9c0249834919dd372e8fb3344cd7bd Cluster:dc1c772ff615f0b094c13f822ebffd71 Name:vol2 Id:ab65b7e5834e8ddf856afb7bfe1c588f Cluster:dc1c772ff615f0b094c13f822ebffd71 Name:vol1 Id:be3468334af0dfad99539bb30db00c5f Cluster:dc1c772ff615f0b094c13f822ebffd71 Name:heketidbstorage [root@dhcp46-81 home]# heketi-cli volume delete ab65b7e5834e8ddf856afb7bfe1c588f Error: Unable to execute command on glusterfs-storage-2ddzp: umount: /var/lib/heketi/mounts/vg_7bd0718ec9b17e44856735b60d662d17/brick_e706f784a6791cb397d02064827a5217: target is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1))
Logs and sosreports. http://rhsqe-repo.lab.eng.blr.redhat.com/cns/sosreports/BZ-1565977/
From the log of the brick process: - glusterfs-storage-2ddzp - /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_7bd0718ec9b17e44856735b60d662d17-brick_e706f784a6791cb397d02064827a5217-brick.log [glusterfsd-mgmt.c:262:glusterfs_handle_terminate] 0-glusterfs: detaching not-only child /var/lib/heketi/mounts/vg_7bd0718ec9b17e44856735b60d662d17/brick_e706f784a6791cb397d02064827a5217/brick [glusterfsd-mgmt.c:234:glusterfs_handle_terminate] 0-glusterfs: can't terminate /var/lib/heketi/mounts/vg_7bd0718ec9b17e44856735b60d662d17/brick_e706f784a6791cb397d02064827a5217/brick - not found Failing to unmount a brick might be the problem from bz#1524336: - glusterfs-server-3.8.4-54.8.el7rhgs.x86_64 while testing - glusterfs-3.12.2-8 should have the fix (RHGS-3.4) The heketi database still contains the bricks (all 3) that are part of the volume. I think this is the expected behaviour for 'failing to delete a volume'. The space should not be free'd in the database, as the bricks are still in use too. I guess that deleting the volume a 2nd time will fail (the volume does not exist in Gluster anymore). The database is consistent with the space accounting, but not with the volumes. John, what is the intended behaviour for this? Nitin, this problem is quite different from the description in comment #0 and summary. Does it really block verification of the fix for the initial issue?
Niels de Vos, volume creation if failed because of unmounted bricks that time. Now again bricks are not unmounted successfully that's why i think this is a same issue.
Niels, your summary appears correct to me. The heketi database needs to retain the bricks in order to try the delete again in the future. If we "forgot" those brick entries we would not know what needed to be deleted on a future retry of the delete. The inability to unmount the brick is indeed a known gluster (rather than heketi) issue that is unlikely to be fixed for CNS 3.10. What we decided to do in Heketi was not to "lie" and say that the volume was successfully deleted when it was not. --- (In reply to Nitin Goyal from comment #18) > Niels de Vos, > > volume creation if failed because of unmounted bricks that time. Now again > bricks are not unmounted successfully that's why i think this is a same > issue. Nitin, I don't understand what you mean by this. If you try to delete the volume again am I correct that it also fails? --- Michael, As noted above the Heketi DB retains the information we need to clean up but does not reflect the state in gluster. I think this was OK when we last discussed it but it would lead to needing manual cleanup. We can try to squeeze making delete more "idempotent" or we can just declare that failure is an option and do that work as a future enhancement. Any opinions?
> > volume creation if failed because of unmounted bricks that time. Now again > > bricks are not unmounted successfully that's why i think this is a same > > issue. > > Nitin, I don't understand what you mean by this. This is a reply to Niels that this is a same issue > If you try to delete the volume again am I correct that it also fails? John, Yes it is failing again. Because of fail we can not create new volume. [root@dhcp46-81 home]# heketi-cli volume list Id:4f9c0249834919dd372e8fb3344cd7bd Cluster:dc1c772ff615f0b094c13f822ebffd71 Name:vol2 Id:63d905ff4bf00223215f44ccdc429121 Cluster:dc1c772ff615f0b094c13f822ebffd71 Name:vol_63d905ff4bf00223215f44ccdc429121 Id:ab65b7e5834e8ddf856afb7bfe1c588f Cluster:dc1c772ff615f0b094c13f822ebffd71 Name:vol1 Id:be3468334af0dfad99539bb30db00c5f Cluster:dc1c772ff615f0b094c13f822ebffd71 Name:heketidbstorage [root@dhcp46-81 home]# heketi-cli volume delete ab65b7e5834e8ddf856afb7bfe1c588f Error: Unable to delete volume vol1: Unable to execute command on glusterfs-storage-nhmmx: volume delete: vol1: failed: Volume vol1 does not exist
https://github.com/heketi/heketi/pull/1222
I tried to reproduce this issue in heketi-7.0.0-7.el7rhgs.x86_64, But not able to reproduce the issue. So marking this as verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2686