Bug 1568521 - shard files present even after deleting vm from ovirt UI
Summary: shard files present even after deleting vm from ovirt UI
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: sharding
Version: mainline
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Krutika Dhananjay
QA Contact: bugs@gluster.org
URL:
Whiteboard:
Depends On:
Blocks: 1520882 1522624
TreeView+ depends on / blocked
 
Reported: 2018-04-17 16:47 UTC by Krutika Dhananjay
Modified: 2018-10-23 15:06 UTC (History)
8 users (show)

Fixed In Version: glusterfs-5.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1520882
Environment:
Last Closed: 2018-06-20 18:05:09 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Krutika Dhananjay 2018-04-17 16:47:41 UTC
+++ This bug was initially created as a clone of Bug #1520882 +++

Description of problem:

ghost shard files present even after deleting vm from the rhev resulting more space utilized than it should actually have.

Version-Release number of selected component (if applicable):

glusterfs-3.8.4-18.6.el7rhgs.x86_64 

Actual results:

shard files available even after deleting vm

Expected results:

shard files should have deleted after vm deletion.

--- Additional comment from Abhishek Kumar on 2017-12-05 07:11:29 EST ---

Root cause of the issue :

Presence of 3 orphaned shard set in .shard location caused less free available space on the volume.

All these 3 shard set almost consist of around 630 GB of space.

Reason for these shard not getting deleted after VM got deleted :
~~~
[2017-11-20 11:52:48.788824] W [fuse-bridge.c:1355:fuse_unlink_cbk] 0-glusterfs-fuse: 1725: UNLINK() /1d7b3d24-88d4-4eba-a0cf-9bd3625acbf6/images/_remove_me_d9a3fbce-1abf-46b4-ab88-ae38a61bb9f9/e47a4bfe-84d8-4f29-af78-7730e7ec1008 => -1 (Transport endpoint is not connected)
~~~
and a lot of errors of the kind 'Transport endpoint is not connected' in the fuse mount log from both the replicas, it is clear that the UNLINK operation (which is nothing but disk deletion) failed midway before all shards could be cleaned up.

Comment 1 Worker Ant 2018-04-17 16:49:17 UTC
REVIEW: https://review.gluster.org/19892 (features/shard: Make operations on internal directories generic) posted (#1) for review on master by Krutika Dhananjay

Comment 2 Krutika Dhananjay 2018-04-17 16:51:18 UTC
(In reply to Worker Ant from comment #1)
> REVIEW: https://review.gluster.org/19892 (features/shard: Make operations on
> internal directories generic) posted (#1) for review on master by Krutika
> Dhananjay

This is still only first of the many patches to come that are needed to fix this bug. Splitting it into multiple patches for easier review and incremental testing.

Comment 3 Worker Ant 2018-04-18 15:50:55 UTC
COMMIT: https://review.gluster.org/19892 committed in master by "Pranith Kumar Karampuri" <pkarampu@redhat.com> with a commit message- features/shard: Make operations on internal directories generic

Change-Id: Iea7ad2102220c6d415909f8caef84167ce2d6818
updates: bz#1568521
Signed-off-by: Krutika Dhananjay <kdhananj@redhat.com>

Comment 4 Worker Ant 2018-04-22 16:48:25 UTC
REVIEW: https://review.gluster.org/19915 (features/shard: Add option to barrier parallel lookup and unlink of shards) posted (#1) for review on master by Krutika Dhananjay

Comment 5 Worker Ant 2018-04-23 15:53:54 UTC
COMMIT: https://review.gluster.org/19915 committed in master by "Pranith Kumar Karampuri" <pkarampu@redhat.com> with a commit message- features/shard: Add option to barrier parallel lookup and unlink of shards

Also move the common parallel unlink callback for GF_FOP_TRUNCATE and
GF_FOP_FTRUNCATE into a separate function.

Change-Id: Ib0f90a5f62abdfa89cda7bef9f3ff99f349ec332
updates: bz#1568521
Signed-off-by: Krutika Dhananjay <kdhananj@redhat.com>

Comment 6 Worker Ant 2018-04-23 15:55:28 UTC
REVIEW: https://review.gluster.org/19927 (libglusterfs/syncop: Handle barrier_{init/destroy} in error cases) posted (#1) for review on master by Pranith Kumar Karampuri

Comment 7 Worker Ant 2018-04-24 04:06:46 UTC
REVIEW: https://review.gluster.org/19929 (features/shard: Introducing .shard_remove_me for atomic shard deletion (part 1)) posted (#1) for review on master by Krutika Dhananjay

Comment 8 Worker Ant 2018-04-25 01:53:32 UTC
COMMIT: https://review.gluster.org/19927 committed in master by "Pranith Kumar Karampuri" <pkarampu@redhat.com> with a commit message- libglusterfs/syncop: Handle barrier_{init/destroy} in error cases

BUG: 1568521
updates: bz#1568521
Change-Id: I53e60cfcaa7f8edfa5eca47307fa99f10ee64505
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>

Comment 9 Worker Ant 2018-04-25 09:28:45 UTC
REVIEW: https://review.gluster.org/19937 (features/shard: Make sure .shard_remove_me is not exposed to mount point) posted (#1) for review on master by Krutika Dhananjay

Comment 10 Worker Ant 2018-05-07 10:56:20 UTC
REVIEW: https://review.gluster.org/19970 (features/shard: Perform shards deletion in the background) posted (#2) for review on master by Krutika Dhananjay

Comment 11 Worker Ant 2018-06-13 09:57:43 UTC
COMMIT: https://review.gluster.org/19929 committed in master by "Pranith Kumar Karampuri" <pkarampu@redhat.com> with a commit message- features/shard: Introducing ".shard/.remove_me" for atomic shard deletion (part 1)

PROBLEM:
Shards are deleted synchronously when a sharded file is unlinked or
when a sharded file participating as the dst in a rename() is going to
be replaced. The problem with this approach is it makes the operation
really slow, sometimes causing the application to time out, especially
with large files.

SOLUTION:
To make this operation atomic, we introduce a ".remove_me" directory.
Now renames and unlinks will simply involve two steps:
1. creating an empty file under .remove_me named after the gfid of the file
participating in unlink/rename
2. carrying out the actual rename/unlink
A synctask is created (more on that in part 2) to scan this directory
after every unlink/rename operation (or upon a volume mount) and clean
up all shards associated with it. All of this happens in the background.
The task takes care to delete the shards associated with the gfid in
.remove_me only if this gfid doesn't exist in backend, ensuring that the
file was successfully renamed/unlinked and its shards can be discarded now
safely.

Change-Id: Ia1d238b721a3e99f951a73abbe199e4245f51a3a
updates: bz#1568521
Signed-off-by: Krutika Dhananjay <kdhananj@redhat.com>

Comment 12 Worker Ant 2018-06-20 15:04:22 UTC
COMMIT: https://review.gluster.org/19970 committed in master by "Krutika Dhananjay" <kdhananj@redhat.com> with a commit message- features/shard: Perform shards deletion in the background

A synctask is created that would scan the indices from
.shard/.remove_me, to delete the shards associated with the
gfid corresponding to the index bname and the rate of deletion
is controlled by the option features.shard-deletion-rate whose
default value is 100.
The task is launched on two accounts:
1. when shard receives its first-ever lookup on the volume
2. when a rename or unlink deleted an inode

Change-Id: Ia83117230c9dd7d0d9cae05235644f8475e97bc3
updates: bz#1568521
Signed-off-by: Krutika Dhananjay <kdhananj@redhat.com>

Comment 13 Shyamsundar 2018-06-20 18:05:09 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-v4.1.0, please open a new bug report.

glusterfs-v4.1.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://lists.gluster.org/pipermail/announce/2018-June/000102.html
[2] https://www.gluster.org/pipermail/gluster-users/

Comment 14 Shyamsundar 2018-10-23 15:06:59 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.0, please open a new bug report.

glusterfs-5.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://lists.gluster.org/pipermail/announce/2018-October/000115.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.