Bug 1337839

Summary: Files present in the .shard folder even after deleting all the vms from the UI
Product: [Community] GlusterFS Reporter: Krutika Dhananjay <kdhananj>
Component: shardingAssignee: bugs <bugs>
Status: CLOSED CURRENTRELEASE QA Contact: bugs <bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.8.0CC: bugs, knarra, pkarampu, sabose
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.8rc2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1335858 Environment:
Last Closed: 2016-06-16 14:07:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1333643, 1335858    
Bug Blocks: 1337837    

Description Krutika Dhananjay 2016-05-20 08:17:54 UTC 
+++ This bug was initially created as a clone of Bug #1335858 +++

+++ This bug was initially created as a clone of Bug #1333643 +++

Description of problem:
I have a HC setup with templates created for both windows and linux vms. I have created a pool out of those templates and launched vms . Now i went a head and removed pool and the vms associated to it except the HostedEngine. I see that there are some files still present under .shard folder plus in the mount point too.

Files present under data storage domain:
===============================================
[root@sulphur images]# ll
total 12
drwxr-xr-x. 2 vdsm kvm 4096 May  5 11:45 4ec5c4d3-89b5-4b75-93e7-3cb54773850c
drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 11:03 a0685750-13af-4723-a257-e483138675ce
drwxr-xr-x. 2 vdsm kvm 4096 May  5 11:44 f1ac00a8-836c-488f-8131-78e72911495e

.shard files present inside the data brick of the volume:
=========================================================
[root@sulphur .shard]# ll
total 7529900
-rw-rw----. 2 root root         0 Apr 26 19:34 71ce5baa-60c6-4015-846b-9d721b899ab2.1
-rw-rw----. 2 root root         0 Apr 26 19:38 71ce5baa-60c6-4015-846b-9d721b899ab2.10
-rw-rw----. 2 root root         0 Apr 26 19:39 71ce5baa-60c6-4015-846b-9d721b899ab2.11
-rw-rw----. 2 root root         0 Apr 26 19:39 71ce5baa-60c6-4015-846b-9d721b899ab2.12
-rw-rw----. 2 root root         0 Apr 26 19:40 71ce5baa-60c6-4015-846b-9d721b899ab2.13
-rw-rw----. 2 root root         0 Apr 26 19:40 71ce5baa-60c6-4015-846b-9d721b899ab2.14
-rw-rw----. 2 root root         0 Apr 26 19:41 71ce5baa-60c6-4015-846b-9d721b899ab2.15
-rw-rw----. 2 root root         0 Apr 26 19:41 71ce5baa-60c6-4015-846b-9d721b899ab2.16
-rw-rw----. 2 root root         0 Apr 26 19:42 71ce5baa-60c6-4015-846b-9d721b899ab2.17
-rw-rw----. 2 root root         0 Apr 26 19:42 71ce5baa-60c6-4015-846b-9d721b899ab2.18
-rw-rw----. 2 root root         0 Apr 26 19:42 71ce5baa-60c6-4015-846b-9d721b899ab2.19
-rw-rw----. 2 root root         0 Apr 26 19:34 71ce5baa-60c6-4015-846b-9d721b899ab2.2
-rw-rw----. 2 root root         0 Apr 26 19:43 71ce5baa-60c6-4015-846b-9d721b899ab2.20
-rw-rw----. 2 root root         0 Apr 26 19:43 71ce5baa-60c6-4015-846b-9d721b899ab2.21
-rw-rw----. 2 root root         0 Apr 26 19:44 71ce5baa-60c6-4015-846b-9d721b899ab2.22
-rw-rw----. 2 root root         0 Apr 26 19:44 71ce5baa-60c6-4015-846b-9d721b899ab2.23
-rw-rw----. 2 root root         0 Apr 26 19:45 71ce5baa-60c6-4015-846b-9d721b899ab2.24
-rw-rw----. 2 root root         0 Apr 26 19:21 71ce5baa-60c6-4015-846b-9d721b899ab2.25
-rw-rw----. 2 root root         0 Apr 26 19:46 71ce5baa-60c6-4015-846b-9d721b899ab2.26
-rw-rw----. 2 root root         0 Apr 26 19:46 71ce5baa-60c6-4015-846b-9d721b899ab2.27
-rw-rw----. 2 root root         0 Apr 26 19:47 71ce5baa-60c6-4015-846b-9d721b899ab2.28
-rw-rw----. 2 root root         0 Apr 26 19:47 71ce5baa-60c6-4015-846b-9d721b899ab2.29
-rw-rw----. 2 root root         0 Apr 26 19:35 71ce5baa-60c6-4015-846b-9d721b899ab2.3
-rw-rw----. 2 root root         0 Apr 26 19:48 71ce5baa-60c6-4015-846b-9d721b899ab2.30
-rw-rw----. 2 root root         0 Apr 26 19:48 71ce5baa-60c6-4015-846b-9d721b899ab2.31
-rw-rw----. 2 root root         0 Apr 26 19:49 71ce5baa-60c6-4015-846b-9d721b899ab2.32
-rw-rw----. 2 root root         0 Apr 26 19:49 71ce5baa-60c6-4015-846b-9d721b899ab2.33
-rw-rw----. 2 root root         0 Apr 26 19:50 71ce5baa-60c6-4015-846b-9d721b899ab2.34
-rw-rw----. 2 root root         0 Apr 26 19:50 71ce5baa-60c6-4015-846b-9d721b899ab2.35
-rw-rw----. 2 root root         0 Apr 26 19:50 71ce5baa-60c6-4015-846b-9d721b899ab2.36
-rw-rw----. 2 root root         0 Apr 26 19:51 71ce5baa-60c6-4015-846b-9d721b899ab2.37
-rw-rw----. 2 root root         0 Apr 26 19:51 71ce5baa-60c6-4015-846b-9d721b899ab2.38
-rw-rw----. 2 root root         0 Apr 26 19:52 71ce5baa-60c6-4015-846b-9d721b899ab2.39
-rw-rw----. 2 root root         0 Apr 26 19:35 71ce5baa-60c6-4015-846b-9d721b899ab2.4
-rw-rw----. 2 root root         0 Apr 26 19:52 71ce5baa-60c6-4015-846b-9d721b899ab2.40
-rw-rw----. 2 root root         0 Apr 26 19:53 71ce5baa-60c6-4015-846b-9d721b899ab2.41
-rw-rw----. 2 root root         0 Apr 26 19:53 71ce5baa-60c6-4015-846b-9d721b899ab2.42
-rw-rw----. 2 root root         0 Apr 26 19:54 71ce5baa-60c6-4015-846b-9d721b899ab2.43
-rw-rw----. 2 root root         0 Apr 26 19:54 71ce5baa-60c6-4015-846b-9d721b899ab2.44
-rw-rw----. 2 root root         0 Apr 26 19:55 71ce5baa-60c6-4015-846b-9d721b899ab2.45
-rw-rw----. 2 root root         0 Apr 26 19:55 71ce5baa-60c6-4015-846b-9d721b899ab2.46
-rw-rw----. 2 root root         0 Apr 26 19:56 71ce5baa-60c6-4015-846b-9d721b899ab2.47
-rw-rw----. 2 root root         0 Apr 26 19:56 71ce5baa-60c6-4015-846b-9d721b899ab2.48
-rw-rw----. 2 root root         0 Apr 26 19:57 71ce5baa-60c6-4015-846b-9d721b899ab2.49
-rw-rw----. 2 root root         0 Apr 26 19:36 71ce5baa-60c6-4015-846b-9d721b899ab2.5
-rw-rw----. 2 root root         0 Apr 26 19:21 71ce5baa-60c6-4015-846b-9d721b899ab2.50
-rw-rw----. 2 root root         0 Apr 26 19:57 71ce5baa-60c6-4015-846b-9d721b899ab2.51
-rw-rw----. 2 root root         0 Apr 26 19:58 71ce5baa-60c6-4015-846b-9d721b899ab2.52
-rw-rw----. 2 root root         0 Apr 26 19:58 71ce5baa-60c6-4015-846b-9d721b899ab2.53
-rw-rw----. 2 root root         0 Apr 26 19:59 71ce5baa-60c6-4015-846b-9d721b899ab2.54
-rw-rw----. 2 root root         0 Apr 26 19:59 71ce5baa-60c6-4015-846b-9d721b899ab2.55
-rw-rw----. 2 root root         0 Apr 26 20:00 71ce5baa-60c6-4015-846b-9d721b899ab2.56
-rw-rw----. 2 root root         0 Apr 26 20:00 71ce5baa-60c6-4015-846b-9d721b899ab2.57
-rw-rw----. 2 root root         0 Apr 26 20:01 71ce5baa-60c6-4015-846b-9d721b899ab2.58
-rw-rw----. 2 root root         0 Apr 26 20:01 71ce5baa-60c6-4015-846b-9d721b899ab2.59
-rw-rw----. 2 root root         0 Apr 26 19:36 71ce5baa-60c6-4015-846b-9d721b899ab2.6
-rw-rw----. 2 root root         0 Apr 26 20:02 71ce5baa-60c6-4015-846b-9d721b899ab2.60
-rw-rw----. 2 root root         0 Apr 26 20:02 71ce5baa-60c6-4015-846b-9d721b899ab2.61
-rw-rw----. 2 root root         0 Apr 26 20:03 71ce5baa-60c6-4015-846b-9d721b899ab2.62
-rw-rw----. 2 root root         0 Apr 26 20:03 71ce5baa-60c6-4015-846b-9d721b899ab2.63
-rw-rw----. 2 root root         0 Apr 26 20:04 71ce5baa-60c6-4015-846b-9d721b899ab2.64
-rw-rw----. 2 root root         0 Apr 26 20:04 71ce5baa-60c6-4015-846b-9d721b899ab2.65
-rw-rw----. 2 root root         0 Apr 26 20:04 71ce5baa-60c6-4015-846b-9d721b899ab2.66
-rw-rw----. 2 root root         0 Apr 26 20:05 71ce5baa-60c6-4015-846b-9d721b899ab2.67
-rw-rw----. 2 root root         0 Apr 26 20:05 71ce5baa-60c6-4015-846b-9d721b899ab2.68
-rw-rw----. 2 root root         0 Apr 26 20:06 71ce5baa-60c6-4015-846b-9d721b899ab2.69
-rw-rw----. 2 root root         0 Apr 26 19:37 71ce5baa-60c6-4015-846b-9d721b899ab2.7
-rw-rw----. 2 root root         0 Apr 26 20:06 71ce5baa-60c6-4015-846b-9d721b899ab2.70
-rw-rw----. 2 root root         0 Apr 26 20:07 71ce5baa-60c6-4015-846b-9d721b899ab2.71
-rw-rw----. 2 root root         0 Apr 26 20:07 71ce5baa-60c6-4015-846b-9d721b899ab2.72
-rw-rw----. 2 root root         0 Apr 26 20:08 71ce5baa-60c6-4015-846b-9d721b899ab2.73
-rw-rw----. 2 root root         0 Apr 26 20:08 71ce5baa-60c6-4015-846b-9d721b899ab2.74
-rw-rw----. 2 root root         0 Apr 26 19:21 71ce5baa-60c6-4015-846b-9d721b899ab2.75
-rw-rw----. 2 root root         0 Apr 26 20:09 71ce5baa-60c6-4015-846b-9d721b899ab2.76
-rw-rw----. 2 root root         0 Apr 26 20:10 71ce5baa-60c6-4015-846b-9d721b899ab2.77
-rw-rw----. 2 root root         0 Apr 26 20:10 71ce5baa-60c6-4015-846b-9d721b899ab2.78
-rw-rw----. 2 root root         0 Apr 26 20:11 71ce5baa-60c6-4015-846b-9d721b899ab2.79
-rw-rw----. 2 root root         0 Apr 26 19:37 71ce5baa-60c6-4015-846b-9d721b899ab2.8
-rw-rw----. 2 root root         0 Apr 26 20:11 71ce5baa-60c6-4015-846b-9d721b899ab2.80
-rw-rw----. 2 root root         0 Apr 26 20:11 71ce5baa-60c6-4015-846b-9d721b899ab2.81
-rw-rw----. 2 root root         0 Apr 26 20:12 71ce5baa-60c6-4015-846b-9d721b899ab2.82
-rw-rw----. 2 root root         0 Apr 26 20:12 71ce5baa-60c6-4015-846b-9d721b899ab2.83
-rw-rw----. 2 root root         0 Apr 26 20:13 71ce5baa-60c6-4015-846b-9d721b899ab2.84
-rw-rw----. 2 root root         0 Apr 26 20:13 71ce5baa-60c6-4015-846b-9d721b899ab2.85
-rw-rw----. 2 root root         0 Apr 26 20:14 71ce5baa-60c6-4015-846b-9d721b899ab2.86
-rw-rw----. 2 root root         0 Apr 26 20:14 71ce5baa-60c6-4015-846b-9d721b899ab2.87
-rw-rw----. 2 root root         0 Apr 26 20:15 71ce5baa-60c6-4015-846b-9d721b899ab2.88
-rw-rw----. 2 root root         0 Apr 26 20:15 71ce5baa-60c6-4015-846b-9d721b899ab2.89
-rw-rw----. 2 root root         0 Apr 26 19:38 71ce5baa-60c6-4015-846b-9d721b899ab2.9
-rw-rw----. 2 root root         0 Apr 26 20:16 71ce5baa-60c6-4015-846b-9d721b899ab2.90
-rw-rw----. 2 root root         0 Apr 26 20:16 71ce5baa-60c6-4015-846b-9d721b899ab2.91
-rw-rw----. 2 root root         0 Apr 26 20:17 71ce5baa-60c6-4015-846b-9d721b899ab2.92
-rw-rw----. 2 root root         0 Apr 26 20:17 71ce5baa-60c6-4015-846b-9d721b899ab2.93
-rw-rw----. 2 root root         0 Apr 26 20:18 71ce5baa-60c6-4015-846b-9d721b899ab2.94
-rw-rw----. 2 root root         0 Apr 26 20:18 71ce5baa-60c6-4015-846b-9d721b899ab2.95
-rw-rw----. 2 root root         0 Apr 26 20:19 71ce5baa-60c6-4015-846b-9d721b899ab2.96
-rw-rw----. 2 root root         0 Apr 26 20:19 71ce5baa-60c6-4015-846b-9d721b899ab2.97
-rw-rw----. 2 root root         0 Apr 26 20:19 71ce5baa-60c6-4015-846b-9d721b899ab2.98
-rw-rw----. 2 root root         0 Apr 13 11:15 71ce5baa-60c6-4015-846b-9d721b899ab2.99
-rw-rw----. 2 root root 536870912 May  4 17:58 f9333cb0-eab1-4899-be93-86d03d75ee0e.1
-rw-rw----. 2 root root 536870912 May  4 17:27 f9333cb0-eab1-4899-be93-86d03d75ee0e.10
-rw-rw----. 2 root root 536870912 May  4 17:26 f9333cb0-eab1-4899-be93-86d03d75ee0e.11
-rw-rw----. 2 root root 536870912 May  4 17:26 f9333cb0-eab1-4899-be93-86d03d75ee0e.12
-rw-rw----. 2 root root  47284224 May  4 17:27 f9333cb0-eab1-4899-be93-86d03d75ee0e.13
-rw-rw----. 2 root root         0 May  4 17:11 f9333cb0-eab1-4899-be93-86d03d75ee0e.14
-rw-rw----. 2 root root         0 May  4 17:11 f9333cb0-eab1-4899-be93-86d03d75ee0e.15
-rw-rw----. 2 root root 536870912 May  4 17:53 f9333cb0-eab1-4899-be93-86d03d75ee0e.16
-rw-rw----. 2 root root 536870912 May  4 17:54 f9333cb0-eab1-4899-be93-86d03d75ee0e.17
-rw-rw----. 2 root root         0 May  4 17:11 f9333cb0-eab1-4899-be93-86d03d75ee0e.18
-rw-rw----. 2 root root 143769600 May  4 17:55 f9333cb0-eab1-4899-be93-86d03d75ee0e.19
-rw-rw----. 2 root root 536870912 May  5 10:15 f9333cb0-eab1-4899-be93-86d03d75ee0e.2
-rw-rw----. 2 root root    802816 May  4 16:46 f9333cb0-eab1-4899-be93-86d03d75ee0e.25
-rw-rw----. 2 root root 536870912 May  4 18:00 f9333cb0-eab1-4899-be93-86d03d75ee0e.3
-rw-rw----. 2 root root 536870912 May  4 18:01 f9333cb0-eab1-4899-be93-86d03d75ee0e.4
-rw-rw----. 2 root root 536870912 May  4 16:57 f9333cb0-eab1-4899-be93-86d03d75ee0e.5
-rw-rw----. 2 root root  26750976 May  4 18:33 f9333cb0-eab1-4899-be93-86d03d75ee0e.50
-rw-rw----. 2 root root 536870912 May  4 17:20 f9333cb0-eab1-4899-be93-86d03d75ee0e.6
-rw-rw----. 2 root root 536870912 May  5 10:15 f9333cb0-eab1-4899-be93-86d03d75ee0e.7
-rw-rw----. 2 root root    278528 May  4 16:46 f9333cb0-eab1-4899-be93-86d03d75ee0e.75
-rw-rw----. 2 root root 536870912 May  5 10:15 f9333cb0-eab1-4899-be93-86d03d75ee0e.8
-rw-rw----. 2 root root 536870912 May  4 17:27 f9333cb0-eab1-4899-be93-86d03d75ee0e.9
-rw-rw----. 2 root root 536870912 May  4 16:46 f9333cb0-eab1-4899-be93-86d03d75ee0e.99


Images under vmstore storage domain:
======================================
[root@sulphur images]# ll
total 16
drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 11:02 210ca44a-d0e3-4750-b3d3-4b1358df280b
drwxr-xr-x. 2 vdsm kvm 4096 May  5 11:45 5dc92a56-ee20-4d2e-a6bd-9f3753263029
drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 12:14 8d0b02a5-d0f1-42e4-bc10-9baa9d0d5506
drwxr-xr-x. 2 vdsm kvm 4096 May  5 11:45 bf3ed8c9-fac0-4f42-931b-9c92c9af2fe5


.shard files under vmstore brick:
======================================
[root@sulphur .shard]# ll
total 9055212
-rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.1
-rw-rw----. 2 root root 536870912 Apr 13 12:13 026de0b6-9bea-4b14-8b6f-b7261eb6c030.10
-rw-rw----. 2 root root 536870912 Apr 13 12:13 026de0b6-9bea-4b14-8b6f-b7261eb6c030.11
-rw-rw----. 2 root root 536870912 Apr 13 12:13 026de0b6-9bea-4b14-8b6f-b7261eb6c030.12
-rw-rw----. 2 root root 536870912 Apr 13 12:13 026de0b6-9bea-4b14-8b6f-b7261eb6c030.13
-rw-rw----. 2 root root 536870912 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.14
-rw-rw----. 2 root root 375697408 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.15
-rw-rw----. 2 root root         0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.16
-rw-rw----. 2 root root         0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.17
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.18
-rw-rw----. 2 root root         0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.19
-rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.2
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.20
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.21
-rw-rw----. 2 root root         0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.22
-rw-rw----. 2 root root 536870912 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.23
-rw-rw----. 2 root root 536870912 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.24
-rw-rw----. 2 root root  77815808 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.25
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.26
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.27
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.28
-rw-rw----. 2 root root         0 Apr 15 21:50 026de0b6-9bea-4b14-8b6f-b7261eb6c030.29
-rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.3
-rw-rw----. 2 root root         0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.30
-rw-rw----. 2 root root         0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.31
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.32
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.33
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.34
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.35
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.36
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.37
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.38
-rw-rw----. 2 root root 535822336 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.39
-rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.4
-rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.5
-rw-rw----. 2 root root 536870912 Apr 13 12:12 026de0b6-9bea-4b14-8b6f-b7261eb6c030.6
-rw-rw----. 2 root root 536870912 Apr 13 12:12 026de0b6-9bea-4b14-8b6f-b7261eb6c030.7
-rw-rw----. 2 root root 536870912 Apr 13 12:12 026de0b6-9bea-4b14-8b6f-b7261eb6c030.8
-rw-rw----. 2 root root 536870912 Apr 13 12:12 026de0b6-9bea-4b14-8b6f-b7261eb6c030.9
-rw-rw----. 2 root root         0 May  2 20:19 14215e87-1c82-45ec-9a34-4a8bb028d86f.1
-rw-rw----. 2 root root         0 May  2 20:23 14215e87-1c82-45ec-9a34-4a8bb028d86f.10
-rw-rw----. 2 root root         0 May  2 20:23 14215e87-1c82-45ec-9a34-4a8bb028d86f.11
-rw-rw----. 2 root root         0 May  2 20:24 14215e87-1c82-45ec-9a34-4a8bb028d86f.12
-rw-rw----. 2 root root 536870912 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.13
-rw-rw----. 2 root root  94326784 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.14
-rw-rw----. 2 root root         0 Apr 29 16:54 14215e87-1c82-45ec-9a34-4a8bb028d86f.15
-rw-rw----. 2 root root         0 Apr 29 16:55 14215e87-1c82-45ec-9a34-4a8bb028d86f.16
-rw-rw----. 2 root root         0 Apr 29 16:55 14215e87-1c82-45ec-9a34-4a8bb028d86f.17
-rw-rw----. 2 root root         0 May  2 20:26 14215e87-1c82-45ec-9a34-4a8bb028d86f.18
-rw-rw----. 2 root root         0 May  2 20:26 14215e87-1c82-45ec-9a34-4a8bb028d86f.19
-rw-rw----. 2 root root         0 May  2 20:20 14215e87-1c82-45ec-9a34-4a8bb028d86f.2
-rw-rw----. 2 root root         0 May  2 20:26 14215e87-1c82-45ec-9a34-4a8bb028d86f.20
-rw-rw----. 2 root root         0 May  2 20:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.21
-rw-rw----. 2 root root 499495424 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.22
-rw-rw----. 2 root root         0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.23
-rw-rw----. 2 root root         0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.24
-rw-rw----. 2 root root         0 Apr 26 19:28 14215e87-1c82-45ec-9a34-4a8bb028d86f.25
-rw-rw----. 2 root root         0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.26
-rw-rw----. 2 root root         0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.27
-rw-rw----. 2 root root         0 Apr 26 19:28 14215e87-1c82-45ec-9a34-4a8bb028d86f.28
-rw-rw----. 2 root root         0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.29
-rw-rw----. 2 root root         0 May  2 20:20 14215e87-1c82-45ec-9a34-4a8bb028d86f.3
-rw-rw----. 2 root root         0 Apr 26 19:28 14215e87-1c82-45ec-9a34-4a8bb028d86f.30
-rw-rw----. 2 root root 372343296 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.31
-rw-rw----. 2 root root         0 May  2 20:31 14215e87-1c82-45ec-9a34-4a8bb028d86f.32
-rw-rw----. 2 root root         0 May  2 20:31 14215e87-1c82-45ec-9a34-4a8bb028d86f.33
-rw-rw----. 2 root root         0 May  2 20:32 14215e87-1c82-45ec-9a34-4a8bb028d86f.34
-rw-rw----. 2 root root         0 May  2 20:32 14215e87-1c82-45ec-9a34-4a8bb028d86f.35
-rw-rw----. 2 root root         0 May  2 20:32 14215e87-1c82-45ec-9a34-4a8bb028d86f.36
-rw-rw----. 2 root root         0 May  2 20:33 14215e87-1c82-45ec-9a34-4a8bb028d86f.37
-rw-rw----. 2 root root         0 May  2 20:33 14215e87-1c82-45ec-9a34-4a8bb028d86f.38
-rw-rw----. 2 root root         0 Apr 13 11:15 14215e87-1c82-45ec-9a34-4a8bb028d86f.39
-rw-rw----. 2 root root 536869888 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.4
-rw-rw----. 2 root root 214117376 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.5
-rw-rw----. 2 root root         0 Apr 29 19:24 14215e87-1c82-45ec-9a34-4a8bb028d86f.6
-rw-rw----. 2 root root         0 Apr 29 19:24 14215e87-1c82-45ec-9a34-4a8bb028d86f.7
-rw-rw----. 2 root root         0 May  2 20:22 14215e87-1c82-45ec-9a34-4a8bb028d86f.8
-rw-rw----. 2 root root         0 May  2 20:22 14215e87-1c82-45ec-9a34-4a8bb028d86f.9


I ran the command virsh -r list command to make sure that there are no other vms running other than HostedEngine from all the nodes:

[root@sulphur ~]# virsh -r list
 Id    Name                           State
----------------------------------------------------
 616   HostedEngine                   running


[root@tettnang ~]# virsh -r list
 Id    Name                           State
----------------------------------------------------

[root@zod ~]# virsh -r list
 Id    Name                           State
----------------------------------------------------






Version-Release number of selected component (if applicable):
glusterfs-3.7.9-2.el7rhgs.x86_64

How reproducible:
Tried it once

Steps to Reproduce:
1. Create vms in HC setup using RHEV templates and Pools.
2. Now delete the pool so that all the vms are removed except HostedEngine
3.

Actual results:
I still see that there are some image files present inside the storage domain plus entries under .shard folder for both data and vmstore volumes.

Expected results:
When all the virtual machines are removed ideally there should not be any entries present in the .shard folder and in the storage domain. 

Additional info:


--- Additional comment from Pranith Kumar K on 2016-05-12 06:02:21 EDT ---

RCA:

Handling of hardlinks in sharding xlator:
Sharding translator performs hardlinks only on the base-file. When the last link is deleted, it deletes all the shard files associated with that base-file. So when an unlink fop comes to shard, it does a lookup (Equivalent of stat for user applications) to find out the number of links. If the number of links is > 1 then it just deletes that link but doesn't shards. If the number of links is 1, then it goes ahead and deletes all the shards as well.

What happens in the rhev-setup on VM deletion:
Kasturi (QE) selected all the VMs at once and performed delete. All these vm data will be moved to remove_me_xyz directory and host which is the owner of that VM will start doing unlinks in the background.

It seems like there are hardlinks on the setup where VMs are stored, where links are shared between different hosts.
[root@sulphur vmstore-brick1]# find . -type f | xargs getfattr -d -m. -e hex | grep trusted.gfid | sort | uniq -c | grep -v " 2 "
     17 trusted.gfid=0x026de0b69bea4b148b6fb7261eb6c030
      1 trusted.gfid=0x122c529ce3ab440f87edc2cf9ef79051
      1 trusted.gfid=0x1383cf29af214231bbaafd3120fd4c27
     16 trusted.gfid=0x13fd032531be41d4a5e8cff2aff02aca
     16 trusted.gfid=0x14215e871c8245ec9a344a8bb028d86f
      1 trusted.gfid=0x14807a3c2c0b4f6b92cea4a042856971
     16 trusted.gfid=0x2b294ed3423c4e43afee404315a316ab
      1 trusted.gfid=0x445a3b7cfb444fb290352b9087dc105b
      1 trusted.gfid=0x4a21a35d946d4fe88283f3a2a29992cd
      1 trusted.gfid=0x4ae3729419894b3bbb08467e32a088e7
      1 trusted.gfid=0x655f5fdf9a59499bb3d329a7ab23ff5e
      1 trusted.gfid=0x710d1bbe94e54a60ac64d916b1df672c
      1 trusted.gfid=0x97abfcae67ff420797ad3014ce084e2e
      1 trusted.gfid=0xc5816599efb547cca48a4a58a9751844
     17 trusted.gfid=0xd939b5a7aa29453e814e32c0cf6e2608
      1 trusted.gfid=0xd94592fb54ef46dbbf3da881ac94e465
     17 trusted.gfid=0xdec2404ec7574494a78598ea4bd31381
      1 trusted.gfid=0xfff5f116ef9140078c6536f23197695f
[root@sulphur vmstore-brick1]# ls -lih .glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030
469762591 -rw-rw----. 17 vdsm kvm 512M Apr 13 12:10 .glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030
[root@sulphur vmstore-brick1]# find . -inum 469762591
./.glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/8d0b02a5-d0f1-42e4-bc10-9baa9d0d5506/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/5308d74e-28e2-4158-b2f9-af898e14e6d0/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/7ca57bac-c614-402e-9bb4-0d172d69c476/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/5ee9dc38-f4a5-41ec-bd95-878165c022b1/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/99f4c43a-dc2f-46f1-b4b4-2cd289f43757/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/bedfd824-6560-454a-9ed1-6baf3e592428/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/9965825b-3e79-434a-8eec-b85994901ce5/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/548b241a-7958-40f4-b84a-161628f4e0e5/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/394f07f9-baab-4299-ba76-a13b9d2a949f/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/6e97619b-bbfb-4856-b63c-05727f84556f/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/ff21f8ba-5038-42f6-9b29-d56d87d87b21/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/3f61feef-0eb3-4cee-875b-d2e22dc8e7e3/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/a96768b4-548c-4c9d-80ac-fb4552e97a94/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/ae9715b8-c6a7-4125-948f-877fba1a7757/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/f8de7123-c638-4ee7-9ce0-c87bad52141c/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/280f77cd-63b8-4cc9-9449-b10d1c5c36e1/fe3a369e-567b-4d46-a82d-5dd309af0f5d


When all the VMs are deleted together, it can lead to different hosts trying to delete hardlinks in parallel in their respective remove_me_xxx directory. i.e. host1 can delete "./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/280f77cd-63b8-4cc9-9449-b10d1c5c36e1/fe3a369e-567b-4d46-a82d-5dd309af0f5d" where as host2 can delete "./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/f8de7123-c638-4ee7-9ce0-c87bad52141c/fe3a369e-567b-4d46-a82d-5dd309af0f5d". which are hardlinks to each other. So at the time when shard queries how many links the file has by doing lookup both of them will get a number greater than 1, so they both go ahead and delete the file. But one of those unlinks will be the last one but shard has no clue about this because at the time when it queried number of links is more than 1. This leads to shard not deleting shards. This is the reason for the bug.

We have seen files with hardlink count '1' which suggests there is a bug even in posix-xlator which is not deleting gfid-link thinking it is not the last link.

Pranith

--- Additional comment from Vijay Bellur on 2016-05-13 08:07:18 EDT ---

REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in {unlink,rename}_cbk before deleting shards) posted (#1) for review on master by Krutika Dhananjay (kdhananj)

--- Additional comment from Vijay Bellur on 2016-05-16 06:47:42 EDT ---

REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in {unlink,rename}_cbk before deleting shards) posted (#2) for review on master by Krutika Dhananjay (kdhananj)

--- Additional comment from Vijay Bellur on 2016-05-18 02:18:20 EDT ---

REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in {unlink,rename}_cbk before deleting shards) posted (#3) for review on master by Krutika Dhananjay (kdhananj)

--- Additional comment from Vijay Bellur on 2016-05-20 02:56:54 EDT ---

REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in {unlink,rename}_cbk before deleting shards) posted (#4) for review on master by Krutika Dhananjay (kdhananj)

--- Additional comment from Vijay Bellur on 2016-05-20 03:38:32 EDT ---

REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in {unlink,rename}_cbk before deleting shards) posted (#5) for review on master by Krutika Dhananjay (kdhananj)
Comment 1 Vijay Bellur 2016-05-20 08:18:37 UTC 
REVIEW: http://review.gluster.org/14451 (features/shard: Get hard-link-count in {unlink,rename}_cbk before deleting shards) posted (#1) for review on release-3.8 by Krutika Dhananjay (kdhananj)
Comment 2 Vijay Bellur 2016-05-24 08:57:50 UTC 
COMMIT: http://review.gluster.org/14451 committed in release-3.8 by Niels de Vos (ndevos) 
------
commit 8d03c3a8caf292d1b509f7fe9b4c7c4da4dfb504
Author: Krutika Dhananjay <kdhananj>
Date:   Thu May 12 15:06:59 2016 +0530

    features/shard: Get hard-link-count in {unlink,rename}_cbk before deleting shards
    
            Backport of http://review.gluster.org/#/c/14334/
    
    Change-Id: Iff0e90bee22e20c309eaea6c6a19e4fa6e101ed7
    BUG: 1337839
    Signed-off-by: Krutika Dhananjay <kdhananj>
    Reviewed-on: http://review.gluster.org/14451
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.com>
    Reviewed-by: Niels de Vos <ndevos>
Comment 3 Niels de Vos 2016-06-16 14:07:54 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user