Bug 1333643

Summary: Files present in the .shard folder even after deleting all the vms from the UI
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: RamaKasturi <knarra>
Component: shardingAssignee: Krutika Dhananjay <kdhananj>
Status: CLOSED ERRATA QA Contact: RamaKasturi <knarra>
Severity: medium Docs Contact:
Priority: high    
Version: rhgs-3.1CC: asrivast, pkarampu, rhinduja, sabose
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.1.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.7.9-6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1335858 (view as bug list) Environment:
Last Closed: 2016-06-23 05:21:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1258386, 1311817, 1335858, 1337837, 1337839    

Description RamaKasturi 2016-05-06 05:25:17 UTC 
Description of problem:
I have a HC setup with templates created for both windows and linux vms. I have created a pool out of those templates and launched vms . Now i went a head and removed pool and the vms associated to it except the HostedEngine. I see that there are some files still present under .shard folder plus in the mount point too.

Files present under data storage domain:
===============================================
[root@sulphur images]# ll
total 12
drwxr-xr-x. 2 vdsm kvm 4096 May  5 11:45 4ec5c4d3-89b5-4b75-93e7-3cb54773850c
drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 11:03 a0685750-13af-4723-a257-e483138675ce
drwxr-xr-x. 2 vdsm kvm 4096 May  5 11:44 f1ac00a8-836c-488f-8131-78e72911495e

.shard files present inside the data brick of the volume:
=========================================================
[root@sulphur .shard]# ll
total 7529900
-rw-rw----. 2 root root         0 Apr 26 19:34 71ce5baa-60c6-4015-846b-9d721b899ab2.1
-rw-rw----. 2 root root         0 Apr 26 19:38 71ce5baa-60c6-4015-846b-9d721b899ab2.10
-rw-rw----. 2 root root         0 Apr 26 19:39 71ce5baa-60c6-4015-846b-9d721b899ab2.11
-rw-rw----. 2 root root         0 Apr 26 19:39 71ce5baa-60c6-4015-846b-9d721b899ab2.12
-rw-rw----. 2 root root         0 Apr 26 19:40 71ce5baa-60c6-4015-846b-9d721b899ab2.13
-rw-rw----. 2 root root         0 Apr 26 19:40 71ce5baa-60c6-4015-846b-9d721b899ab2.14
-rw-rw----. 2 root root         0 Apr 26 19:41 71ce5baa-60c6-4015-846b-9d721b899ab2.15
-rw-rw----. 2 root root         0 Apr 26 19:41 71ce5baa-60c6-4015-846b-9d721b899ab2.16
-rw-rw----. 2 root root         0 Apr 26 19:42 71ce5baa-60c6-4015-846b-9d721b899ab2.17
-rw-rw----. 2 root root         0 Apr 26 19:42 71ce5baa-60c6-4015-846b-9d721b899ab2.18
-rw-rw----. 2 root root         0 Apr 26 19:42 71ce5baa-60c6-4015-846b-9d721b899ab2.19
-rw-rw----. 2 root root         0 Apr 26 19:34 71ce5baa-60c6-4015-846b-9d721b899ab2.2
-rw-rw----. 2 root root         0 Apr 26 19:43 71ce5baa-60c6-4015-846b-9d721b899ab2.20
-rw-rw----. 2 root root         0 Apr 26 19:43 71ce5baa-60c6-4015-846b-9d721b899ab2.21
-rw-rw----. 2 root root         0 Apr 26 19:44 71ce5baa-60c6-4015-846b-9d721b899ab2.22
-rw-rw----. 2 root root         0 Apr 26 19:44 71ce5baa-60c6-4015-846b-9d721b899ab2.23
-rw-rw----. 2 root root         0 Apr 26 19:45 71ce5baa-60c6-4015-846b-9d721b899ab2.24
-rw-rw----. 2 root root         0 Apr 26 19:21 71ce5baa-60c6-4015-846b-9d721b899ab2.25
-rw-rw----. 2 root root         0 Apr 26 19:46 71ce5baa-60c6-4015-846b-9d721b899ab2.26
-rw-rw----. 2 root root         0 Apr 26 19:46 71ce5baa-60c6-4015-846b-9d721b899ab2.27
-rw-rw----. 2 root root         0 Apr 26 19:47 71ce5baa-60c6-4015-846b-9d721b899ab2.28
-rw-rw----. 2 root root         0 Apr 26 19:47 71ce5baa-60c6-4015-846b-9d721b899ab2.29
-rw-rw----. 2 root root         0 Apr 26 19:35 71ce5baa-60c6-4015-846b-9d721b899ab2.3
-rw-rw----. 2 root root         0 Apr 26 19:48 71ce5baa-60c6-4015-846b-9d721b899ab2.30
-rw-rw----. 2 root root         0 Apr 26 19:48 71ce5baa-60c6-4015-846b-9d721b899ab2.31
-rw-rw----. 2 root root         0 Apr 26 19:49 71ce5baa-60c6-4015-846b-9d721b899ab2.32
-rw-rw----. 2 root root         0 Apr 26 19:49 71ce5baa-60c6-4015-846b-9d721b899ab2.33
-rw-rw----. 2 root root         0 Apr 26 19:50 71ce5baa-60c6-4015-846b-9d721b899ab2.34
-rw-rw----. 2 root root         0 Apr 26 19:50 71ce5baa-60c6-4015-846b-9d721b899ab2.35
-rw-rw----. 2 root root         0 Apr 26 19:50 71ce5baa-60c6-4015-846b-9d721b899ab2.36
-rw-rw----. 2 root root         0 Apr 26 19:51 71ce5baa-60c6-4015-846b-9d721b899ab2.37
-rw-rw----. 2 root root         0 Apr 26 19:51 71ce5baa-60c6-4015-846b-9d721b899ab2.38
-rw-rw----. 2 root root         0 Apr 26 19:52 71ce5baa-60c6-4015-846b-9d721b899ab2.39
-rw-rw----. 2 root root         0 Apr 26 19:35 71ce5baa-60c6-4015-846b-9d721b899ab2.4
-rw-rw----. 2 root root         0 Apr 26 19:52 71ce5baa-60c6-4015-846b-9d721b899ab2.40
-rw-rw----. 2 root root         0 Apr 26 19:53 71ce5baa-60c6-4015-846b-9d721b899ab2.41
-rw-rw----. 2 root root         0 Apr 26 19:53 71ce5baa-60c6-4015-846b-9d721b899ab2.42
-rw-rw----. 2 root root         0 Apr 26 19:54 71ce5baa-60c6-4015-846b-9d721b899ab2.43
-rw-rw----. 2 root root         0 Apr 26 19:54 71ce5baa-60c6-4015-846b-9d721b899ab2.44
-rw-rw----. 2 root root         0 Apr 26 19:55 71ce5baa-60c6-4015-846b-9d721b899ab2.45
-rw-rw----. 2 root root         0 Apr 26 19:55 71ce5baa-60c6-4015-846b-9d721b899ab2.46
-rw-rw----. 2 root root         0 Apr 26 19:56 71ce5baa-60c6-4015-846b-9d721b899ab2.47
-rw-rw----. 2 root root         0 Apr 26 19:56 71ce5baa-60c6-4015-846b-9d721b899ab2.48
-rw-rw----. 2 root root         0 Apr 26 19:57 71ce5baa-60c6-4015-846b-9d721b899ab2.49
-rw-rw----. 2 root root         0 Apr 26 19:36 71ce5baa-60c6-4015-846b-9d721b899ab2.5
-rw-rw----. 2 root root         0 Apr 26 19:21 71ce5baa-60c6-4015-846b-9d721b899ab2.50
-rw-rw----. 2 root root         0 Apr 26 19:57 71ce5baa-60c6-4015-846b-9d721b899ab2.51
-rw-rw----. 2 root root         0 Apr 26 19:58 71ce5baa-60c6-4015-846b-9d721b899ab2.52
-rw-rw----. 2 root root         0 Apr 26 19:58 71ce5baa-60c6-4015-846b-9d721b899ab2.53
-rw-rw----. 2 root root         0 Apr 26 19:59 71ce5baa-60c6-4015-846b-9d721b899ab2.54
-rw-rw----. 2 root root         0 Apr 26 19:59 71ce5baa-60c6-4015-846b-9d721b899ab2.55
-rw-rw----. 2 root root         0 Apr 26 20:00 71ce5baa-60c6-4015-846b-9d721b899ab2.56
-rw-rw----. 2 root root         0 Apr 26 20:00 71ce5baa-60c6-4015-846b-9d721b899ab2.57
-rw-rw----. 2 root root         0 Apr 26 20:01 71ce5baa-60c6-4015-846b-9d721b899ab2.58
-rw-rw----. 2 root root         0 Apr 26 20:01 71ce5baa-60c6-4015-846b-9d721b899ab2.59
-rw-rw----. 2 root root         0 Apr 26 19:36 71ce5baa-60c6-4015-846b-9d721b899ab2.6
-rw-rw----. 2 root root         0 Apr 26 20:02 71ce5baa-60c6-4015-846b-9d721b899ab2.60
-rw-rw----. 2 root root         0 Apr 26 20:02 71ce5baa-60c6-4015-846b-9d721b899ab2.61
-rw-rw----. 2 root root         0 Apr 26 20:03 71ce5baa-60c6-4015-846b-9d721b899ab2.62
-rw-rw----. 2 root root         0 Apr 26 20:03 71ce5baa-60c6-4015-846b-9d721b899ab2.63
-rw-rw----. 2 root root         0 Apr 26 20:04 71ce5baa-60c6-4015-846b-9d721b899ab2.64
-rw-rw----. 2 root root         0 Apr 26 20:04 71ce5baa-60c6-4015-846b-9d721b899ab2.65
-rw-rw----. 2 root root         0 Apr 26 20:04 71ce5baa-60c6-4015-846b-9d721b899ab2.66
-rw-rw----. 2 root root         0 Apr 26 20:05 71ce5baa-60c6-4015-846b-9d721b899ab2.67
-rw-rw----. 2 root root         0 Apr 26 20:05 71ce5baa-60c6-4015-846b-9d721b899ab2.68
-rw-rw----. 2 root root         0 Apr 26 20:06 71ce5baa-60c6-4015-846b-9d721b899ab2.69
-rw-rw----. 2 root root         0 Apr 26 19:37 71ce5baa-60c6-4015-846b-9d721b899ab2.7
-rw-rw----. 2 root root         0 Apr 26 20:06 71ce5baa-60c6-4015-846b-9d721b899ab2.70
-rw-rw----. 2 root root         0 Apr 26 20:07 71ce5baa-60c6-4015-846b-9d721b899ab2.71
-rw-rw----. 2 root root         0 Apr 26 20:07 71ce5baa-60c6-4015-846b-9d721b899ab2.72
-rw-rw----. 2 root root         0 Apr 26 20:08 71ce5baa-60c6-4015-846b-9d721b899ab2.73
-rw-rw----. 2 root root         0 Apr 26 20:08 71ce5baa-60c6-4015-846b-9d721b899ab2.74
-rw-rw----. 2 root root         0 Apr 26 19:21 71ce5baa-60c6-4015-846b-9d721b899ab2.75
-rw-rw----. 2 root root         0 Apr 26 20:09 71ce5baa-60c6-4015-846b-9d721b899ab2.76
-rw-rw----. 2 root root         0 Apr 26 20:10 71ce5baa-60c6-4015-846b-9d721b899ab2.77
-rw-rw----. 2 root root         0 Apr 26 20:10 71ce5baa-60c6-4015-846b-9d721b899ab2.78
-rw-rw----. 2 root root         0 Apr 26 20:11 71ce5baa-60c6-4015-846b-9d721b899ab2.79
-rw-rw----. 2 root root         0 Apr 26 19:37 71ce5baa-60c6-4015-846b-9d721b899ab2.8
-rw-rw----. 2 root root         0 Apr 26 20:11 71ce5baa-60c6-4015-846b-9d721b899ab2.80
-rw-rw----. 2 root root         0 Apr 26 20:11 71ce5baa-60c6-4015-846b-9d721b899ab2.81
-rw-rw----. 2 root root         0 Apr 26 20:12 71ce5baa-60c6-4015-846b-9d721b899ab2.82
-rw-rw----. 2 root root         0 Apr 26 20:12 71ce5baa-60c6-4015-846b-9d721b899ab2.83
-rw-rw----. 2 root root         0 Apr 26 20:13 71ce5baa-60c6-4015-846b-9d721b899ab2.84
-rw-rw----. 2 root root         0 Apr 26 20:13 71ce5baa-60c6-4015-846b-9d721b899ab2.85
-rw-rw----. 2 root root         0 Apr 26 20:14 71ce5baa-60c6-4015-846b-9d721b899ab2.86
-rw-rw----. 2 root root         0 Apr 26 20:14 71ce5baa-60c6-4015-846b-9d721b899ab2.87
-rw-rw----. 2 root root         0 Apr 26 20:15 71ce5baa-60c6-4015-846b-9d721b899ab2.88
-rw-rw----. 2 root root         0 Apr 26 20:15 71ce5baa-60c6-4015-846b-9d721b899ab2.89
-rw-rw----. 2 root root         0 Apr 26 19:38 71ce5baa-60c6-4015-846b-9d721b899ab2.9
-rw-rw----. 2 root root         0 Apr 26 20:16 71ce5baa-60c6-4015-846b-9d721b899ab2.90
-rw-rw----. 2 root root         0 Apr 26 20:16 71ce5baa-60c6-4015-846b-9d721b899ab2.91
-rw-rw----. 2 root root         0 Apr 26 20:17 71ce5baa-60c6-4015-846b-9d721b899ab2.92
-rw-rw----. 2 root root         0 Apr 26 20:17 71ce5baa-60c6-4015-846b-9d721b899ab2.93
-rw-rw----. 2 root root         0 Apr 26 20:18 71ce5baa-60c6-4015-846b-9d721b899ab2.94
-rw-rw----. 2 root root         0 Apr 26 20:18 71ce5baa-60c6-4015-846b-9d721b899ab2.95
-rw-rw----. 2 root root         0 Apr 26 20:19 71ce5baa-60c6-4015-846b-9d721b899ab2.96
-rw-rw----. 2 root root         0 Apr 26 20:19 71ce5baa-60c6-4015-846b-9d721b899ab2.97
-rw-rw----. 2 root root         0 Apr 26 20:19 71ce5baa-60c6-4015-846b-9d721b899ab2.98
-rw-rw----. 2 root root         0 Apr 13 11:15 71ce5baa-60c6-4015-846b-9d721b899ab2.99
-rw-rw----. 2 root root 536870912 May  4 17:58 f9333cb0-eab1-4899-be93-86d03d75ee0e.1
-rw-rw----. 2 root root 536870912 May  4 17:27 f9333cb0-eab1-4899-be93-86d03d75ee0e.10
-rw-rw----. 2 root root 536870912 May  4 17:26 f9333cb0-eab1-4899-be93-86d03d75ee0e.11
-rw-rw----. 2 root root 536870912 May  4 17:26 f9333cb0-eab1-4899-be93-86d03d75ee0e.12
-rw-rw----. 2 root root  47284224 May  4 17:27 f9333cb0-eab1-4899-be93-86d03d75ee0e.13
-rw-rw----. 2 root root         0 May  4 17:11 f9333cb0-eab1-4899-be93-86d03d75ee0e.14
-rw-rw----. 2 root root         0 May  4 17:11 f9333cb0-eab1-4899-be93-86d03d75ee0e.15
-rw-rw----. 2 root root 536870912 May  4 17:53 f9333cb0-eab1-4899-be93-86d03d75ee0e.16
-rw-rw----. 2 root root 536870912 May  4 17:54 f9333cb0-eab1-4899-be93-86d03d75ee0e.17
-rw-rw----. 2 root root         0 May  4 17:11 f9333cb0-eab1-4899-be93-86d03d75ee0e.18
-rw-rw----. 2 root root 143769600 May  4 17:55 f9333cb0-eab1-4899-be93-86d03d75ee0e.19
-rw-rw----. 2 root root 536870912 May  5 10:15 f9333cb0-eab1-4899-be93-86d03d75ee0e.2
-rw-rw----. 2 root root    802816 May  4 16:46 f9333cb0-eab1-4899-be93-86d03d75ee0e.25
-rw-rw----. 2 root root 536870912 May  4 18:00 f9333cb0-eab1-4899-be93-86d03d75ee0e.3
-rw-rw----. 2 root root 536870912 May  4 18:01 f9333cb0-eab1-4899-be93-86d03d75ee0e.4
-rw-rw----. 2 root root 536870912 May  4 16:57 f9333cb0-eab1-4899-be93-86d03d75ee0e.5
-rw-rw----. 2 root root  26750976 May  4 18:33 f9333cb0-eab1-4899-be93-86d03d75ee0e.50
-rw-rw----. 2 root root 536870912 May  4 17:20 f9333cb0-eab1-4899-be93-86d03d75ee0e.6
-rw-rw----. 2 root root 536870912 May  5 10:15 f9333cb0-eab1-4899-be93-86d03d75ee0e.7
-rw-rw----. 2 root root    278528 May  4 16:46 f9333cb0-eab1-4899-be93-86d03d75ee0e.75
-rw-rw----. 2 root root 536870912 May  5 10:15 f9333cb0-eab1-4899-be93-86d03d75ee0e.8
-rw-rw----. 2 root root 536870912 May  4 17:27 f9333cb0-eab1-4899-be93-86d03d75ee0e.9
-rw-rw----. 2 root root 536870912 May  4 16:46 f9333cb0-eab1-4899-be93-86d03d75ee0e.99


Images under vmstore storage domain:
======================================
[root@sulphur images]# ll
total 16
drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 11:02 210ca44a-d0e3-4750-b3d3-4b1358df280b
drwxr-xr-x. 2 vdsm kvm 4096 May  5 11:45 5dc92a56-ee20-4d2e-a6bd-9f3753263029
drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 12:14 8d0b02a5-d0f1-42e4-bc10-9baa9d0d5506
drwxr-xr-x. 2 vdsm kvm 4096 May  5 11:45 bf3ed8c9-fac0-4f42-931b-9c92c9af2fe5


.shard files under vmstore brick:
======================================
[root@sulphur .shard]# ll
total 9055212
-rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.1
-rw-rw----. 2 root root 536870912 Apr 13 12:13 026de0b6-9bea-4b14-8b6f-b7261eb6c030.10
-rw-rw----. 2 root root 536870912 Apr 13 12:13 026de0b6-9bea-4b14-8b6f-b7261eb6c030.11
-rw-rw----. 2 root root 536870912 Apr 13 12:13 026de0b6-9bea-4b14-8b6f-b7261eb6c030.12
-rw-rw----. 2 root root 536870912 Apr 13 12:13 026de0b6-9bea-4b14-8b6f-b7261eb6c030.13
-rw-rw----. 2 root root 536870912 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.14
-rw-rw----. 2 root root 375697408 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.15
-rw-rw----. 2 root root         0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.16
-rw-rw----. 2 root root         0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.17
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.18
-rw-rw----. 2 root root         0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.19
-rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.2
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.20
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.21
-rw-rw----. 2 root root         0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.22
-rw-rw----. 2 root root 536870912 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.23
-rw-rw----. 2 root root 536870912 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.24
-rw-rw----. 2 root root  77815808 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.25
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.26
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.27
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.28
-rw-rw----. 2 root root         0 Apr 15 21:50 026de0b6-9bea-4b14-8b6f-b7261eb6c030.29
-rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.3
-rw-rw----. 2 root root         0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.30
-rw-rw----. 2 root root         0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.31
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.32
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.33
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.34
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.35
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.36
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.37
-rw-rw----. 2 root root         0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.38
-rw-rw----. 2 root root 535822336 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.39
-rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.4
-rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.5
-rw-rw----. 2 root root 536870912 Apr 13 12:12 026de0b6-9bea-4b14-8b6f-b7261eb6c030.6
-rw-rw----. 2 root root 536870912 Apr 13 12:12 026de0b6-9bea-4b14-8b6f-b7261eb6c030.7
-rw-rw----. 2 root root 536870912 Apr 13 12:12 026de0b6-9bea-4b14-8b6f-b7261eb6c030.8
-rw-rw----. 2 root root 536870912 Apr 13 12:12 026de0b6-9bea-4b14-8b6f-b7261eb6c030.9
-rw-rw----. 2 root root         0 May  2 20:19 14215e87-1c82-45ec-9a34-4a8bb028d86f.1
-rw-rw----. 2 root root         0 May  2 20:23 14215e87-1c82-45ec-9a34-4a8bb028d86f.10
-rw-rw----. 2 root root         0 May  2 20:23 14215e87-1c82-45ec-9a34-4a8bb028d86f.11
-rw-rw----. 2 root root         0 May  2 20:24 14215e87-1c82-45ec-9a34-4a8bb028d86f.12
-rw-rw----. 2 root root 536870912 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.13
-rw-rw----. 2 root root  94326784 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.14
-rw-rw----. 2 root root         0 Apr 29 16:54 14215e87-1c82-45ec-9a34-4a8bb028d86f.15
-rw-rw----. 2 root root         0 Apr 29 16:55 14215e87-1c82-45ec-9a34-4a8bb028d86f.16
-rw-rw----. 2 root root         0 Apr 29 16:55 14215e87-1c82-45ec-9a34-4a8bb028d86f.17
-rw-rw----. 2 root root         0 May  2 20:26 14215e87-1c82-45ec-9a34-4a8bb028d86f.18
-rw-rw----. 2 root root         0 May  2 20:26 14215e87-1c82-45ec-9a34-4a8bb028d86f.19
-rw-rw----. 2 root root         0 May  2 20:20 14215e87-1c82-45ec-9a34-4a8bb028d86f.2
-rw-rw----. 2 root root         0 May  2 20:26 14215e87-1c82-45ec-9a34-4a8bb028d86f.20
-rw-rw----. 2 root root         0 May  2 20:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.21
-rw-rw----. 2 root root 499495424 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.22
-rw-rw----. 2 root root         0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.23
-rw-rw----. 2 root root         0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.24
-rw-rw----. 2 root root         0 Apr 26 19:28 14215e87-1c82-45ec-9a34-4a8bb028d86f.25
-rw-rw----. 2 root root         0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.26
-rw-rw----. 2 root root         0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.27
-rw-rw----. 2 root root         0 Apr 26 19:28 14215e87-1c82-45ec-9a34-4a8bb028d86f.28
-rw-rw----. 2 root root         0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.29
-rw-rw----. 2 root root         0 May  2 20:20 14215e87-1c82-45ec-9a34-4a8bb028d86f.3
-rw-rw----. 2 root root         0 Apr 26 19:28 14215e87-1c82-45ec-9a34-4a8bb028d86f.30
-rw-rw----. 2 root root 372343296 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.31
-rw-rw----. 2 root root         0 May  2 20:31 14215e87-1c82-45ec-9a34-4a8bb028d86f.32
-rw-rw----. 2 root root         0 May  2 20:31 14215e87-1c82-45ec-9a34-4a8bb028d86f.33
-rw-rw----. 2 root root         0 May  2 20:32 14215e87-1c82-45ec-9a34-4a8bb028d86f.34
-rw-rw----. 2 root root         0 May  2 20:32 14215e87-1c82-45ec-9a34-4a8bb028d86f.35
-rw-rw----. 2 root root         0 May  2 20:32 14215e87-1c82-45ec-9a34-4a8bb028d86f.36
-rw-rw----. 2 root root         0 May  2 20:33 14215e87-1c82-45ec-9a34-4a8bb028d86f.37
-rw-rw----. 2 root root         0 May  2 20:33 14215e87-1c82-45ec-9a34-4a8bb028d86f.38
-rw-rw----. 2 root root         0 Apr 13 11:15 14215e87-1c82-45ec-9a34-4a8bb028d86f.39
-rw-rw----. 2 root root 536869888 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.4
-rw-rw----. 2 root root 214117376 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.5
-rw-rw----. 2 root root         0 Apr 29 19:24 14215e87-1c82-45ec-9a34-4a8bb028d86f.6
-rw-rw----. 2 root root         0 Apr 29 19:24 14215e87-1c82-45ec-9a34-4a8bb028d86f.7
-rw-rw----. 2 root root         0 May  2 20:22 14215e87-1c82-45ec-9a34-4a8bb028d86f.8
-rw-rw----. 2 root root         0 May  2 20:22 14215e87-1c82-45ec-9a34-4a8bb028d86f.9


I ran the command virsh -r list command to make sure that there are no other vms running other than HostedEngine from all the nodes:

[root@sulphur ~]# virsh -r list
 Id    Name                           State
----------------------------------------------------
 616   HostedEngine                   running


[root@tettnang ~]# virsh -r list
 Id    Name                           State
----------------------------------------------------

[root@zod ~]# virsh -r list
 Id    Name                           State
----------------------------------------------------






Version-Release number of selected component (if applicable):
glusterfs-3.7.9-2.el7rhgs.x86_64

How reproducible:
Tried it once

Steps to Reproduce:
1. Create vms in HC setup using RHEV templates and Pools.
2. Now delete the pool so that all the vms are removed except HostedEngine
3.

Actual results:
I still see that there are some image files present inside the storage domain plus entries under .shard folder for both data and vmstore volumes.

Expected results:
When all the virtual machines are removed ideally there should not be any entries present in the .shard folder and in the storage domain. 

Additional info:
Comment 2 RamaKasturi 2016-05-06 05:42:32 UTC 
sosreports can be found in the link below:
===========================================

http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/HC/1333643/
Comment 3 Pranith Kumar K 2016-05-12 10:02:21 UTC 
RCA:

Handling of hardlinks in sharding xlator:
Sharding translator performs hardlinks only on the base-file. When the last link is deleted, it deletes all the shard files associated with that base-file. So when an unlink fop comes to shard, it does a lookup (Equivalent of stat for user applications) to find out the number of links. If the number of links is > 1 then it just deletes that link but doesn't shards. If the number of links is 1, then it goes ahead and deletes all the shards as well.

What happens in the rhev-setup on VM deletion:
Kasturi (QE) selected all the VMs at once and performed delete. All these vm data will be moved to remove_me_xyz directory and host which is the owner of that VM will start doing unlinks in the background.

It seems like there are hardlinks on the setup where VMs are stored, where links are shared between different hosts.
[root@sulphur vmstore-brick1]# find . -type f | xargs getfattr -d -m. -e hex | grep trusted.gfid | sort | uniq -c | grep -v " 2 "
     17 trusted.gfid=0x026de0b69bea4b148b6fb7261eb6c030
      1 trusted.gfid=0x122c529ce3ab440f87edc2cf9ef79051
      1 trusted.gfid=0x1383cf29af214231bbaafd3120fd4c27
     16 trusted.gfid=0x13fd032531be41d4a5e8cff2aff02aca
     16 trusted.gfid=0x14215e871c8245ec9a344a8bb028d86f
      1 trusted.gfid=0x14807a3c2c0b4f6b92cea4a042856971
     16 trusted.gfid=0x2b294ed3423c4e43afee404315a316ab
      1 trusted.gfid=0x445a3b7cfb444fb290352b9087dc105b
      1 trusted.gfid=0x4a21a35d946d4fe88283f3a2a29992cd
      1 trusted.gfid=0x4ae3729419894b3bbb08467e32a088e7
      1 trusted.gfid=0x655f5fdf9a59499bb3d329a7ab23ff5e
      1 trusted.gfid=0x710d1bbe94e54a60ac64d916b1df672c
      1 trusted.gfid=0x97abfcae67ff420797ad3014ce084e2e
      1 trusted.gfid=0xc5816599efb547cca48a4a58a9751844
     17 trusted.gfid=0xd939b5a7aa29453e814e32c0cf6e2608
      1 trusted.gfid=0xd94592fb54ef46dbbf3da881ac94e465
     17 trusted.gfid=0xdec2404ec7574494a78598ea4bd31381
      1 trusted.gfid=0xfff5f116ef9140078c6536f23197695f
[root@sulphur vmstore-brick1]# ls -lih .glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030
469762591 -rw-rw----. 17 vdsm kvm 512M Apr 13 12:10 .glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030
[root@sulphur vmstore-brick1]# find . -inum 469762591
./.glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/8d0b02a5-d0f1-42e4-bc10-9baa9d0d5506/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/5308d74e-28e2-4158-b2f9-af898e14e6d0/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/7ca57bac-c614-402e-9bb4-0d172d69c476/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/5ee9dc38-f4a5-41ec-bd95-878165c022b1/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/99f4c43a-dc2f-46f1-b4b4-2cd289f43757/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/bedfd824-6560-454a-9ed1-6baf3e592428/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/9965825b-3e79-434a-8eec-b85994901ce5/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/548b241a-7958-40f4-b84a-161628f4e0e5/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/394f07f9-baab-4299-ba76-a13b9d2a949f/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/6e97619b-bbfb-4856-b63c-05727f84556f/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/ff21f8ba-5038-42f6-9b29-d56d87d87b21/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/3f61feef-0eb3-4cee-875b-d2e22dc8e7e3/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/a96768b4-548c-4c9d-80ac-fb4552e97a94/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/ae9715b8-c6a7-4125-948f-877fba1a7757/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/f8de7123-c638-4ee7-9ce0-c87bad52141c/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/280f77cd-63b8-4cc9-9449-b10d1c5c36e1/fe3a369e-567b-4d46-a82d-5dd309af0f5d


When all the VMs are deleted together, it can lead to different hosts trying to delete hardlinks in parallel in their respective remove_me_xxx directory. i.e. host1 can delete "./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/280f77cd-63b8-4cc9-9449-b10d1c5c36e1/fe3a369e-567b-4d46-a82d-5dd309af0f5d" where as host2 can delete "./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/f8de7123-c638-4ee7-9ce0-c87bad52141c/fe3a369e-567b-4d46-a82d-5dd309af0f5d". which are hardlinks to each other. So at the time when shard queries how many links the file has by doing lookup both of them will get a number greater than 1, so they both go ahead and delete the file. But one of those unlinks will be the last one but shard has no clue about this because at the time when it queried number of links is more than 1. This leads to shard not deleting shards. This is the reason for the bug.

We have seen files with hardlink count '1' which suggests there is a bug even in posix-xlator which is not deleting gfid-link thinking it is not the last link.

Pranith
Comment 4 Krutika Dhananjay 2016-05-17 09:17:03 UTC 
Upstream patch: http://review.gluster.org/14334
Moving hte bug to POST.
Comment 5 Krutika Dhananjay 2016-05-20 08:55:59 UTC 
https://code.engineering.redhat.com/gerrit/74766 <-- d/s patch
Comment 8 RamaKasturi 2016-06-03 14:29:00 UTC 
Verified and works fine with build glusterfs-3.7.9-6.el7rhgs.x86_64.

Ran the following tests:

1) Directly created vms from template and deleted them from UI. All the .shards corresponding to the vm images got deleted except the shards for template since template is not deleted.

2) Created a pool from template and then launched vms and deleted them from UI. All the .shards corresponding to the vm images got deleted except the .shards for template since template is not deleted.

Moving this to verified, will re open this bug in case hit again.
Comment 10 errata-xmlrpc 2016-06-23 05:21:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240