Description of problem: I have a HC setup with templates created for both windows and linux vms. I have created a pool out of those templates and launched vms . Now i went a head and removed pool and the vms associated to it except the HostedEngine. I see that there are some files still present under .shard folder plus in the mount point too. Files present under data storage domain: =============================================== [root@sulphur images]# ll total 12 drwxr-xr-x. 2 vdsm kvm 4096 May 5 11:45 4ec5c4d3-89b5-4b75-93e7-3cb54773850c drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 11:03 a0685750-13af-4723-a257-e483138675ce drwxr-xr-x. 2 vdsm kvm 4096 May 5 11:44 f1ac00a8-836c-488f-8131-78e72911495e .shard files present inside the data brick of the volume: ========================================================= [root@sulphur .shard]# ll total 7529900 -rw-rw----. 2 root root 0 Apr 26 19:34 71ce5baa-60c6-4015-846b-9d721b899ab2.1 -rw-rw----. 2 root root 0 Apr 26 19:38 71ce5baa-60c6-4015-846b-9d721b899ab2.10 -rw-rw----. 2 root root 0 Apr 26 19:39 71ce5baa-60c6-4015-846b-9d721b899ab2.11 -rw-rw----. 2 root root 0 Apr 26 19:39 71ce5baa-60c6-4015-846b-9d721b899ab2.12 -rw-rw----. 2 root root 0 Apr 26 19:40 71ce5baa-60c6-4015-846b-9d721b899ab2.13 -rw-rw----. 2 root root 0 Apr 26 19:40 71ce5baa-60c6-4015-846b-9d721b899ab2.14 -rw-rw----. 2 root root 0 Apr 26 19:41 71ce5baa-60c6-4015-846b-9d721b899ab2.15 -rw-rw----. 2 root root 0 Apr 26 19:41 71ce5baa-60c6-4015-846b-9d721b899ab2.16 -rw-rw----. 2 root root 0 Apr 26 19:42 71ce5baa-60c6-4015-846b-9d721b899ab2.17 -rw-rw----. 2 root root 0 Apr 26 19:42 71ce5baa-60c6-4015-846b-9d721b899ab2.18 -rw-rw----. 2 root root 0 Apr 26 19:42 71ce5baa-60c6-4015-846b-9d721b899ab2.19 -rw-rw----. 2 root root 0 Apr 26 19:34 71ce5baa-60c6-4015-846b-9d721b899ab2.2 -rw-rw----. 2 root root 0 Apr 26 19:43 71ce5baa-60c6-4015-846b-9d721b899ab2.20 -rw-rw----. 2 root root 0 Apr 26 19:43 71ce5baa-60c6-4015-846b-9d721b899ab2.21 -rw-rw----. 2 root root 0 Apr 26 19:44 71ce5baa-60c6-4015-846b-9d721b899ab2.22 -rw-rw----. 2 root root 0 Apr 26 19:44 71ce5baa-60c6-4015-846b-9d721b899ab2.23 -rw-rw----. 2 root root 0 Apr 26 19:45 71ce5baa-60c6-4015-846b-9d721b899ab2.24 -rw-rw----. 2 root root 0 Apr 26 19:21 71ce5baa-60c6-4015-846b-9d721b899ab2.25 -rw-rw----. 2 root root 0 Apr 26 19:46 71ce5baa-60c6-4015-846b-9d721b899ab2.26 -rw-rw----. 2 root root 0 Apr 26 19:46 71ce5baa-60c6-4015-846b-9d721b899ab2.27 -rw-rw----. 2 root root 0 Apr 26 19:47 71ce5baa-60c6-4015-846b-9d721b899ab2.28 -rw-rw----. 2 root root 0 Apr 26 19:47 71ce5baa-60c6-4015-846b-9d721b899ab2.29 -rw-rw----. 2 root root 0 Apr 26 19:35 71ce5baa-60c6-4015-846b-9d721b899ab2.3 -rw-rw----. 2 root root 0 Apr 26 19:48 71ce5baa-60c6-4015-846b-9d721b899ab2.30 -rw-rw----. 2 root root 0 Apr 26 19:48 71ce5baa-60c6-4015-846b-9d721b899ab2.31 -rw-rw----. 2 root root 0 Apr 26 19:49 71ce5baa-60c6-4015-846b-9d721b899ab2.32 -rw-rw----. 2 root root 0 Apr 26 19:49 71ce5baa-60c6-4015-846b-9d721b899ab2.33 -rw-rw----. 2 root root 0 Apr 26 19:50 71ce5baa-60c6-4015-846b-9d721b899ab2.34 -rw-rw----. 2 root root 0 Apr 26 19:50 71ce5baa-60c6-4015-846b-9d721b899ab2.35 -rw-rw----. 2 root root 0 Apr 26 19:50 71ce5baa-60c6-4015-846b-9d721b899ab2.36 -rw-rw----. 2 root root 0 Apr 26 19:51 71ce5baa-60c6-4015-846b-9d721b899ab2.37 -rw-rw----. 2 root root 0 Apr 26 19:51 71ce5baa-60c6-4015-846b-9d721b899ab2.38 -rw-rw----. 2 root root 0 Apr 26 19:52 71ce5baa-60c6-4015-846b-9d721b899ab2.39 -rw-rw----. 2 root root 0 Apr 26 19:35 71ce5baa-60c6-4015-846b-9d721b899ab2.4 -rw-rw----. 2 root root 0 Apr 26 19:52 71ce5baa-60c6-4015-846b-9d721b899ab2.40 -rw-rw----. 2 root root 0 Apr 26 19:53 71ce5baa-60c6-4015-846b-9d721b899ab2.41 -rw-rw----. 2 root root 0 Apr 26 19:53 71ce5baa-60c6-4015-846b-9d721b899ab2.42 -rw-rw----. 2 root root 0 Apr 26 19:54 71ce5baa-60c6-4015-846b-9d721b899ab2.43 -rw-rw----. 2 root root 0 Apr 26 19:54 71ce5baa-60c6-4015-846b-9d721b899ab2.44 -rw-rw----. 2 root root 0 Apr 26 19:55 71ce5baa-60c6-4015-846b-9d721b899ab2.45 -rw-rw----. 2 root root 0 Apr 26 19:55 71ce5baa-60c6-4015-846b-9d721b899ab2.46 -rw-rw----. 2 root root 0 Apr 26 19:56 71ce5baa-60c6-4015-846b-9d721b899ab2.47 -rw-rw----. 2 root root 0 Apr 26 19:56 71ce5baa-60c6-4015-846b-9d721b899ab2.48 -rw-rw----. 2 root root 0 Apr 26 19:57 71ce5baa-60c6-4015-846b-9d721b899ab2.49 -rw-rw----. 2 root root 0 Apr 26 19:36 71ce5baa-60c6-4015-846b-9d721b899ab2.5 -rw-rw----. 2 root root 0 Apr 26 19:21 71ce5baa-60c6-4015-846b-9d721b899ab2.50 -rw-rw----. 2 root root 0 Apr 26 19:57 71ce5baa-60c6-4015-846b-9d721b899ab2.51 -rw-rw----. 2 root root 0 Apr 26 19:58 71ce5baa-60c6-4015-846b-9d721b899ab2.52 -rw-rw----. 2 root root 0 Apr 26 19:58 71ce5baa-60c6-4015-846b-9d721b899ab2.53 -rw-rw----. 2 root root 0 Apr 26 19:59 71ce5baa-60c6-4015-846b-9d721b899ab2.54 -rw-rw----. 2 root root 0 Apr 26 19:59 71ce5baa-60c6-4015-846b-9d721b899ab2.55 -rw-rw----. 2 root root 0 Apr 26 20:00 71ce5baa-60c6-4015-846b-9d721b899ab2.56 -rw-rw----. 2 root root 0 Apr 26 20:00 71ce5baa-60c6-4015-846b-9d721b899ab2.57 -rw-rw----. 2 root root 0 Apr 26 20:01 71ce5baa-60c6-4015-846b-9d721b899ab2.58 -rw-rw----. 2 root root 0 Apr 26 20:01 71ce5baa-60c6-4015-846b-9d721b899ab2.59 -rw-rw----. 2 root root 0 Apr 26 19:36 71ce5baa-60c6-4015-846b-9d721b899ab2.6 -rw-rw----. 2 root root 0 Apr 26 20:02 71ce5baa-60c6-4015-846b-9d721b899ab2.60 -rw-rw----. 2 root root 0 Apr 26 20:02 71ce5baa-60c6-4015-846b-9d721b899ab2.61 -rw-rw----. 2 root root 0 Apr 26 20:03 71ce5baa-60c6-4015-846b-9d721b899ab2.62 -rw-rw----. 2 root root 0 Apr 26 20:03 71ce5baa-60c6-4015-846b-9d721b899ab2.63 -rw-rw----. 2 root root 0 Apr 26 20:04 71ce5baa-60c6-4015-846b-9d721b899ab2.64 -rw-rw----. 2 root root 0 Apr 26 20:04 71ce5baa-60c6-4015-846b-9d721b899ab2.65 -rw-rw----. 2 root root 0 Apr 26 20:04 71ce5baa-60c6-4015-846b-9d721b899ab2.66 -rw-rw----. 2 root root 0 Apr 26 20:05 71ce5baa-60c6-4015-846b-9d721b899ab2.67 -rw-rw----. 2 root root 0 Apr 26 20:05 71ce5baa-60c6-4015-846b-9d721b899ab2.68 -rw-rw----. 2 root root 0 Apr 26 20:06 71ce5baa-60c6-4015-846b-9d721b899ab2.69 -rw-rw----. 2 root root 0 Apr 26 19:37 71ce5baa-60c6-4015-846b-9d721b899ab2.7 -rw-rw----. 2 root root 0 Apr 26 20:06 71ce5baa-60c6-4015-846b-9d721b899ab2.70 -rw-rw----. 2 root root 0 Apr 26 20:07 71ce5baa-60c6-4015-846b-9d721b899ab2.71 -rw-rw----. 2 root root 0 Apr 26 20:07 71ce5baa-60c6-4015-846b-9d721b899ab2.72 -rw-rw----. 2 root root 0 Apr 26 20:08 71ce5baa-60c6-4015-846b-9d721b899ab2.73 -rw-rw----. 2 root root 0 Apr 26 20:08 71ce5baa-60c6-4015-846b-9d721b899ab2.74 -rw-rw----. 2 root root 0 Apr 26 19:21 71ce5baa-60c6-4015-846b-9d721b899ab2.75 -rw-rw----. 2 root root 0 Apr 26 20:09 71ce5baa-60c6-4015-846b-9d721b899ab2.76 -rw-rw----. 2 root root 0 Apr 26 20:10 71ce5baa-60c6-4015-846b-9d721b899ab2.77 -rw-rw----. 2 root root 0 Apr 26 20:10 71ce5baa-60c6-4015-846b-9d721b899ab2.78 -rw-rw----. 2 root root 0 Apr 26 20:11 71ce5baa-60c6-4015-846b-9d721b899ab2.79 -rw-rw----. 2 root root 0 Apr 26 19:37 71ce5baa-60c6-4015-846b-9d721b899ab2.8 -rw-rw----. 2 root root 0 Apr 26 20:11 71ce5baa-60c6-4015-846b-9d721b899ab2.80 -rw-rw----. 2 root root 0 Apr 26 20:11 71ce5baa-60c6-4015-846b-9d721b899ab2.81 -rw-rw----. 2 root root 0 Apr 26 20:12 71ce5baa-60c6-4015-846b-9d721b899ab2.82 -rw-rw----. 2 root root 0 Apr 26 20:12 71ce5baa-60c6-4015-846b-9d721b899ab2.83 -rw-rw----. 2 root root 0 Apr 26 20:13 71ce5baa-60c6-4015-846b-9d721b899ab2.84 -rw-rw----. 2 root root 0 Apr 26 20:13 71ce5baa-60c6-4015-846b-9d721b899ab2.85 -rw-rw----. 2 root root 0 Apr 26 20:14 71ce5baa-60c6-4015-846b-9d721b899ab2.86 -rw-rw----. 2 root root 0 Apr 26 20:14 71ce5baa-60c6-4015-846b-9d721b899ab2.87 -rw-rw----. 2 root root 0 Apr 26 20:15 71ce5baa-60c6-4015-846b-9d721b899ab2.88 -rw-rw----. 2 root root 0 Apr 26 20:15 71ce5baa-60c6-4015-846b-9d721b899ab2.89 -rw-rw----. 2 root root 0 Apr 26 19:38 71ce5baa-60c6-4015-846b-9d721b899ab2.9 -rw-rw----. 2 root root 0 Apr 26 20:16 71ce5baa-60c6-4015-846b-9d721b899ab2.90 -rw-rw----. 2 root root 0 Apr 26 20:16 71ce5baa-60c6-4015-846b-9d721b899ab2.91 -rw-rw----. 2 root root 0 Apr 26 20:17 71ce5baa-60c6-4015-846b-9d721b899ab2.92 -rw-rw----. 2 root root 0 Apr 26 20:17 71ce5baa-60c6-4015-846b-9d721b899ab2.93 -rw-rw----. 2 root root 0 Apr 26 20:18 71ce5baa-60c6-4015-846b-9d721b899ab2.94 -rw-rw----. 2 root root 0 Apr 26 20:18 71ce5baa-60c6-4015-846b-9d721b899ab2.95 -rw-rw----. 2 root root 0 Apr 26 20:19 71ce5baa-60c6-4015-846b-9d721b899ab2.96 -rw-rw----. 2 root root 0 Apr 26 20:19 71ce5baa-60c6-4015-846b-9d721b899ab2.97 -rw-rw----. 2 root root 0 Apr 26 20:19 71ce5baa-60c6-4015-846b-9d721b899ab2.98 -rw-rw----. 2 root root 0 Apr 13 11:15 71ce5baa-60c6-4015-846b-9d721b899ab2.99 -rw-rw----. 2 root root 536870912 May 4 17:58 f9333cb0-eab1-4899-be93-86d03d75ee0e.1 -rw-rw----. 2 root root 536870912 May 4 17:27 f9333cb0-eab1-4899-be93-86d03d75ee0e.10 -rw-rw----. 2 root root 536870912 May 4 17:26 f9333cb0-eab1-4899-be93-86d03d75ee0e.11 -rw-rw----. 2 root root 536870912 May 4 17:26 f9333cb0-eab1-4899-be93-86d03d75ee0e.12 -rw-rw----. 2 root root 47284224 May 4 17:27 f9333cb0-eab1-4899-be93-86d03d75ee0e.13 -rw-rw----. 2 root root 0 May 4 17:11 f9333cb0-eab1-4899-be93-86d03d75ee0e.14 -rw-rw----. 2 root root 0 May 4 17:11 f9333cb0-eab1-4899-be93-86d03d75ee0e.15 -rw-rw----. 2 root root 536870912 May 4 17:53 f9333cb0-eab1-4899-be93-86d03d75ee0e.16 -rw-rw----. 2 root root 536870912 May 4 17:54 f9333cb0-eab1-4899-be93-86d03d75ee0e.17 -rw-rw----. 2 root root 0 May 4 17:11 f9333cb0-eab1-4899-be93-86d03d75ee0e.18 -rw-rw----. 2 root root 143769600 May 4 17:55 f9333cb0-eab1-4899-be93-86d03d75ee0e.19 -rw-rw----. 2 root root 536870912 May 5 10:15 f9333cb0-eab1-4899-be93-86d03d75ee0e.2 -rw-rw----. 2 root root 802816 May 4 16:46 f9333cb0-eab1-4899-be93-86d03d75ee0e.25 -rw-rw----. 2 root root 536870912 May 4 18:00 f9333cb0-eab1-4899-be93-86d03d75ee0e.3 -rw-rw----. 2 root root 536870912 May 4 18:01 f9333cb0-eab1-4899-be93-86d03d75ee0e.4 -rw-rw----. 2 root root 536870912 May 4 16:57 f9333cb0-eab1-4899-be93-86d03d75ee0e.5 -rw-rw----. 2 root root 26750976 May 4 18:33 f9333cb0-eab1-4899-be93-86d03d75ee0e.50 -rw-rw----. 2 root root 536870912 May 4 17:20 f9333cb0-eab1-4899-be93-86d03d75ee0e.6 -rw-rw----. 2 root root 536870912 May 5 10:15 f9333cb0-eab1-4899-be93-86d03d75ee0e.7 -rw-rw----. 2 root root 278528 May 4 16:46 f9333cb0-eab1-4899-be93-86d03d75ee0e.75 -rw-rw----. 2 root root 536870912 May 5 10:15 f9333cb0-eab1-4899-be93-86d03d75ee0e.8 -rw-rw----. 2 root root 536870912 May 4 17:27 f9333cb0-eab1-4899-be93-86d03d75ee0e.9 -rw-rw----. 2 root root 536870912 May 4 16:46 f9333cb0-eab1-4899-be93-86d03d75ee0e.99 Images under vmstore storage domain: ====================================== [root@sulphur images]# ll total 16 drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 11:02 210ca44a-d0e3-4750-b3d3-4b1358df280b drwxr-xr-x. 2 vdsm kvm 4096 May 5 11:45 5dc92a56-ee20-4d2e-a6bd-9f3753263029 drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 12:14 8d0b02a5-d0f1-42e4-bc10-9baa9d0d5506 drwxr-xr-x. 2 vdsm kvm 4096 May 5 11:45 bf3ed8c9-fac0-4f42-931b-9c92c9af2fe5 .shard files under vmstore brick: ====================================== [root@sulphur .shard]# ll total 9055212 -rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.1 -rw-rw----. 2 root root 536870912 Apr 13 12:13 026de0b6-9bea-4b14-8b6f-b7261eb6c030.10 -rw-rw----. 2 root root 536870912 Apr 13 12:13 026de0b6-9bea-4b14-8b6f-b7261eb6c030.11 -rw-rw----. 2 root root 536870912 Apr 13 12:13 026de0b6-9bea-4b14-8b6f-b7261eb6c030.12 -rw-rw----. 2 root root 536870912 Apr 13 12:13 026de0b6-9bea-4b14-8b6f-b7261eb6c030.13 -rw-rw----. 2 root root 536870912 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.14 -rw-rw----. 2 root root 375697408 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.15 -rw-rw----. 2 root root 0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.16 -rw-rw----. 2 root root 0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.17 -rw-rw----. 2 root root 0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.18 -rw-rw----. 2 root root 0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.19 -rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.2 -rw-rw----. 2 root root 0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.20 -rw-rw----. 2 root root 0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.21 -rw-rw----. 2 root root 0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.22 -rw-rw----. 2 root root 536870912 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.23 -rw-rw----. 2 root root 536870912 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.24 -rw-rw----. 2 root root 77815808 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.25 -rw-rw----. 2 root root 0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.26 -rw-rw----. 2 root root 0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.27 -rw-rw----. 2 root root 0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.28 -rw-rw----. 2 root root 0 Apr 15 21:50 026de0b6-9bea-4b14-8b6f-b7261eb6c030.29 -rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.3 -rw-rw----. 2 root root 0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.30 -rw-rw----. 2 root root 0 Apr 13 12:25 026de0b6-9bea-4b14-8b6f-b7261eb6c030.31 -rw-rw----. 2 root root 0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.32 -rw-rw----. 2 root root 0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.33 -rw-rw----. 2 root root 0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.34 -rw-rw----. 2 root root 0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.35 -rw-rw----. 2 root root 0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.36 -rw-rw----. 2 root root 0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.37 -rw-rw----. 2 root root 0 Apr 13 12:30 026de0b6-9bea-4b14-8b6f-b7261eb6c030.38 -rw-rw----. 2 root root 535822336 Apr 13 12:14 026de0b6-9bea-4b14-8b6f-b7261eb6c030.39 -rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.4 -rw-rw----. 2 root root 536870912 Apr 13 12:11 026de0b6-9bea-4b14-8b6f-b7261eb6c030.5 -rw-rw----. 2 root root 536870912 Apr 13 12:12 026de0b6-9bea-4b14-8b6f-b7261eb6c030.6 -rw-rw----. 2 root root 536870912 Apr 13 12:12 026de0b6-9bea-4b14-8b6f-b7261eb6c030.7 -rw-rw----. 2 root root 536870912 Apr 13 12:12 026de0b6-9bea-4b14-8b6f-b7261eb6c030.8 -rw-rw----. 2 root root 536870912 Apr 13 12:12 026de0b6-9bea-4b14-8b6f-b7261eb6c030.9 -rw-rw----. 2 root root 0 May 2 20:19 14215e87-1c82-45ec-9a34-4a8bb028d86f.1 -rw-rw----. 2 root root 0 May 2 20:23 14215e87-1c82-45ec-9a34-4a8bb028d86f.10 -rw-rw----. 2 root root 0 May 2 20:23 14215e87-1c82-45ec-9a34-4a8bb028d86f.11 -rw-rw----. 2 root root 0 May 2 20:24 14215e87-1c82-45ec-9a34-4a8bb028d86f.12 -rw-rw----. 2 root root 536870912 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.13 -rw-rw----. 2 root root 94326784 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.14 -rw-rw----. 2 root root 0 Apr 29 16:54 14215e87-1c82-45ec-9a34-4a8bb028d86f.15 -rw-rw----. 2 root root 0 Apr 29 16:55 14215e87-1c82-45ec-9a34-4a8bb028d86f.16 -rw-rw----. 2 root root 0 Apr 29 16:55 14215e87-1c82-45ec-9a34-4a8bb028d86f.17 -rw-rw----. 2 root root 0 May 2 20:26 14215e87-1c82-45ec-9a34-4a8bb028d86f.18 -rw-rw----. 2 root root 0 May 2 20:26 14215e87-1c82-45ec-9a34-4a8bb028d86f.19 -rw-rw----. 2 root root 0 May 2 20:20 14215e87-1c82-45ec-9a34-4a8bb028d86f.2 -rw-rw----. 2 root root 0 May 2 20:26 14215e87-1c82-45ec-9a34-4a8bb028d86f.20 -rw-rw----. 2 root root 0 May 2 20:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.21 -rw-rw----. 2 root root 499495424 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.22 -rw-rw----. 2 root root 0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.23 -rw-rw----. 2 root root 0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.24 -rw-rw----. 2 root root 0 Apr 26 19:28 14215e87-1c82-45ec-9a34-4a8bb028d86f.25 -rw-rw----. 2 root root 0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.26 -rw-rw----. 2 root root 0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.27 -rw-rw----. 2 root root 0 Apr 26 19:28 14215e87-1c82-45ec-9a34-4a8bb028d86f.28 -rw-rw----. 2 root root 0 Apr 26 19:27 14215e87-1c82-45ec-9a34-4a8bb028d86f.29 -rw-rw----. 2 root root 0 May 2 20:20 14215e87-1c82-45ec-9a34-4a8bb028d86f.3 -rw-rw----. 2 root root 0 Apr 26 19:28 14215e87-1c82-45ec-9a34-4a8bb028d86f.30 -rw-rw----. 2 root root 372343296 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.31 -rw-rw----. 2 root root 0 May 2 20:31 14215e87-1c82-45ec-9a34-4a8bb028d86f.32 -rw-rw----. 2 root root 0 May 2 20:31 14215e87-1c82-45ec-9a34-4a8bb028d86f.33 -rw-rw----. 2 root root 0 May 2 20:32 14215e87-1c82-45ec-9a34-4a8bb028d86f.34 -rw-rw----. 2 root root 0 May 2 20:32 14215e87-1c82-45ec-9a34-4a8bb028d86f.35 -rw-rw----. 2 root root 0 May 2 20:32 14215e87-1c82-45ec-9a34-4a8bb028d86f.36 -rw-rw----. 2 root root 0 May 2 20:33 14215e87-1c82-45ec-9a34-4a8bb028d86f.37 -rw-rw----. 2 root root 0 May 2 20:33 14215e87-1c82-45ec-9a34-4a8bb028d86f.38 -rw-rw----. 2 root root 0 Apr 13 11:15 14215e87-1c82-45ec-9a34-4a8bb028d86f.39 -rw-rw----. 2 root root 536869888 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.4 -rw-rw----. 2 root root 214117376 Apr 13 11:02 14215e87-1c82-45ec-9a34-4a8bb028d86f.5 -rw-rw----. 2 root root 0 Apr 29 19:24 14215e87-1c82-45ec-9a34-4a8bb028d86f.6 -rw-rw----. 2 root root 0 Apr 29 19:24 14215e87-1c82-45ec-9a34-4a8bb028d86f.7 -rw-rw----. 2 root root 0 May 2 20:22 14215e87-1c82-45ec-9a34-4a8bb028d86f.8 -rw-rw----. 2 root root 0 May 2 20:22 14215e87-1c82-45ec-9a34-4a8bb028d86f.9 I ran the command virsh -r list command to make sure that there are no other vms running other than HostedEngine from all the nodes: [root@sulphur ~]# virsh -r list Id Name State ---------------------------------------------------- 616 HostedEngine running [root@tettnang ~]# virsh -r list Id Name State ---------------------------------------------------- [root@zod ~]# virsh -r list Id Name State ---------------------------------------------------- Version-Release number of selected component (if applicable): glusterfs-3.7.9-2.el7rhgs.x86_64 How reproducible: Tried it once Steps to Reproduce: 1. Create vms in HC setup using RHEV templates and Pools. 2. Now delete the pool so that all the vms are removed except HostedEngine 3. Actual results: I still see that there are some image files present inside the storage domain plus entries under .shard folder for both data and vmstore volumes. Expected results: When all the virtual machines are removed ideally there should not be any entries present in the .shard folder and in the storage domain. Additional info:
sosreports can be found in the link below: =========================================== http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/HC/1333643/
RCA: Handling of hardlinks in sharding xlator: Sharding translator performs hardlinks only on the base-file. When the last link is deleted, it deletes all the shard files associated with that base-file. So when an unlink fop comes to shard, it does a lookup (Equivalent of stat for user applications) to find out the number of links. If the number of links is > 1 then it just deletes that link but doesn't shards. If the number of links is 1, then it goes ahead and deletes all the shards as well. What happens in the rhev-setup on VM deletion: Kasturi (QE) selected all the VMs at once and performed delete. All these vm data will be moved to remove_me_xyz directory and host which is the owner of that VM will start doing unlinks in the background. It seems like there are hardlinks on the setup where VMs are stored, where links are shared between different hosts. [root@sulphur vmstore-brick1]# find . -type f | xargs getfattr -d -m. -e hex | grep trusted.gfid | sort | uniq -c | grep -v " 2 " 17 trusted.gfid=0x026de0b69bea4b148b6fb7261eb6c030 1 trusted.gfid=0x122c529ce3ab440f87edc2cf9ef79051 1 trusted.gfid=0x1383cf29af214231bbaafd3120fd4c27 16 trusted.gfid=0x13fd032531be41d4a5e8cff2aff02aca 16 trusted.gfid=0x14215e871c8245ec9a344a8bb028d86f 1 trusted.gfid=0x14807a3c2c0b4f6b92cea4a042856971 16 trusted.gfid=0x2b294ed3423c4e43afee404315a316ab 1 trusted.gfid=0x445a3b7cfb444fb290352b9087dc105b 1 trusted.gfid=0x4a21a35d946d4fe88283f3a2a29992cd 1 trusted.gfid=0x4ae3729419894b3bbb08467e32a088e7 1 trusted.gfid=0x655f5fdf9a59499bb3d329a7ab23ff5e 1 trusted.gfid=0x710d1bbe94e54a60ac64d916b1df672c 1 trusted.gfid=0x97abfcae67ff420797ad3014ce084e2e 1 trusted.gfid=0xc5816599efb547cca48a4a58a9751844 17 trusted.gfid=0xd939b5a7aa29453e814e32c0cf6e2608 1 trusted.gfid=0xd94592fb54ef46dbbf3da881ac94e465 17 trusted.gfid=0xdec2404ec7574494a78598ea4bd31381 1 trusted.gfid=0xfff5f116ef9140078c6536f23197695f [root@sulphur vmstore-brick1]# ls -lih .glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030 469762591 -rw-rw----. 17 vdsm kvm 512M Apr 13 12:10 .glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030 [root@sulphur vmstore-brick1]# find . -inum 469762591 ./.glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030 ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/8d0b02a5-d0f1-42e4-bc10-9baa9d0d5506/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/5308d74e-28e2-4158-b2f9-af898e14e6d0/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/7ca57bac-c614-402e-9bb4-0d172d69c476/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/5ee9dc38-f4a5-41ec-bd95-878165c022b1/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/99f4c43a-dc2f-46f1-b4b4-2cd289f43757/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/bedfd824-6560-454a-9ed1-6baf3e592428/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/9965825b-3e79-434a-8eec-b85994901ce5/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/548b241a-7958-40f4-b84a-161628f4e0e5/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/394f07f9-baab-4299-ba76-a13b9d2a949f/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/6e97619b-bbfb-4856-b63c-05727f84556f/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/ff21f8ba-5038-42f6-9b29-d56d87d87b21/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/3f61feef-0eb3-4cee-875b-d2e22dc8e7e3/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/a96768b4-548c-4c9d-80ac-fb4552e97a94/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/ae9715b8-c6a7-4125-948f-877fba1a7757/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/f8de7123-c638-4ee7-9ce0-c87bad52141c/fe3a369e-567b-4d46-a82d-5dd309af0f5d ./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/280f77cd-63b8-4cc9-9449-b10d1c5c36e1/fe3a369e-567b-4d46-a82d-5dd309af0f5d When all the VMs are deleted together, it can lead to different hosts trying to delete hardlinks in parallel in their respective remove_me_xxx directory. i.e. host1 can delete "./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/280f77cd-63b8-4cc9-9449-b10d1c5c36e1/fe3a369e-567b-4d46-a82d-5dd309af0f5d" where as host2 can delete "./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/f8de7123-c638-4ee7-9ce0-c87bad52141c/fe3a369e-567b-4d46-a82d-5dd309af0f5d". which are hardlinks to each other. So at the time when shard queries how many links the file has by doing lookup both of them will get a number greater than 1, so they both go ahead and delete the file. But one of those unlinks will be the last one but shard has no clue about this because at the time when it queried number of links is more than 1. This leads to shard not deleting shards. This is the reason for the bug. We have seen files with hardlink count '1' which suggests there is a bug even in posix-xlator which is not deleting gfid-link thinking it is not the last link. Pranith
Upstream patch: http://review.gluster.org/14334 Moving hte bug to POST.
https://code.engineering.redhat.com/gerrit/74766 <-- d/s patch
Verified and works fine with build glusterfs-3.7.9-6.el7rhgs.x86_64. Ran the following tests: 1) Directly created vms from template and deleted them from UI. All the .shards corresponding to the vm images got deleted except the shards for template since template is not deleted. 2) Created a pool from template and then launched vms and deleted them from UI. All the .shards corresponding to the vm images got deleted except the .shards for template since template is not deleted. Moving this to verified, will re open this bug in case hit again.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1240