Created attachment 621958 [details] dmesg output of server1 and server2 Description of problem: ----------------------- Self-heal of a single virtual machine (30GB) in a pure replicate volume (1x2) is not complete even after 20hours of self-heal on that VM. It is taking longer time than expected. Version-Release number of selected component (if applicable): -------------------------------------------------------------- [10/05/12 - 12:14:18 root@rhs-client7 ~]# rpm -qa | grep gluster glusterfs-3.3.0rhsvirt1-6.el6rhs.x86_64 glusterfs-rdma-3.3.0rhsvirt1-6.el6rhs.x86_64 vdsm-gluster-4.9.6-14.el6rhs.noarch gluster-swift-plugin-1.0-5.noarch gluster-swift-container-1.4.8-4.el6.noarch org.apache.hadoop.fs.glusterfs-glusterfs-0.20.2_0.2-1.noarch glusterfs-fuse-3.3.0rhsvirt1-6.el6rhs.x86_64 glusterfs-geo-replication-3.3.0rhsvirt1-6.el6rhs.x86_64 gluster-swift-proxy-1.4.8-4.el6.noarch gluster-swift-account-1.4.8-4.el6.noarch gluster-swift-doc-1.4.8-4.el6.noarch glusterfs-server-3.3.0rhsvirt1-6.el6rhs.x86_64 gluster-swift-1.4.8-4.el6.noarch gluster-swift-object-1.4.8-4.el6.noarch [10/05/12 - 12:14:24 root@rhs-client7 ~]# gluster --version glusterfs 3.3.0rhsvirt1 built on Sep 25 2012 14:53:06 Steps to Reproduce: ----------------------- 1.Create a pure replicate volume (1x2) with 2 servers and 1 brick on each server. This is the storage for the VM's. start the volume. 2.Set-up the KVM to use the volume as VM store. 3.Bring down brick1. 4.Create a VM. 5.Bring back brick1. 6.self-heal daemon process starts the self-heal of VM to brick1. Actual results: --------------- [10/05/12 - 12:08:53 root@rhs-client7 ~]# gluster volume heal replicate-rhevh2 info Heal operation on volume replicate-rhevh2 has been successful Brick rhs-client6.lab.eng.blr.redhat.com:/replicate-disk Number of entries: 0 Brick rhs-client7.lab.eng.blr.redhat.com:/replicate-disk Number of entries: 2 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids [10/05/12 - 12:08:59 root@rhs-client7 ~]# gluster volume heal replicate-rhevh2 info healedHeal operation on volume replicate-rhevh2 has been successful Brick rhs-client6.lab.eng.blr.redhat.com:/replicate-disk Number of entries: 0 Brick rhs-client7.lab.eng.blr.redhat.com:/replicate-disk Number of entries: 260 at path on brick ----------------------------------- 2012-10-05 12:06:29 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 12:06:29 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 11:56:31 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 11:56:31 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 11:46:34 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 11:46:34 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 11:36:30 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 11:36:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 11:26:33 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 11:26:33 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 11:16:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 11:16:25 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 11:06:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 11:06:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 10:56:29 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 10:56:29 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 10:46:30 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 10:46:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 10:36:30 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 10:36:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 10:26:31 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 10:26:31 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 10:16:30 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 10:16:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 10:06:30 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 10:06:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 09:56:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 09:56:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 09:46:29 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 09:46:29 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 09:36:31 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 09:36:31 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 09:26:28 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 09:26:28 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 09:16:31 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 09:16:31 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 09:06:28 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 09:06:28 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 08:56:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 08:56:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 08:46:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 08:46:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 08:36:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 08:36:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 08:26:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 08:26:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 08:16:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 08:16:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 08:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 08:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 07:56:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 07:56:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 07:46:29 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 07:46:29 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 07:36:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 07:36:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 07:26:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 07:26:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 07:16:30 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 07:16:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 07:06:29 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 07:06:29 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 06:56:29 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 06:56:29 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 06:46:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 06:46:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 06:36:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 06:36:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 06:26:28 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 06:26:28 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 06:16:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 06:16:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 06:06:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 06:06:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 05:56:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 05:56:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 05:46:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 05:46:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 05:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 05:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 05:26:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 05:26:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 05:16:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 05:16:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 05:06:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 05:06:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 04:56:28 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 04:56:28 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 04:46:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 04:46:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 04:36:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 04:36:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 04:26:28 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 04:26:28 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 04:16:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 04:16:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 04:06:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 04:06:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 03:56:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 03:56:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 03:46:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 03:46:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 03:36:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 03:36:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 03:26:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 03:26:25 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 03:16:20 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 03:16:20 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 03:06:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 03:06:25 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 02:56:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 02:56:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 02:46:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 02:46:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 02:36:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 02:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 02:26:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 02:26:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 02:16:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 02:16:25 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 02:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 02:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 01:56:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 01:56:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 01:46:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 01:46:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 01:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 01:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 01:26:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 01:26:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 01:16:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 01:16:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 01:06:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 01:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 00:56:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 00:56:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 00:46:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 00:46:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 00:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 00:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 00:26:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 00:26:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 00:16:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 00:16:25 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-05 00:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-05 00:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 23:56:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 23:56:25 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 23:46:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 23:46:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 23:36:20 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 23:36:20 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 23:26:18 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 23:26:18 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 23:16:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 23:16:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 23:06:20 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 23:06:20 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 22:56:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 22:56:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 22:46:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 22:46:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 22:36:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 22:36:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 22:26:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 22:26:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 22:16:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 22:16:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 22:06:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 22:06:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 21:56:18 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 21:56:18 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 21:46:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 21:46:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 21:36:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 21:36:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 21:26:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 21:26:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 21:16:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 21:16:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 21:06:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 21:06:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 20:56:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 20:56:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 20:46:20 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 20:46:20 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 20:36:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 20:36:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 20:26:18 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 20:26:18 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 20:16:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 20:16:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 20:06:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 20:06:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 19:56:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 19:56:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 19:46:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 19:46:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 19:36:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 19:36:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 19:26:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 19:26:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 19:16:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 19:16:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 19:06:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 19:06:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 18:56:46 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 18:56:46 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 18:46:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 18:46:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 18:36:20 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 18:36:20 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 18:26:18 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 18:26:18 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 18:16:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 18:16:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 18:06:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 18:06:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 17:56:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 17:56:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 17:46:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 17:46:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 17:36:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 17:36:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 17:26:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 17:26:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 17:16:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 17:16:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 17:06:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 17:06:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 16:58:53 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 16:58:53 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 16:46:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 16:46:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 16:36:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 16:36:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 16:26:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 16:26:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 16:16:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 16:16:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 16:06:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 16:06:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 15:56:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 15:56:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 15:49:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 15:45:04 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 15:45:04 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 15:40:00 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 15:26:35 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 15:26:35 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 15:06:20 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 15:06:20 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 14:56:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids 2012-10-04 14:56:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 14:46:56 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624.meta 2012-10-04 14:46:56 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624.lease 2012-10-04 14:46:56 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 2012-10-04 14:42:41 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9 2012-10-04 14:42:41 /baa779f1-7935-4083-bd62-c33d43b242c3/images 2012-10-04 14:42:41 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids [10/05/12 - 12:10:11 root@rhs-client7 ~]# getfattr -d -e hex -m . /replicate-disk//baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 getfattr: Removing leading '/' from absolute path names # file: replicate-disk//baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624 security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.replicate-rhevh2-client-0=0x0000014a0000000000000000 trusted.afr.replicate-rhevh2-client-1=0x000000000000000000000000 trusted.gfid=0xca65569309fa47698742e29d884ceca9 [10/05/12 - 12:10:15 root@rhs-client7 ~]# getfattr -d -e hex -m . /replicate-disk//baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids getfattr: Removing leading '/' from absolute path names # file: replicate-disk//baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000 trusted.afr.replicate-rhevh2-client-0=0x000000160000000000000000 trusted.afr.replicate-rhevh2-client-1=0x000000000000000000000000 trusted.gfid=0xc874c72408534f2da6e3087549630ec7 [10/05/12 - 12:10:31 root@rhs-client7 ~]# ls /replicate-disk/.glusterfs/indices/xattrop/ c874c724-0853-4f2d-a6e3-087549630ec7 ca655693-09fa-4769-8742-e29d884ceca9 xattrop-2b78b486-50dd-48e9-968b-090d8183f150 Additional info:- ------------------- [10/05/12 - 12:20:38 root@rhs-client7 ~]# gluster volume status replicate-rhevh2 Status of volume: replicate-rhevh2 Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick rhs-client6.lab.eng.blr.redhat.com:/replicate-dis k 24014 Y 3613 Brick rhs-client7.lab.eng.blr.redhat.com:/replicate-dis k 24013 Y 23353 NFS Server on localhost 38467 Y 26451 Self-heal Daemon on localhost N/A Y 26457 NFS Server on rhs-client6.lab.eng.blr.redhat.com 38467 Y 3619 Self-heal Daemon on rhs-client6.lab.eng.blr.redhat.com N/A Y 3625 NFS Server on rhs-client8.lab.eng.blr.redhat.com 38467 Y 26104 Self-heal Daemon on rhs-client8.lab.eng.blr.redhat.com N/A Y 26110
[10/05/12 - 12:24:48 root@rhs-client7 ~]# gluster volume info replicate-rhevh2 Volume Name: replicate-rhevh2 Type: Replicate Volume ID: 1e697968-2e90-4589-8225-f596fee8af97 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: rhs-client6.lab.eng.blr.redhat.com:/replicate-disk Brick2: rhs-client7.lab.eng.blr.redhat.com:/replicate-disk Options Reconfigured: performance.quick-read: disable performance.io-cache: disable performance.stat-prefetch: disable performance.read-ahead: disable cluster.eager-lock: enable storage.linux-aio: disable
we could able to recreate the issue once again. Server2:- ---------- [10/18/12 - 11:30:27 root@rhs-client7 ~]# ls /disk0/.glusterfs/indices/xattrop/ 14d300c0-c558-4b78-afcf-63a4a8e2dc73 ccededf4-6be4-47af-aff2-973a538fb50f xattrop-10eb7b69-338a-4e14-8d65-96a2c649f520 [10/18/12 - 12:22:48 root@rhs-client7 ~]# gluster volume heal replicate info Heal operation on volume replicate has been successful Brick rhs-client6.lab.eng.blr.redhat.com:/disk0 Number of entries: 1 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids Brick rhs-client7.lab.eng.blr.redhat.com:/disk0 Number of entries: 3 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids [10/18/12 - 12:24:01 root@rhs-client7 ~]# gluster volume heal replicate info healed Heal operation on volume replicate has been successful Brick rhs-client6.lab.eng.blr.redhat.com:/disk0 Number of entries: 38 at path on brick ----------------------------------- 2012-10-18 11:59:15 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 10:19:14 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 10:09:14 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 09:39:14 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 08:59:13 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 08:19:13 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 07:29:13 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 06:49:12 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 04:19:11 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 04:09:11 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 03:59:11 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 03:59:11 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 03:49:11 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 03:19:11 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 01:19:10 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 23:59:09 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 21:09:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 19:49:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 19:29:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 19:19:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 19:09:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 19:09:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 18:59:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 18:49:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 18:39:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 18:29:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 18:29:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 18:19:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 18:09:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 17:49:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 16:59:05 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 16:49:05 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 14:49:04 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 14:19:04 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 14:09:04 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 13:59:04 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 13:49:04 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 12:59:03 <gfid:feafd34b-833a-47ec-9b75-5f2c9777e562>/0812e8dd-e7f9-4f91-a872-5235499db4cb Brick rhs-client7.lab.eng.blr.redhat.com:/disk0 Number of entries: 83 at path on brick ----------------------------------- 2012-10-18 12:23:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 12:13:25 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 11:53:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 11:43:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 11:33:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 11:23:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 11:13:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 11:03:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 11:03:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 10:33:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 10:23:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 10:13:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 10:03:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 09:53:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 09:43:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 09:33:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 09:23:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 09:13:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 09:03:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 08:53:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 08:53:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 08:43:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 08:43:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 08:33:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 08:33:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 08:23:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 08:23:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 08:13:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 08:13:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 08:03:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 08:03:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 07:53:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 07:53:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 07:43:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 07:33:18 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 07:23:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 07:03:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-18 04:13:19 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 03:33:18 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 03:23:04 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-18 03:03:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 02:53:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 02:13:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 01:33:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 00:53:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-18 00:13:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 22:33:08 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 22:23:09 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 22:13:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 22:03:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 21:53:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 21:43:10 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 21:33:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 21:23:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 21:23:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 21:23:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 21:13:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 21:13:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 20:53:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 20:43:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 20:43:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 20:33:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 20:33:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 20:23:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb 2012-10-17 20:23:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 20:13:00 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 20:03:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 20:03:00 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 19:53:00 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 19:43:00 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 19:23:12 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 18:43:13 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 18:23:13 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 17:23:14 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids 2012-10-17 15:02:58 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 14:52:58 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 14:42:58 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 14:32:58 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 14:22:58 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 13:52:58 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 13:32:57 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 13:22:57 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c 2012-10-17 13:02:57 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c [10/18/12 - 12:24:04 root@rhs-client7 ~]# Server1:- --------- [10/18/12 - 11:31:38 root@rhs-client6 ~]# ls /disk0/.glusterfs/indices/xattrop/ xattrop-338b90ec-6977-4271-baa3-79d2c8a33aaa
I did a quick test of a similar (non-VM but similar size) scenario. Self-heal seemed to do about 1600 network writes per second. 83% were very short (probably changelog operations) and the remainder were just over 128KB (data writes). That works out to 36MB/s and, sure enough, the self-heal completed in under 15 minutes. Therefore, the difference is either environment-related or something that VMs specifically do that interferes with the self-heal process. I'll experiment some more along the second line of inquiry. Meanwhile, it would be very helpful if someone could do the same sort of strace that I did to get an idea of how the self-heal daemon is actually spending its time. Here's the command I used. strace -o /tmp/healcalls -tt -T -p 30775 Here, 30775 is the PID for my glustershd process that's doing all the I/O. Please collect about 10s of data, hit ^C to detach strace from the process, and then add the resulting "healcalls" file here as an attachment.
I ran a similar test with a client generating I/O during the self-heal. The self-heal seemed unaffected and completed in approximately the same amount of time as before. The only semi-interesting thing about the test is that I restarted the client I/O process and it blocked on its first write until the self-heal was complete - i.e. granular self-heal wasn't seeming very granular in that case.
Same tests on 3.3.0 (had been using master) yielded exactly the same result.
For some reason client6.lab.eng.blr.redhat.com says the brick is online according to the output in bug-description, but: In the sos_commands on client7.lab.eng.blr.redhat.com this is what I see. Status of volume: replicate-rhevh2 Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick rhs-client6.lab.eng.blr.redhat.com:/replicate-dis k 24014 N 3613 Brick rhs-client7.lab.eng.blr.redhat.com:/replicate-dis k 24013 Y 23353 The brick is down. Shwetha, Is the network fine between the two machines? This is not for this volume but in the logs I also see the following messages for other volume: [2012-10-08 15:26:02.113251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-08 15:22:36.490612 (xid=0x1682x) [2012-10-08 15:26:02.113273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error [2012-10-08 15:26:02.113290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected [2012-10-08 15:27:30.051150] W [socket.c:1512:__socket_proto_state_machine] 0-replicate-rhevh2-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.36.30:24014) Status of volume: dist-rep-rhevh Gluster process Port Online Pid ------------------------------------------------------------------------------ Brick rhs-client6.lab.eng.blr.redhat.com:/disk1 N/A Y 2926 Brick rhs-client7.lab.eng.blr.redhat.com:/disk1 24011 Y 23358 The brick is online but the port is N/A. It seems a lot like either glusterd port-map issues or network problems between the client6/client7 is causing the issue.
There are no network problems b/w the machines. Also , the issue is not happening any more after the fix went in for the bug. https://bugzilla.redhat.com/show_bug.cgi?id=866459
We were never able to re-create this bug in our setup. Please feel free to re-open with relevant logs if observed again.