Bug 863333

Summary:

self-heal daemon taking more time to self-heal a 30GB VM. ( > than 20 hours )

Product:

[Red Hat Storage] Red Hat Gluster Storage

Reporter:

spandura

Component:

glusterfs

Assignee:

Jeff Darcy <jdarcy>

Status:

CLOSED WORKSFORME

QA Contact:

spandura

Severity:

high

Docs Contact:

Priority:

high

Version:

2.0

CC:

grajaiya, jdarcy, pkarampu, rhs-bugs, sdharane, shaines, vbellur

Target Milestone:

---

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2012-11-15 07:33:05 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
dmesg output of server1 and server2	none

Description spandura 2012-10-05 06:54:32 UTC

Created attachment 621958 [details]
dmesg output of server1 and server2

Description of problem:
-----------------------
Self-heal of a single virtual machine (30GB) in a pure replicate volume (1x2) is not complete even after 20hours of self-heal on that VM. 

It is taking longer time than expected. 

Version-Release number of selected component (if applicable):
--------------------------------------------------------------
[10/05/12 - 12:14:18 root@rhs-client7 ~]# rpm -qa | grep gluster
glusterfs-3.3.0rhsvirt1-6.el6rhs.x86_64
glusterfs-rdma-3.3.0rhsvirt1-6.el6rhs.x86_64
vdsm-gluster-4.9.6-14.el6rhs.noarch
gluster-swift-plugin-1.0-5.noarch
gluster-swift-container-1.4.8-4.el6.noarch
org.apache.hadoop.fs.glusterfs-glusterfs-0.20.2_0.2-1.noarch
glusterfs-fuse-3.3.0rhsvirt1-6.el6rhs.x86_64
glusterfs-geo-replication-3.3.0rhsvirt1-6.el6rhs.x86_64
gluster-swift-proxy-1.4.8-4.el6.noarch
gluster-swift-account-1.4.8-4.el6.noarch
gluster-swift-doc-1.4.8-4.el6.noarch
glusterfs-server-3.3.0rhsvirt1-6.el6rhs.x86_64
gluster-swift-1.4.8-4.el6.noarch
gluster-swift-object-1.4.8-4.el6.noarch
[10/05/12 - 12:14:24 root@rhs-client7 ~]# gluster --version
glusterfs 3.3.0rhsvirt1 built on Sep 25 2012 14:53:06

Steps to Reproduce:
-----------------------
1.Create a pure replicate volume (1x2) with 2 servers and 1 brick on each server. This is the storage for the VM's. start the volume.
2.Set-up the KVM to use the volume as VM store. 
3.Bring down brick1.
4.Create a VM. 
5.Bring back brick1.
6.self-heal daemon process starts the self-heal of VM to brick1. 
  
Actual results:
---------------
[10/05/12 - 12:08:53 root@rhs-client7 ~]# gluster volume heal replicate-rhevh2 info
Heal operation on volume replicate-rhevh2 has been successful

Brick rhs-client6.lab.eng.blr.redhat.com:/replicate-disk
Number of entries: 0

Brick rhs-client7.lab.eng.blr.redhat.com:/replicate-disk
Number of entries: 2
/baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
/baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids


[10/05/12 - 12:08:59 root@rhs-client7 ~]# gluster volume heal replicate-rhevh2 info healedHeal operation on volume replicate-rhevh2 has been successful

Brick rhs-client6.lab.eng.blr.redhat.com:/replicate-disk
Number of entries: 0

Brick rhs-client7.lab.eng.blr.redhat.com:/replicate-disk
Number of entries: 260
at                    path on brick
-----------------------------------
2012-10-05 12:06:29 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 12:06:29 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 11:56:31 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 11:56:31 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 11:46:34 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 11:46:34 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 11:36:30 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 11:36:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 11:26:33 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 11:26:33 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 11:16:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 11:16:25 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 11:06:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 11:06:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 10:56:29 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 10:56:29 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 10:46:30 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 10:46:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 10:36:30 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 10:36:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 10:26:31 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 10:26:31 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 10:16:30 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 10:16:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 10:06:30 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 10:06:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 09:56:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 09:56:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 09:46:29 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 09:46:29 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 09:36:31 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 09:36:31 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 09:26:28 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 09:26:28 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 09:16:31 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 09:16:31 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 09:06:28 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 09:06:28 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 08:56:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 08:56:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 08:46:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 08:46:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 08:36:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 08:36:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 08:26:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 08:26:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 08:16:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 08:16:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 08:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 08:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 07:56:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 07:56:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 07:46:29 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 07:46:29 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 07:36:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 07:36:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 07:26:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 07:26:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 07:16:30 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 07:16:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 07:06:29 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 07:06:29 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 06:56:29 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 06:56:29 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 06:46:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 06:46:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 06:36:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 06:36:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 06:26:28 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 06:26:28 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 06:16:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 06:16:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 06:06:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 06:06:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 05:56:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 05:56:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 05:46:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 05:46:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 05:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 05:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 05:26:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 05:26:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 05:16:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 05:16:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 05:06:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 05:06:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 04:56:28 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 04:56:28 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 04:46:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 04:46:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 04:36:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 04:36:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 04:26:28 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 04:26:28 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 04:16:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 04:16:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 04:06:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 04:06:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 03:56:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 03:56:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 03:46:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 03:46:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 03:36:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 03:36:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 03:26:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 03:26:25 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 03:16:20 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 03:16:20 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 03:06:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 03:06:25 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 02:56:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 02:56:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 02:46:26 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 02:46:26 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 02:36:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 02:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 02:26:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 02:26:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 02:16:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 02:16:25 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 02:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 02:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 01:56:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 01:56:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 01:46:27 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 01:46:27 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 01:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 01:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 01:26:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 01:26:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 01:16:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 01:16:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 01:06:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 01:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 00:56:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 00:56:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 00:46:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 00:46:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 00:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 00:36:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 00:26:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 00:26:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 00:16:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 00:16:25 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-05 00:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-05 00:06:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 23:56:25 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 23:56:25 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 23:46:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 23:46:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 23:36:20 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 23:36:20 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 23:26:18 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 23:26:18 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 23:16:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 23:16:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 23:06:20 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 23:06:20 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 22:56:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 22:56:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 22:46:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 22:46:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 22:36:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 22:36:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 22:26:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 22:26:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 22:16:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 22:16:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 22:06:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 22:06:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 21:56:18 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 21:56:18 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 21:46:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 21:46:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 21:36:24 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 21:36:24 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 21:26:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 21:26:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 21:16:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 21:16:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 21:06:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 21:06:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 20:56:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 20:56:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 20:46:20 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 20:46:20 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 20:36:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 20:36:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 20:26:18 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 20:26:18 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 20:16:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 20:16:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 20:06:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 20:06:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 19:56:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 19:56:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 19:46:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 19:46:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 19:36:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 19:36:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 19:26:21 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 19:26:21 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 19:16:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 19:16:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 19:06:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 19:06:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 18:56:46 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 18:56:46 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 18:46:19 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 18:46:19 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 18:36:20 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 18:36:20 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 18:26:18 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 18:26:18 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 18:16:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 18:16:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 18:06:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 18:06:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 17:56:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 17:56:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 17:46:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 17:46:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 17:36:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 17:36:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 17:26:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 17:26:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 17:16:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 17:16:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 17:06:22 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 17:06:22 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 16:58:53 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 16:58:53 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 16:46:17 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 16:46:17 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 16:36:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 16:36:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 16:26:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 16:26:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 16:16:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 16:16:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 16:06:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 16:06:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 15:56:16 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 15:56:16 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 15:49:30 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 15:45:04 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 15:45:04 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 15:40:00 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 15:26:35 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 15:26:35 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 15:06:20 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 15:06:20 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 14:56:23 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
2012-10-04 14:56:23 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 14:46:56 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624.meta
2012-10-04 14:46:56 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624.lease
2012-10-04 14:46:56 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
2012-10-04 14:42:41 /baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9
2012-10-04 14:42:41 /baa779f1-7935-4083-bd62-c33d43b242c3/images
2012-10-04 14:42:41 /baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids


[10/05/12 - 12:10:11 root@rhs-client7 ~]# getfattr -d -e hex -m . /replicate-disk//baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
getfattr: Removing leading '/' from absolute path names
# file: replicate-disk//baa779f1-7935-4083-bd62-c33d43b242c3/images/1f41c42f-d8c8-4869-9466-9c0294f735c9/a77e5a15-3492-4832-988d-40c96e71f624
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.replicate-rhevh2-client-0=0x0000014a0000000000000000
trusted.afr.replicate-rhevh2-client-1=0x000000000000000000000000
trusted.gfid=0xca65569309fa47698742e29d884ceca9


[10/05/12 - 12:10:15 root@rhs-client7 ~]# getfattr -d -e hex -m . /replicate-disk//baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
getfattr: Removing leading '/' from absolute path names
# file: replicate-disk//baa779f1-7935-4083-bd62-c33d43b242c3/dom_md/ids
security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.replicate-rhevh2-client-0=0x000000160000000000000000
trusted.afr.replicate-rhevh2-client-1=0x000000000000000000000000
trusted.gfid=0xc874c72408534f2da6e3087549630ec7

[10/05/12 - 12:10:31 root@rhs-client7 ~]# ls /replicate-disk/.glusterfs/indices/xattrop/
c874c724-0853-4f2d-a6e3-087549630ec7  ca655693-09fa-4769-8742-e29d884ceca9  xattrop-2b78b486-50dd-48e9-968b-090d8183f150

Additional info:-
-------------------

[10/05/12 - 12:20:38 root@rhs-client7 ~]# gluster volume status replicate-rhevh2
Status of volume: replicate-rhevh2
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick rhs-client6.lab.eng.blr.redhat.com:/replicate-dis
k							24014	Y	3613
Brick rhs-client7.lab.eng.blr.redhat.com:/replicate-dis
k							24013	Y	23353
NFS Server on localhost					38467	Y	26451
Self-heal Daemon on localhost				N/A	Y	26457
NFS Server on rhs-client6.lab.eng.blr.redhat.com	38467	Y	3619
Self-heal Daemon on rhs-client6.lab.eng.blr.redhat.com	N/A	Y	3625
NFS Server on rhs-client8.lab.eng.blr.redhat.com	38467	Y	26104
Self-heal Daemon on rhs-client8.lab.eng.blr.redhat.com	N/A	Y	26110

Comment 1 spandura 2012-10-05 06:55:17 UTC

[10/05/12 - 12:24:48 root@rhs-client7 ~]# gluster volume info replicate-rhevh2
 
Volume Name: replicate-rhevh2
Type: Replicate
Volume ID: 1e697968-2e90-4589-8225-f596fee8af97
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: rhs-client6.lab.eng.blr.redhat.com:/replicate-disk
Brick2: rhs-client7.lab.eng.blr.redhat.com:/replicate-disk
Options Reconfigured:
performance.quick-read: disable
performance.io-cache: disable
performance.stat-prefetch: disable
performance.read-ahead: disable
cluster.eager-lock: enable
storage.linux-aio: disable

Comment 5 spandura 2012-10-18 06:55:40 UTC

we could able to recreate the issue once again. 

Server2:-
----------

[10/18/12 - 11:30:27 root@rhs-client7 ~]# ls /disk0/.glusterfs/indices/xattrop/
14d300c0-c558-4b78-afcf-63a4a8e2dc73  ccededf4-6be4-47af-aff2-973a538fb50f  xattrop-10eb7b69-338a-4e14-8d65-96a2c649f520

[10/18/12 - 12:22:48 root@rhs-client7 ~]# gluster volume heal replicate info
Heal operation on volume replicate has been successful

Brick rhs-client6.lab.eng.blr.redhat.com:/disk0
Number of entries: 1
/2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids

Brick rhs-client7.lab.eng.blr.redhat.com:/disk0
Number of entries: 3
/2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
/2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
/2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids

[10/18/12 - 12:24:01 root@rhs-client7 ~]# gluster volume heal replicate info healed
Heal operation on volume replicate has been successful

Brick rhs-client6.lab.eng.blr.redhat.com:/disk0
Number of entries: 38
at                    path on brick
-----------------------------------
2012-10-18 11:59:15 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 10:19:14 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 10:09:14 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 09:39:14 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 08:59:13 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 08:19:13 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 07:29:13 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 06:49:12 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 04:19:11 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 04:09:11 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 03:59:11 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 03:59:11 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 03:49:11 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 03:19:11 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 01:19:10 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 23:59:09 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 21:09:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 19:49:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 19:29:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 19:19:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 19:09:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 19:09:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 18:59:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 18:49:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 18:39:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 18:29:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 18:29:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 18:19:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 18:09:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 17:49:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 16:59:05 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 16:49:05 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 14:49:04 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 14:19:04 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 14:09:04 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 13:59:04 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 13:49:04 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 12:59:03 <gfid:feafd34b-833a-47ec-9b75-5f2c9777e562>/0812e8dd-e7f9-4f91-a872-5235499db4cb

Brick rhs-client7.lab.eng.blr.redhat.com:/disk0
Number of entries: 83
at                    path on brick
-----------------------------------
2012-10-18 12:23:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 12:13:25 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 11:53:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 11:43:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 11:33:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 11:23:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 11:13:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 11:03:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 11:03:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 10:33:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 10:23:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 10:13:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 10:03:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 09:53:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 09:43:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 09:33:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 09:23:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 09:13:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 09:03:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 08:53:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 08:53:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 08:43:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 08:43:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 08:33:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 08:33:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 08:23:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 08:23:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 08:13:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 08:13:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 08:03:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 08:03:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 07:53:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 07:53:06 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 07:43:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 07:33:18 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 07:23:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 07:03:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-18 04:13:19 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 03:33:18 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 03:23:04 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-18 03:03:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 02:53:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 02:13:17 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 01:33:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 00:53:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-18 00:13:16 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 22:33:08 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 22:23:09 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 22:13:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 22:03:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 21:53:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 21:43:10 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 21:33:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 21:23:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 21:23:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 21:23:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 21:13:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 21:13:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 20:53:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 20:43:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 20:43:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 20:33:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 20:33:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 20:23:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/75029cd5-a871-4c14-969f-7d7b3363a1a8/0812e8dd-e7f9-4f91-a872-5235499db4cb
2012-10-17 20:23:01 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 20:13:00 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 20:03:07 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 20:03:00 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 19:53:00 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 19:43:00 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 19:23:12 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 18:43:13 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 18:23:13 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 17:23:14 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/dom_md/ids
2012-10-17 15:02:58 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 14:52:58 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 14:42:58 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 14:32:58 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 14:22:58 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 13:52:58 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 13:32:57 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 13:22:57 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
2012-10-17 13:02:57 /2a1bdd6f-8de1-46bb-acba-5f2cfe282b38/images/d44cf496-ef59-4b12-94b3-84c91587bb9e/28b639ca-efe6-4cf3-a2ff-47392df14b0c
[10/18/12 - 12:24:04 root@rhs-client7 ~]# 

Server1:-
---------

[10/18/12 - 11:31:38 root@rhs-client6 ~]# ls /disk0/.glusterfs/indices/xattrop/
xattrop-338b90ec-6977-4271-baa3-79d2c8a33aaa

Comment 6 Jeff Darcy 2012-10-23 21:27:34 UTC

I did a quick test of a similar (non-VM but similar size) scenario.  Self-heal seemed to do about 1600 network writes per second.  83% were very short (probably changelog operations) and the remainder were just over 128KB (data writes).  That works out to 36MB/s and, sure enough, the self-heal completed in under 15 minutes.  Therefore, the difference is either environment-related or something that VMs specifically do that interferes with the self-heal process.  I'll experiment some more along the second line of inquiry.  Meanwhile, it would be very helpful if someone could do the same sort of strace that I did to get an idea of how the self-heal daemon is actually spending its time.  Here's the command I used.

    strace -o /tmp/healcalls -tt -T -p 30775

Here, 30775 is the PID for my glustershd process that's doing all the I/O.  Please collect about 10s of data, hit ^C to detach strace from the process, and then add the resulting "healcalls" file here as an attachment.

Comment 7 Jeff Darcy 2012-10-24 16:01:55 UTC

I ran a similar test with a client generating I/O during the self-heal.  The self-heal seemed unaffected and completed in approximately the same amount of time as before.  The only semi-interesting thing about the test is that I restarted the client I/O process and it blocked on its first write until the self-heal was complete - i.e. granular self-heal wasn't seeming very granular in that case.

Comment 8 Jeff Darcy 2012-10-24 19:49:20 UTC

Same tests on 3.3.0 (had been using master) yielded exactly the same result.

Comment 9 Pranith Kumar K 2012-10-25 18:21:50 UTC

For some reason client6.lab.eng.blr.redhat.com says the brick is online according to the output in bug-description, but:
In the sos_commands on client7.lab.eng.blr.redhat.com this is what I see. 
Status of volume: replicate-rhevh2
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick rhs-client6.lab.eng.blr.redhat.com:/replicate-dis
k                                                       24014   N       3613
Brick rhs-client7.lab.eng.blr.redhat.com:/replicate-dis
k                                                       24013   Y       23353
The brick is down.

Shwetha,
     Is the network fine between the two machines?

This is not for this volume but in the logs I also see the following messages for other volume:
[2012-10-08 15:26:02.113251] E [rpc-clnt.c:373:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x78) [0x3a9f60f818] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xb0) [0x3a9f60f4d0] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x3a9f60ef3e]))) 0-dist-rep-rhevh-client-3: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2012-10-08 15:22:36.490612 (xid=0x1682x)
[2012-10-08 15:26:02.113273] W [client-handshake.c:1797:client_dump_version_cbk] 0-dist-rep-rhevh-client-3: received RPC status error
[2012-10-08 15:26:02.113290] I [client.c:2090:client_rpc_notify] 0-dist-rep-rhevh-client-3: disconnected
[2012-10-08 15:27:30.051150] W [socket.c:1512:__socket_proto_state_machine] 0-replicate-rhevh2-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (10.70.36.30:24014)

Status of volume: dist-rep-rhevh
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick rhs-client6.lab.eng.blr.redhat.com:/disk1         N/A     Y       2926
Brick rhs-client7.lab.eng.blr.redhat.com:/disk1         24011   Y       23358

The brick is online but the port is N/A.
It seems a lot like either glusterd port-map issues or network problems between the client6/client7 is causing the issue.

Comment 10 spandura 2012-11-05 05:13:52 UTC

There are no network problems b/w the machines. 

Also , the issue is not happening any more after the fix went in for the bug. https://bugzilla.redhat.com/show_bug.cgi?id=866459

Comment 11 Pranith Kumar K 2012-11-15 07:33:05 UTC

We were never able to re-create this bug in our setup. Please feel free to re-open with relevant logs if observed again.