Bug 916934

Summary:	Rebalance failures/Very slow
Product:	[Community] GlusterFS	Reporter:	Philip <flips01>
Component:	core	Assignee:	Nagaprasad Sathyanarayana <nsathyan>
Status:	CLOSED DEFERRED	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	medium
Version:	3.3.1	CC:	bugs, gluster-bugs, smohan
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-12-14 19:40:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Philip 2013-03-01 09:41:21 UTC

Hi,

I have added 11 new bricks to my gluster system and perform a rebalance right now. The rebalance itself is really slow and will take more than a week to complete and causes lots of failures:

Node    Rebalanced-files          size       scanned      failures         status
---------      -----------   -----------   -----------   -----------   ------------
s003                0        0Bytes         26025             0    in progress
s004                0        0Bytes         26026             0    in progress
s001                0        0Bytes         26025             0    in progress
s002             4173        16.2GB          9461          1344    in progress

My volume info looks as follows:

Volume Name: data
Type: Distributed-Replicate
Volume ID: 3a3ce21e-c07e-486a-b0ca-5e93f68152d5
Status: Started
Number of Bricks: 22 x 2 = 44
Transport-type: tcp
Bricks:
Brick1: s002:/var/gfs/sdb/data-s002-1
Brick2: s003:/var/gfs/sdb/data-s003-1
Brick3: s002:/var/gfs/sdc/data-s002-2
Brick4: s003:/var/gfs/sdc/data-s003-2
Brick5: s002:/var/gfs/sdd/data-s002-3
Brick6: s003:/var/gfs/sdd/data-s003-3
Brick7: s002:/var/gfs/sde/data-s002-4
Brick8: s003:/var/gfs/sde/data-s003-4
Brick9: s002:/var/gfs/sdf/data-s002-5
Brick10: s003:/var/gfs/sdf/data-s003-5
Brick11: s002:/var/gfs/sdg/data-s002-6
Brick12: s003:/var/gfs/sdg/data-s003-6
Brick13: s002:/var/gfs/sdh/data-s002-7
Brick14: s003:/var/gfs/sdh/data-s003-7
Brick15: s002:/var/gfs/sdi/data-s002-8
Brick16: s003:/var/gfs/sdi/data-s003-8
Brick17: s002:/var/gfs/sdj/data-s002-9
Brick18: s003:/var/gfs/sdj/data-s003-9
Brick19: s002:/var/gfs/sdk/data-s002-10
Brick20: s003:/var/gfs/sdk/data-s003-10
Brick21: s002:/var/gfs/sdl/data-s002-11
Brick22: s003:/var/gfs/sdl/data-s003-11
Brick23: s001:/var/gfs/sdb/data-s001-1
Brick24: s004:/var/gfs/sdb/data-s004-1
Brick25: s001:/var/gfs/sdc/data-s001-2
Brick26: s004:/var/gfs/sdc/data-s004-2
Brick27: s001:/var/gfs/sdd/data-s001-3
Brick28: s004:/var/gfs/sdd/data-s004-3
Brick29: s001:/var/gfs/sde/data-s001-4
Brick30: s004:/var/gfs/sde/data-s004-4
Brick31: s001:/var/gfs/sdf/data-s001-5
Brick32: s004:/var/gfs/sdf/data-s004-5
Brick33: s001:/var/gfs/sdg/data-s001-6
Brick34: s004:/var/gfs/sdg/data-s004-6
Brick35: s001:/var/gfs/sdh/data-s001-7
Brick36: s004:/var/gfs/sdh/data-s004-7
Brick37: s001:/var/gfs/sdi/data-s001-8
Brick38: s004:/var/gfs/sdi/data-s004-8
Brick39: s001:/var/gfs/sdj/data-s001-9
Brick40: s004:/var/gfs/sdj/data-s004-9
Brick41: s001:/var/gfs/sdk/data-s001-10
Brick42: s004:/var/gfs/sdk/data-s004-10
Brick43: s001:/var/gfs/sdl/data-s001-11
Brick44: s004:/var/gfs/sdl/data-s004-11
Options Reconfigured:
network.ping-timeout: 15
cluster.data-self-heal-algorithm: full


The error messages in the rebalance log are as follows:
[2013-03-01 10:27:37.300801] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/cS6ENja-GV4.dat
[2013-03-01 10:27:37.316505] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/dnU-DG0u4ZI.dat on data-replicate-10
[2013-03-01 10:27:37.317639] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.317656] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.320451] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-28: remote operation failed: File exists. Path: /files/1a/2/dnU-DG0u4ZI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.320544] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-29: remote operation failed: File exists. Path: /files/1a/2/dnU-DG0u4ZI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.320615] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.320650] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/dnU-DG0u4ZI.dat
[2013-03-01 10:27:37.341623] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/i9It0K0tRXA.dat on data-replicate-10
[2013-03-01 10:27:37.342987] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.343010] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.343606] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-27: remote operation failed: File exists. Path: /files/1a/2/i9It0K0tRXA.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.343636] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-26: remote operation failed: File exists. Path: /files/1a/2/i9It0K0tRXA.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.343774] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.343794] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/i9It0K0tRXA.dat
[2013-03-01 10:27:37.366354] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/ld0ie2PRNSc.dat on data-replicate-10
[2013-03-01 10:27:37.367343] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.367454] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.368632] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-26: remote operation failed: File exists. Path: /files/1a/2/ld0ie2PRNSc.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.368659] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-27: remote operation failed: File exists. Path: /files/1a/2/ld0ie2PRNSc.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.368690] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.368712] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/ld0ie2PRNSc.dat
[2013-03-01 10:27:37.390409] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/liDZQzxeDGY.dat on data-replicate-10
[2013-03-01 10:27:37.391500] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.391713] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.392124] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-27: remote operation failed: File exists. Path: /files/1a/2/liDZQzxeDGY.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.392143] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-26: remote operation failed: File exists. Path: /files/1a/2/liDZQzxeDGY.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.392188] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.392209] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/liDZQzxeDGY.dat
[2013-03-01 10:27:37.419169] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/mfwAlZkj6v0.dat on data-replicate-10
[2013-03-01 10:27:37.420555] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.420756] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.421516] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-28: remote operation failed: File exists. Path: /files/1a/2/mfwAlZkj6v0.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.421542] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-29: remote operation failed: File exists. Path: /files/1a/2/mfwAlZkj6v0.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.421598] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.421620] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/mfwAlZkj6v0.dat
[2013-03-01 10:27:37.442289] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/q3wpOjnWe0Q.dat on data-replicate-10
[2013-03-01 10:27:37.443264] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.443462] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.443969] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-28: remote operation failed: File exists. Path: /files/1a/2/q3wpOjnWe0Q.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.443999] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-29: remote operation failed: File exists. Path: /files/1a/2/q3wpOjnWe0Q.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.444028] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.444046] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/q3wpOjnWe0Q.dat
[2013-03-01 10:27:37.454595] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/qMGjHEOhJPQ.dat on data-replicate-10
[2013-03-01 10:27:37.456899] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.457633] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.464561] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-26: remote operation failed: File exists. Path: /files/1a/2/qMGjHEOhJPQ.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.464597] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-27: remote operation failed: File exists. Path: /files/1a/2/qMGjHEOhJPQ.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.464657] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.464677] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/qMGjHEOhJPQ.dat
[2013-03-01 10:27:37.492396] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/rPIv3I-3-j4.dat on data-replicate-10
[2013-03-01 10:27:37.531518] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.531758] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.532503] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-29: remote operation failed: File exists. Path: /files/1a/2/rPIv3I-3-j4.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.532775] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-28: remote operation failed: File exists. Path: /files/1a/2/rPIv3I-3-j4.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.532832] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.532853] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/rPIv3I-3-j4.dat
[2013-03-01 10:27:37.551058] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/tO684wfg8-Y.dat on data-replicate-10
[2013-03-01 10:27:37.574459] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.574625] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.575155] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-28: remote operation failed: File exists. Path: /files/1a/2/tO684wfg8-Y.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.575206] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-29: remote operation failed: File exists. Path: /files/1a/2/tO684wfg8-Y.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.575270] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.575287] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/tO684wfg8-Y.dat
[2013-03-01 10:27:37.608492] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/yvz_HUAY_FQ.dat on data-replicate-10
[2013-03-01 10:27:37.610902] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.611186] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.611650] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-28: remote operation failed: File exists. Path: /files/1a/2/yvz_HUAY_FQ.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.611695] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-29: remote operation failed: File exists. Path: /files/1a/2/yvz_HUAY_FQ.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.611728] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.611749] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/yvz_HUAY_FQ.dat
[2013-03-01 10:27:37.626040] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/zVZd1Y_YuUI.dat on data-replicate-10
[2013-03-01 10:27:37.628373] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.628640] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.629964] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-26: remote operation failed: File exists. Path: /files/1a/2/zVZd1Y_YuUI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.630062] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-27: remote operation failed: File exists. Path: /files/1a/2/zVZd1Y_YuUI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.630108] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.630125] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/zVZd1Y_YuUI.dat
[2013-03-01 10:27:37.893793] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/0q7Evq9lCTs.dat on data-replicate-2
[2013-03-01 10:27:37.916575] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:37.916852] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:37.918104] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/0q7Evq9lCTs.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.918131] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/0q7Evq9lCTs.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.918188] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.918216] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/0q7Evq9lCTs.dat
[2013-03-01 10:27:37.937777] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/3N7ZX5zbtkg.dat on data-replicate-2
[2013-03-01 10:27:37.939034] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:37.939171] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:37.940404] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/3N7ZX5zbtkg.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.940450] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/3N7ZX5zbtkg.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.940487] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.940504] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/3N7ZX5zbtkg.dat
[2013-03-01 10:27:37.958643] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/3bcndcvhC4Y.dat on data-replicate-2
[2013-03-01 10:27:37.959568] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:37.959616] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:37.960164] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/3bcndcvhC4Y.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.960201] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/3bcndcvhC4Y.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.960243] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.960263] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/3bcndcvhC4Y.dat
[2013-03-01 10:27:37.973902] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/3i6qX9pfgUI.dat on data-replicate-2
[2013-03-01 10:27:37.975015] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:37.975134] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:37.975696] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/3i6qX9pfgUI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.975715] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/3i6qX9pfgUI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.975750] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.975769] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/3i6qX9pfgUI.dat
[2013-03-01 10:27:37.985571] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/44CfKD344Gw.dat on data-replicate-2
[2013-03-01 10:27:37.986343] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:37.986394] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:37.989849] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/44CfKD344Gw.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.989893] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/44CfKD344Gw.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.989941] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.989959] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/44CfKD344Gw.dat
[2013-03-01 10:27:38.003915] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/5g9hLeiCYy8.dat on data-replicate-2
[2013-03-01 10:27:38.005187] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.005235] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.018538] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/5g9hLeiCYy8.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.018576] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/5g9hLeiCYy8.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.018606] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.018624] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/5g9hLeiCYy8.dat
[2013-03-01 10:27:38.043932] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/61Wsx0msQlY.dat on data-replicate-2
[2013-03-01 10:27:38.065831] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/61Wsx0msQlY.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.065861] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/61Wsx0msQlY.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.065921] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.065941] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/61Wsx0msQlY.dat
[2013-03-01 10:27:38.084469] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/7ktOT-xb_7E.dat on data-replicate-2
[2013-03-01 10:27:38.085807] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.085892] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.086701] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/7ktOT-xb_7E.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.086734] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/7ktOT-xb_7E.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.086785] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.086811] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/7ktOT-xb_7E.dat
[2013-03-01 10:27:38.103915] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/9YfIFIZHhxY.dat on data-replicate-2
[2013-03-01 10:27:38.105229] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.105380] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.106734] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/9YfIFIZHhxY.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.106768] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/9YfIFIZHhxY.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.106833] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.106855] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/9YfIFIZHhxY.dat
[2013-03-01 10:27:38.128055] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/AM3TNrsQ8YE.dat on data-replicate-2
[2013-03-01 10:27:38.142427] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.152652] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.153130] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/AM3TNrsQ8YE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.153164] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/AM3TNrsQ8YE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.153214] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.153240] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/AM3TNrsQ8YE.dat
[2013-03-01 10:27:38.156575] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/D54-8RCh1So.dat on data-replicate-2
[2013-03-01 10:27:38.157237] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.157444] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.157951] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/D54-8RCh1So.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.157972] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/D54-8RCh1So.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.158001] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.158027] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/D54-8RCh1So.dat
[2013-03-01 10:27:38.167871] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/E6YqBh9R1qQ.dat on data-replicate-2
[2013-03-01 10:27:38.169668] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.169723] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.170146] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/E6YqBh9R1qQ.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.170175] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/E6YqBh9R1qQ.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.170222] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.170239] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/E6YqBh9R1qQ.dat
[2013-03-01 10:27:38.195240] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/G58vaV_l9OU.dat on data-replicate-2
[2013-03-01 10:27:38.196620] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.196830] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.197465] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/G58vaV_l9OU.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.197671] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/G58vaV_l9OU.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.197780] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.197801] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/G58vaV_l9OU.dat
[2013-03-01 10:27:38.207458] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/Hp1YHFPDbyE.dat on data-replicate-2
[2013-03-01 10:27:38.208331] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.208425] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.218089] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/Hp1YHFPDbyE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.218117] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/Hp1YHFPDbyE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.218158] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.218176] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/Hp1YHFPDbyE.dat
[2013-03-01 10:27:38.226427] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/IeE7J2rsWk0.dat on data-replicate-2
[2013-03-01 10:27:38.227471] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.227549] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.267233] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/JiW9vV7QU_M.dat on data-replicate-2
[2013-03-01 10:27:38.290792] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/NdQDgC6vyZo.dat on data-replicate-2
[2013-03-01 10:27:38.301196] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/OJf3bBclwZQ.dat on data-replicate-2
[2013-03-01 10:27:38.323468] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/RLywGyXrqtw.dat on data-replicate-2
[2013-03-01 10:27:38.349233] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/RlnJSWEdwg0.dat on data-replicate-2
[2013-03-01 10:27:38.370195] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/SrSFTTP9ops.dat on data-replicate-2
[2013-03-01 10:27:38.423762] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/UVTwzUWXiks.dat on data-replicate-2
[2013-03-01 10:27:38.427444] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/UfrYx8K9tno.dat on data-replicate-2
[2013-03-01 10:27:38.452374] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/WZH8fo_VXRc.dat on data-replicate-2
[2013-03-01 10:27:38.476780] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/XHUTgIMDG5g.dat on data-replicate-2
[2013-03-01 10:27:38.524987] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/Y7YNj8VodcU.dat on data-replicate-2
[2013-03-01 10:27:38.545167] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/Z306AiSHYmE.dat on data-replicate-2
[2013-03-01 10:27:38.569919] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/_UIZI19t2Ks.dat on data-replicate-2
[2013-03-01 10:27:38.571080] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.571125] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.572417] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/_UIZI19t2Ks.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.572450] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/_UIZI19t2Ks.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.572496] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.572513] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/_UIZI19t2Ks.dat
[2013-03-01 10:27:38.589345] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/_k1UrniQSaU.dat on data-replicate-2
[2013-03-01 10:27:38.590302] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.590488] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.591536] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/_k1UrniQSaU.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.591565] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/_k1UrniQSaU.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.591597] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.591617] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/_k1UrniQSaU.dat
[2013-03-01 10:27:38.619167] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/aWPnAOxC_Ps.dat on data-replicate-2
[2013-03-01 10:27:38.620226] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.620405] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.622126] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/aWPnAOxC_Ps.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.622310] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/aWPnAOxC_Ps.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.622435] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.622456] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/aWPnAOxC_Ps.dat
[2013-03-01 10:27:38.635740] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/a_TjWLUgwpA.dat on data-replicate-2
[2013-03-01 10:27:38.638781] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.661265] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.661979] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/a_TjWLUgwpA.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.662023] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/a_TjWLUgwpA.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.662091] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.662113] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/a_TjWLUgwpA.dat
[2013-03-01 10:27:38.679597] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/cLkTVRtwmSc.dat on data-replicate-2
[2013-03-01 10:27:38.723479] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/iEpXb8AacBQ.dat on data-replicate-2
[2013-03-01 10:27:38.750615] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/ibfRXJZOGag.dat on data-replicate-2
[2013-03-01 10:27:38.767441] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.767672] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.768178] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/ibfRXJZOGag.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.768296] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/ibfRXJZOGag.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.768341] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.768364] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/ibfRXJZOGag.dat
[2013-03-01 10:27:38.778052] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/j72MzttuX2k.dat on data-replicate-2
[2013-03-01 10:27:38.788032] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.792178] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.792574] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/j72MzttuX2k.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.792711] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/j72MzttuX2k.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.792781] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.792801] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/j72MzttuX2k.dat
[2013-03-01 10:27:38.798114] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/kD_EPbCQb00.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.798244] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/kD_EPbCQb00.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.798285] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.798317] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/kD_EPbCQb00.dat
[2013-03-01 10:27:38.804628] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/qjAqe3C_hic.dat on data-replicate-2
[2013-03-01 10:27:38.808652] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.809218] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.809701] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/qjAqe3C_hic.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.809724] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/qjAqe3C_hic.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.809754] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.809773] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/qjAqe3C_hic.dat
[2013-03-01 10:27:38.836722] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/uwo6cvyV9jE.dat on data-replicate-2
[2013-03-01 10:27:38.859479] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/uwo6cvyV9jE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.859598] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/uwo6cvyV9jE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.859677] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.859713] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/uwo6cvyV9jE.dat
[2013-03-01 10:27:38.884404] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/yVwCvlj7R-U.dat on data-replicate-2
[2013-03-01 10:27:38.885804] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.885884] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.886558] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/yVwCvlj7R-U.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.886607] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/yVwCvlj7R-U.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.886652] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.886671] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/yVwCvlj7R-U.dat
[2013-03-01 10:27:39.197136] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/-3s0xx_RP84.dat on data-replicate-3
[2013-03-01 10:27:39.198407] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.198525] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.212786] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/-3s0xx_RP84.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.212814] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/-3s0xx_RP84.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.212848] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.212887] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/-3s0xx_RP84.dat
[2013-03-01 10:27:39.226500] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/-N__gdV0flg.dat on data-replicate-3
[2013-03-01 10:27:39.228736] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.228900] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.229393] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/-N__gdV0flg.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.229420] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/-N__gdV0flg.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.229447] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.229464] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/-N__gdV0flg.dat
[2013-03-01 10:27:39.231395] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/2wlWLMjuLGk.dat on data-replicate-3
[2013-03-01 10:27:39.242450] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.242503] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.243080] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-36: remote operation failed: File exists. Path: /files/1a/2/2wlWLMjuLGk.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.243121] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-37: remote operation failed: File exists. Path: /files/1a/2/2wlWLMjuLGk.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.243159] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.243177] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/2wlWLMjuLGk.dat
[2013-03-01 10:27:39.268901] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/7rFj27fc_DQ.dat on data-replicate-17
[2013-03-01 10:27:39.288698] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/8tx65uEM4y0.dat on data-replicate-3
[2013-03-01 10:27:39.290549] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.290688] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.291299] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/8tx65uEM4y0.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.291382] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/8tx65uEM4y0.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.291436] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.291466] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/8tx65uEM4y0.dat
[2013-03-01 10:27:39.293349] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/95LcZ74f2cU.dat on data-replicate-17
[2013-03-01 10:27:39.328745] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/Cz8nKS8gvEs.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.328816] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/Cz8nKS8gvEs.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.328898] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.328915] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/Cz8nKS8gvEs.dat
[2013-03-01 10:27:39.339408] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/D0YjtWDkpjM.dat on data-replicate-3
[2013-03-01 10:27:39.341313] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.341555] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.348838] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/D0YjtWDkpjM.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.348860] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/D0YjtWDkpjM.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.348915] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.348936] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/D0YjtWDkpjM.dat
[2013-03-01 10:27:39.350772] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/E_9q6kwFzoI.dat on data-replicate-3
[2013-03-01 10:27:39.351325] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.351512] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.352411] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-36: remote operation failed: File exists. Path: /files/1a/2/E_9q6kwFzoI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.352585] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-37: remote operation failed: File exists. Path: /files/1a/2/E_9q6kwFzoI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.352628] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.352653] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/E_9q6kwFzoI.dat
[2013-03-01 10:27:39.369601] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/F1EBIVNwBX4.dat on data-replicate-3
[2013-03-01 10:27:39.370961] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.370995] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.372277] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-36: remote operation failed: File exists. Path: /files/1a/2/F1EBIVNwBX4.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.372308] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-37: remote operation failed: File exists. Path: /files/1a/2/F1EBIVNwBX4.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.372348] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.372373] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/F1EBIVNwBX4.dat
[2013-03-01 10:27:39.390213] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/F5A_Xh3uHPE.dat on data-replicate-3
[2013-03-01 10:27:39.391736] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.391902] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.392726] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/F5A_Xh3uHPE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.392812] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/F5A_Xh3uHPE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.392848] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.392864] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/F5A_Xh3uHPE.dat
[2013-03-01 10:27:39.406628] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/H1fFwLwhrOE.dat on data-replicate-3
[2013-03-01 10:27:39.407360] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.407408] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.414772] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/H1fFwLwhrOE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.414798] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/H1fFwLwhrOE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.414827] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.414846] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/H1fFwLwhrOE.dat
[2013-03-01 10:27:39.417673] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/LtP5An7sQso.dat on data-replicate-3
[2013-03-01 10:27:39.418554] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.418635] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.442202] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-37: remote operation failed: File exists. Path: /files/1a/2/LtP5An7sQso.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.442237] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-36: remote operation failed: File exists. Path: /files/1a/2/LtP5An7sQso.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.442296] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL

Comment 1 Philip 2013-03-01 10:42:30 UTC

Now my original gluster servers seem to be crashed. They are not accessable via ssh anymore. I had this problem before and posted it in the mailing list: http://www.gluster.org/pipermail/gluster-users/2013-February/035547.html

The last messages in the glustershd.log are:
[2013-03-01 10:43:05.923426] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:cb1ee73f-c362-4db2-9066-dc0d3ebab5fa>: Performing conservative merge
[2013-03-01 10:43:05.923877] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:91612de0-dd11-414e-8a99-d9365240a15e>: Performing conservative merge
[2013-03-01 10:43:05.925518] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:1fc75b91-de57-44e0-ac21-2f30c98efc44>: Performing conservative merge
[2013-03-01 10:43:05.926464] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:d5b77c14-baf9-4b1a-a698-d4703a2d8c57>: Performing conservative merge
[2013-03-01 10:43:05.926536] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:570804f8-1149-4ffe-b1cf-1ec47b6feb1d>: Performing conservative merge
[2013-03-01 10:43:05.928567] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:91135c37-bafa-461a-a404-64abf4684251>: Performing conservative merge
[2013-03-01 10:43:05.929195] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:49787c87-176b-4c54-96ea-c77bf4d53555>: Performing conservative merge
[2013-03-01 10:43:05.929505] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:cb1ee73f-c362-4db2-9066-dc0d3ebab5fa>: Performing conservative merge
[2013-03-01 10:43:05.937388] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:d5b77c14-baf9-4b1a-a698-d4703a2d8c57>: Performing conservative merge
[2013-03-01 10:43:05.937504] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:25f6123c-d305-462c-8fee-0f6f33a95cd3>: Performing conservative merge
[2013-03-01 10:43:05.946168] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:49787c87-176b-4c54-96ea-c77bf4d53555>: Performing conservative merge
[2013-03-01 10:43:05.946320] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:b5a36b38-c036-49ed-9bd9-23f65ecb6635>: Performing conservative merge
[2013-03-01 10:43:05.949335] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:85eb0d1e-304c-4b08-b0d3-6e635518c449>: Performing conservative merge
[2013-03-01 10:43:05.949398] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:91135c37-bafa-461a-a404-64abf4684251>: Performing conservative merge
[2013-03-01 10:43:05.949496] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:91135c37-bafa-461a-a404-64abf4684251>: Performing conservative merge
[2013-03-01 10:43:05.976084] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:85eb0d1e-304c-4b08-b0d3-6e635518c449>: Performing conservative merge
[2013-03-01 10:43:05.980568] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:cc5750c6-49f6-4dab-aaf3-f37623917084>: Performing conservative merge
[2013-03-01 10:43:05.981759] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:85eb0d1e-304c-4b08-b0d3-6e635518c449>: Performing conservative merge
[2013-03-01 10:43:06.010476] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:5ab355e2-bfd5-4504-88f7-b42fda99c616>: Performing conservative merge
[2013-03-01 10:43:06.010568] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:cc5750c6-49f6-4dab-aaf3-f37623917084>: Performing conservative merge
[2013-03-01 10:43:06.010645] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:cc5750c6-49f6-4dab-aaf3-f37623917084>: Performing conservative merge
[2013-03-01 10:43:06.039079] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:93d55cc4-5aac-41b9-a3fa-9f32517ff786>: Performing conservative merge
[2013-03-01 10:43:06.039572] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:5ab355e2-bfd5-4504-88f7-b42fda99c616>: Performing conservative merge
[2013-03-01 10:43:06.044342] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:5ab355e2-bfd5-4504-88f7-b42fda99c616>: Performing conservative merge
[2013-03-01 10:43:06.061237] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:feb30943-22c7-43f4-a6e8-30f751ef173e>: Performing conservative merge
[2013-03-01 10:43:06.062258] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:93d55cc4-5aac-41b9-a3fa-9f32517ff786>: Performing conservative merge
[2013-03-01 10:43:06.074859] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:93d55cc4-5aac-41b9-a3fa-9f32517ff786>: Performing conservative merge
[2013-03-01 10:43:06.088740] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:2c164035-b076-42ad-b496-b65c72518bf7>: Performing conservative merge
[2013-03-01 10:43:06.089071] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:feb30943-22c7-43f4-a6e8-30f751ef173e>: Performing conservative merge
[2013-03-01 10:43:06.115676] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:feb30943-22c7-43f4-a6e8-30f751ef173e>: Performing conservative merge
[2013-03-01 10:43:06.116260] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:2c164035-b076-42ad-b496-b65c72518bf7>: Performing conservative merge
[2013-03-01 10:43:06.133935] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:2c164035-b076-42ad-b496-b65c72518bf7>: Performing conservative merge
[2013-03-01 10:43:06.395975] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:10c5ce16-ad0a-4bb1-b0d4-a25a3372b791>: Performing conservative merge
[2013-03-01 10:43:06.450119] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:349fb70b-03d3-48d2-a807-4b9935fbcfab>: Performing conservative merge
[2013-03-01 10:43:06.462177] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:a55ba301-48ae-4766-8c67-72302552a23f>: Performing conservative merge
[2013-03-01 10:43:06.466253] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:5a7d7456-f9d2-450c-8fda-d0e720f3162d>: Performing conservative merge
[2013-03-01 10:43:06.486670] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:9cbf2dc5-add8-459c-8de9-1780d72f61b5>: Performing conservative merge
[2013-03-01 10:43:06.496520] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:a15c2ad0-dd2e-4df5-b14c-2db3cac5c80b>: Performing conservative merge
[2013-03-01 10:43:06.506454] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:e8c45e36-c4dd-4d0f-9eba-1314920fbdb5>: Performing conservative merge
[2013-03-01 10:43:06.515501] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:ab2d3bce-7ae4-4cb7-b233-dac75fe0cc4f>: Performing conservative merge
[2013-03-01 10:43:06.542925] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:0bbb1590-4af7-4302-9c17-20d482a0852e>: Performing conservative merge
[2013-03-01 10:43:06.552459] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:2d29ee00-90cb-42d3-b4ca-4c4514527e07>: Performing conservative merge
[2013-03-01 10:43:06.585275] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:347d4907-71b6-4b20-a7f7-94e2968e4862>: Performing conservative merge
[2013-03-01 10:43:06.588558] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:e9b5e4a7-22d3-4b41-9543-ae59d8c6f79b>: Performing conservative merge
[2013-03-01 10:43:06.626280] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:b884e51b-0fde-4b3a-ba9e-846cbe53ebf3>: Performing conservative merge
[2013-03-01 10:43:06.638691] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:596a60a4-1f69-4dd9-9fc5-1d8117d2df7c>: Performing conservative merge
[2013-03-01 10:43:06.643488] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:180c503a-137a-489a-88c8-98ce1dd59217>: Performing conservative merge
[2013-03-01 10:43:06.647676] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:852cf3b9-a89f-4bba-b3f4-8ec7e0e1cdc5>: Performing conservative merge
[2013-03-01 10:43:06.650560] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:3a6dbac6-e904-4048-8ea6-539ff4f6f23c>: Performing conservative merge
[2013-03-01 10:43:06.690637] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:fd2f9cac-653d-4f01-9e9e-60dd38ecab95>: Performing conservative merge
[2013-03-01 10:43:06.694432] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:fabc4776-49c7-4bcd-a391-bf8fb45c1785>: Performing conservative merge
[2013-03-01 10:43:06.698631] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:570804f8-1149-4ffe-b1cf-1ec47b6feb1d>: Performing conservative merge
[2013-03-01 10:43:06.701959] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:6c99fbb5-8443-4f3f-ae5b-01541f6bd06a>: Performing conservative merge
[2013-03-01 10:43:06.705097] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:49787c87-176b-4c54-96ea-c77bf4d53555>: Performing conservative merge
[2013-03-01 10:43:06.708874] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:25f6123c-d305-462c-8fee-0f6f33a95cd3>: Performing conservative merge
[2013-03-01 10:43:06.712038] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:b5a36b38-c036-49ed-9bd9-23f65ecb6635>: Performing conservative merge
[2013-03-01 10:43:06.715171] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:f74cd8c3-d17b-4215-83c6-465b192f3b2f>: Performing conservative merge
[2013-03-01 10:43:06.718740] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:91135c37-bafa-461a-a404-64abf4684251>: Performing conservative merge
[2013-03-01 10:43:06.750868] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:85eb0d1e-304c-4b08-b0d3-6e635518c449>: Performing conservative merge
[2013-03-01 10:43:06.777705] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:cc5750c6-49f6-4dab-aaf3-f37623917084>: Performing conservative merge
[2013-03-01 10:43:06.802520] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:5ab355e2-bfd5-4504-88f7-b42fda99c616>: Performing conservative merge
[2013-03-01 10:43:06.862105] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:93d55cc4-5aac-41b9-a3fa-9f32517ff786>: Performing conservative merge
[2013-03-01 10:43:06.877074] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:feb30943-22c7-43f4-a6e8-30f751ef173e>: Performing conservative merge
[2013-03-01 10:43:06.896972] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:2c164035-b076-42ad-b496-b65c72518bf7>: Performing conservative merge
[2013-03-01 10:52:47.241084] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-5: <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>: Performing conservative merge
[2013-03-01 10:52:47.604271] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/-reAWDuEQ7o.dat on data-client-18
[2013-03-01 10:52:47.604356] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/4hypFSAKxC8.dat on data-client-18
[2013-03-01 10:52:47.604412] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/6b3zx_IZhJA.dat on data-client-18
[2013-03-01 10:52:47.604448] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/9ffveVk8uzY.dat on data-client-18
[2013-03-01 10:52:47.604500] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/CXTp9Jsez0g.dat on data-client-18
[2013-03-01 10:52:47.604558] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/D2BudsKiFSs.dat on data-client-18
[2013-03-01 10:52:47.604627] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/N3OPl1BLtXU.dat on data-client-18
[2013-03-01 10:52:47.604663] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/OwZlgJnbAVk.dat on data-client-18
[2013-03-01 10:52:47.604698] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/PchuM81Dp1E.dat on data-client-18
[2013-03-01 10:52:47.604732] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/Q0Fw-BZrq94.dat on data-client-18
[2013-03-01 10:52:47.604767] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/Q3NdMIRUTrQ.dat on data-client-18
[2013-03-01 10:52:47.604818] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/Vy_P7-lRj-s.dat on data-client-18
[2013-03-01 10:52:47.604844] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/X5AYPv5EQwc.dat on data-client-18
[2013-03-01 10:52:47.604875] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/bHL2xZcU6Dk.dat on data-client-18
[2013-03-01 10:52:47.604988] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/pZ1PEtw0bXs.dat on data-client-18
[2013-03-01 10:52:47.605016] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/qVo07JNt1MA.dat on data-client-18
[2013-03-01 10:52:47.605039] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/rP_teS6y3e8.dat on data-client-18
[2013-03-01 10:52:47.605061] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/s-p16BACRdc.dat on data-client-18
[2013-03-01 10:52:47.605084] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/v8TJFOFK3NE.dat on data-client-18
[2013-03-01 10:52:47.605122] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/wZvX5wVmDyo.dat on data-client-18
[2013-03-01 10:52:47.605155] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/yuric2csiA0.dat on data-client-18
[2013-03-01 10:52:47.895622] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-10: <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>: Performing conservative merge
[2013-03-01 10:52:58.624185] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-4: <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>: Performing conservative merge
[2013-03-01 11:02:59.255375] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-1: server 172.16.5.3:24009 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255469] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-3: server 172.16.5.3:24010 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255500] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-7: server 172.16.5.3:24012 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255546] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-5: server 172.16.5.3:24011 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255573] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-9: server 172.16.5.3:24013 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255600] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-11: server 172.16.5.3:24014 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255626] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-13: server 172.16.5.3:24015 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255650] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-19: server 172.16.5.3:24018 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255674] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-17: server 172.16.5.3:24017 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255703] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-21: server 172.16.5.3:24019 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255728] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-15: server 172.16.5.3:24016 has not responded in the last 15 seconds, disconnecting.

Comment 2 shishir gowda 2013-03-05 03:37:06 UTC

Hi,

Can you please provide the following:
output from:
1. gluster peer status
2. gluster volume status data

logs from servers/bricks. I see that the node-uuid xattr requests are failing.

Comment 4 Niels de Vos 2014-11-27 14:54:11 UTC

The version that this bug has been reported against, does not get any updates from the Gluster Community anymore. Please verify if this report is still valid against a current (3.4, 3.5 or 3.6) release and update the version, or close this bug.

If there has been no update before 9 December 2014, this bug will get automatocally closed.

Comment 5 Niels de Vos 2015-05-26 12:33:52 UTC

This bug has been CLOSED, and there has not been a response to the requested NEEDINFO in more than 4 weeks. The NEEDINFO flag is now getting cleared so that our Bugzilla household is getting more in order.