916934 – Rebalance failures/Very slow

Bug 916934 - Rebalance failures/Very slow

Summary: Rebalance failures/Very slow

Keywords:
Status:	CLOSED DEFERRED
Alias:	None
Product:	GlusterFS
Classification:	Community
Component:	core
Sub Component:
Version:	3.3.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	unspecified
Target Milestone:	---
Assignee:	Nagaprasad Sathyanarayana
QA Contact:
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2013-03-01 09:41 UTC by Philip
Modified:	2016-02-18 00:19 UTC (History)
CC List:	3 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2014-12-14 19:40:30 UTC
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Philip 2013-03-01 09:41:21 UTC

Hi,

I have added 11 new bricks to my gluster system and perform a rebalance right now. The rebalance itself is really slow and will take more than a week to complete and causes lots of failures:

Node    Rebalanced-files          size       scanned      failures         status
---------      -----------   -----------   -----------   -----------   ------------
s003                0        0Bytes         26025             0    in progress
s004                0        0Bytes         26026             0    in progress
s001                0        0Bytes         26025             0    in progress
s002             4173        16.2GB          9461          1344    in progress

My volume info looks as follows:

Volume Name: data
Type: Distributed-Replicate
Volume ID: 3a3ce21e-c07e-486a-b0ca-5e93f68152d5
Status: Started
Number of Bricks: 22 x 2 = 44
Transport-type: tcp
Bricks:
Brick1: s002:/var/gfs/sdb/data-s002-1
Brick2: s003:/var/gfs/sdb/data-s003-1
Brick3: s002:/var/gfs/sdc/data-s002-2
Brick4: s003:/var/gfs/sdc/data-s003-2
Brick5: s002:/var/gfs/sdd/data-s002-3
Brick6: s003:/var/gfs/sdd/data-s003-3
Brick7: s002:/var/gfs/sde/data-s002-4
Brick8: s003:/var/gfs/sde/data-s003-4
Brick9: s002:/var/gfs/sdf/data-s002-5
Brick10: s003:/var/gfs/sdf/data-s003-5
Brick11: s002:/var/gfs/sdg/data-s002-6
Brick12: s003:/var/gfs/sdg/data-s003-6
Brick13: s002:/var/gfs/sdh/data-s002-7
Brick14: s003:/var/gfs/sdh/data-s003-7
Brick15: s002:/var/gfs/sdi/data-s002-8
Brick16: s003:/var/gfs/sdi/data-s003-8
Brick17: s002:/var/gfs/sdj/data-s002-9
Brick18: s003:/var/gfs/sdj/data-s003-9
Brick19: s002:/var/gfs/sdk/data-s002-10
Brick20: s003:/var/gfs/sdk/data-s003-10
Brick21: s002:/var/gfs/sdl/data-s002-11
Brick22: s003:/var/gfs/sdl/data-s003-11
Brick23: s001:/var/gfs/sdb/data-s001-1
Brick24: s004:/var/gfs/sdb/data-s004-1
Brick25: s001:/var/gfs/sdc/data-s001-2
Brick26: s004:/var/gfs/sdc/data-s004-2
Brick27: s001:/var/gfs/sdd/data-s001-3
Brick28: s004:/var/gfs/sdd/data-s004-3
Brick29: s001:/var/gfs/sde/data-s001-4
Brick30: s004:/var/gfs/sde/data-s004-4
Brick31: s001:/var/gfs/sdf/data-s001-5
Brick32: s004:/var/gfs/sdf/data-s004-5
Brick33: s001:/var/gfs/sdg/data-s001-6
Brick34: s004:/var/gfs/sdg/data-s004-6
Brick35: s001:/var/gfs/sdh/data-s001-7
Brick36: s004:/var/gfs/sdh/data-s004-7
Brick37: s001:/var/gfs/sdi/data-s001-8
Brick38: s004:/var/gfs/sdi/data-s004-8
Brick39: s001:/var/gfs/sdj/data-s001-9
Brick40: s004:/var/gfs/sdj/data-s004-9
Brick41: s001:/var/gfs/sdk/data-s001-10
Brick42: s004:/var/gfs/sdk/data-s004-10
Brick43: s001:/var/gfs/sdl/data-s001-11
Brick44: s004:/var/gfs/sdl/data-s004-11
Options Reconfigured:
network.ping-timeout: 15
cluster.data-self-heal-algorithm: full


The error messages in the rebalance log are as follows:
[2013-03-01 10:27:37.300801] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/cS6ENja-GV4.dat
[2013-03-01 10:27:37.316505] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/dnU-DG0u4ZI.dat on data-replicate-10
[2013-03-01 10:27:37.317639] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.317656] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.320451] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-28: remote operation failed: File exists. Path: /files/1a/2/dnU-DG0u4ZI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.320544] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-29: remote operation failed: File exists. Path: /files/1a/2/dnU-DG0u4ZI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.320615] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.320650] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/dnU-DG0u4ZI.dat
[2013-03-01 10:27:37.341623] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/i9It0K0tRXA.dat on data-replicate-10
[2013-03-01 10:27:37.342987] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.343010] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.343606] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-27: remote operation failed: File exists. Path: /files/1a/2/i9It0K0tRXA.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.343636] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-26: remote operation failed: File exists. Path: /files/1a/2/i9It0K0tRXA.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.343774] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.343794] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/i9It0K0tRXA.dat
[2013-03-01 10:27:37.366354] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/ld0ie2PRNSc.dat on data-replicate-10
[2013-03-01 10:27:37.367343] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.367454] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.368632] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-26: remote operation failed: File exists. Path: /files/1a/2/ld0ie2PRNSc.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.368659] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-27: remote operation failed: File exists. Path: /files/1a/2/ld0ie2PRNSc.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.368690] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.368712] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/ld0ie2PRNSc.dat
[2013-03-01 10:27:37.390409] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/liDZQzxeDGY.dat on data-replicate-10
[2013-03-01 10:27:37.391500] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.391713] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.392124] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-27: remote operation failed: File exists. Path: /files/1a/2/liDZQzxeDGY.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.392143] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-26: remote operation failed: File exists. Path: /files/1a/2/liDZQzxeDGY.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.392188] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.392209] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/liDZQzxeDGY.dat
[2013-03-01 10:27:37.419169] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/mfwAlZkj6v0.dat on data-replicate-10
[2013-03-01 10:27:37.420555] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.420756] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.421516] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-28: remote operation failed: File exists. Path: /files/1a/2/mfwAlZkj6v0.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.421542] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-29: remote operation failed: File exists. Path: /files/1a/2/mfwAlZkj6v0.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.421598] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.421620] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/mfwAlZkj6v0.dat
[2013-03-01 10:27:37.442289] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/q3wpOjnWe0Q.dat on data-replicate-10
[2013-03-01 10:27:37.443264] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.443462] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.443969] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-28: remote operation failed: File exists. Path: /files/1a/2/q3wpOjnWe0Q.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.443999] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-29: remote operation failed: File exists. Path: /files/1a/2/q3wpOjnWe0Q.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.444028] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.444046] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/q3wpOjnWe0Q.dat
[2013-03-01 10:27:37.454595] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/qMGjHEOhJPQ.dat on data-replicate-10
[2013-03-01 10:27:37.456899] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.457633] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.464561] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-26: remote operation failed: File exists. Path: /files/1a/2/qMGjHEOhJPQ.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.464597] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-27: remote operation failed: File exists. Path: /files/1a/2/qMGjHEOhJPQ.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.464657] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.464677] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/qMGjHEOhJPQ.dat
[2013-03-01 10:27:37.492396] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/rPIv3I-3-j4.dat on data-replicate-10
[2013-03-01 10:27:37.531518] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.531758] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.532503] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-29: remote operation failed: File exists. Path: /files/1a/2/rPIv3I-3-j4.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.532775] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-28: remote operation failed: File exists. Path: /files/1a/2/rPIv3I-3-j4.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.532832] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.532853] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/rPIv3I-3-j4.dat
[2013-03-01 10:27:37.551058] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/tO684wfg8-Y.dat on data-replicate-10
[2013-03-01 10:27:37.574459] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.574625] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.575155] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-28: remote operation failed: File exists. Path: /files/1a/2/tO684wfg8-Y.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.575206] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-29: remote operation failed: File exists. Path: /files/1a/2/tO684wfg8-Y.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.575270] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.575287] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/tO684wfg8-Y.dat
[2013-03-01 10:27:37.608492] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/yvz_HUAY_FQ.dat on data-replicate-10
[2013-03-01 10:27:37.610902] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.611186] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.611650] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-28: remote operation failed: File exists. Path: /files/1a/2/yvz_HUAY_FQ.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.611695] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-29: remote operation failed: File exists. Path: /files/1a/2/yvz_HUAY_FQ.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.611728] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.611749] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/yvz_HUAY_FQ.dat
[2013-03-01 10:27:37.626040] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/zVZd1Y_YuUI.dat on data-replicate-10
[2013-03-01 10:27:37.628373] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-21: remote operation failed: No such file or directory
[2013-03-01 10:27:37.628640] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-20: remote operation failed: No such file or directory
[2013-03-01 10:27:37.629964] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-26: remote operation failed: File exists. Path: /files/1a/2/zVZd1Y_YuUI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.630062] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-27: remote operation failed: File exists. Path: /files/1a/2/zVZd1Y_YuUI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.630108] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.630125] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/zVZd1Y_YuUI.dat
[2013-03-01 10:27:37.893793] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/0q7Evq9lCTs.dat on data-replicate-2
[2013-03-01 10:27:37.916575] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:37.916852] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:37.918104] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/0q7Evq9lCTs.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.918131] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/0q7Evq9lCTs.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.918188] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.918216] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/0q7Evq9lCTs.dat
[2013-03-01 10:27:37.937777] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/3N7ZX5zbtkg.dat on data-replicate-2
[2013-03-01 10:27:37.939034] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:37.939171] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:37.940404] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/3N7ZX5zbtkg.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.940450] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/3N7ZX5zbtkg.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.940487] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.940504] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/3N7ZX5zbtkg.dat
[2013-03-01 10:27:37.958643] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/3bcndcvhC4Y.dat on data-replicate-2
[2013-03-01 10:27:37.959568] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:37.959616] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:37.960164] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/3bcndcvhC4Y.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.960201] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/3bcndcvhC4Y.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.960243] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.960263] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/3bcndcvhC4Y.dat
[2013-03-01 10:27:37.973902] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/3i6qX9pfgUI.dat on data-replicate-2
[2013-03-01 10:27:37.975015] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:37.975134] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:37.975696] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/3i6qX9pfgUI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.975715] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/3i6qX9pfgUI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.975750] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.975769] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/3i6qX9pfgUI.dat
[2013-03-01 10:27:37.985571] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/44CfKD344Gw.dat on data-replicate-2
[2013-03-01 10:27:37.986343] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:37.986394] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:37.989849] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/44CfKD344Gw.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.989893] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/44CfKD344Gw.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:37.989941] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:37.989959] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/44CfKD344Gw.dat
[2013-03-01 10:27:38.003915] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/5g9hLeiCYy8.dat on data-replicate-2
[2013-03-01 10:27:38.005187] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.005235] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.018538] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/5g9hLeiCYy8.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.018576] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/5g9hLeiCYy8.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.018606] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.018624] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/5g9hLeiCYy8.dat
[2013-03-01 10:27:38.043932] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/61Wsx0msQlY.dat on data-replicate-2
[2013-03-01 10:27:38.065831] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/61Wsx0msQlY.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.065861] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/61Wsx0msQlY.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.065921] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.065941] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/61Wsx0msQlY.dat
[2013-03-01 10:27:38.084469] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/7ktOT-xb_7E.dat on data-replicate-2
[2013-03-01 10:27:38.085807] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.085892] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.086701] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/7ktOT-xb_7E.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.086734] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/7ktOT-xb_7E.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.086785] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.086811] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/7ktOT-xb_7E.dat
[2013-03-01 10:27:38.103915] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/9YfIFIZHhxY.dat on data-replicate-2
[2013-03-01 10:27:38.105229] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.105380] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.106734] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/9YfIFIZHhxY.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.106768] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/9YfIFIZHhxY.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.106833] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.106855] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/9YfIFIZHhxY.dat
[2013-03-01 10:27:38.128055] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/AM3TNrsQ8YE.dat on data-replicate-2
[2013-03-01 10:27:38.142427] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.152652] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.153130] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/AM3TNrsQ8YE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.153164] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/AM3TNrsQ8YE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.153214] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.153240] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/AM3TNrsQ8YE.dat
[2013-03-01 10:27:38.156575] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/D54-8RCh1So.dat on data-replicate-2
[2013-03-01 10:27:38.157237] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.157444] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.157951] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/D54-8RCh1So.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.157972] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/D54-8RCh1So.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.158001] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.158027] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/D54-8RCh1So.dat
[2013-03-01 10:27:38.167871] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/E6YqBh9R1qQ.dat on data-replicate-2
[2013-03-01 10:27:38.169668] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.169723] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.170146] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/E6YqBh9R1qQ.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.170175] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/E6YqBh9R1qQ.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.170222] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.170239] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/E6YqBh9R1qQ.dat
[2013-03-01 10:27:38.195240] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/G58vaV_l9OU.dat on data-replicate-2
[2013-03-01 10:27:38.196620] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.196830] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.197465] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/G58vaV_l9OU.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.197671] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/G58vaV_l9OU.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.197780] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.197801] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/G58vaV_l9OU.dat
[2013-03-01 10:27:38.207458] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/Hp1YHFPDbyE.dat on data-replicate-2
[2013-03-01 10:27:38.208331] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.208425] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.218089] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/Hp1YHFPDbyE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.218117] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/Hp1YHFPDbyE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.218158] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.218176] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/Hp1YHFPDbyE.dat
[2013-03-01 10:27:38.226427] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/IeE7J2rsWk0.dat on data-replicate-2
[2013-03-01 10:27:38.227471] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.227549] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.267233] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/JiW9vV7QU_M.dat on data-replicate-2
[2013-03-01 10:27:38.290792] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/NdQDgC6vyZo.dat on data-replicate-2
[2013-03-01 10:27:38.301196] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/OJf3bBclwZQ.dat on data-replicate-2
[2013-03-01 10:27:38.323468] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/RLywGyXrqtw.dat on data-replicate-2
[2013-03-01 10:27:38.349233] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/RlnJSWEdwg0.dat on data-replicate-2
[2013-03-01 10:27:38.370195] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/SrSFTTP9ops.dat on data-replicate-2
[2013-03-01 10:27:38.423762] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/UVTwzUWXiks.dat on data-replicate-2
[2013-03-01 10:27:38.427444] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/UfrYx8K9tno.dat on data-replicate-2
[2013-03-01 10:27:38.452374] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/WZH8fo_VXRc.dat on data-replicate-2
[2013-03-01 10:27:38.476780] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/XHUTgIMDG5g.dat on data-replicate-2
[2013-03-01 10:27:38.524987] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/Y7YNj8VodcU.dat on data-replicate-2
[2013-03-01 10:27:38.545167] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/Z306AiSHYmE.dat on data-replicate-2
[2013-03-01 10:27:38.569919] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/_UIZI19t2Ks.dat on data-replicate-2
[2013-03-01 10:27:38.571080] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.571125] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.572417] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/_UIZI19t2Ks.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.572450] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/_UIZI19t2Ks.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.572496] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.572513] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/_UIZI19t2Ks.dat
[2013-03-01 10:27:38.589345] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/_k1UrniQSaU.dat on data-replicate-2
[2013-03-01 10:27:38.590302] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.590488] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.591536] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/_k1UrniQSaU.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.591565] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/_k1UrniQSaU.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.591597] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.591617] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/_k1UrniQSaU.dat
[2013-03-01 10:27:38.619167] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/aWPnAOxC_Ps.dat on data-replicate-2
[2013-03-01 10:27:38.620226] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.620405] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.622126] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/aWPnAOxC_Ps.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.622310] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/aWPnAOxC_Ps.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.622435] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.622456] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/aWPnAOxC_Ps.dat
[2013-03-01 10:27:38.635740] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/a_TjWLUgwpA.dat on data-replicate-2
[2013-03-01 10:27:38.638781] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.661265] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.661979] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/a_TjWLUgwpA.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.662023] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/a_TjWLUgwpA.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.662091] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.662113] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/a_TjWLUgwpA.dat
[2013-03-01 10:27:38.679597] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/cLkTVRtwmSc.dat on data-replicate-2
[2013-03-01 10:27:38.723479] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/iEpXb8AacBQ.dat on data-replicate-2
[2013-03-01 10:27:38.750615] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/ibfRXJZOGag.dat on data-replicate-2
[2013-03-01 10:27:38.767441] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.767672] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.768178] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/ibfRXJZOGag.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.768296] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/ibfRXJZOGag.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.768341] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.768364] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/ibfRXJZOGag.dat
[2013-03-01 10:27:38.778052] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/j72MzttuX2k.dat on data-replicate-2
[2013-03-01 10:27:38.788032] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.792178] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.792574] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-30: remote operation failed: File exists. Path: /files/1a/2/j72MzttuX2k.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.792711] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-31: remote operation failed: File exists. Path: /files/1a/2/j72MzttuX2k.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.792781] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.792801] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/j72MzttuX2k.dat
[2013-03-01 10:27:38.798114] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/kD_EPbCQb00.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.798244] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/kD_EPbCQb00.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.798285] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.798317] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/kD_EPbCQb00.dat
[2013-03-01 10:27:38.804628] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/qjAqe3C_hic.dat on data-replicate-2
[2013-03-01 10:27:38.808652] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.809218] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.809701] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/qjAqe3C_hic.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.809724] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/qjAqe3C_hic.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.809754] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.809773] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/qjAqe3C_hic.dat
[2013-03-01 10:27:38.836722] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/uwo6cvyV9jE.dat on data-replicate-2
[2013-03-01 10:27:38.859479] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/uwo6cvyV9jE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.859598] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/uwo6cvyV9jE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.859677] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.859713] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/uwo6cvyV9jE.dat
[2013-03-01 10:27:38.884404] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/yVwCvlj7R-U.dat on data-replicate-2
[2013-03-01 10:27:38.885804] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-5: remote operation failed: No such file or directory
[2013-03-01 10:27:38.885884] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-4: remote operation failed: No such file or directory
[2013-03-01 10:27:38.886558] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-32: remote operation failed: File exists. Path: /files/1a/2/yVwCvlj7R-U.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.886607] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-33: remote operation failed: File exists. Path: /files/1a/2/yVwCvlj7R-U.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:38.886652] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:38.886671] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/yVwCvlj7R-U.dat
[2013-03-01 10:27:39.197136] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/-3s0xx_RP84.dat on data-replicate-3
[2013-03-01 10:27:39.198407] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.198525] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.212786] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/-3s0xx_RP84.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.212814] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/-3s0xx_RP84.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.212848] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.212887] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/-3s0xx_RP84.dat
[2013-03-01 10:27:39.226500] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/-N__gdV0flg.dat on data-replicate-3
[2013-03-01 10:27:39.228736] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.228900] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.229393] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/-N__gdV0flg.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.229420] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/-N__gdV0flg.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.229447] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.229464] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/-N__gdV0flg.dat
[2013-03-01 10:27:39.231395] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/2wlWLMjuLGk.dat on data-replicate-3
[2013-03-01 10:27:39.242450] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.242503] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.243080] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-36: remote operation failed: File exists. Path: /files/1a/2/2wlWLMjuLGk.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.243121] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-37: remote operation failed: File exists. Path: /files/1a/2/2wlWLMjuLGk.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.243159] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.243177] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/2wlWLMjuLGk.dat
[2013-03-01 10:27:39.268901] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/7rFj27fc_DQ.dat on data-replicate-17
[2013-03-01 10:27:39.288698] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/8tx65uEM4y0.dat on data-replicate-3
[2013-03-01 10:27:39.290549] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.290688] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.291299] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/8tx65uEM4y0.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.291382] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/8tx65uEM4y0.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.291436] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.291466] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/8tx65uEM4y0.dat
[2013-03-01 10:27:39.293349] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/95LcZ74f2cU.dat on data-replicate-17
[2013-03-01 10:27:39.328745] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/Cz8nKS8gvEs.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.328816] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/Cz8nKS8gvEs.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.328898] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.328915] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/Cz8nKS8gvEs.dat
[2013-03-01 10:27:39.339408] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/D0YjtWDkpjM.dat on data-replicate-3
[2013-03-01 10:27:39.341313] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.341555] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.348838] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/D0YjtWDkpjM.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.348860] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/D0YjtWDkpjM.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.348915] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.348936] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/D0YjtWDkpjM.dat
[2013-03-01 10:27:39.350772] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/E_9q6kwFzoI.dat on data-replicate-3
[2013-03-01 10:27:39.351325] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.351512] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.352411] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-36: remote operation failed: File exists. Path: /files/1a/2/E_9q6kwFzoI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.352585] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-37: remote operation failed: File exists. Path: /files/1a/2/E_9q6kwFzoI.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.352628] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.352653] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/E_9q6kwFzoI.dat
[2013-03-01 10:27:39.369601] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/F1EBIVNwBX4.dat on data-replicate-3
[2013-03-01 10:27:39.370961] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.370995] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.372277] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-36: remote operation failed: File exists. Path: /files/1a/2/F1EBIVNwBX4.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.372308] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-37: remote operation failed: File exists. Path: /files/1a/2/F1EBIVNwBX4.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.372348] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.372373] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/F1EBIVNwBX4.dat
[2013-03-01 10:27:39.390213] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/F5A_Xh3uHPE.dat on data-replicate-3
[2013-03-01 10:27:39.391736] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.391902] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.392726] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/F5A_Xh3uHPE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.392812] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/F5A_Xh3uHPE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.392848] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.392864] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/F5A_Xh3uHPE.dat
[2013-03-01 10:27:39.406628] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/H1fFwLwhrOE.dat on data-replicate-3
[2013-03-01 10:27:39.407360] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.407408] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.414772] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-34: remote operation failed: File exists. Path: /files/1a/2/H1fFwLwhrOE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.414798] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-35: remote operation failed: File exists. Path: /files/1a/2/H1fFwLwhrOE.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.414827] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL
[2013-03-01 10:27:39.414846] E [dht-rebalance.c:1150:gf_defrag_migrate_data] 0-data-dht: Failed to get node-uuid for /files/1a/2/H1fFwLwhrOE.dat
[2013-03-01 10:27:39.417673] I [dht-common.c:954:dht_lookup_everywhere_cbk] 0-data-dht: deleting stale linkfile /files/1a/2/LtP5An7sQso.dat on data-replicate-3
[2013-03-01 10:27:39.418554] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-7: remote operation failed: No such file or directory
[2013-03-01 10:27:39.418635] W [client3_1-fops.c:592:client3_1_unlink_cbk] 0-data-client-6: remote operation failed: No such file or directory
[2013-03-01 10:27:39.442202] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-37: remote operation failed: File exists. Path: /files/1a/2/LtP5An7sQso.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.442237] W [client3_1-fops.c:258:client3_1_mknod_cbk] 0-data-client-36: remote operation failed: File exists. Path: /files/1a/2/LtP5An7sQso.dat (00000000-0000-0000-0000-000000000000)
[2013-03-01 10:27:39.442296] E [dht-common.c:1911:dht_getxattr] 0-data-dht: layout is NULL

Comment 1 Philip 2013-03-01 10:42:30 UTC

Now my original gluster servers seem to be crashed. They are not accessable via ssh anymore. I had this problem before and posted it in the mailing list: http://www.gluster.org/pipermail/gluster-users/2013-February/035547.html

The last messages in the glustershd.log are:
[2013-03-01 10:43:05.923426] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:cb1ee73f-c362-4db2-9066-dc0d3ebab5fa>: Performing conservative merge
[2013-03-01 10:43:05.923877] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:91612de0-dd11-414e-8a99-d9365240a15e>: Performing conservative merge
[2013-03-01 10:43:05.925518] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:1fc75b91-de57-44e0-ac21-2f30c98efc44>: Performing conservative merge
[2013-03-01 10:43:05.926464] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:d5b77c14-baf9-4b1a-a698-d4703a2d8c57>: Performing conservative merge
[2013-03-01 10:43:05.926536] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:570804f8-1149-4ffe-b1cf-1ec47b6feb1d>: Performing conservative merge
[2013-03-01 10:43:05.928567] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:91135c37-bafa-461a-a404-64abf4684251>: Performing conservative merge
[2013-03-01 10:43:05.929195] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:49787c87-176b-4c54-96ea-c77bf4d53555>: Performing conservative merge
[2013-03-01 10:43:05.929505] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:cb1ee73f-c362-4db2-9066-dc0d3ebab5fa>: Performing conservative merge
[2013-03-01 10:43:05.937388] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:d5b77c14-baf9-4b1a-a698-d4703a2d8c57>: Performing conservative merge
[2013-03-01 10:43:05.937504] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:25f6123c-d305-462c-8fee-0f6f33a95cd3>: Performing conservative merge
[2013-03-01 10:43:05.946168] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:49787c87-176b-4c54-96ea-c77bf4d53555>: Performing conservative merge
[2013-03-01 10:43:05.946320] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:b5a36b38-c036-49ed-9bd9-23f65ecb6635>: Performing conservative merge
[2013-03-01 10:43:05.949335] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:85eb0d1e-304c-4b08-b0d3-6e635518c449>: Performing conservative merge
[2013-03-01 10:43:05.949398] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:91135c37-bafa-461a-a404-64abf4684251>: Performing conservative merge
[2013-03-01 10:43:05.949496] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:91135c37-bafa-461a-a404-64abf4684251>: Performing conservative merge
[2013-03-01 10:43:05.976084] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:85eb0d1e-304c-4b08-b0d3-6e635518c449>: Performing conservative merge
[2013-03-01 10:43:05.980568] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:cc5750c6-49f6-4dab-aaf3-f37623917084>: Performing conservative merge
[2013-03-01 10:43:05.981759] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:85eb0d1e-304c-4b08-b0d3-6e635518c449>: Performing conservative merge
[2013-03-01 10:43:06.010476] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:5ab355e2-bfd5-4504-88f7-b42fda99c616>: Performing conservative merge
[2013-03-01 10:43:06.010568] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:cc5750c6-49f6-4dab-aaf3-f37623917084>: Performing conservative merge
[2013-03-01 10:43:06.010645] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:cc5750c6-49f6-4dab-aaf3-f37623917084>: Performing conservative merge
[2013-03-01 10:43:06.039079] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:93d55cc4-5aac-41b9-a3fa-9f32517ff786>: Performing conservative merge
[2013-03-01 10:43:06.039572] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:5ab355e2-bfd5-4504-88f7-b42fda99c616>: Performing conservative merge
[2013-03-01 10:43:06.044342] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:5ab355e2-bfd5-4504-88f7-b42fda99c616>: Performing conservative merge
[2013-03-01 10:43:06.061237] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:feb30943-22c7-43f4-a6e8-30f751ef173e>: Performing conservative merge
[2013-03-01 10:43:06.062258] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:93d55cc4-5aac-41b9-a3fa-9f32517ff786>: Performing conservative merge
[2013-03-01 10:43:06.074859] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:93d55cc4-5aac-41b9-a3fa-9f32517ff786>: Performing conservative merge
[2013-03-01 10:43:06.088740] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-2: <gfid:2c164035-b076-42ad-b496-b65c72518bf7>: Performing conservative merge
[2013-03-01 10:43:06.089071] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:feb30943-22c7-43f4-a6e8-30f751ef173e>: Performing conservative merge
[2013-03-01 10:43:06.115676] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:feb30943-22c7-43f4-a6e8-30f751ef173e>: Performing conservative merge
[2013-03-01 10:43:06.116260] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-1: <gfid:2c164035-b076-42ad-b496-b65c72518bf7>: Performing conservative merge
[2013-03-01 10:43:06.133935] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-9: <gfid:2c164035-b076-42ad-b496-b65c72518bf7>: Performing conservative merge
[2013-03-01 10:43:06.395975] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:10c5ce16-ad0a-4bb1-b0d4-a25a3372b791>: Performing conservative merge
[2013-03-01 10:43:06.450119] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:349fb70b-03d3-48d2-a807-4b9935fbcfab>: Performing conservative merge
[2013-03-01 10:43:06.462177] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:a55ba301-48ae-4766-8c67-72302552a23f>: Performing conservative merge
[2013-03-01 10:43:06.466253] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:5a7d7456-f9d2-450c-8fda-d0e720f3162d>: Performing conservative merge
[2013-03-01 10:43:06.486670] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:9cbf2dc5-add8-459c-8de9-1780d72f61b5>: Performing conservative merge
[2013-03-01 10:43:06.496520] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:a15c2ad0-dd2e-4df5-b14c-2db3cac5c80b>: Performing conservative merge
[2013-03-01 10:43:06.506454] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:e8c45e36-c4dd-4d0f-9eba-1314920fbdb5>: Performing conservative merge
[2013-03-01 10:43:06.515501] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:ab2d3bce-7ae4-4cb7-b233-dac75fe0cc4f>: Performing conservative merge
[2013-03-01 10:43:06.542925] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:0bbb1590-4af7-4302-9c17-20d482a0852e>: Performing conservative merge
[2013-03-01 10:43:06.552459] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:2d29ee00-90cb-42d3-b4ca-4c4514527e07>: Performing conservative merge
[2013-03-01 10:43:06.585275] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:347d4907-71b6-4b20-a7f7-94e2968e4862>: Performing conservative merge
[2013-03-01 10:43:06.588558] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:e9b5e4a7-22d3-4b41-9543-ae59d8c6f79b>: Performing conservative merge
[2013-03-01 10:43:06.626280] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:b884e51b-0fde-4b3a-ba9e-846cbe53ebf3>: Performing conservative merge
[2013-03-01 10:43:06.638691] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:596a60a4-1f69-4dd9-9fc5-1d8117d2df7c>: Performing conservative merge
[2013-03-01 10:43:06.643488] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:180c503a-137a-489a-88c8-98ce1dd59217>: Performing conservative merge
[2013-03-01 10:43:06.647676] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:852cf3b9-a89f-4bba-b3f4-8ec7e0e1cdc5>: Performing conservative merge
[2013-03-01 10:43:06.650560] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:3a6dbac6-e904-4048-8ea6-539ff4f6f23c>: Performing conservative merge
[2013-03-01 10:43:06.690637] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:fd2f9cac-653d-4f01-9e9e-60dd38ecab95>: Performing conservative merge
[2013-03-01 10:43:06.694432] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:fabc4776-49c7-4bcd-a391-bf8fb45c1785>: Performing conservative merge
[2013-03-01 10:43:06.698631] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:570804f8-1149-4ffe-b1cf-1ec47b6feb1d>: Performing conservative merge
[2013-03-01 10:43:06.701959] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:6c99fbb5-8443-4f3f-ae5b-01541f6bd06a>: Performing conservative merge
[2013-03-01 10:43:06.705097] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:49787c87-176b-4c54-96ea-c77bf4d53555>: Performing conservative merge
[2013-03-01 10:43:06.708874] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:25f6123c-d305-462c-8fee-0f6f33a95cd3>: Performing conservative merge
[2013-03-01 10:43:06.712038] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:b5a36b38-c036-49ed-9bd9-23f65ecb6635>: Performing conservative merge
[2013-03-01 10:43:06.715171] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:f74cd8c3-d17b-4215-83c6-465b192f3b2f>: Performing conservative merge
[2013-03-01 10:43:06.718740] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:91135c37-bafa-461a-a404-64abf4684251>: Performing conservative merge
[2013-03-01 10:43:06.750868] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:85eb0d1e-304c-4b08-b0d3-6e635518c449>: Performing conservative merge
[2013-03-01 10:43:06.777705] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:cc5750c6-49f6-4dab-aaf3-f37623917084>: Performing conservative merge
[2013-03-01 10:43:06.802520] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:5ab355e2-bfd5-4504-88f7-b42fda99c616>: Performing conservative merge
[2013-03-01 10:43:06.862105] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:93d55cc4-5aac-41b9-a3fa-9f32517ff786>: Performing conservative merge
[2013-03-01 10:43:06.877074] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:feb30943-22c7-43f4-a6e8-30f751ef173e>: Performing conservative merge
[2013-03-01 10:43:06.896972] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-7: <gfid:2c164035-b076-42ad-b496-b65c72518bf7>: Performing conservative merge
[2013-03-01 10:52:47.241084] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-5: <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>: Performing conservative merge
[2013-03-01 10:52:47.604271] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/-reAWDuEQ7o.dat on data-client-18
[2013-03-01 10:52:47.604356] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/4hypFSAKxC8.dat on data-client-18
[2013-03-01 10:52:47.604412] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/6b3zx_IZhJA.dat on data-client-18
[2013-03-01 10:52:47.604448] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/9ffveVk8uzY.dat on data-client-18
[2013-03-01 10:52:47.604500] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/CXTp9Jsez0g.dat on data-client-18
[2013-03-01 10:52:47.604558] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/D2BudsKiFSs.dat on data-client-18
[2013-03-01 10:52:47.604627] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/N3OPl1BLtXU.dat on data-client-18
[2013-03-01 10:52:47.604663] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/OwZlgJnbAVk.dat on data-client-18
[2013-03-01 10:52:47.604698] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/PchuM81Dp1E.dat on data-client-18
[2013-03-01 10:52:47.604732] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/Q0Fw-BZrq94.dat on data-client-18
[2013-03-01 10:52:47.604767] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/Q3NdMIRUTrQ.dat on data-client-18
[2013-03-01 10:52:47.604818] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/Vy_P7-lRj-s.dat on data-client-18
[2013-03-01 10:52:47.604844] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/X5AYPv5EQwc.dat on data-client-18
[2013-03-01 10:52:47.604875] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/bHL2xZcU6Dk.dat on data-client-18
[2013-03-01 10:52:47.604988] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/pZ1PEtw0bXs.dat on data-client-18
[2013-03-01 10:52:47.605016] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/qVo07JNt1MA.dat on data-client-18
[2013-03-01 10:52:47.605039] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/rP_teS6y3e8.dat on data-client-18
[2013-03-01 10:52:47.605061] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/s-p16BACRdc.dat on data-client-18
[2013-03-01 10:52:47.605084] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/v8TJFOFK3NE.dat on data-client-18
[2013-03-01 10:52:47.605122] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/wZvX5wVmDyo.dat on data-client-18
[2013-03-01 10:52:47.605155] I [afr-self-heal-entry.c:638:afr_sh_entry_expunge_entry_cbk] 0-data-replicate-9: missing entry <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>/yuric2csiA0.dat on data-client-18
[2013-03-01 10:52:47.895622] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-10: <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>: Performing conservative merge
[2013-03-01 10:52:58.624185] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 0-data-replicate-4: <gfid:1af423e8-5e22-4abc-ad55-29874f6e176b>: Performing conservative merge
[2013-03-01 11:02:59.255375] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-1: server 172.16.5.3:24009 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255469] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-3: server 172.16.5.3:24010 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255500] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-7: server 172.16.5.3:24012 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255546] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-5: server 172.16.5.3:24011 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255573] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-9: server 172.16.5.3:24013 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255600] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-11: server 172.16.5.3:24014 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255626] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-13: server 172.16.5.3:24015 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255650] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-19: server 172.16.5.3:24018 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255674] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-17: server 172.16.5.3:24017 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255703] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-21: server 172.16.5.3:24019 has not responded in the last 15 seconds, disconnecting.
[2013-03-01 11:02:59.255728] C [client-handshake.c:126:rpc_client_ping_timer_expired] 0-data-client-15: server 172.16.5.3:24016 has not responded in the last 15 seconds, disconnecting.

Comment 2 shishir gowda 2013-03-05 03:37:06 UTC

Hi,

Can you please provide the following:
output from:
1. gluster peer status
2. gluster volume status data

logs from servers/bricks. I see that the node-uuid xattr requests are failing.

Comment 4 Niels de Vos 2014-11-27 14:54:11 UTC

The version that this bug has been reported against, does not get any updates from the Gluster Community anymore. Please verify if this report is still valid against a current (3.4, 3.5 or 3.6) release and update the version, or close this bug.

If there has been no update before 9 December 2014, this bug will get automatocally closed.

Comment 5 Niels de Vos 2015-05-26 12:33:52 UTC

This bug has been CLOSED, and there has not been a response to the requested NEEDINFO in more than 4 weeks. The NEEDINFO flag is now getting cleared so that our Bugzilla household is getting more in order.

Note You need to log in before you can comment on or make changes to this bug.