Description of problem: Adding a brick and running rebalance leads to migration failures , log message says " remote operation failed: File exists" Version-Release number of selected component (if applicable): glusterfs-geo-replication-3.6.0.18-1.el6rhs.x86_64 gluster-swift-object-1.10.0-2.el6rhs.noarch glusterfs-fuse-3.6.0.18-1.el6rhs.x86_64 gluster-swift-1.10.0-2.el6rhs.noarch gluster-nagios-common-0.1.0-26.git2b35b66.el6rhs.x86_64 glusterfs-server-3.6.0.18-1.el6rhs.x86_64 glusterfs-rdma-3.6.0.18-1.el6rhs.x86_64 gluster-swift-proxy-1.10.0-2.el6rhs.noarch gluster-swift-account-1.10.0-2.el6rhs.noarch glusterfs-libs-3.6.0.18-1.el6rhs.x86_64 glusterfs-3.6.0.18-1.el6rhs.x86_64 glusterfs-cli-3.6.0.18-1.el6rhs.x86_64 glusterfs-debuginfo-3.6.0.18-1.el6rhs.x86_64 gluster-swift-container-1.10.0-2.el6rhs.noarch gluster-swift-plugin-1.10.0-5.el6rhs.noarch gluster-nagios-addons-0.1.0-57.git9d252a3.el6rhs.x86_64 samba-glusterfs-3.6.9-168.1.el6rhs.x86_64 vdsm-gluster-4.14.5-21.git7a3d0f0.el6rhs.noarch glusterfs-api-3.6.0.18-1.el6rhs.x86_64 How reproducible: Tried once Steps to Reproduce: 1.created a 2 brick distribute volume 2. created some data by for i in {1..10} do mkdir $i ; cd $i; cp -R /etc/* .; done 3. add one more brick and run rebalance Actual results: Rebalance failures are seen Additional info: --------------- Rebalance logs ================ [2014-06-18 06:44:50.298216] W [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-new-client-2: remote operation failed: File exists. Path: /1/2/3/4/5/6/7/8/9/10/xdg [2014-06-18 06:44:50.298252] D [dht-selfheal.c:419:dht_selfheal_dir_mkdir_cbk] 0-new-dht: selfhealing directory /1/2/3/4/5/6/7/8/9/10/xdg failed: File exists Volume Name: new Type: Distribute Volume ID: 5d5b5cdf-6c77-4200-a7d0-9f4ac5828a0c Status: Started Snap Volume: no Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: rhs-client4.lab.eng.blr.redhat.com:/home/n0 Brick2: rhs-client39.lab.eng.blr.redhat.com:/home/n1 Brick3: rhs-client4.lab.eng.blr.redhat.com:/home/n2 Options Reconfigured: diagnostics.client-log-level: DEBUG diagnostics.client-log-buf-size: 20 diagnostics.brick-log-buf-size: 10 diagnostics.brick-log-flush-timeout: 200 diagnostics.client-log-flush-timeout: 300 diagnostics.client-log-format: with-msg-id [root@rhs-client4 mnt]# gluster v rebalance new status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 1306 313.8KB 20503 17 2395 completed 93.00 rhs-client39.lab.eng.blr.redhat.com 1001 18.6KB 20912 15 2106 completed 93.00 volume rebalance: new: success: cluster info ============== rhs-client4.lab.eng.blr.redhat.com rhs-client39.lab.eng.blr.redhat.com attached the sosreports
Created attachment 911931 [details] Rebalane-Race
Added an attachment which describes the race condition. From the logs, it seems to be a race condition between two rebalance prcocess. STATE 1: BRICK-1 only one brick Cached File in the system STATE 2: Add brick-2 BRICK-1 BRICK-2 STATE 3: Lookup of File on brick-2 by this node's rebalance will fail because hashed file is not created yet. So dht_lookup_everywhere is about to get called. STATE 4: As part of lookup link file at brick-2 will be created. STATE 5: getxattr to check that cached file belongs to this node is done STATE 6: dht_lookup_everywhere_cbk detects the link created by rebalance-1. It will unlink it. STATE 7: getxattr at the link file with "pathinfo" key will be called will fail as the link file is deleted by rebalance on node-2
With the release: I see much more failures reported during remove-brick: Node Rebalanced-files size scanned failures skipped status r un time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 0 0Bytes 458684 0 0 in progress 5790.00 172.17.69.1 114615 3.4MB 293904 30561 0 in progress 5790.00 30561 and counting... However, I do not see any errors reported in the logs.
Sent on downstream branch: https://code.engineering.redhat.com/gerrit/#/c/29357/
Additional patches need to be merged to complete this fix bug. They are currently being reviewed. Moving this back to POST.
Gluster-server version ====================== [root@rhssvm-swift2 ~]# gluster --version glusterfs 3.6.0.28 built on Sep 3 2014 10:13:12 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. glusterfs-client version ======================== [root@rhs-client10 10]# glusterfs --version glusterfs 3.6.0.28 built on Sep 3 2014 10:13:11 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. Tested the following steps: 1. Created a distributed volume with 2 bricks 2. From mount point ran for i in {1..10} do mkdir $i ; cd $i; cp -R /etc/* .; done to create some data 3. Added a brick and then executed rebalance & waited till the status became completed. 4. grep "remote operation failed: File exists" /var/log/glusterfs/vol1-rebalance.log | wc -l 0 Hence this bug is verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-1278.html