Description of problem: Dist-geo-rep : add brick + rebalance + geo rep - frequent unmouting on slave cluster, status becomes faulty and after that process is started again Version-Release number of selected component (if applicable): 3.4.0.32rhs-1.el6rhs.x86_64 How reproducible: always Steps to Reproduce: Case 1:- -create data on master volume(around 30+GB, make sure that no bricks were down while creating data) - add-new bricks to master volume.(Brick7: 10.70.37.148:/rhs/brick4/3 Brick8: 10.70.37.210:/rhs/brick4/3) - start rebalance process for that volume - start geo rep session between master and slave volume. master v info :- [root@4VM4 ~]# gluster v info add_before Volume Name: add_before Type: Distributed-Replicate Volume ID: 22c821a8-87fd-4469-8cc4-5387ea21403b Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: 10.70.37.148:/rhs/brick4/1 Brick2: 10.70.37.210:/rhs/brick4/1 Brick3: 10.70.37.202:/rhs/brick4/1 Brick4: 10.70.37.148:/rhs/brick4/2 Brick5: 10.70.37.210:/rhs/brick4/2 Brick6: 10.70.37.202:/rhs/brick4/2 Brick7: 10.70.37.148:/rhs/brick4/3 Brick8: 10.70.37.210:/rhs/brick4/3 Options Reconfigured: geo-replication.indexing: on geo-replication.ignore-pid-check: on changelog.changelog: on slave v info:- [root@rhsauto027 ~]# gluster v info xsync_before Volume Name: xsync_before Type: Distribute Volume ID: 1f187623-07d4-4cde-afde-0e2b5021c252 Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: rhsauto027.lab.eng.blr.redhat.com:/rhs/brick5/1 Brick2: rhsauto026.lab.eng.blr.redhat.com:/rhs/brick5/1 status after 36 hour:- [root@4VM4 ~]# gluster volume geo add_before status NODE MASTER SLAVE HEALTH UPTIME --------------------------------------------------------------------------------------------------------------------------- 4VM1.lab.eng.blr.redhat.com add_before ssh://rhsauto031.lab.eng.blr.redhat.com::xsync_before Stable 00:47:55 4VM3.lab.eng.blr.redhat.com add_before ssh://rhsauto031.lab.eng.blr.redhat.com::xsync_before Stable 00:48:11 4VM2.lab.eng.blr.redhat.com add_before ssh://rhsauto031.lab.eng.blr.redhat.com::xsync_before Stable 20:53:10 [root@4VM1 ~]# less /var/log/glusterfs/geo-replication/add_before/ssh%3A%2F%2Froot%4010.70.37.6%3Agluster%3A%2F%2F127.0.0.1%3Axsync_before.log | grep 'faulty' [2013-09-09 15:15:40.593316] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 15:18:10.272254] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 15:22:25.415704] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 15:24:45.472109] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 15:41:56.719897] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 15:48:34.908493] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 15:54:29.453197] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 15:59:11.747431] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:00:42.73467] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:02:31.353377] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:07:59.994769] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:14:17.958304] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:15:56.755346] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:20:03.353142] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:21:19.831585] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:26:58.856230] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:29:33.310067] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:35:52.539582] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:38:57.242283] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:44:02.236695] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:50:01.983565] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:57:29.182040] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 17:10:05.655935] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 17:25:11.452641] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 17:40:46.431274] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 17:46:03.445842] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 17:57:09.696702] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 18:02:40.137281] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 18:21:28.261570] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 18:39:11.361285] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 18:50:19.921430] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 18:56:46.713002] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 19:21:36.547723] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 19:50:06.923304] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 19:52:35.459363] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 20:23:57.729643] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 20:35:23.747926] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 21:04:57.62361] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 21:22:51.817504] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 21:55:07.62434] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 22:07:31.91154] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 22:59:09.382849] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 23:11:16.534081] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-10 00:09:16.758229] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-10 00:22:48.432632] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-10 01:30:10.686233] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-10 01:45:01.178625] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-10 02:58:20.215467] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-10 03:30:14.587718] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-10 04:54:06.143501] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-10 05:32:56.848484] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-10 07:17:05.945013] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-10 08:26:58.803406] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-10 11:01:03.986064] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-11 04:21:42.966079] I [monitor(monitor):81:set_state] Monitor: new state: faulty [root@4VM1 ~]# less /var/log/glusterfs/geo-replication/add_before/ssh%3A%2F%2Froot%4010.70.37.6%3Agluster%3A%2F%2F127.0.0.1%3Axsync_before.%2Frhs%2Fbrick4%2F1.gluster.log | grep unmounting [2013-09-09 09:45:40.592740] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-Sk06i0 [2013-09-09 10:24:29.451755] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-X0gp9c [2013-09-09 10:30:42.072866] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-4Xl_dH [2013-09-09 10:44:17.957847] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-9ROXiK [2013-09-09 10:50:03.348410] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-vtIROH [2013-09-09 10:56:58.855502] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-3BvJ3w [2013-09-09 11:05:52.534539] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-vzRvAj [2013-09-09 11:14:02.235149] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-qP2tcN [2013-09-09 12:10:46.430734] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-YLzO6H [2013-09-09 12:27:09.693658] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-x9Vnki [2013-09-09 12:51:43.550592] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-ZP_PZR [2013-09-09 13:20:19.920812] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-UbIy8g [2013-09-09 14:20:06.921042] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-XXdvOj [2013-09-09 15:05:23.747104] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-c56nCF [2013-09-09 15:52:51.816892] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-C10FZh [2013-09-09 16:37:31.089595] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-jQrJV3 [2013-09-09 17:41:16.533608] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-ylq2yP [2013-09-09 18:52:48.431676] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-yH4sgF [2013-09-09 20:15:01.177935] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-qklfXx [2013-09-09 22:00:14.586722] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-g4t6sw [2013-09-10 00:02:56.847900] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-TzrozE [2013-09-10 02:56:58.748539] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-S2VQaZ Case 2:- -create data on master volume(around 30+GB, make sure that no bricks were down while creating data) - start geo rep session between master and slave volume - add-new node to master volume(10.70.37.106) and bricks to master volume from that node.(Brick7: 10.70.37.106:/rhs/brick5/1 Brick8: 10.70.37.106:/rhs/brick5/2) - start rebalance process for that volume. . [root@4VM4 ~]# gluster v info add_xsync Volume Name: add_xsync Type: Distributed-Replicate Volume ID: 408c059a-4d45-42e4-8af5-46e240bae1eb Status: Started Number of Bricks: 4 x 2 = 8 Transport-type: tcp Bricks: Brick1: 10.70.37.148:/rhs/brick5/1 Brick2: 10.70.37.210:/rhs/brick5/1 Brick3: 10.70.37.202:/rhs/brick5/1 Brick4: 10.70.37.148:/rhs/brick5/2 Brick5: 10.70.37.210:/rhs/brick5/2 Brick6: 10.70.37.202:/rhs/brick5/2 Brick7: 10.70.37.106:/rhs/brick5/1 Brick8: 10.70.37.106:/rhs/brick5/2 Options Reconfigured: geo-replication.indexing: on geo-replication.ignore-pid-check: on changelog.changelog: on slave volume:- [root@rhsauto027 ~]# gluster v info xsync Volume Name: xsync Type: Distribute Volume ID: 1ded1f29-95bd-4aa8-910e-2804846ab7fb Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: rhsauto027.lab.eng.blr.redhat.com:/rhs/brick5/2 Brick2: rhsauto026.lab.eng.blr.redhat.com:/rhs/brick5/2 status after 36 hour:- [root@4VM4 ~]# gluster volume geo add_xsync status NODE MASTER SLAVE HEALTH UPTIME ------------------------------------------------------------------------------------------------------------------------- 4VM4.lab.eng.blr.redhat.com add_xsync ssh://rhsauto031.lab.eng.blr.redhat.com::xsync Stable 1 day 13:38:11 4VM1.lab.eng.blr.redhat.com add_xsync ssh://rhsauto031.lab.eng.blr.redhat.com::xsync Stable 1 day 02:58:25 4VM2.lab.eng.blr.redhat.com add_xsync ssh://rhsauto031.lab.eng.blr.redhat.com::xsync Stable 23:36:25 4VM3.lab.eng.blr.redhat.com add_xsync ssh://rhsauto031.lab.eng.blr.redhat.com::xsync Stable 1 day 04:10:22 log snippet:- [root@4VM1 ~]# less /var/log/glusterfs/geo-replication/add_xsync/ssh%3A%2F%2Froot%4010.70.37.6%3Agluster%3A%2F%2F127.0.0.1%3Axsync.log | grep 'faulty' [2013-09-09 15:34:04.802366] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 15:48:58.782192] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 15:55:56.583132] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 15:58:38.26252] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:04:25.723999] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:15:30.932065] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:18:43.637089] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 16:41:26.201895] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 18:22:53.361649] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 18:50:27.961016] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 19:15:32.916991] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 22:06:56.695663] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-09 23:51:58.291364] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-10 01:07:10.433307] I [monitor(monitor):81:set_state] Monitor: new state: faulty [2013-09-10 02:16:29.444293] I [monitor(monitor):81:set_state] Monitor: new state: faulty [root@4VM1 ~]# less /var/log/glusterfs/geo-replication/add_xsync/ssh%3A%2F%2Froot%4010.70.37.6%3Agluster%3A%2F%2F127.0.0.1%3Axsync.%2Frhs%2Fbrick5%2F1.gluster.log | grep -B 6 'unmounting' [2013-09-09 10:18:56.413837] I [dht-layout.c:633:dht_layout_normalize] 0-add_xsync-dht: found anomalies in /flat/1/etc4/cluster/cman-notify.d. holes=1 overlaps=0 missing=1 down=0 misc=0 [2013-09-09 10:18:56.483299] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-add_xsync-client-7: remote operation failed: File exists. Path: /flat/1/etc4/cluster/cman-notify.d [2013-09-09 10:18:56.483371] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-add_xsync-client-6: remote operation failed: File exists. Path: /flat/1/etc4/cluster/cman-notify.d [2013-09-09 10:18:57.736476] I [dht-layout.c:633:dht_layout_normalize] 0-add_xsync-dht: found anomalies in /flat/1/etc4/ntp. holes=1 overlaps=0 missing=1 down=0 misc=0 [2013-09-09 10:18:58.364601] I [dht-layout.c:633:dht_layout_normalize] 0-add_xsync-dht: found anomalies in /flat/1/etc4/ntp/crypto. holes=1 overlaps=1 missing=0 down=0 misc=0 [2013-09-09 10:18:58.612947] I [dht-common.c:1035:dht_lookup_everywhere_cbk] 0-add_xsync-dht: deleting stale linkfile /flat/1/etc4/ntp/crypto/pw on add_xsync-replicate-0 [2013-09-09 10:18:58.781583] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-r9P3SN -- [2013-09-09 10:23:16.956405] I [dht-layout.c:720:dht_layout_dir_mismatch] 0-add_xsync-dht: /flat/1/etc5 - disk layout missing [2013-09-09 10:23:16.956459] I [dht-common.c:638:dht_revalidate_cbk] 0-add_xsync-dht: mismatching layouts for /flat/1/etc5 [2013-09-09 10:23:21.760133] I [dht-layout.c:720:dht_layout_dir_mismatch] 0-add_xsync-dht: /flat/1/etc5 - disk layout missing [2013-09-09 10:23:21.760183] I [dht-common.c:638:dht_revalidate_cbk] 0-add_xsync-dht: mismatching layouts for /flat/1/etc5 [2013-09-09 10:23:26.620804] I [dht-layout.c:720:dht_layout_dir_mismatch] 0-add_xsync-dht: /flat/1/etc5 - disk layout missing [2013-09-09 10:23:26.620858] I [dht-common.c:638:dht_revalidate_cbk] 0-add_xsync-dht: mismatching layouts for /flat/1/etc5 [2013-09-09 10:25:56.581343] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-CNcXec -- [2013-09-09 10:26:09.192225] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-3: added root inode [2013-09-09 10:26:09.192519] I [afr-common.c:2184:afr_discovery_cbk] 0-add_xsync-replicate-0: selecting local read_child add_xsync-client-0 [2013-09-09 10:26:09.193733] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-2: added root inode [2013-09-09 10:27:15.224986] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-add_xsync-client-4: remote operation failed: File exists. Path: /flat/1/etc2/X11/fontpath.d/default-ghostscript [2013-09-09 10:27:15.225042] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-add_xsync-client-5: remote operation failed: File exists. Path: /flat/1/etc2/X11/fontpath.d/default-ghostscript [2013-09-09 10:27:15.229058] W [dht-linkfile.c:44:dht_linkfile_lookup_cbk] 0-add_xsync-dht: got non-linkfile add_xsync-replicate-2:/flat/1/etc2/X11/fontpath.d/default-ghostscript [2013-09-09 10:28:38.025439] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-H3aPiW -- [2013-09-09 12:52:40.196671] I [dht-layout.c:633:dht_layout_normalize] 0-add_xsync-dht: found anomalies in /flat/1/etc57/kdump-adv-conf/kdump_initscripts. holes=1 overlaps=0 missing=1 down=0 misc=0 [2013-09-09 12:52:40.211725] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-add_xsync-client-6: remote operation failed: File exists. Path: /flat/1/etc57/kdump-adv-conf/kdump_initscripts [2013-09-09 12:52:40.211775] W [client-rpc-fops.c:322:client3_3_mkdir_cbk] 0-add_xsync-client-7: remote operation failed: File exists. Path: /flat/1/etc57/kdump-adv-conf/kdump_initscripts [2013-09-09 12:52:40.831716] I [dht-layout.c:633:dht_layout_normalize] 0-add_xsync-dht: found anomalies in /flat/1/etc57/gtk-2.0. holes=1 overlaps=0 missing=1 down=0 misc=0 [2013-09-09 12:52:53.001802] I [dht-layout.c:633:dht_layout_normalize] 0-add_xsync-dht: found anomalies in /flat/1/etc57/audisp/plugins.d. holes=2 overlaps=0 missing=0 down=0 misc=0 [2013-09-09 12:52:53.171243] I [dht-common.c:1035:dht_lookup_everywhere_cbk] 0-add_xsync-dht: deleting stale linkfile /flat/1/etc57/audisp/plugins.d/syslog.conf on add_xsync-replicate-0 [2013-09-09 12:52:53.360690] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-CTBoey -- [2013-09-09 12:53:05.364749] I [afr-common.c:2184:afr_discovery_cbk] 0-add_xsync-replicate-0: selecting local read_child add_xsync-client-0 [2013-09-09 12:53:05.366916] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-1: added root inode [2013-09-09 12:53:05.367050] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-3: added root inode [2013-09-09 12:53:05.367193] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-2: added root inode [2013-09-09 12:53:05.367320] I [afr-common.c:2184:afr_discovery_cbk] 0-add_xsync-replicate-1: selecting local read_child add_xsync-client-3 [2013-09-09 13:20:27.812719] I [dht-common.c:1035:dht_lookup_everywhere_cbk] 0-add_xsync-dht: deleting stale linkfile /flat/1/etc39/sysconfig/ha/lvs.cf on add_xsync-replicate-0 [2013-09-09 13:20:27.960701] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-l4jxwp -- [2013-09-09 13:20:40.083565] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-0: added root inode [2013-09-09 13:20:40.083848] I [afr-common.c:2184:afr_discovery_cbk] 0-add_xsync-replicate-0: selecting local read_child add_xsync-client-0 [2013-09-09 13:20:40.097408] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-1: added root inode [2013-09-09 13:20:40.097591] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-2: added root inode [2013-09-09 13:20:40.097738] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-3: added root inode [2013-09-09 13:20:40.097817] I [afr-common.c:2184:afr_discovery_cbk] 0-add_xsync-replicate-1: selecting local read_child add_xsync-client-3 [2013-09-09 13:45:32.914347] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-kSXDrs -- [2013-09-09 16:36:54.804144] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-add_xsync-client-4: remote operation failed: File exists. Path: /flat/flat/flat/2/etc3/makedev.d/02linux-2.6.x [2013-09-09 16:36:55.122001] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-add_xsync-client-4: remote operation failed: File exists. Path: /flat/flat/flat/2/etc3/makedev.d/01sound [2013-09-09 16:36:55.122643] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-add_xsync-client-5: remote operation failed: File exists. Path: /flat/flat/flat/2/etc3/makedev.d/01sound [2013-09-09 16:36:55.812452] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-add_xsync-client-5: remote operation failed: File exists. Path: /flat/flat/flat/2/etc3/makedev.d/01linux1394 [2013-09-09 16:36:55.813097] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-add_xsync-client-4: remote operation failed: File exists. Path: /flat/flat/flat/2/etc3/makedev.d/01linux1394 [2013-09-09 16:36:56.428212] I [dht-common.c:1035:dht_lookup_everywhere_cbk] 0-add_xsync-dht: deleting stale linkfile /flat/flat/flat/2/etc3/host.conf on add_xsync-replicate-0 [2013-09-09 16:36:56.693821] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-mTVDON -- [2013-09-09 18:19:03.199296] I [dht-common.c:1035:dht_lookup_everywhere_cbk] 0-add_xsync-dht: deleting stale linkfile /flat/flat/flat/2/etc2/.pwd.lock on add_xsync-replicate-3 [2013-09-09 18:19:04.318893] I [dht-common.c:1035:dht_lookup_everywhere_cbk] 0-add_xsync-dht: deleting stale linkfile /flat/flat/flat/2/etc2/redhat-release on add_xsync-replicate-3 [2013-09-09 18:19:04.378022] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-add_xsync-client-5: remote operation failed: File exists. Path: /flat/flat/flat/2/etc2/redhat-release [2013-09-09 18:19:04.378615] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-add_xsync-client-4: remote operation failed: File exists. Path: /flat/flat/flat/2/etc2/redhat-release [2013-09-09 18:19:11.392492] I [dht-common.c:1035:dht_lookup_everywhere_cbk] 0-add_xsync-dht: deleting stale linkfile /flat/flat/flat/2/etc2/fstab on add_xsync-replicate-3 [2013-09-09 18:21:58.144908] I [dht-common.c:1035:dht_lookup_everywhere_cbk] 0-add_xsync-dht: deleting stale linkfile /flat/flat/flat/2/etc3/aliases on add_xsync-replicate-0 [2013-09-09 18:21:58.290901] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-PPAxiu -- [2013-09-09 18:22:15.118493] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-3: added root inode [2013-09-09 18:22:15.118752] I [afr-common.c:2184:afr_discovery_cbk] 0-add_xsync-replicate-1: selecting local read_child add_xsync-client-3 [2013-09-09 18:22:15.118823] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-2: added root inode [2013-09-09 19:37:09.958214] I [dht-common.c:1035:dht_lookup_everywhere_cbk] 0-add_xsync-dht: deleting stale linkfile /flat/flat/flat/2/etc3/shadow- on add_xsync-replicate-0 [2013-09-09 19:37:10.029724] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-add_xsync-client-6: remote operation failed: File exists. Path: /flat/flat/flat/2/etc3/shadow- [2013-09-09 19:37:10.030219] W [client-rpc-fops.c:256:client3_3_mknod_cbk] 0-add_xsync-client-7: remote operation failed: File exists. Path: /flat/flat/flat/2/etc3/shadow- [2013-09-09 19:37:10.432798] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-cRxHEQ -- [2013-09-09 19:37:26.823808] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-0: added root inode [2013-09-09 19:37:26.825667] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-1: added root inode [2013-09-09 19:37:26.825750] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-2: added root inode [2013-09-09 19:37:26.825960] I [afr-common.c:2121:afr_set_root_inode_on_first_lookup] 0-add_xsync-replicate-3: added root inode [2013-09-09 19:37:26.826101] I [afr-common.c:2184:afr_discovery_cbk] 0-add_xsync-replicate-0: selecting local read_child add_xsync-client-0 [2013-09-09 19:37:26.828432] I [afr-common.c:2184:afr_discovery_cbk] 0-add_xsync-replicate-1: selecting local read_child add_xsync-client-3 [2013-09-09 20:46:29.443695] I [fuse-bridge.c:5714:fuse_thread_proc] 0-fuse: unmounting /tmp/gsyncd-aux-mount-MKn0vR Actual results: due to frequent faulty status, all data is not synced to slave Expected results: Additional info: after 36 hours: rebalance status:- [root@4VM4 ~]# gluster volume rebalance add_xsync status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 0 0Bytes 1121097 0 0 completed 122819.00 10.70.37.148 197622 3.1GB 995033 0 1 in progress 135602.00 10.70.37.202 198642 2.5GB 943085 0 0 in progress 135602.00 10.70.37.210 184583 2.8GB 809732 0 0 in progress 135603.00 volume rebalance: add_xsync: success: [root@4VM4 ~]# gluster volume rebalance add_before status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 0 0Bytes 0 0 0 not started 0.00 10.70.37.148 170959 2.6GB 857639 6 106 completed 110078.00 10.70.37.202 170823 2.1GB 870464 3 131 completed 110011.00 10.70.37.210 171041 2.5GB 804526 0 40 completed 110124.00 volume rebalance: add_before: success: (for add_before volume it took around 24 hours)
Closing this bug since RHGS 2.1 release reached EOL. Required bugs are cloned to RHGS 3.1. Please re-open this issue if found again.