Bug 763429 (GLUSTER-1697)

added 2 bricks to existing 2 afr servers  and did a rebalance  - file 3 and 2 are moved to newly added server after rebalance.- and these files are not accessible from nfs mount. but from fuse there is no issue.

nfsmount# ls -trltr
ls: cannot access 3: No such file or directory
ls: cannot access 2: No such file or directory
total 20
?????????? ? ?    ?    ?                ? 3
?????????? ? ?    ?    ?                ? 2
-rw-r--r-- 1 root root 0 2010-09-24 02:26 1
-rw-r--r-- 1 root root 0 2010-09-24 02:29 9
-rw-r--r-- 1 root root 0 2010-09-24 03:29 8
-rw-r--r-- 1 root root 0 2010-09-24 03:29 7
-rw-r--r-- 1 root root 0 2010-09-24 03:29 5


nfs-server log -

[2010-09-24 03:31:34.64134] E [afr-common.c:2654:afr_notify] betaafr-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2010-09-24 03:31:34.65174] E [afr-common.c:2654:afr_notify] betaafr-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2010-09-24 03:31:34.656453] E [client-handshake.c:730:client_query_portmap_cbk] betaafr-client-3: failed to get the port number for remote subvolume
[2010-09-24 03:31:34.656530] E [afr-common.c:2654:afr_notify] betaafr-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
[2010-09-24 03:31:34.721498] E [afr-common.c:2654:afr_notify] betaafr-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
[2010-09-24 03:31:37.67814] I [client-handshake.c:660:select_server_supported_programs] betaafr-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-24 03:31:37.68072] I [client-handshake.c:496:client_setvolume_cbk] betaafr-client-0: Connected to 10.192.141.187:6971, attached to remote volume '/mnt/beta'.
[2010-09-24 03:31:37.68094] I [afr-common.c:2624:afr_notify] betaafr-replicate-0: Subvolume 'betaafr-client-0' came back up; going online.
[2010-09-24 03:31:37.84603] I [client-handshake.c:660:select_server_supported_programs] betaafr-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-24 03:31:37.85134] I [client-handshake.c:496:client_setvolume_cbk] betaafr-client-1: Connected to 10.192.134.144:6971, attached to remote volume '/mnt/beta'.
[2010-09-24 03:31:38.75108] E [afr-common.c:2654:afr_notify] betaafr-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
[2010-09-24 03:31:38.77052] I [client-handshake.c:660:select_server_supported_programs] betaafr-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-24 03:31:38.77546] I [client-handshake.c:496:client_setvolume_cbk] betaafr-client-2: Connected to 10.214.231.112:6971, attached to remote volume '/mnt/beta'.
[2010-09-24 03:31:38.77572] I [afr-common.c:2624:afr_notify] betaafr-replicate-1: Subvolume 'betaafr-client-2' came back up; going online.
[2010-09-24 03:31:38.78168] I [afr-common.c:827:afr_fresh_lookup_cbk] betaafr-replicate-0: added root inode
[2010-09-24 03:31:38.78372] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:31:38.79427] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:31:38.79803] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:31:38.80347] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:31:38.80372] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:31:38.80514] I [afr-common.c:827:afr_fresh_lookup_cbk] betaafr-replicate-1: added root inode
[2010-09-24 03:31:38.81230] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:31:38.81600] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:31:38.81668] I [nfs.c:315:__nfs_subvolume_start] nfs: All exports up
[2010-09-24 03:31:41.81784] I [client-handshake.c:660:select_server_supported_programs] betaafr-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-24 03:31:41.82497] I [client-handshake.c:496:client_setvolume_cbk] betaafr-client-3: Connected to 10.198.110.16:6971, attached to remote volume '/mnt/beta'.
[2010-09-24 03:31:47.527950] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:31:47.528017] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:31:47.528453] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-1: background  meta-data self-heal triggered. path: /
[2010-09-24 03:31:47.538796] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:31:47.539280] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:31:47.539800] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-1: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:31:47.549793] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-1: background  meta-data self-heal completed on /
[2010-09-24 03:33:23.304648] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:33:23.304705] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:33:23.305048] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-1: split brain detected during lookup of /.
[2010-09-24 03:33:23.305073] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-1: background  data self-heal triggered. path: /
[2010-09-24 03:33:23.305145] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-1: background  data self-heal completed on /
[2010-09-24 03:33:23.305619] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:33:23.306000] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:33:25.639876] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:33:25.639928] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:33:25.640779] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:33:25.641154] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:33:32.397096] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:33:32.397174] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:33:32.398034] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:33:32.398412] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:34:41.137883] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:34:41.137957] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:34:41.138795] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:34:41.139190] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:34:54.402184] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:34:54.402259] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:34:54.403162] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:34:54.403545] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:34:55.682175] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:34:55.682236] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:34:55.683137] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:34:55.683520] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:35:26.711633] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:35:26.711714] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:35:26.723124] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:35:26.723499] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:35:45.525570] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:35:45.525635] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:35:45.526491] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:35:45.526936] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:35:45.678801] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:35:45.678831] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:35:45.679656] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:35:45.680023] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:35:45.854238] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:35:45.854287] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:35:45.855108] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:35:45.855484] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:38:17.918331] E [nfs3.c:4336:nfs3_fsstat] nfs-nfsv3: Failed to map FH to vol
[2010-09-24 03:38:17.940138] E [nfs3.c:4336:nfs3_fsstat] nfs-nfsv3: Failed to map FH to vol
[2010-09-24 03:39:16.469653] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:39:16.469730] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:39:16.470641] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:39:16.471042] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:39:56.728298] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:39:56.728412] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:39:56.728591] I [dht-layout.c:676:dht_layout_dir_mismatch] betaafr-dht: subvol: betaafr-replicate-1; inode layout - 2147483647 - 4294967295; disk layout - 1431655765 - 2863311529
[2010-09-24 03:39:56.728612] I [dht-common.c:412:dht_revalidate_cbk] betaafr-dht: mismatching layouts for /
[2010-09-24 03:39:56.729307] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:39:56.729685] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:39:56.729711] I [dht-layout.c:676:dht_layout_dir_mismatch] betaafr-dht: subvol: betaafr-replicate-0; inode layout - 0 - 2147483646; disk layout - 0 - 1431655764
[2010-09-24 03:39:56.729738] I [dht-common.c:412:dht_revalidate_cbk] betaafr-dht: mismatching layouts for /
[2010-09-24 03:39:56.730266] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 03:39:56.730292] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 03:39:56.731125] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 03:39:56.731500] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 03:39:56.731522] I [dht-layout.c:575:dht_layout_normalize] betaafr-dht: found anomalies in /. holes=1 overlaps=0
[2010-09-24 03:39:56.731538] I [dht-common.c:273:dht_lookup_root_dir_cbk] betaafr-dht: fixing assignment on /
[2010-09-24 04:31:20.896225] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 04:31:20.896337] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 04:31:20.897255] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 04:31:20.897622] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 05:14:10.138384] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 05:14:10.138685] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 05:14:10.139602] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 05:14:10.140005] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /
[2010-09-24 05:14:11.184618] I [afr-common.c:678:afr_lookup_done] betaafr-replicate-0: split brain detected during lookup of /.
[2010-09-24 05:14:11.184681] I [afr-common.c:724:afr_lookup_done] betaafr-replicate-0: background  meta-data data self-heal triggered. path: /
[2010-09-24 05:14:11.185519] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] betaafr-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes
[2010-09-24 05:14:11.185893] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] betaafr-replicate-0: background  meta-data data self-heal completed on /