Description of problem: ======================= after multiple rename and rebalnce, output of ls -l on parents has '?' in attribute field -????????? ? ? ? ? ? zero9-112 -rw-r--r-- 3 root root 0 Oct 17 14:22 zero9-200 -????????? ? ? ? ? ? zero9-201 version:- ======== 3.4.0.69rhs-1.el6rhs.x86_64 How reproducible: ================= always Steps to Reproduce: =================== 1. create, start and mount distributed volume. 2. create few files inside it 3. start renaming files from multiple mount and start rebalance also 4. repeat step 3 for few times 5. verify all files. Actual results: ============== NFS mount:- for few files lookup on parents shows that files attributes are not listed -????????? ? ? ? ? ? zero1-110 -????????? ? ? ? ? ? zero11-116 -rw-r--r-- 9 root root 0 Oct 17 14:22 zero11-149 -????????? ? ? ? ? ? zero11-201 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero12-200 -rw-r--r-- 7 root root 0 Oct 17 14:22 zero12-201 -rw-r--r-- 2 root root 20971520 Oct 17 14:24 zero13-200 -rw-r--r-- 9 root root 0 Oct 17 14:22 zero13-201 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero14-201 -rw-r--r-- 9 root root 0 Oct 17 14:22 zero15-127 -rw-r--r-- 7 root root 0 Oct 17 14:22 zero16-168 -rw-r--r-- 9 root root 0 Oct 17 14:22 zero16-201 -rw-r--r-- 2 root root 0 Oct 17 14:22 zero16-67 -rw-r--r-- 2 root root 20971520 Oct 17 14:21 zero17-166 -rw-r--r-- 3 root root 0 Oct 17 14:22 zero17-201 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero18-200 -rw-r--r-- 9 root root 0 Oct 17 14:22 zero19-127 -rw-r--r-- 3 root root 0 Oct 17 14:22 zero19-200 -rw-r--r-- 2 root root 0 Oct 17 14:22 zero19-201 -rw-r--r-- 2 root root 0 Oct 17 14:22 zero19-41 -????????? ? ? ? ? ? zero20-101 -rw-r--r-- 9 root root 0 Oct 17 14:22 zero20-201 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero21-1 -rw-r--r-- 5 root root 20971520 Oct 17 14:21 zero2-154 -rw-r--r-- 7 root root 0 Oct 17 14:22 zero2-169 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero22-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero23-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero24-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero25-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero26-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero27-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero28-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero29-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero30-1 -????????? ? ? ? ? ? zero3-101 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero31-1 -????????? ? ? ? ? ? zero3-110 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero32-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero33-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero34-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero35-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero36-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero37-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero38-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero39-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero40-1 -rw-r--r-- 4 root root 0 Oct 17 14:22 zero4-201 -rw-r--r-- 3 root root 0 Oct 17 14:22 zero5-201 -rw-r--r-- 3 root root 0 Oct 17 14:22 zero6-201 -rw-r--r-- 4 root root 0 Oct 17 14:22 zero7-201 -rw-r--r-- 5 root root 20971520 Oct 17 14:21 zero8-140 -rw-r--r-- 0 root root 0 Oct 17 14:22 zero8-201 -????????? ? ? ? ? ? zero9-112 -rw-r--r-- 3 root root 0 Oct 17 14:22 zero9-200 -????????? ? ? ? ? ? zero9-201 FUSE mount:- -rw-r--r-- 5 root root 20971520 Oct 17 14:21 zero2-154 -rw-r--r-- 7 root root 0 Oct 17 14:22 zero2-169 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero22-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero23-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero24-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero25-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero26-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero27-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero28-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero29-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero30-1 -rw-r--r-- 4 root root 0 Oct 17 14:22 zero3-101 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero31-1 -rw-r--r-- 4 root root 0 Oct 17 14:22 zero3-110 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero32-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero33-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero34-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero35-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero36-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero37-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero38-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero39-1 -rw-r--r-- 1 root root 0 Oct 17 14:22 zero40-1 -rw-r--r-- 4 root root 0 Oct 17 14:22 zero4-201 -rw-r--r-- 3 root root 0 Oct 17 14:22 zero5-201 -rw-r--r-- 3 root root 0 Oct 17 14:22 zero6-201 -rw-r--r-- 4 root root 0 Oct 17 14:22 zero7-201 -rw-r--r-- 5 root root 20971520 Oct 17 14:21 zero8-140 -rw-r--r-- 0 root root 0 Oct 17 14:22 zero8-201 -rw-r--r-- 4 root root 0 Oct 17 14:22 zero9-112 -rw-r--r-- 3 root root 0 Oct 17 14:22 zero9-200 -rw-r--r-- 4 root root 0 Oct 17 14:22 zero9-201 Expected results: ================= NFS mount should show file attributes Additional info: log snippet:- [2014-10-27 17:54:59.307808] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:3e650fd4-bbc2-4cd6-8372-b50ebbc0772e>. holes=1 overlaps=0 missing=2 down=0 misc=0 [2014-10-27 17:54:59.307932] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 3e650fd4-bbc2-4cd6-8372-b50ebbc0772e [2014-10-27 17:54:59.307964] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: ed73a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address) [2014-10-27 17:54:59.310734] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:d42016a9-6f5e-4ee3-8a2e-92c922eb016f>. holes=1 overlaps=0 missing=2 down=0 misc=0 [2014-10-27 17:54:59.310825] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : d42016a9-6f5e-4ee3-8a2e-92c922eb016f [2014-10-27 17:54:59.310872] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: ef73a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address) [2014-10-27 17:54:59.312554] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:3e650fd4-bbc2-4cd6-8372-b50ebbc0772e>. holes=1 overlaps=0 missing=2 down=0 misc=0 [2014-10-27 17:54:59.312672] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 3e650fd4-bbc2-4cd6-8372-b50ebbc0772e [2014-10-27 17:54:59.312700] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f073a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address) [2014-10-27 17:54:59.314103] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:3e650fd4-bbc2-4cd6-8372-b50ebbc0772e>. holes=1 overlaps=0 missing=2 down=0 misc=0 [2014-10-27 17:54:59.314188] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 3e650fd4-bbc2-4cd6-8372-b50ebbc0772e [2014-10-27 17:54:59.314216] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f173a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address) [2014-10-27 17:54:59.315837] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:d42016a9-6f5e-4ee3-8a2e-92c922eb016f>. holes=1 overlaps=0 missing=2 down=0 misc=0 [2014-10-27 17:54:59.315894] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : d42016a9-6f5e-4ee3-8a2e-92c922eb016f [2014-10-27 17:54:59.315917] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f273a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address) [2014-10-27 17:54:59.317878] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 1d770502-8406-411e-ac41-80e1442cb37a [2014-10-27 17:54:59.317921] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f373a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address) [2014-10-27 17:54:59.319345] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:d42016a9-6f5e-4ee3-8a2e-92c922eb016f>. holes=1 overlaps=0 missing=2 down=0 misc=0 [2014-10-27 17:54:59.319402] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : d42016a9-6f5e-4ee3-8a2e-92c922eb016f [2014-10-27 17:54:59.319424] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f473a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address) [2014-10-27 17:54:59.321152] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:3de423ac-8d98-434a-b3fb-93d1ed76ecb0>. holes=1 overlaps=0 missing=5 down=0 misc=0 [2014-10-27 17:54:59.321220] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 3de423ac-8d98-434a-b3fb-93d1ed76ecb0 [2014-10-27 17:54:59.321246] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f573a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address) [2014-10-27 17:54:59.323104] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 1d770502-8406-411e-ac41-80e1442cb37a [2014-10-27 17:54:59.323156] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f673a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address) [2014-10-27 17:54:59.324780] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:3de423ac-8d98-434a-b3fb-93d1ed76ecb0>. holes=1 overlaps=0 missing=5 down=0 misc=0 [2014-10-27 17:54:59.324857] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 3de423ac-8d98-434a-b3fb-93d1ed76ecb0 [2014-10-27 17:54:59.324881] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f773a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address) [2014-10-27 17:54:59.326409] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:d42016a9-6f5e-4ee3-8a2e-92c922eb016f>. holes=1 overlaps=0 missing=2 down=0 misc=0 [2014-10-27 17:54:59.326473] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : d42016a9-6f5e-4ee3-8a2e-92c922eb016f [2014-10-27 17:54:59.326498] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f873a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address) [2014-10-27 17:54:59.328262] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:3e650fd4-bbc2-4cd6-8372-b50ebbc0772e>. holes=1 overlaps=0 missing=2 down=0 misc=0 [2014-10-27 17:54:59.328351] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 3e650fd4-bbc2-4cd6-8372-b50ebbc0772e [2014-10-27 17:54:59.328377] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f973a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address)
Cloning to 3.1. To be fixed in future release.