Bug 1157465 - DHT + Rebalance + rename +NFS mount :- after multiple rename and rebalance , ls -l on parents shows '?' for few file attributes
Summary: DHT + Rebalance + rename +NFS mount :- after multiple rename and rebalance ,...
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: distribute
Version: 2.1
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Susant Kumar Palai
QA Contact: storage-qa-internal@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1286096
TreeView+ depends on / blocked
 
Reported: 2014-10-27 10:09 UTC by Rachana Patel
Modified: 2015-11-27 10:46 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1286096 (view as bug list)
Environment:
Last Closed: 2015-11-27 10:46:26 UTC
Embargoed:


Attachments (Terms of Use)

Description Rachana Patel 2014-10-27 10:09:29 UTC
Description of problem:
=======================
after multiple rename and rebalnce, output of ls -l on parents has '?' in attribute field 
-????????? ? ?    ?           ?            ? zero9-112
-rw-r--r-- 3 root root        0 Oct 17 14:22 zero9-200
-????????? ? ?    ?           ?            ? zero9-201


version:-
========
3.4.0.69rhs-1.el6rhs.x86_64


How reproducible:
=================
always


Steps to Reproduce:
===================
1. create, start and mount distributed volume. 
2. create few files inside it
3. start renaming files from multiple mount and start rebalance also
4. repeat step 3 for few times
5. verify all files.



Actual results:
==============

NFS mount:-
for few files lookup on parents shows that files attributes are not listed

-????????? ? ?    ?           ?            ? zero1-110
-????????? ? ?    ?           ?            ? zero11-116
-rw-r--r-- 9 root root        0 Oct 17 14:22 zero11-149
-????????? ? ?    ?           ?            ? zero11-201
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero12-200
-rw-r--r-- 7 root root        0 Oct 17 14:22 zero12-201
-rw-r--r-- 2 root root 20971520 Oct 17 14:24 zero13-200
-rw-r--r-- 9 root root        0 Oct 17 14:22 zero13-201
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero14-201
-rw-r--r-- 9 root root        0 Oct 17 14:22 zero15-127
-rw-r--r-- 7 root root        0 Oct 17 14:22 zero16-168
-rw-r--r-- 9 root root        0 Oct 17 14:22 zero16-201
-rw-r--r-- 2 root root        0 Oct 17 14:22 zero16-67
-rw-r--r-- 2 root root 20971520 Oct 17 14:21 zero17-166
-rw-r--r-- 3 root root        0 Oct 17 14:22 zero17-201
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero18-200
-rw-r--r-- 9 root root        0 Oct 17 14:22 zero19-127
-rw-r--r-- 3 root root        0 Oct 17 14:22 zero19-200
-rw-r--r-- 2 root root        0 Oct 17 14:22 zero19-201
-rw-r--r-- 2 root root        0 Oct 17 14:22 zero19-41
-????????? ? ?    ?           ?            ? zero20-101
-rw-r--r-- 9 root root        0 Oct 17 14:22 zero20-201
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero21-1
-rw-r--r-- 5 root root 20971520 Oct 17 14:21 zero2-154
-rw-r--r-- 7 root root        0 Oct 17 14:22 zero2-169
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero22-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero23-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero24-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero25-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero26-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero27-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero28-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero29-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero30-1
-????????? ? ?    ?           ?            ? zero3-101
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero31-1
-????????? ? ?    ?           ?            ? zero3-110
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero32-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero33-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero34-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero35-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero36-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero37-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero38-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero39-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero40-1
-rw-r--r-- 4 root root        0 Oct 17 14:22 zero4-201
-rw-r--r-- 3 root root        0 Oct 17 14:22 zero5-201
-rw-r--r-- 3 root root        0 Oct 17 14:22 zero6-201
-rw-r--r-- 4 root root        0 Oct 17 14:22 zero7-201
-rw-r--r-- 5 root root 20971520 Oct 17 14:21 zero8-140
-rw-r--r-- 0 root root        0 Oct 17 14:22 zero8-201
-????????? ? ?    ?           ?            ? zero9-112
-rw-r--r-- 3 root root        0 Oct 17 14:22 zero9-200
-????????? ? ?    ?           ?            ? zero9-201


FUSE mount:-
-rw-r--r-- 5 root root 20971520 Oct 17 14:21 zero2-154
-rw-r--r-- 7 root root        0 Oct 17 14:22 zero2-169
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero22-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero23-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero24-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero25-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero26-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero27-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero28-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero29-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero30-1
-rw-r--r-- 4 root root        0 Oct 17 14:22 zero3-101
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero31-1
-rw-r--r-- 4 root root        0 Oct 17 14:22 zero3-110
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero32-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero33-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero34-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero35-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero36-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero37-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero38-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero39-1
-rw-r--r-- 1 root root        0 Oct 17 14:22 zero40-1
-rw-r--r-- 4 root root        0 Oct 17 14:22 zero4-201
-rw-r--r-- 3 root root        0 Oct 17 14:22 zero5-201
-rw-r--r-- 3 root root        0 Oct 17 14:22 zero6-201
-rw-r--r-- 4 root root        0 Oct 17 14:22 zero7-201
-rw-r--r-- 5 root root 20971520 Oct 17 14:21 zero8-140
-rw-r--r-- 0 root root        0 Oct 17 14:22 zero8-201
-rw-r--r-- 4 root root        0 Oct 17 14:22 zero9-112
-rw-r--r-- 3 root root        0 Oct 17 14:22 zero9-200
-rw-r--r-- 4 root root        0 Oct 17 14:22 zero9-201



Expected results:
=================
NFS mount should show file attributes


Additional info:
log snippet:-

[2014-10-27 17:54:59.307808] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:3e650fd4-bbc2-4cd6-8372-b50ebbc0772e>. holes=1 overlaps=0 missing=2 down=0 misc=0
[2014-10-27 17:54:59.307932] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 3e650fd4-bbc2-4cd6-8372-b50ebbc0772e
[2014-10-27 17:54:59.307964] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: ed73a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address)
[2014-10-27 17:54:59.310734] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:d42016a9-6f5e-4ee3-8a2e-92c922eb016f>. holes=1 overlaps=0 missing=2 down=0 misc=0
[2014-10-27 17:54:59.310825] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : d42016a9-6f5e-4ee3-8a2e-92c922eb016f
[2014-10-27 17:54:59.310872] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: ef73a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address)
[2014-10-27 17:54:59.312554] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:3e650fd4-bbc2-4cd6-8372-b50ebbc0772e>. holes=1 overlaps=0 missing=2 down=0 misc=0
[2014-10-27 17:54:59.312672] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 3e650fd4-bbc2-4cd6-8372-b50ebbc0772e
[2014-10-27 17:54:59.312700] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f073a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address)
[2014-10-27 17:54:59.314103] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:3e650fd4-bbc2-4cd6-8372-b50ebbc0772e>. holes=1 overlaps=0 missing=2 down=0 misc=0
[2014-10-27 17:54:59.314188] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 3e650fd4-bbc2-4cd6-8372-b50ebbc0772e
[2014-10-27 17:54:59.314216] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f173a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address)
[2014-10-27 17:54:59.315837] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:d42016a9-6f5e-4ee3-8a2e-92c922eb016f>. holes=1 overlaps=0 missing=2 down=0 misc=0
[2014-10-27 17:54:59.315894] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : d42016a9-6f5e-4ee3-8a2e-92c922eb016f
[2014-10-27 17:54:59.315917] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f273a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address)
[2014-10-27 17:54:59.317878] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 1d770502-8406-411e-ac41-80e1442cb37a
[2014-10-27 17:54:59.317921] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f373a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address)
[2014-10-27 17:54:59.319345] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:d42016a9-6f5e-4ee3-8a2e-92c922eb016f>. holes=1 overlaps=0 missing=2 down=0 misc=0
[2014-10-27 17:54:59.319402] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : d42016a9-6f5e-4ee3-8a2e-92c922eb016f
[2014-10-27 17:54:59.319424] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f473a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address)
[2014-10-27 17:54:59.321152] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:3de423ac-8d98-434a-b3fb-93d1ed76ecb0>. holes=1 overlaps=0 missing=5 down=0 misc=0
[2014-10-27 17:54:59.321220] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 3de423ac-8d98-434a-b3fb-93d1ed76ecb0
[2014-10-27 17:54:59.321246] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f573a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address)
[2014-10-27 17:54:59.323104] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 1d770502-8406-411e-ac41-80e1442cb37a
[2014-10-27 17:54:59.323156] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f673a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address)
[2014-10-27 17:54:59.324780] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:3de423ac-8d98-434a-b3fb-93d1ed76ecb0>. holes=1 overlaps=0 missing=5 down=0 misc=0
[2014-10-27 17:54:59.324857] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 3de423ac-8d98-434a-b3fb-93d1ed76ecb0
[2014-10-27 17:54:59.324881] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f773a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address)
[2014-10-27 17:54:59.326409] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:d42016a9-6f5e-4ee3-8a2e-92c922eb016f>. holes=1 overlaps=0 missing=2 down=0 misc=0
[2014-10-27 17:54:59.326473] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : d42016a9-6f5e-4ee3-8a2e-92c922eb016f
[2014-10-27 17:54:59.326498] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f873a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address)
[2014-10-27 17:54:59.328262] I [dht-layout.c:645:dht_layout_normalize] 0-manhoos-dht: found anomalies in <gfid:3e650fd4-bbc2-4cd6-8372-b50ebbc0772e>. holes=1 overlaps=0 missing=2 down=0 misc=0
[2014-10-27 17:54:59.328351] E [nfs3.c:768:nfs3_getattr_resume] 0-nfs-nfsv3: No such file or directory: (10.70.44.64:880) manhoos : 3e650fd4-bbc2-4cd6-8372-b50ebbc0772e
[2014-10-27 17:54:59.328377] W [nfs3-helpers.c:3409:nfs3_log_common_res] 0-nfs-nfsv3: XID: f973a42a, GETATTR: NFS: 2(No such file or directory), POSIX: 14(Bad address)

Comment 4 Susant Kumar Palai 2015-11-27 10:46:26 UTC
Cloning to 3.1. To be fixed in future release.


Note You need to log in before you can comment on or make changes to this bug.