Description of problem: Presently, brick logs are throwing "E" messages while "ls -R" is executed from client basically from nfs mount-point The volume is Version-Release number of selected component (if applicable): glusterfs-3.6.0.28-1.el6rhs.x86_64 How reproducible: seen for this time Steps to Reproduce: 1. create a volume for 6x2 type, start it 2. enable and set quota on the volume, lets say 10GB for "/" and 5GB for some dir 3. mount the volume over nfs 4. start creating data inside dir having 5GB limit, till limit is reached 5. add-brick and start rebalance(force) 6. start creating data inside another dir, till "/" limit is reached. 7. while step 5 and step 6 are going on, start "ls -R" on same mount-point 8. check brick logs 9. even if the data creation is finished, a subsequent "ls -R" still throws the same error messages. Actual results: bricks logs report, [2014-09-05 00:10:44.619073] E [inode.c:1135:__inode_path] (-->/usr/lib64/glusterfs/3.6.0.28/xlator/features/marker.so(marker_readdirp_cbk+0x1e1) [0x7f79ed743031] (-->/usr/lib64/glusterfs/3.6.0.28/xlator/features/marker.so(marker_inode_loc_fill+0x7a) [0x7f79ed742e0a] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f79fcdeb8ea]))) 0-: Assertion failed: 0 [2014-09-05 00:10:44.619120] W [inode.c:1136:__inode_path] (-->/usr/lib64/glusterfs/3.6.0.28/xlator/features/marker.so(marker_readdirp_cbk+0x1e1) [0x7f79ed743031] (-->/usr/lib64/glusterfs/3.6.0.28/xlator/features/marker.so(marker_inode_loc_fill+0x7a) [0x7f79ed742e0a] (-->/usr/lib64/libglusterfs.so.0(inode_path+0x4a) [0x7f79fcdeb8ea]))) 0-dist-rep-marker: invalid inode [2014-09-05 00:10:44.619130] W [marker.c:2752:marker_readdirp_cbk] 0-dist-rep-marker: Couln't build loc for 61498ac2-f68b-4e18-aea6-8e5051ff682e/file_310 Expected results: No Error msg is expected Additional info:
Issue is even when rebalance has completed and we are just executing "ls -R" from the mount-point
RCA: ==== In marker_readdirp_cbk, we try to construct path from entry->inode. However, entry->inode might be a new-inode not linked yet. In this case, inode_path fails with the error msg shown in logs. To fix this completely, we should be building ancestry for the entry->inode. However, the inode on which readdirp is happening (the parent directory) itself might've been looked up through nameless lookup and hence ancestry might not be present in inode-table. Impact: ======= Accounting of all of the parent directories might be incorrect as mq_xattr_state basically does healing of accounting that might be gone bad. Healing is also done during lookup, but in case of NFS we might never get a named-lookup on the dentry. FIX: ==== The complete fix should construct the ancestry for each of the dentries. Ancestry building code is already present in quota-enforcement xlator (quota.c). That needs to be borrowed into marker.
*** Bug 1173086 has been marked as a duplicate of this bug. ***
*** Bug 1175116 has been marked as a duplicate of this bug. ***
Bug# 1189792 and Bug# 1138646 are same. Bug# 1189792 has customer escalation attached, so closing bug# 1138646 *** This bug has been marked as a duplicate of bug 1189792 ***