Bug 1545570

Summary: DHT calls dht_lookup_everywhere for 1xn volumes
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Nithya Balachandran <nbalacha>
Component: distributeAssignee: Nithya Balachandran <nbalacha>
Status: CLOSED ERRATA QA Contact: Prasad Desala <tdesala>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rhgs-3.4CC: rhinduja, rhs-bugs, storage-qa-internal
Target Milestone: ---   
Target Release: RHGS 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.12.2-5 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1546620 (view as bug list) Environment:
Last Closed: 2018-09-04 06:42:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1546620, 1548270, 1548271    
Bug Blocks: 1503137    

Description Nithya Balachandran 2018-02-15 09:58:23 UTC
Description of problem:

DHT lookup does not handle lookups on 1xn volumes optimally. It calls a dht_lookup_everywhere even though it has already checked the only subvol leading to repeated, redundant lookups for entries.

Version-Release number of selected component (if applicable):


How reproducible:
Consistently

Steps to Reproduce:
1. Create a 1x3 volume and set client-log-level to DEBUG
2. Fuse mount the volume and create some files (touch file-{1..10})
3. Check the mnt log for dht messages

Actual results:
[2018-02-15 09:53:04.799711] D [MSGID: 0] [dht-common.c:2761:dht_lookup] 0-gitmo-dht: Calling fresh lookup for /file-10 on gitmo-replicate-0
[2018-02-15 09:53:04.800179] D [MSGID: 0] [dht-common.c:2344:dht_lookup_cbk] 0-gitmo-dht: fresh_lookup returned for /file-10 with op_ret -1 [No such file or directory]
[2018-02-15 09:53:04.800188] D [MSGID: 0] [dht-common.c:2357:dht_lookup_cbk] 0-gitmo-dht: Entry /file-10 missing on subvol gitmo-replicate-0
[2018-02-15 09:53:04.800196] D [MSGID: 0] [dht-common.c:2128:dht_lookup_everywhere] 0-gitmo-dht: winding lookup call to 1 subvols
[2018-02-15 09:53:04.800670] D [MSGID: 0] [dht-common.c:1930:dht_lookup_everywhere_cbk] 0-gitmo-dht: returned with op_ret -1 and op_errno 2 (/file-10) from subvol gitmo-replicate-0
[2018-02-15 09:53:04.800679] D [MSGID: 0] [dht-common.c:1594:dht_lookup_everywhere_done] 0-gitmo-dht: STATUS: hashed_subvol gitmo-replicate-0 cached_subvol null
[2018-02-15 09:53:04.800684] D [MSGID: 0] [dht-common.c:1655:dht_lookup_everywhere_done] 0-gitmo-dht: There was no cached file and  unlink on hashed is not skipped /file-10
[2018-02-15 09:53:04.800691] D [MSGID: 0] [dht-common.c:1658:dht_lookup_everywhere_done] 0-stack-trace: stack-address: 0x7f03e4000d50, gitmo-dht returned -1 error: No such file or directory [No such file or directory]


Expected results:

DHT need not call dht_lookup_everywhere for 1xn volumes

Additional info:

Comment 5 Nithya Balachandran 2018-02-23 09:24:15 UTC
This patch is very specific to lookups on files that do not exist (data or linkto). It does not handle directories or linkto files.

Comment 7 Prasad Desala 2018-04-16 10:39:31 UTC
Verified this BZ on glusterfs version 3.12.2-7.el7rhgs.x86_64. 
Now, dht_lookup_everywhere is not called for 1xn volumes.

[2018-04-16 10:25:11.829200] D [MSGID: 0] [dht-common.c:3419:dht_lookup] 0-1x3-dht: Calling fresh lookup for /file-10 on 1x3-replicate-0
[2018-04-16 10:25:11.831602] D [MSGID: 0] [dht-common.c:2972:dht_lookup_cbk] 0-1x3-dht: fresh_lookup returned for /file-10 with op_ret -1 [No such file or directory]
[2018-04-16 10:25:11.831918] D [fuse-resolve.c:61:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/file-10: failed to resolve (No such file or directory)
[2018-04-16 10:25:11.832104] D [MSGID: 0] [dht-common.c:3419:dht_lookup] 0-1x3-dht: Calling fresh lookup for /file-10 on 1x3-replicate-0
[2018-04-16 10:25:11.834616] D [MSGID: 0] [dht-common.c:2972:dht_lookup_cbk] 0-1x3-dht: fresh_lookup returned for /file-10 with op_ret -1 [No such file or directory]
[2018-04-16 10:25:11.835342] D [MSGID: 0] [dht-common.c:3419:dht_lookup] 0-1x3-dht: Calling fresh lookup for /file-10 on 1x3-replicate-0
[2018-04-16 10:25:11.837554] D [MSGID: 0] [dht-common.c:2972:dht_lookup_cbk] 0-1x3-dht: fresh_lookup returned for /file-10 with op_ret -1 [No such file or directory]
[2018-04-16 10:25:11.837859] D [fuse-resolve.c:61:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/file-10: failed to resolve (No such file or directory)
[2018-04-16 10:25:11.837961] D [MSGID: 0] [dht-common.c:8074:dht_create_wind_to_avail_subvol] 0-1x3-dht: creating /file-10 on 1x3-replicate-0
[2018-04-16 10:25:11.854286] D [fuse-bridge.c:1359:fuse_removexattr_cbk] 0-glusterfs-fuse: 87: REMOVEXATTR() of security.ima on /file-10 => -1 (No data available)

Moving this BZ to Verified.

Comment 9 errata-xmlrpc 2018-09-04 06:42:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607