Bug 1282319

Summary: DHT : for many operation directory/file path is '(null)' in brick log
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Sakshi <sabansal>
Component: coreAssignee: Vijay Bellur <vbellur>
Status: CLOSED WONTFIX QA Contact: Rahul Hinduja <rhinduja>
Severity: medium Docs Contact:
Priority: low    
Version: rhgs-3.1CC: atumball, mzywusko, nbalacha, racpatel, rhs-bugs, rwheeler, smohan, storage-qa-internal, tdesala, vbellur
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: 3.7.9-10 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1031016
: 1456361 (view as bug list) Environment:
Last Closed: 2018-04-16 03:05:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1031016    
Bug Blocks: 1456361    

Description Sakshi 2015-11-16 05:39:01 UTC
+++ This bug was initially created as a clone of Bug #1031016 +++

Description of problem:
In case of DHT volume, brick logs has '(null)' as a path for operations like OPENDIR, MKDIR, MKNOD, SETXATTR

e.g
log snippet from brick

[2013-11-14 01:52:01.592994] I [server-rpc-fops.c:705:server_opendir_cbk] 0-dht-server: 61510: OPENDIR (null) (16bb3787-361d-4977-8c66-
a62b8d2f70a3) ==> (No such file or directory)
[2013-11-14 03:14:56.011608] I [server-rpc-fops.c:528:server_mkdir_cbk] 0-dht-server: 32254: MKDIR (null) (00000000-0000-0000-0000-0000
00000001/{1.1000000}) ==> (File exists)
[2013-11-14 03:18:18.304290] I [server-rpc-fops.c:438:server_access_cbk] 0-dht-server: 47194: ACCESS (null) (2d0887f8-358d-4758-949f-db
bd06d67265) ==> (No such file or directory)
[2013-11-14 03:18:18.317609] I [server-rpc-fops.c:705:server_opendir_cbk] 0-dht-server: 47197: OPENDIR (null) (2d0887f8-358d-4758-949f-
dbbd06d67265) ==> (No such file or directory)
...
[2013-11-14 06:32:14.172144] I [server-rpc-fops.c:898:_gf_server_log_setxattr_failure] 0-dht-server: 164153: SETXATTR (null) (85258d5e-
fde6-4452-a4cf-71ef0568129a) ==> trusted.glusterfs.dht
[2013-11-14 06:32:14.174551] I [server-rpc-fops.c:528:server_mkdir_cbk] 0-dht-server: 285540: MKDIR (null) (00000000-0000-0000-0000-000
000000001/118) ==> (File exists)
[2013-11-14 06:32:14.202874] I [server-rpc-fops.c:575:server_mknod_cbk] 0-dht-server: 285551: MKNOD (null) (bf070eaf-2822-40b7-ab44-24d
91926db45/f7) ==> (File exists)

In most of the cases reason  is '(File exists)' or ' (No such file or directory)'


Version-Release number of selected component (if applicable):
============================================================= 
3.4.0.44rhs-1.el6rhs.x86_64



How reproducible:
always


Steps to Reproduce:
1.create DHT volume and mout it using FUSE and NFS
2. create, delete files/Directories from both mount point simultaneously.
3. Keep checking logs.

Actual results:
for many operation directory/file path is  '(null)' in brick  log 

Expected results:
It should show path of that Dir/file

Additional info:

--- Additional comment from Rachana Patel on 2013-11-15 08:34:31 EST ---

sosreport @ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1030309

volume info:-
[root@7-VM3 glusterfs]# gluster v info dht
 
Volume Name: dht
Type: Distribute
Volume ID: 29b35eed-9b1a-443b-914a-e77d9452eb47
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: 10.70.36.132:/rhs/brick4/d1
Brick2: 10.70.36.132:/rhs/brick4/d2
Brick3: 10.70.36.132:/rhs/brick4/d3
Brick4: 10.70.36.132:/rhs/brick4/d4



mount info:-

[root@rhs-client22 dht-nfs]# mount | grep dht
10.70.36.132:/dht on /mnt/dht type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
10.70.36.132:/dht on /mnt/dht-nfs type nfs (rw,addr=10.70.36.132)


--- Additional comment from Amar Tumballi on 2013-12-02 05:09:24 EST ---

(null) in here is the 'path' of the file. Yes it could have not been printed at all as a proper fix.

Above there are two type of operations:

inode ops (Access, opendir etc), where there is a log of 'gfid' which can help people to debug. (Also, these calls may be also path less, gfid only fops)..

in entry ops: (mkdir and mknod), the gfid of parent, and basename is logged to help debugging.

Comment 2 Susant Kumar Palai 2015-11-27 12:21:58 UTC
*** Bug 1031016 has been marked as a duplicate of this bug. ***

Comment 6 Nithya Balachandran 2016-09-21 04:52:17 UTC
Most of these messages are now logged only in case of errors.

Comment 7 Prasad Desala 2016-09-21 06:17:55 UTC
Tested this BZ with glusterfs version 3.7.9-12.el7rhgs.x86_64. 

Brick logs has '(null)' as path for ACCESS, STAT, LOOKUP.

Below are brick logs output snippets:
=====================================
[2016-09-21 05:05:41.320200] I [MSGID: 115055] [server-rpc-fops.c:423:server_access_cbk] 0-distrep-server: 1421998: ACCESS (null) (2bb72069-3217-4c38-b660-abc90b40f118) ==> (Stale file handle) [Stale file handle]
rpc-fops.c:1748:server_stat_cbk] 0-distrep-server: 1405862: STAT (null) (5c449912-1199-48c1-8fbc-996ec6a2962a) ==> (Stale file handle) [Stale file handle]
[2016-09-21 05:03:34.570429] D [MSGID: 115050] [server-rpc-fops.c:179:server_lookup_cbk] 0-distrep-server: 1131709: LOOKUP /linux-4.6.4/Documentation/RCU/rculist_nulls.txt (fa65c27b-d7db-41ab-8459-6b7c827dd78e/rculist_nulls.txt) ==> (No such file or directory) [No such file or directory]

Moving this bug to ASSIGNED.

Comment 11 Amar Tumballi 2018-04-16 03:05:59 UTC
This is expected to be null in some calls as there can be anon-fd (ie, open on gfid) type of operation. Closing as WONTFIX, feel free to open this if found required.