Bug 1027174

+++ This bug was initially created as a clone of Bug #1007033 +++

Description of problem:

The below log message filled 99% of the /var/log/samba/glusterfs-dis-rep3.log file for a particular fops i.e. remove file and recreate the same file.

[2013-09-11 18:19:07.521255] W [client-rpc-fops.c:1170:client3_3_fgetxattr_cbk] 9-dis-rep3-client-5: remote operation failed: No data available
[2013-09-11 18:19:07.521724] W [client-rpc-fops.c:1170:client3_3_fgetxattr_cbk] 9-dis-rep3-client-4: remote operation failed: No data available
[2013-09-11 18:19:07.636170] W [client-rpc-fops.c:1170:client3_3_fgetxattr_cbk] 0-dis-rep3-client-11: remote operation failed: No data available
[2013-09-11 18:19:07.636739] W [client-rpc-fops.c:1170:client3_3_fgetxattr_cbk] 0-dis-rep3-client-10: remote operation failed: No data available
[2013-09-11 18:19:08.059362] W [client-rpc-fops.c:1170:client3_3_fgetxattr_cbk] 0-dis-rep3-client-7: remote operation failed: No data available
[2013-09-11 18:19:08.059707] W [client-rpc-fops.c:1170:client3_3_fgetxattr_cbk] 0-dis-rep3-client-6: remote operation failed: No data available
[2013-09-11 18:19:08.286463] W [client-rpc-fops.c:1170:client3_3_fgetxattr_cbk] 9-dis-rep3-client-5: remote operation failed: No data available
[2013-09-11 18:19:08.286937] W [client-rpc-fops.c:1170:client3_3_fgetxattr_cbk] 9-dis-rep3-client-4: remote operation failed: No data available
[2013-09-11 18:19:08.334356] W [client-rpc-fops.c:1170:client3_3_fgetxattr_cbk] 9-dis-rep3-client-5: remote operation failed: No data available
[2013-09-11 18:19:08.334903] W [client-rpc-fops.c:1170:client3_3_fgetxattr_cbk] 9-dis-rep3-client-4: remote operation failed: No data available


[root@dhcp159-136 samba]# cat glusterfs-dis-rep3.log | wc -l
314820
[root@dhcp159-136 samba]# cat glusterfs-dis-rep3.log | grep 'remote operation failed' | wc -l
308636

How reproducible:

Always

Steps to Reproduce:
1. Mount a gluster samba share on Windows client
2. Run a script to do below steps, in an infinite loop
   a. create a file
   b. write in to the file
   c. delete the file
   d. go to step a

3. Check the /var/log/samba/gluster-<volume name>.log in the samba server

Actual results:

The log file is continuously filled with the mentioned log message.

Expected results:

We should not log same messages so much , else it will fill up the /var/log/samba directory

Additional info:

We can see from the log message the message is coming from client3_3_fgetxattr_cbk function present in xlators/protocol/client/src/client-rpc-fops.c

Though this means it is not a bug from gluster-smb. But it seems only samba users are getting effected by this as the log file in /var/log/samba is getting filled by it. So kept the component as gluster-smb

--- Additional comment from Jose A. Rivera on 2013-10-18 15:13:23 EDT ---

We have almost enough information to do a proper analysis on this. We need a network capture (Wireshark-compatible) of the traffic that is associated with this error.

--- Additional comment from Krutika Dhananjay on 2013-10-31 04:48:20 EDT ---

Posting my observations on this issue:

1. These log messages are seen when stat-prefetch on the volume is turned off.

2. The log messages mentioned in comment #1 are due to getxattr failures on 'system.posix_acl_access' on the filename after it is created.

3. Also, one must note that there are 4 (innocuous) warning messages in the brick logfiles for every file that is created. This happens because of the get_real_filename requests sent by samba BEFORE creating the file, leading to a getxattr failure with ENOENT on the bricks in posix xlator.

4. Also present in the client logs are messages of the kind '[2013-11-05 15:18:23.847545] E [afr-common.c:2109:afr_lookup_done] 1-dis-rep-replicate-1: /..: No gfid present' for every file that is created.