Bug 1188064 - log files get flooded when removexattr() can't find a specified key or value
Summary: log files get flooded when removexattr() can't find a specified key or value
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: logging
Version: 3.6.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: bugs@gluster.org
QA Contact:
URL:
Whiteboard:
Depends On: 1144527 1192832 1205715
Blocks: glusterfs-3.6.3
TreeView+ depends on / blocked
 
Reported: 2015-02-01 22:59 UTC by Vijay Bellur
Modified: 2016-02-04 15:21 UTC (History)
5 users (show)

Fixed In Version: glusterfs-v3.6.3
Doc Type: Bug Fix
Doc Text:
Clone Of: 1144527
Environment:
Last Closed: 2016-02-04 15:21:33 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)

Description Vijay Bellur 2015-02-01 22:59:24 UTC
+++ This bug was initially created as a clone of Bug #1144527 +++

Description of problem:

I'm running 
#: glusterfs -V
#: glusterfs 3.4.5 built on Aug  6 2014 19:15:07
on ubuntu 14.04 from the semiosis ppa

I have a replica 2 with 2 servers.
another client does a fuse mount of a volume.
On rsyncing a bit of data onto the fuse mount,
 I get an entry like the below one on the client - for each file that is copied onto the volume

[2014-09-19 07:57:39.877806] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-0: remote operation failed: No data available
[2014-09-19 07:57:39.877963] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-1: remote operation failed: No data available
[2014-09-19 07:57:39.878462] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 21741144: REMOVEXATTR() /<path_to_file>/.<file_name_with_a_leading_dot> => -1 (No data available)

The data itself is present and accessible on the volume and on both bricks.

So three questions:
a.) what kind of data is not available? what is the client complaining about?
b.) since it is a warning and the data seems to be okay - is there anything I need to fix?
c.) How can I get rid of the amount of log lines? it's more than 3GB/day..


Version-Release number of selected component (if applicable):

glusterfs 3.4.5 built on Aug  6 2014 19:15:07

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
an - so far unknown - xattribute key or value that should be removed couldn't be found for a specific file


Expected results:
As it is a warning message and the data SEEMS to be okay 
logging of this event should be reduced by a meaningful algory like:
if message density bigger than X;print short version.

Additional info:
bricks are running on zol, not xfs

--- Additional comment from Anand Avati on 2014-09-19 12:45:12 EDT ---

REVIEW: http://review.gluster.org/8781 (protocol: Log ENODATA & ENOATTR logs at DEBUG loglevel in removexattr_cbk.) posted (#4) for review on master by Vijay Bellur (vbellur@redhat.com)

--- Additional comment from Anand Avati on 2014-09-24 03:49:03 EDT ---

COMMIT: http://review.gluster.org/8781 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit bd592f8b8379087604f35c3b377f6e94b9e1697d
Author: Vijay Bellur <vbellur@redhat.com>
Date:   Fri Sep 19 19:08:05 2014 +0530

    protocol: Log ENODATA & ENOATTR logs at DEBUG loglevel in removexattr_cbk.
    
    Prevents messages of the following type from being seen by default in the log files:
    
    [2014-09-19 07:57:39.877806] W
    [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-0:
    remote operation failed: No data available
    [2014-09-19 07:57:39.877963] W
    [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-1:
    remote operation failed: No data available
    
    Change-Id: I3b1a121b0fc272eb772547275bb8085ed19db5a1
    BUG: 1144527
    Signed-off-by: Vijay Bellur <vbellur@redhat.com>
    Reviewed-on: http://review.gluster.org/8781
    Reviewed-by: Niels de Vos <ndevos@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Jeff Darcy <jdarcy@redhat.com>

Comment 1 Anand Avati 2015-02-01 23:01:18 UTC
REVIEW: http://review.gluster.org/9524 (protocol: Log ENODATA & ENOATTR logs at DEBUG loglevel in removexattr_cbk.) posted (#1) for review on release-3.6 by Vijay Bellur (vbellur@redhat.com)

Comment 2 Anand Avati 2015-02-25 07:38:03 UTC
COMMIT: http://review.gluster.org/9524 committed in release-3.6 by Raghavendra Bhat (raghavendra@redhat.com) 
------
commit 642355b50e792aa7b309122c871731ad887a97c0
Author: Vijay Bellur <vbellur@redhat.com>
Date:   Fri Sep 19 19:08:05 2014 +0530

    protocol: Log ENODATA & ENOATTR logs at DEBUG loglevel in removexattr_cbk.
    
    Prevents messages of the following type from being seen by default in the log files:
    
    [2014-09-19 07:57:39.877806] W
    [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-0:
    remote operation failed: No data available
    [2014-09-19 07:57:39.877963] W
    [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-1:
    remote operation failed: No data available
    
    Change-Id: I3b1a121b0fc272eb772547275bb8085ed19db5a1
    BUG: 1188064
    Signed-off-by: Vijay Bellur <vbellur@redhat.com>
    Reviewed-on: http://review.gluster.org/8781
    Reviewed-by: Niels de Vos <ndevos@redhat.com>
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
    Reviewed-on: http://review.gluster.org/9524
    Reviewed-by: Raghavendra Bhat <raghavendra@redhat.com>

Comment 3 André Bauer 2015-07-16 13:52:15 UTC
Will this be backported also to 3.5.x?

Have the same problem on Ubuntu 14.04, Glusterfs 3.5.5 (Launchpad Ubuntu Packages) and EXT4 bricks...

Comment 4 Niels de Vos 2015-07-16 15:45:57 UTC
(In reply to André Bauer from comment #3)
> Will this be backported also to 3.5.x?
> 
> Have the same problem on Ubuntu 14.04, Glusterfs 3.5.5 (Launchpad Ubuntu
> Packages) and EXT4 bricks...

Please open a new bug for 3.5.x and attach the logs with the exact messages. Bug 1192832 was used for backporting and the fix should be in 3.5.4 already. If you are seeing similar messages, we can address that.

Comment 5 Kaushal 2016-02-04 15:21:33 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-v3.6.3, please open a new bug report.

glusterfs-v3.6.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://www.gluster.org/pipermail/gluster-users/2015-April/021669.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.