Bug 1205715 - log files get flooded when removexattr() can't find a specified key or value
Summary: log files get flooded when removexattr() can't find a specified key or value
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: logging
Version: 3.4.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kaleb KEITHLEY
QA Contact:
URL:
Whiteboard:
Depends On: 1144527
Blocks: glusterfs-3.4.7 1188064 1192832
TreeView+ depends on / blocked
 
Reported: 2015-03-25 13:58 UTC by Kaleb KEITHLEY
Modified: 2015-12-01 16:45 UTC (History)
5 users (show)

Fixed In Version:
Clone Of: 1144527
Environment:
Last Closed: 2015-04-13 06:59:59 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description Kaleb KEITHLEY 2015-03-25 13:58:45 UTC
+++ This bug was initially created as a clone of Bug #1144527 +++

Description of problem:

I'm running 
#: glusterfs -V
#: glusterfs 3.4.5 built on Aug  6 2014 19:15:07
on ubuntu 14.04 from the semiosis ppa

I have a replica 2 with 2 servers.
another client does a fuse mount of a volume.
On rsyncing a bit of data onto the fuse mount,
 I get an entry like the below one on the client - for each file that is copied onto the volume

[2014-09-19 07:57:39.877806] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-0: remote operation failed: No data available
[2014-09-19 07:57:39.877963] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-1: remote operation failed: No data available
[2014-09-19 07:57:39.878462] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 21741144: REMOVEXATTR() /<path_to_file>/.<file_name_with_a_leading_dot> => -1 (No data available)

The data itself is present and accessible on the volume and on both bricks.

So three questions:
a.) what kind of data is not available? what is the client complaining about?
b.) since it is a warning and the data seems to be okay - is there anything I need to fix?
c.) How can I get rid of the amount of log lines? it's more than 3GB/day..


Version-Release number of selected component (if applicable):

glusterfs 3.4.5 built on Aug  6 2014 19:15:07

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
an - so far unknown - xattribute key or value that should be removed couldn't be found for a specific file


Expected results:
As it is a warning message and the data SEEMS to be okay 
logging of this event should be reduced by a meaningful algory like:
if message density bigger than X;print short version.

Additional info:
bricks are running on zol, not xfs

--- Additional comment from Anand Avati on 2014-09-19 12:45:12 EDT ---

REVIEW: http://review.gluster.org/8781 (protocol: Log ENODATA & ENOATTR logs at DEBUG loglevel in removexattr_cbk.) posted (#4) for review on master by Vijay Bellur (vbellur)

--- Additional comment from Anand Avati on 2014-09-24 03:49:03 EDT ---

COMMIT: http://review.gluster.org/8781 committed in master by Vijay Bellur (vbellur) 
------
commit bd592f8b8379087604f35c3b377f6e94b9e1697d
Author: Vijay Bellur <vbellur>
Date:   Fri Sep 19 19:08:05 2014 +0530

    protocol: Log ENODATA & ENOATTR logs at DEBUG loglevel in removexattr_cbk.
    
    Prevents messages of the following type from being seen by default in the log files:
    
    [2014-09-19 07:57:39.877806] W
    [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-0:
    remote operation failed: No data available
    [2014-09-19 07:57:39.877963] W
    [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-1:
    remote operation failed: No data available
    
    Change-Id: I3b1a121b0fc272eb772547275bb8085ed19db5a1
    BUG: 1144527
    Signed-off-by: Vijay Bellur <vbellur>
    Reviewed-on: http://review.gluster.org/8781
    Reviewed-by: Niels de Vos <ndevos>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Jeff Darcy <jdarcy>

Comment 1 Anand Avati 2015-03-25 14:21:12 UTC
REVIEW: http://review.gluster.org/9996 (protocol: Log ENODATA & ENOATTR logs at DEBUG loglevel in removexattr_cbk) posted (#1) for review on release-3.4 by Kaleb KEITHLEY (kkeithle)

Comment 2 Anand Avati 2015-03-25 17:13:06 UTC
COMMIT: http://review.gluster.org/9996 committed in release-3.4 by Kaleb KEITHLEY (kkeithle) 
------
commit 076143f52e47a2a733a02cfb05a7d780553b147d
Author: Kaleb S. KEITHLEY <kkeithle>
Date:   Wed Mar 25 10:18:22 2015 -0400

    protocol: Log ENODATA & ENOATTR logs at DEBUG loglevel in removexattr_cbk
    
    Prevents messages of the following type from being seen by default in
    the log files:
    
    [2014-09-19 07:57:39.877806] W
    [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-0:
    remote operation failed: No data available
    [2014-09-19 07:57:39.877963] W
    [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-1:
    remote operation failed: No data available
    
    Backport of http://review.gluster.org/#/c/8781/, see BZ 1144527
    
    Change-Id: I814e12c56f5b95fcc24a6f0d46c7240d7abd16e4
    BUG: 1205715
    Signed-off-by: Kaleb S. KEITHLEY <kkeithle>
    Reviewed-on: http://review.gluster.org/9996
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Niels de Vos <ndevos>


Note You need to log in before you can comment on or make changes to this bug.