Bug 1144527 - log files get flooded when removexattr() can't find a specified key or value
Summary: log files get flooded when removexattr() can't find a specified key or value
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: logging
Version: mainline
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Vijay Bellur
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: 1188064 1192832 1205715
TreeView+ depends on / blocked
 
Reported: 2014-09-19 16:22 UTC by bernhard.glomm
Modified: 2015-07-31 07:54 UTC (History)
3 users (show)

Fixed In Version: glusterfs-3.7.0
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1188064 1192832 1205715 (view as bug list)
Environment:
Last Closed: 2015-05-14 17:27:46 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description bernhard.glomm 2014-09-19 16:22:14 UTC
Description of problem:

I'm running 
#: glusterfs -V
#: glusterfs 3.4.5 built on Aug  6 2014 19:15:07
on ubuntu 14.04 from the semiosis ppa

I have a replica 2 with 2 servers.
another client does a fuse mount of a volume.
On rsyncing a bit of data onto the fuse mount,
 I get an entry like the below one on the client - for each file that is copied onto the volume

[2014-09-19 07:57:39.877806] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-0: remote operation failed: No data available
[2014-09-19 07:57:39.877963] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-1: remote operation failed: No data available
[2014-09-19 07:57:39.878462] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 21741144: REMOVEXATTR() /<path_to_file>/.<file_name_with_a_leading_dot> => -1 (No data available)

The data itself is present and accessible on the volume and on both bricks.

So three questions:
a.) what kind of data is not available? what is the client complaining about?
b.) since it is a warning and the data seems to be okay - is there anything I need to fix?
c.) How can I get rid of the amount of log lines? it's more than 3GB/day..


Version-Release number of selected component (if applicable):

glusterfs 3.4.5 built on Aug  6 2014 19:15:07

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:
an - so far unknown - xattribute key or value that should be removed couldn't be found for a specific file


Expected results:
As it is a warning message and the data SEEMS to be okay 
logging of this event should be reduced by a meaningful algory like:
if message density bigger than X;print short version.

Additional info:
bricks are running on zol, not xfs

Comment 1 Anand Avati 2014-09-19 16:45:12 UTC
REVIEW: http://review.gluster.org/8781 (protocol: Log ENODATA & ENOATTR logs at DEBUG loglevel in removexattr_cbk.) posted (#4) for review on master by Vijay Bellur (vbellur)

Comment 2 Anand Avati 2014-09-24 07:49:03 UTC
COMMIT: http://review.gluster.org/8781 committed in master by Vijay Bellur (vbellur) 
------
commit bd592f8b8379087604f35c3b377f6e94b9e1697d
Author: Vijay Bellur <vbellur>
Date:   Fri Sep 19 19:08:05 2014 +0530

    protocol: Log ENODATA & ENOATTR logs at DEBUG loglevel in removexattr_cbk.
    
    Prevents messages of the following type from being seen by default in the log files:
    
    [2014-09-19 07:57:39.877806] W
    [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-0:
    remote operation failed: No data available
    [2014-09-19 07:57:39.877963] W
    [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-<volume_name>-client-1:
    remote operation failed: No data available
    
    Change-Id: I3b1a121b0fc272eb772547275bb8085ed19db5a1
    BUG: 1144527
    Signed-off-by: Vijay Bellur <vbellur>
    Reviewed-on: http://review.gluster.org/8781
    Reviewed-by: Niels de Vos <ndevos>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Jeff Darcy <jdarcy>

Comment 3 Niels de Vos 2015-05-14 17:27:46 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 4 Niels de Vos 2015-05-14 17:35:37 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 5 Niels de Vos 2015-05-14 17:37:58 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 6 Niels de Vos 2015-05-14 17:43:54 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 7 Ivo Petrov 2015-07-30 13:42:59 UTC
I'm on 3.7.2 - Issue is still present - here's paste from my logs:
[2015-07-30 13:41:14.413181] W [fuse-bridge.c:1263:fuse_err_cbk] 0-glusterfs-fuse: 356407119: REMOVEXATTR() /***FILE***.JPG => -1 (No data available)

Comment 8 Niels de Vos 2015-07-30 20:50:18 UTC
Hi Ivo,

the message is different, the bit between the [square brackets] is important. Could you file a new bug for the version and exact message that you get? 

  https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS&component=logging&version=3.7.2&blocked=glusterfs-3.7.4

Thanks!

Comment 9 Ivo Petrov 2015-07-31 07:54:13 UTC
Hi Niels,
created https://bugzilla.redhat.com/show_bug.cgi?id=1248941

Cheers,
Ivo


Note You need to log in before you can comment on or make changes to this bug.