Bug 1212172 - excessive logging for glusterfs mountpoint
Summary: excessive logging for glusterfs mountpoint
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: fuse
Version: rhgs-3.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Nithya Balachandran
QA Contact: storage-qa-internal@redhat.com
URL:
Whiteboard: dht-try-latest-build
Depends On: 1090220
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-04-15 17:32 UTC by Ben England
Modified: 2016-06-09 10:19 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-06-09 10:19:38 UTC
Embargoed:


Attachments (Terms of Use)

Description Ben England 2015-04-15 17:32:57 UTC
Description of problem:

For small-file workloads such as untarring this file:

http://perf1.perf.lab.eng.bos.redhat.com/bengland/tmp/example-pbench-tarball.tar.xz

The logfile for the Glusterfs FUSE mountpoint process grows at an alarming rate, 40 KB/sec = 3 GB/day.  See bottom of this post for the messages.  The application, tar xJf, reports no errors.

background: the Red Hat perf team is using Gluster to store its performance data and has encountered this situation because much of this data is "small-file" (example: sosreport data, /proc fs, etc).  

Version-Release number of selected component (if applicable):

server:
glusterfs-3.6.0.53-1.el6rhs.x86_64 (RHS 3.0.4)
kernel-2.6.32-504.3.3.el6.x86_64 (RHEL6.6)

client:
glusterfs-3.6.0.53-1.el7rhs.x86_64 (RHS 3.0.4)
kernel-3.10.0-229.el7.x86_64 (RHEL7.1)

How reproducible:

every time

Steps to Reproduce:
1. cd /your/mountpoint ; tar xJf example.tar.xz &
2. tail -f /var/log/glusterfs/your-mountpoint.log

Actual results:

see above

Expected results:

No logging of messages that do not correspond to real problems in Gluster, unless severity level is raised to INFO or DEBUG.  

The error ENODATA = "No data available" is returned when getxattr() system call does not find an xattr with the specified name associated with the specified file.  In general, the absence of an xattr is not in itself an error, unless Gluster has some reason to believe that it should have been present in the first place.

Additional info:

[2015-04-15 17:10:58.566443] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 4788006: REMOVEXATTR() /ben/uperf_baseline_no_tuned_2015-04-09_13:24:01/2-tcp_stream-1B-8i/sample1/tools-default/dell-r620-01/proc-interrupts/csv/proc-interrupts-by-irq_IRQ_142_IR-PCI-MSI-edge_em2-TxRx-0.csv => -1 (No data available)
[2015-04-15 17:10:58.569275] W [client-rpc-fops.c:1213:client3_3_removexattr_cbk] 0-pbench-client-12: remote operation failed: No data available
[2015-04-15 17:10:58.569325] W [client-rpc-fops.c:1213:client3_3_removexattr_cbk] 0-pbench-client-13: remote operation failed: No data available
[2015-04-15 17:10:58.569327] W [client-rpc-fops.c:1213:client3_3_removexattr_cbk] 0-pbench-client-14: remote operation failed: No data available
[2015-04-15 17:10:58.570311] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 4788009: REMOVEXATTR() /ben/uperf_baseline_no_tuned_2015-04-09_13:24:01/2-tcp_stream-1B-8i/sample1/tools-default/dell-r620-01/proc-interrupts/csv/proc-interrupts-by-irq_IRQ_142_IR-PCI-MSI-edge_em2-TxRx-0.csv => -1 (No data available)
[2015-04-15 17:10:58.572517] W [client-rpc-fops.c:1213:client3_3_removexattr_cbk] 0-pbench-client-12: remote operation failed: No data available
[2015-04-15 17:10:58.572564] W [client-rpc-fops.c:1213:client3_3_removexattr_cbk] 0-pbench-client-14: remote operation failed: No data available
[2015-04-15 17:10:58.572584] W [client-rpc-fops.c:1213:client3_3_removexattr_cbk] 0-pbench-client-13: remote operation failed: No data available
[


Note You need to log in before you can comment on or make changes to this bug.