Bug 1212172

Summary: excessive logging for glusterfs mountpoint
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Ben England <bengland>
Component: fuseAssignee: Nithya Balachandran <nbalacha>
Status: CLOSED CURRENTRELEASE QA Contact: storage-qa-internal <storage-qa-internal>
Severity: high Docs Contact:
Priority: high    
Version: rhgs-3.0CC: chrisw, csaba, mpillai, nbalacha, ndokos, nlevinki, pportant, sankarshan
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: dht-try-latest-build
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-09 10:19:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1090220    
Bug Blocks:    

Description Ben England 2015-04-15 17:32:57 UTC 
Description of problem:

For small-file workloads such as untarring this file:

http://perf1.perf.lab.eng.bos.redhat.com/bengland/tmp/example-pbench-tarball.tar.xz

The logfile for the Glusterfs FUSE mountpoint process grows at an alarming rate, 40 KB/sec = 3 GB/day.  See bottom of this post for the messages.  The application, tar xJf, reports no errors.

background: the Red Hat perf team is using Gluster to store its performance data and has encountered this situation because much of this data is "small-file" (example: sosreport data, /proc fs, etc).  

Version-Release number of selected component (if applicable):

server:
glusterfs-3.6.0.53-1.el6rhs.x86_64 (RHS 3.0.4)
kernel-2.6.32-504.3.3.el6.x86_64 (RHEL6.6)

client:
glusterfs-3.6.0.53-1.el7rhs.x86_64 (RHS 3.0.4)
kernel-3.10.0-229.el7.x86_64 (RHEL7.1)

How reproducible:

every time

Steps to Reproduce:
1. cd /your/mountpoint ; tar xJf example.tar.xz &
2. tail -f /var/log/glusterfs/your-mountpoint.log

Actual results:

see above

Expected results:

No logging of messages that do not correspond to real problems in Gluster, unless severity level is raised to INFO or DEBUG.  

The error ENODATA = "No data available" is returned when getxattr() system call does not find an xattr with the specified name associated with the specified file.  In general, the absence of an xattr is not in itself an error, unless Gluster has some reason to believe that it should have been present in the first place.

Additional info:

[2015-04-15 17:10:58.566443] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 4788006: REMOVEXATTR() /ben/uperf_baseline_no_tuned_2015-04-09_13:24:01/2-tcp_stream-1B-8i/sample1/tools-default/dell-r620-01/proc-interrupts/csv/proc-interrupts-by-irq_IRQ_142_IR-PCI-MSI-edge_em2-TxRx-0.csv => -1 (No data available)
[2015-04-15 17:10:58.569275] W [client-rpc-fops.c:1213:client3_3_removexattr_cbk] 0-pbench-client-12: remote operation failed: No data available
[2015-04-15 17:10:58.569325] W [client-rpc-fops.c:1213:client3_3_removexattr_cbk] 0-pbench-client-13: remote operation failed: No data available
[2015-04-15 17:10:58.569327] W [client-rpc-fops.c:1213:client3_3_removexattr_cbk] 0-pbench-client-14: remote operation failed: No data available
[2015-04-15 17:10:58.570311] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 4788009: REMOVEXATTR() /ben/uperf_baseline_no_tuned_2015-04-09_13:24:01/2-tcp_stream-1B-8i/sample1/tools-default/dell-r620-01/proc-interrupts/csv/proc-interrupts-by-irq_IRQ_142_IR-PCI-MSI-edge_em2-TxRx-0.csv => -1 (No data available)
[2015-04-15 17:10:58.572517] W [client-rpc-fops.c:1213:client3_3_removexattr_cbk] 0-pbench-client-12: remote operation failed: No data available
[2015-04-15 17:10:58.572564] W [client-rpc-fops.c:1213:client3_3_removexattr_cbk] 0-pbench-client-14: remote operation failed: No data available
[2015-04-15 17:10:58.572584] W [client-rpc-fops.c:1213:client3_3_removexattr_cbk] 0-pbench-client-13: remote operation failed: No data available
[