Bug 1544451 - [GSS] log-level=ERROR mount option not working, W level messages rapidly filling up storage
Summary: [GSS] log-level=ERROR mount option not working, W level messages rapidly fill...
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: core
Version: rhgs-3.3
Hardware: All
OS: Linux
Target Milestone: ---
: RHGS 3.4.0
Assignee: Csaba Henk
QA Contact: Vinayak Papnoi
Depends On:
Blocks: 1501439 1503138
TreeView+ depends on / blocked
Reported: 2018-02-12 14:26 UTC by Pan Ousley
Modified: 2018-09-10 07:05 UTC (History)
10 users (show)

Fixed In Version: glusterfs-3.12.2-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2018-09-04 06:42:41 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3364121 None None None 2018-02-26 14:08:34 UTC
Red Hat Product Errata RHSA-2018:2607 None None None 2018-09-04 06:43:59 UTC

Description Pan Ousley 2018-02-12 14:26:48 UTC
Description of problem:

The customer is getting a very high volume of EBADF errors that seem to be coming from a defect in one of their applications. However, while figuring this out, they need to suppress the messages, because they are filling up the storage very rapidly (1GB/hour in logs). This is heavily impacting production.

The client log is filling up with W level messages:

[2018-02-09 19:34:38.225605] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 0-home-client-3: remote operation failed [Bad file descriptor]
[2018-02-09 19:34:38.226525] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 0-home-client-2: remote operation failed [Bad file descriptor]
[2018-02-09 19:34:38.226633] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 7223: READ => -1 gfid=ef78d2ed-382a-4c22-a410-b100a3dc6ecd fd=0x7fa16401206c (Bad file descriptor)
[2018-02-09 19:34:38.227251] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 0-home-client-3: remote operation failed [Bad file descriptor]

The customer has tried using the log-level=ERROR mount option, but it does not get rid of these errors. They also tried "log-level=NONE" which also did not work.

Normally, the customer uses autofs, but they have also tried mounting manually which did not work either:

mount -t glusterfs -o rw,root-squash=0,nosuid,nodev,noatime,log-file=/var/log/gluster-mount2.log,log-level=ERROR <redacted>:/home /mnt/gluster-test

mount -t glusterfs -o rw,root-squash=0,nosuid,nodev,noatime,log-file=/var/log/gluster-mount3.log,log-level=NONE <redacted>:/home /mnt/gluster-test

Version-Release number of selected component (if applicable): RHGS 3.3

How reproducible: We have not been able to reproduce this issue using the following mount command:

# mount -t glusterfs -o log-level=ERROR /mnt/fuse_mnt

Additional info: This bug is being opened as requested in: https://bugzilla.redhat.com/show_bug.cgi?id=1540282

I was not sure which component to file this under, so please change it if necessary. I will follow up in a private comment with some more information. Please let me know if any other information is needed. Thank you.

Comment 24 errata-xmlrpc 2018-09-04 06:42:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.