Red Hat Bugzilla – Bug 1544451
[GSS] log-level=ERROR mount option not working, W level messages rapidly filling up storage
Last modified: 2018-09-10 03:05:58 EDT
Description of problem: The customer is getting a very high volume of EBADF errors that seem to be coming from a defect in one of their applications. However, while figuring this out, they need to suppress the messages, because they are filling up the storage very rapidly (1GB/hour in logs). This is heavily impacting production. The client log is filling up with W level messages: [2018-02-09 19:34:38.225605] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 0-home-client-3: remote operation failed [Bad file descriptor] [2018-02-09 19:34:38.226525] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 0-home-client-2: remote operation failed [Bad file descriptor] [2018-02-09 19:34:38.226633] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 7223: READ => -1 gfid=ef78d2ed-382a-4c22-a410-b100a3dc6ecd fd=0x7fa16401206c (Bad file descriptor) [2018-02-09 19:34:38.227251] W [MSGID: 114031] [client-rpc-fops.c:3002:client3_3_readv_cbk] 0-home-client-3: remote operation failed [Bad file descriptor] The customer has tried using the log-level=ERROR mount option, but it does not get rid of these errors. They also tried "log-level=NONE" which also did not work. Normally, the customer uses autofs, but they have also tried mounting manually which did not work either: mount -t glusterfs -o rw,root-squash=0,nosuid,nodev,noatime,log-file=/var/log/gluster-mount2.log,log-level=ERROR <redacted>:/home /mnt/gluster-test mount -t glusterfs -o rw,root-squash=0,nosuid,nodev,noatime,log-file=/var/log/gluster-mount3.log,log-level=NONE <redacted>:/home /mnt/gluster-test Version-Release number of selected component (if applicable): RHGS 3.3 How reproducible: We have not been able to reproduce this issue using the following mount command: # mount -t glusterfs -o log-level=ERROR 127.0.0.2:testvol /mnt/fuse_mnt Additional info: This bug is being opened as requested in: https://bugzilla.redhat.com/show_bug.cgi?id=1540282 I was not sure which component to file this under, so please change it if necessary. I will follow up in a private comment with some more information. Please let me know if any other information is needed. Thank you.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607