Bug 1501439

Summary:	[GSS][Tracker] glusterfs plugin logs Warning\|Info\|Error messages even when --log-level=ERROR
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Nicolas Nosenzo <nnosenzo>
Component:	kubernetes	Assignee:	Humble Chirammal <hchiramm>
Status:	CLOSED ERRATA	QA Contact:	Neha Berry <nberry>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.0	CC:	aivaraslaimikis, akrishna, amukherj, aos-bugs, aos-storage-staff, bkunal, dmoessne, eparis, hchiramm, jarrpa, jkaur, jmulligan, jolee, kramdoss, madam, rcyriac, rgeorge, rhs-bugs, rtalur, suchaudh
Target Milestone:	---
Target Release:	CNS 3.10
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:	Even though glusterfs fuse client log level has been set to ERROR, due to an issue in glusterfs client, it logs Info\|Warn\|Error messages regardless of the log level value set in the client process. This issue has been fixed with latest glusterfs client release.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-09-12 12:27:13 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1544451
Bug Blocks:	1568861, 1573420

Description Nicolas Nosenzo 2017-10-12 15:01:54 UTC

Description of problem:
I have configured a glusterfs volumes in OCP. The plugin clients are running with the following parameters:

[root@ocplab ~]# ps -ef | grep gluster
root       3185      1  0 10:41 ?        00:00:00 /usr/sbin/glusterfs --log-level=ERROR --log-file=/var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/glusterfs/pvc-6740a83a-8e42-11e7-84b7-2cc26032ab56/gluster-pod1-glusterfs.log --volfile-server=10.0.2.14 --volfile-id=vol_4819b368e1daa9bf79068e0728938780 /var/lib/origin/openshift.local.volumes/pods/82041e51-8e43-11e7-84b7-2cc26032ab56/volumes/kubernetes.io~glusterfs/pvc-6740a83a-8e42-11e7-84b7-2cc26032ab56
root       3587      1  0 10:41 ?        00:00:00 /usr/sbin/glusterfs --log-level=ERROR --log-file=/var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/glusterfs/gluster-pv/gluster-pod1-glusterfs.log --volfile-server=10.0.2.15 --volfile-id=/HadoopVol /var/lib/origin/openshift.local.volumes/pods/6ec2f4f3-88bb-11e7-b38f-2cc26032ab56/volumes/kubernetes.io~glusterfs/gluster-pv
root       4814   2615  0 10:44 pts/0    00:00:00 grep --color=auto gluster
[root@ocplab ~]# 

As you can see, all glusterfs clients run with --log-level=ERROR. Even though, within the log we can see Info and Warn messages also:

[root@ocplab ~]# grep " I " /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/glusterfs/pvc-6740a83a-8e42-11e7-84b7-2cc26032ab56/gluster-pod1-glusterfs.log | wc -l
144
[root@ocplab ~]# grep " W " /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/glusterfs/pvc-6740a83a-8e42-11e7-84b7-2cc26032ab56/gluster-pod1-glusterfs.log | wc -l
18
[root@ocplab ~]# grep " E " /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/glusterfs/pvc-6740a83a-8e42-11e7-84b7-2cc26032ab56/gluster-pod1-glusterfs.log | wc -l
2
[root@ocplab ~]#

Version-Release number of selected component (if applicable):
OCP 3.6
# rpm -qa | grep gluster
glusterfs-client-xlators-3.7.9-12.el7.x86_64
glusterfs-libs-3.7.9-12.el7.x86_64
glusterfs-3.7.9-12.el7.x86_64
glusterfs-fuse-3.7.9-12.el7.x86_64

How reproducible:

100%

Steps to Reproduce:
1. Install OCP
2. Configure glusterFS plugin


Actual results:

Glusterfs client logs Info|Warn|Error messages no matter which --log-level it runs with.
It eventually causes that the /var FS run out of space.
Expected results:


Additional info:

Comment 16 Humble Chirammal 2018-07-03 03:42:35 UTC

As an additional info there is also a KCS getting published to mitigate/workaround the issue we are seeing with this bug report.

Comment 22 Michael Adam 2018-08-23 05:18:24 UTC

Should be fixed in the latest RHGS gluster client update.

Comment 25 Anjana KD 2018-09-07 17:32:05 UTC

Updated the Doc Text field. kindly review

Comment 26 John Mulligan 2018-09-07 19:13:19 UTC

Doc Text looks OK

Comment 28 errata-xmlrpc 2018-09-12 12:27:13 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2697