Bug 1103413

Summary:	Failure in gf_log_init reopening stderr
Product:	[Community] GlusterFS	Reporter:	Niels de Vos <ndevos>
Component:	libgfapi	Assignee:	Niels de Vos <ndevos>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Sudhir D <sdharane>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	3.5.0	CC:	gluster-bugs, pkarampu
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.5.1beta2	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1088589	Environment:
Last Closed:	2014-06-24 11:05:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1088589, 1150188
Bug Blocks:

Description Niels de Vos 2014-05-31 08:26:21 UTC

+++ This bug was initially created as a clone of Bug #1088589 +++

The qemu gluster block driver attempts to set gluster logging to stderr as such:

    ret = glfs_set_logging(glfs, "-", 4)

And then inside gf_log_init, "-" is matched, file is set to "/dev/stderr", and the file is attempted to be reopened:

    fd = open (file, O_CREAT | O_RDONLY, S_IRUSR | S_IWUSR);

However this fails:

    ERROR: failed to create logfile "/dev/stderr" (Permission denied)

The reason is that libvirt has already redirected stderr into the logfile for the VM and that file has restricted permissions:

[root@compute tmp]# ls -l /proc/29981/fd/2
l-wx------. 1 qemu qemu 64 Apr 16 15:11 /proc/29981/fd/2 -> /var/log/libvirt/qemu/instance-0000000b.log
[root@compute tmp]# ls -l /var/log/libvirt/qemu/instance-0000000b.log
-rw-------. 1 root root 16768 Apr 16 15:11 /var/log/libvirt/qemu/instance-0000000b.log

So the process is unable to reopen the file, but there is already an open descriptor that can be used.  In this case perhaps fall back to the old behavior that was changed by this commit?:

https://forge.gluster.org/glusterfs-core/glusterfs/commit/00b159840added72099f61f45b38133e856d23bf

--- Additional comment from Anand Avati on 2014-04-29 22:59:19 CEST ---

REVIEW: http://review.gluster.org/7607 (logging: use duplicate stderr, instead of re-opening) posted (#1) for review on master by Anand Avati (avati)

--- Additional comment from Anand Avati on 2014-05-01 08:58:22 CEST ---

COMMIT: http://review.gluster.org/7607 committed in master by Vijay Bellur (vbellur) 
------
commit 15ea78fffd63756fecf2f15887d3cad6a13d2a34
Author: Anand Avati <avati>
Date:   Tue Apr 29 13:54:53 2014 -0700

    logging: use duplicate stderr, instead of re-opening
    
    The special filename "-" is supposed to log to stderr. Instead of
    trying to explictly open "/dev/stderr" again (which may not be possible
    as permissions might have changed by then), dup the stderr and use
    the copy.
    
    It is not a good idea to use @stderr global variable directly, as
    ctx->log.logfile is fclose()d in glfs_fini() (was fixed in
    http://review.gluster.org/6452)
    
    Change-Id: Ia6c538fe363905588dcf4fc4783804073956a586
    BUG: 1088589
    Signed-off-by: Anand Avati <avati>
    Reviewed-on: http://review.gluster.org/7607
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 1 Anand Avati 2014-05-31 08:33:54 UTC

REVIEW: http://review.gluster.org/7938 (logging: use duplicate stderr, instead of re-opening) posted (#1) for review on release-3.5 by Niels de Vos (ndevos)

Comment 2 Anand Avati 2014-06-08 07:28:34 UTC

COMMIT: http://review.gluster.org/7938 committed in release-3.5 by Niels de Vos (ndevos) 
------
commit 47732e9742ae1f898f161b4244e7faed74c98756
Author: Niels de Vos <ndevos>
Date:   Tue Apr 29 13:54:53 2014 -0700

    logging: use duplicate stderr, instead of re-opening
    
    The special filename "-" is supposed to log to stderr. Instead of
    trying to explictly open "/dev/stderr" again (which may not be possible
    as permissions might have changed by then), dup the stderr and use
    the copy.
    
    It is not a good idea to use @stderr global variable directly, as
    ctx->log.logfile is fclose()d in glfs_fini() (was fixed in
    http://review.gluster.org/6452)
    
    Cherry picked from commit 15ea78fffd63756fecf2f15887d3cad6a13d2a34:
    > BUG: 1088589
    > Signed-off-by: Anand Avati <avati>
    > Reviewed-on: http://review.gluster.org/7607
    > Tested-by: Gluster Build System <jenkins.com>
    > Reviewed-by: Vijay Bellur <vbellur>
    
    Change-Id: Ia6c538fe363905588dcf4fc4783804073956a586
    BUG: 1103413
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/7938
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Harshavardhana <harsha>

Comment 3 Niels de Vos 2014-06-10 16:52:02 UTC

The second (and last?) Beta for GlusterFS 3.5.1 has been released [1]. Please verify if the release solves this bug report for you. In case the glusterfs-3.5.1beta2 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update (possibly an "updates-testing" repository) infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-June/040547.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/

Comment 4 Pranith Kumar K 2014-06-18 02:16:18 UTC

Justin,

I am testing 3.5.1beta2 in CentOS 6.5 and bug #1103413 is gone.
3.5.1beta2 is the best working release for me so far.
Thank you.

Jae

Based on the mail from jpark, moving this bug to VERIFIED.

Comment 5 Niels de Vos 2014-06-24 11:05:32 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.1, please reopen this bug report.

glusterfs-3.5.1 has been announced on the Gluster Users mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-June/040723.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user