Bug 1445570 - Provide a correct way to save the statedump generated by gfapi application
Summary: Provide a correct way to save the statedump generated by gfapi application
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs
Version: rhgs-3.2
Hardware: x86_64
OS: Linux
Target Milestone: ---
: RHGS 3.3.0
Assignee: Niels de Vos
Depends On: 1445569 1456199
Blocks: Gluster-HC-3 1417151 RHHI-1.1-Approved-Backlog-BZs
TreeView+ depends on / blocked
Reported: 2017-04-26 01:50 UTC by SATHEESARAN
Modified: 2019-04-27 02:13 UTC (History)
8 users (show)

Fixed In Version: glusterfs-3.8.4-25
Doc Type: Bug Fix
Doc Text:
Access to the /var/run/gluster/ directory is typically restricted. As a result, attempting to write a state dump to this directory fails. To write to this location, ensure that the user that runs the application is added to the 'gluster' user group, and restart gluster processes so that the new group is applied.
Clone Of:
Last Closed: 2017-09-21 04:39:40 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1378085 high CLOSED Unable to take Statedump for gfapi applications 2020-10-14 00:28:05 UTC
Red Hat Product Errata RHBA-2017:2774 normal SHIPPED_LIVE glusterfs bug fix and enhancement update 2017-09-21 08:16:29 UTC

Internal Links: 1378085

Description SATHEESARAN 2017-04-26 01:50:21 UTC
Description of problem:
QEMU uses native glusterfs driver ( which uses libgfapi ) to provides VM ability to  access to the disks that resides on gluster volume 

When generating statedump of gfapi application (QEMU), QEMU doesn't have permission to write to statedump path ( /var/run/gluster ) and doesn't generates statedump. Only the applications that runs with 'root' privilege could write to the directory  - /var/run/gluster

There should be a proper way for applications that doesn't have root privilege to write to /var/run/gluster

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Create a VM that runs with disk accessed using QEMU's native driver for glusterfs (gfapi)

2. Trigger statedump from gluster node
# gluster volume statedump <vol> client <hypervisor-IP/FQDN>:<QEMU-PID>

Actual results:
No statedump dumped by QEMU process under /var/run/gluster
QEMU user doesn't have privilege to write to /var/run/gluster

Expected results:
statedump available under /var/run/gluster

Comment 1 SATHEESARAN 2017-04-26 01:53:30 UTC
Thanks Neils for figuring out the actual problem that QEMU doesn't have permission to write to /var/run/gluster directory.

Possible solutions as suggested by Neils

1. One approach would be to create a "gluster" group and give the group
permissions to write to /var/run/gluster/... 

2. Other 'fixes' include setting ACLs on the directory so that specified users can write there.because many daemons have a "home directory" that does not exist, it
probably is not a good idea to use $HOME to store statedumps.

Neils suggested the following in the comment https://bugzilla.redhat.com/show_bug.cgi?id=1378085#c15
The QEMU process runs as user "qemu". This user does not have write permissions on /var/run/gluster/ and can therefore not write the statedump there...

Adjusting the permissions with an ACL makes it work:

    # setfacl -m u:qemu:rwx /var/run/gluster

What the correct approach is to have any application work with this needs to be thought out. Samba (I think?) and NFS-Ganesha run as root, and are not limited by the write permissions.

Comment 2 Atin Mukherjee 2017-04-26 12:22:49 UTC
upstream patch : https://review.gluster.org/#/c/17122/

Comment 5 Niels de Vos 2017-05-03 12:24:54 UTC
Hi Miroslav,

QEMU is unable to store Gluster specific debugging logs under /var/run/gluster/... This is because QEMU runs as user "qemu" and does not have write permissions there. This debugging (called 'statedump') can be triggered from the Gluster CLI which sends an event to QEMU (when libgfapi is used).

When the glusterfs packages make sure that a group "gluster" has write permissions to /var/run/gluster/, could the qemu(-kvm-rhev) package be adjusted to have the "qemu" user in the "gluster" group? This would allow more debugging options that help with potential issues when QEMU runs with disk-images over libgfapi.so.

An other option would be to add an API that QEMU can call to configure a different path. But that has its own other complexities...


Comment 6 Miroslav Rezanina 2017-05-03 13:18:36 UTC
Hi Niels,

sorry for late answer. Had other tasks on plate. 

As I check the options, I think we can add qemu user to gluster group. It's probably the easiest way to solve this issue and allow us to update solution to
modularization handling in 7.5.


Comment 8 Niels de Vos 2017-05-03 21:22:17 UTC
The request to get the "gluster" group added to the "qemu" user in the Fedora QEMU package has been reported as bug 1447799.

Comment 14 SATHEESARAN 2017-06-19 18:16:27 UTC
Tested with RHGS 3.3.0 interim build ( glusterfs-3.8.4-28.el7rhgs ) and selinux-policy-3.13.1-160.el7

1. 'gluster' group is created
# getent group gluster

2. 'qemu' user is added manually to this group
# usermod -a -G gluster qemu

3. The generated gfapi statedump is written to /var/run/gluster/ directory.

Comment 21 errata-xmlrpc 2017-09-21 04:39:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.