Description of problem: ----------------------- QEMU uses native glusterfs driver ( which uses libgfapi ) to provides VM ability to access to the disks that resides on gluster volume When generating statedump of gfapi application (QEMU), QEMU doesn't have permission to write to statedump path ( /var/run/gluster ) and doesn't generates statedump. Only the applications that runs with 'root' privilege could write to the directory - /var/run/gluster There should be a proper way for applications that doesn't have root privilege to write to /var/run/gluster Version-Release number of selected component (if applicable): ------------------------------------------------------------- glusterfs-3.8.4-23.el7rhgs How reproducible: ------------------ Always Steps to Reproduce: -------------------- 1. Create a VM that runs with disk accessed using QEMU's native driver for glusterfs (gfapi) 2. Trigger statedump from gluster node # gluster volume statedump <vol> client <hypervisor-IP/FQDN>:<QEMU-PID> Actual results: --------------- No statedump dumped by QEMU process under /var/run/gluster QEMU user doesn't have privilege to write to /var/run/gluster Expected results: ----------------- statedump available under /var/run/gluster
Thanks Neils for figuring out the actual problem that QEMU doesn't have permission to write to /var/run/gluster directory. Possible solutions as suggested by Neils 1. One approach would be to create a "gluster" group and give the group permissions to write to /var/run/gluster/... 2. Other 'fixes' include setting ACLs on the directory so that specified users can write there.because many daemons have a "home directory" that does not exist, it probably is not a good idea to use $HOME to store statedumps. Neils suggested the following in the comment https://bugzilla.redhat.com/show_bug.cgi?id=1378085#c15 <snip> The QEMU process runs as user "qemu". This user does not have write permissions on /var/run/gluster/ and can therefore not write the statedump there... Adjusting the permissions with an ACL makes it work: # setfacl -m u:qemu:rwx /var/run/gluster What the correct approach is to have any application work with this needs to be thought out. Samba (I think?) and NFS-Ganesha run as root, and are not limited by the write permissions. </snip>
upstream patch : https://review.gluster.org/#/c/17122/
Hi Miroslav, QEMU is unable to store Gluster specific debugging logs under /var/run/gluster/... This is because QEMU runs as user "qemu" and does not have write permissions there. This debugging (called 'statedump') can be triggered from the Gluster CLI which sends an event to QEMU (when libgfapi is used). When the glusterfs packages make sure that a group "gluster" has write permissions to /var/run/gluster/, could the qemu(-kvm-rhev) package be adjusted to have the "qemu" user in the "gluster" group? This would allow more debugging options that help with potential issues when QEMU runs with disk-images over libgfapi.so. An other option would be to add an API that QEMU can call to configure a different path. But that has its own other complexities... Thanks! Niels
Hi Niels, sorry for late answer. Had other tasks on plate. As I check the options, I think we can add qemu user to gluster group. It's probably the easiest way to solve this issue and allow us to update solution to modularization handling in 7.5. Mirek
The request to get the "gluster" group added to the "qemu" user in the Fedora QEMU package has been reported as bug 1447799.
Tested with RHGS 3.3.0 interim build ( glusterfs-3.8.4-28.el7rhgs ) and selinux-policy-3.13.1-160.el7 1. 'gluster' group is created # getent group gluster gluster:x:992:qemu 2. 'qemu' user is added manually to this group # usermod -a -G gluster qemu 3. The generated gfapi statedump is written to /var/run/gluster/ directory.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2774