+++ This bug was initially created as a clone of Bug #1169302 +++ Description of problem: Unable to take Statedump for gfapi applications Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Anand Avati on 2014-12-02 01:54:47 EST --- REVIEW: http://review.gluster.org/9228 (libgfapi: statedump support) posted (#1) for review on master by Poornima G (pgurusid) --- Additional comment from Anand Avati on 2014-12-03 02:11:58 EST --- REVIEW: http://review.gluster.org/9228 (libgfapi: statedump support) posted (#2) for review on master by Poornima G (pgurusid) --- Additional comment from Anand Avati on 2015-01-05 05:54:21 EST --- REVIEW: http://review.gluster.org/9228 (libgfapi: statedump support) posted (#3) for review on master by Poornima G (pgurusid) --- Additional comment from Kaleb KEITHLEY on 2015-10-22 11:40:20 EDT --- pre-release version is ambiguous and about to be removed as a choice. If you believe this is still a bug, please change the status back to NEW and choose the appropriate, applicable version for it. --- Additional comment from Vijay Bellur on 2016-01-28 01:46:57 EST --- REVIEW: http://review.gluster.org/9228 (libgfapi: Implement statedump) posted (#4) for review on master by Poornima G (pgurusid) --- Additional comment from Vijay Bellur on 2016-03-16 05:37:53 EDT --- REVIEW: http://review.gluster.org/9228 (libgfapi: Implement statedump) posted (#5) for review on master by Poornima G (pgurusid) --- Additional comment from Vijay Bellur on 2016-07-20 01:44:13 EDT --- REVIEW: http://review.gluster.org/9228 (libgfapi: Implement statedump) posted (#6) for review on master by Poornima G (pgurusid)
*** Bug 1421137 has been marked as a duplicate of this bug. ***
downstream patches: https://code.engineering.redhat.com/gerrit/#/c/101293 https://code.engineering.redhat.com/gerrit/#/c/101296 https://code.engineering.redhat.com/gerrit/#/c/101297 https://code.engineering.redhat.com/gerrit/#/c/101320/
Tested with RHGS 3.3.0 interim build ( glusterfs-3.8.4-23.el7rhgs ) with the following steps 1. In a 3 node trusted storage pool ( gluster cluster ), created sharded arbitrated replicate volume and optimized it for VM store use case 2. Used another machine outside of trusted storage pool, and created VM image file, and installed the VM with RHEL 7.3 3. Rebooted the VM post OS installation and noted down the pid of QEMU process and also the hostname of this host 4. Created the statedump of gfapi application ( QEMU ) from one the 3 node cluster # gluster volume statedump <vol> <host>:<pid> The command was successful but no statedumps available under /var/run/gluster
I installed a RHGS-3.2 system and upgraded to the latest glusterfs build from Brew. Running the glfs_sysrq.t is successful, showing that gfapi applications can do statedumps: [root@vm019 tests]# rpm -q glusterfs glusterfs-3.8.4-23.el7rhgs.x86_64 [root@vm019 tests]# prove -vv basic/gfapi/glfs_sysrq.t basic/gfapi/glfs_sysrq.t .. 1..10 ok 1, LINENUM:13 ok 2, LINENUM:14 ok 3, LINENUM:15 ok 4, LINENUM:16 ok 5, LINENUM:18 ok 6, LINENUM:19 ok 7, LINENUM:25 ok 8, LINENUM:29 ok 9, LINENUM:32 ok 10, LINENUM:35 ok All tests successful. Files=1, Tests=10, 15 wallclock secs ( 0.04 usr 0.00 sys + 0.75 cusr 0.61 csys = 1.40 CPU) Result: PASS [root@vm019 tests]# ls /var/run/gluster/glusterdump.* /var/run/gluster/glusterdump.21166.dump.1493132392 This test only calls glfs_sysrq() to generate the statedump, and does not test the gluster-cli or the glusterd-mgmt parts. For this, the upstream tests/bugs/cli/bug-1169302.c can be used: # gluster volume create bz1378085 ... # gluster volume start bz1378085 # bugs/cli/bug-1169302 bz1378085 $HOSTNAME \ /var/tmp/bug-1169302.log bug-1169302.bin # pgrep bug-1169302 31043 [root@vm019 ~]# gluster volume statedump bz1378085 client $HOSTNAME:31043 volume statedump: success [root@vm019 ~]# ls /var/run/gluster/glusterdump.* /var/run/gluster/glusterdump.31043.dump.1493133982 Because this works just fine, I suspect that there is a configuration issue on the hypervisor somewhere (SELinux is still a likely candidate).
The QEMU process runs as user "qemu". This user does not have write permissions on /var/run/gluster/ and can therefore not write the statedump there... Adjusting the permissions with an ACL makes it work: # setfacl -m u:qemu:rwx /var/run/gluster What the correct approach is to have any application work with this needs to be thought out. Samba (I think?) and NFS-Ganesha run as root, and are not limited by the write permissions.
(In reply to Niels de Vos from comment #15) > The QEMU process runs as user "qemu". This user does not have write > permissions on /var/run/gluster/ and can therefore not write the statedump > there... > > Adjusting the permissions with an ACL makes it work: > > # setfacl -m u:qemu:rwx /var/run/gluster > > What the correct approach is to have any application work with this needs to > be thought out. Samba (I think?) and NFS-Ganesha run as root, and are not > limited by the write permissions. Thanks Neils. Verified the ability to generate statedump by gfapi application (QEMU) after setting proper acl on /var/run/gluster. Tested with glusterfs-3.8.4-23.el7rhgs New bug BZ1445570 is created for the issue related to gfapi application ( QEMU ) not dumping statedump under /var/run/gluster because of QEMU user doesn't have privilege to write to /var/run/gluster
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2774