Description of problem: Segmentation of libvirtd when starting a VM with volumes directly attached to glusterfs on a grsec kernel 3.14.19. Version-Release number of selected component (if applicable): glusterfs is 3.5.2 + patches queued for 3.5 up to 36fb486b60bf786a8ac55512175c9f81a5f031e1 (hence 3.5.3) + glfs_fini-Fix-a-possible-hang-in-glfs_fini.patch from 3.6 + glfs_fini-Clean-up-all-the-resources-allocated-in-gl.patch backported from 3.6/master from respective glfs_fini (see http://review.gluster.org/7642) How reproducible: Always as stated above Actual results: (gdb) bt #0 0x000003419c219f43 in afr_priv_destroy (priv=0x34180020320) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/xlators/cluster/afr/src/afr-common.c:4620 #1 0x000003419c21b61e in fini (this=<optimized out>) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/xlators/cluster/afr/src/afr.c:479 #2 0x00000341a13f14f1 in xlator_fini_rec (xl=0x34180008b90) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:440 #3 0x00000341a13f1485 in xlator_fini_rec (xl=0x34180009db0) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430 #4 0x00000341a13f1485 in xlator_fini_rec (xl=0x3418000af20) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430 #5 0x00000341a13f1485 in xlator_fini_rec (xl=0x3418000c450) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430 #6 0x00000341a13f1485 in xlator_fini_rec (xl=0x3418000d5f0) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430 #7 0x00000341a13f1485 in xlator_fini_rec (xl=0x3418000e7d0) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430 #8 0x00000341a13f1485 in xlator_fini_rec (xl=0x3418000f9a0) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430 #9 0x00000341a13f1485 in xlator_fini_rec (xl=0x34180010b40) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430 #10 0x00000341a13f1485 in xlator_fini_rec (xl=0x34180011d30) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430 #11 0x00000341a13f2ca2 in xlator_tree_fini (xl=<optimized out>) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:513 #12 0x00000341a1cb59ac in glusterfs_ctx_destroy (ctx=0x341900018a0) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/api/src/glfs.c:718 #13 glfs_fini (fs=0x34190023830) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/api/src/glfs.c:898 #14 0x00000341a2128b38 in virStorageFileBackendGlusterDeinit (src=0x341980f03c0) at storage/storage_backend_gluster.c:562 #15 0x00000341a211dbff in virStorageFileDeinit (src=src@entry=0x341980f03c0) at storage/storage_driver.c:2505 #16 0x00000341a211e508 in virStorageFileGetMetadataRecurse (src=src@entry=0x341980f03c0, uid=uid@entry=107, gid=gid@entry=110, allow_probe=allow_probe@entry=false, cycle=cycle@entry=0x34190001820) at storage/storage_driver.c:2865 #17 0x00000341a211e8e2 in virStorageFileGetMetadata (src=0x341980f03c0, uid=107, gid=110, allow_probe=false) at storage/storage_driver.c:2905 #18 0x00000341a0673d2c in qemuDomainDetermineDiskChain (driver=driver@entry=0x3419800f280, vm=vm@entry=0x3419801b1e0, disk=disk@entry=0x341980f0290, force=force@entry=false) at qemu/qemu_domain.c:2488 #19 0x00000341a0673e6e in qemuDomainCheckDiskPresence (driver=driver@entry=0x3419800f280, vm=vm@entry=0x3419801b1e0, cold_boot=cold_boot@entry=true) at qemu/qemu_domain.c:2321 Additional info: As far as I can tell, the problem is that the codepath in xlators/cluster/afr/src/afr.c permits going out without calling afr_initialise_statistics (line 463), such that when afr_priv_destroy in xlators/cluster/afr/src/afr-common.c is called, priv->shd.statistics gets dereferenced via array indexing in https://github.com/gluster/glusterfs/blob/release-3.5/xlators/cluster/afr/src/afr-common.c#L4620 but the array was never allocated.
REVIEW: http://review.gluster.org/8873 (Only cleanup priv->shd.statistics if created) posted (#1) for review on release-3.5 by Tiziano Müller (tiziano.mueller)
COMMIT: http://review.gluster.org/8873 committed in release-3.5 by Niels de Vos (ndevos) ------ commit 673b1deced72f9325caf0b5d6fb160ad17ec4b35 Author: Tiziano Müller <tiziano.mueller> Date: Sat Sep 27 18:23:24 2014 +0200 Only cleanup priv->shd.statistics if created It is possible that the statistics array was never created and dereferencing it may case a segfault. BUG: 1147156 Change-Id: If905457ba985add62c3ed543bced1313640af762 Signed-off-by: Tiziano Müller <tiziano.mueller> Reviewed-on: http://review.gluster.org/8873 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Pranith Kumar Karampuri <pkarampu> Reviewed-by: Niels de Vos <ndevos>
The second Beta for GlusterFS 3.5.3 has been released [1]. Please verify if the release solves this bug report for you. In case the glusterfs-3.5.3beta2 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED. Packages for several distributions have been made available on [2] to make testing easier. [1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019359.html [2] http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.5.3beta2/
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.3, please reopen this bug report. glusterfs-3.5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://supercolony.gluster.org/pipermail/announce/2014-November/000042.html [2] http://supercolony.gluster.org/pipermail/gluster-users/