Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1147156

Summary:	AFR client segmentation fault in afr_priv_destroy
Product:	[Community] GlusterFS	Reporter:	Tiziano Müller <tm>
Component:	replicate	Assignee:	Tiziano Müller <tm>
Status:	CLOSED CURRENTRELEASE	QA Contact:
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.5.3	CC:	bugs
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	glusterfs-3.5.3	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-11-21 16:03:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Tiziano Müller 2014-09-27 16:13:55 UTC

Description of problem:

Segmentation of libvirtd when starting a VM with volumes directly attached to glusterfs on a grsec kernel 3.14.19.

Version-Release number of selected component (if applicable):

glusterfs is 3.5.2
+ patches queued for 3.5 up to 36fb486b60bf786a8ac55512175c9f81a5f031e1 (hence 3.5.3)
+ glfs_fini-Fix-a-possible-hang-in-glfs_fini.patch from 3.6
+ glfs_fini-Clean-up-all-the-resources-allocated-in-gl.patch backported from 3.6/master from respective glfs_fini (see http://review.gluster.org/7642)

How reproducible:
Always as stated above

Actual results:

(gdb) bt
#0  0x000003419c219f43 in afr_priv_destroy (priv=0x34180020320)
    at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/xlators/cluster/afr/src/afr-common.c:4620
#1  0x000003419c21b61e in fini (this=<optimized out>) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/xlators/cluster/afr/src/afr.c:479
#2  0x00000341a13f14f1 in xlator_fini_rec (xl=0x34180008b90) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:440
#3  0x00000341a13f1485 in xlator_fini_rec (xl=0x34180009db0) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430
#4  0x00000341a13f1485 in xlator_fini_rec (xl=0x3418000af20) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430
#5  0x00000341a13f1485 in xlator_fini_rec (xl=0x3418000c450) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430
#6  0x00000341a13f1485 in xlator_fini_rec (xl=0x3418000d5f0) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430
#7  0x00000341a13f1485 in xlator_fini_rec (xl=0x3418000e7d0) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430
#8  0x00000341a13f1485 in xlator_fini_rec (xl=0x3418000f9a0) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430
#9  0x00000341a13f1485 in xlator_fini_rec (xl=0x34180010b40) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430
#10 0x00000341a13f1485 in xlator_fini_rec (xl=0x34180011d30) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:430
#11 0x00000341a13f2ca2 in xlator_tree_fini (xl=<optimized out>) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/libglusterfs/src/xlator.c:513
#12 0x00000341a1cb59ac in glusterfs_ctx_destroy (ctx=0x341900018a0) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/api/src/glfs.c:718
#13 glfs_fini (fs=0x34190023830) at /var/tmp/portage/sys-cluster/glusterfs-3.5.2/work/glusterfs-3.5.2/api/src/glfs.c:898
#14 0x00000341a2128b38 in virStorageFileBackendGlusterDeinit (src=0x341980f03c0) at storage/storage_backend_gluster.c:562
#15 0x00000341a211dbff in virStorageFileDeinit (src=src@entry=0x341980f03c0) at storage/storage_driver.c:2505
#16 0x00000341a211e508 in virStorageFileGetMetadataRecurse (src=src@entry=0x341980f03c0, uid=uid@entry=107, gid=gid@entry=110, 
    allow_probe=allow_probe@entry=false, cycle=cycle@entry=0x34190001820) at storage/storage_driver.c:2865
#17 0x00000341a211e8e2 in virStorageFileGetMetadata (src=0x341980f03c0, uid=107, gid=110, allow_probe=false) at storage/storage_driver.c:2905
#18 0x00000341a0673d2c in qemuDomainDetermineDiskChain (driver=driver@entry=0x3419800f280, vm=vm@entry=0x3419801b1e0, disk=disk@entry=0x341980f0290, 
    force=force@entry=false) at qemu/qemu_domain.c:2488
#19 0x00000341a0673e6e in qemuDomainCheckDiskPresence (driver=driver@entry=0x3419800f280, vm=vm@entry=0x3419801b1e0, cold_boot=cold_boot@entry=true)
    at qemu/qemu_domain.c:2321


Additional info:

As far as I can tell, the problem is that the codepath in xlators/cluster/afr/src/afr.c permits going out without calling afr_initialise_statistics (line 463), such that when afr_priv_destroy in xlators/cluster/afr/src/afr-common.c is called, priv->shd.statistics gets dereferenced via array indexing in https://github.com/gluster/glusterfs/blob/release-3.5/xlators/cluster/afr/src/afr-common.c#L4620 but the array was never allocated.

Comment 1 Anand Avati 2014-09-27 16:23:57 UTC

REVIEW: http://review.gluster.org/8873 (Only cleanup priv->shd.statistics if created) posted (#1) for review on release-3.5 by Tiziano Müller (tiziano.mueller)

Comment 2 Anand Avati 2014-10-07 07:42:33 UTC

COMMIT: http://review.gluster.org/8873 committed in release-3.5 by Niels de Vos (ndevos) 
------
commit 673b1deced72f9325caf0b5d6fb160ad17ec4b35
Author: Tiziano Müller <tiziano.mueller>
Date:   Sat Sep 27 18:23:24 2014 +0200

    Only cleanup priv->shd.statistics if created
    
    It is possible that the statistics array was never created and
    dereferencing it may case a segfault.
    
    BUG: 1147156
    Change-Id: If905457ba985add62c3ed543bced1313640af762
    Signed-off-by: Tiziano Müller <tiziano.mueller>
    Reviewed-on: http://review.gluster.org/8873
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Reviewed-by: Niels de Vos <ndevos>

Comment 3 Niels de Vos 2014-11-05 09:25:02 UTC

The second Beta for GlusterFS 3.5.3 has been released [1]. Please verify if the release solves this bug report for you. In case the glusterfs-3.5.3beta2 release does not have a resolution for this issue, leave a comment in this bug and move the status to ASSIGNED. If this release fixes the problem for you, leave a note and change the status to VERIFIED.

Packages for several distributions have been made available on [2] to make testing easier.

[1] http://supercolony.gluster.org/pipermail/gluster-users/2014-November/019359.html
[2] http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.5.3beta2/

Comment 4 Niels de Vos 2014-11-21 16:03:06 UTC

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.3, please reopen this bug report.

glusterfs-3.5.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://supercolony.gluster.org/pipermail/announce/2014-November/000042.html
[2] http://supercolony.gluster.org/pipermail/gluster-users/