Bug 1004100 - smbd crashes in libglusterfs under heavy load
Summary: smbd crashes in libglusterfs under heavy load
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: libgfapi
Version: mainline
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: ---
Assignee: GlusterFS Bugs list
QA Contact: Sudhir D
URL:
Whiteboard:
Depends On: 1003917
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-09-03 23:22 UTC by Anand Avati
Modified: 2015-09-01 23:06 UTC (History)
7 users (show)

Fixed In Version: glusterfs-3.5.0
Doc Type: Bug Fix
Doc Text:
Clone Of: 1003917
Environment:
Last Closed: 2014-04-17 11:47:15 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Comment 1 Anand Avati 2013-09-03 23:23:20 UTC
On inspecting the core it was found that the resolver had nested down (towards infinity) while resolving a recursive symlink, and the crash has nothing to do with extended attributes.

Comment 2 Anand Avati 2013-09-03 23:29:12 UTC
REVIEW: http://review.gluster.org/5768 (gfapi: apply an upper bound on nested symlink resolution) posted (#1) for review on master by Anand Avati (avati)

Comment 3 Anand Avati 2013-09-03 23:45:00 UTC
REVIEW: http://review.gluster.org/5768 (gfapi: apply an upper bound on nested symlink resolution) posted (#2) for review on master by Anand Avati (avati)

Comment 4 Christopher R. Hertel 2013-09-04 00:18:10 UTC
Update:  The stack trace given above does not pass through the EA handling.  This is a crash under heavy load.

The load, in this case, is building Gluster itself.  The stack trace shows that the crash occurs in the Gluster layer.  SMBD crashes because it is linked to the Gluster client library.

Comment 5 Anand Avati 2013-09-04 00:59:51 UTC
Chris, the operation was EA related, but the bug was not. Here is the section of the backtrace:


#35694 0x00007f1b76a654d0 in glfs_resolve_at (fs=0x7f1b79d35260, subvol=0x7f1b600105a0, 
    at=<value optimized out>, origpath=<value optimized out>, loc=0x7fffd37a0850, iatt=0x7fffd37a07e0, 
    follow=1, reval=0) at glfs-resolve.c:377
#35695 0x00007f1b76a6655b in glfs_resolve_path (fs=0x7f1b79d35260, subvol=0x7f1b600105a0, 
    origpath=0x7f1b79d7e7f0 "glusterfs.git/conf11752.dir/conf11752.file", loc=0x7fffd37a0850, 
    iatt=0x7fffd37a07e0, follow=<value optimized out>, reval=0) at glfs-resolve.c:454
#35696 0x00007f1b76a665e3 in glfs_resolve (fs=<value optimized out>, subvol=<value optimized out>, 
    origpath=<value optimized out>, loc=<value optimized out>, iatt=<value optimized out>, 
    reval=<value optimized out>) at glfs-resolve.c:469
#35697 0x00007f1b76a60f61 in glfs_getxattr_common (fs=0x7f1b79d35260, 
    path=0x7f1b79d7e7f0 "glusterfs.git/conf11752.dir/conf11752.file", 
    name=0x7f1b795c8b73 "user.DOSATTRIB", value=0x7fffd37a0970, size=256, follow=1) at glfs-fops.c:2230
#35698 0x00007f1b79070f18 in get_ea_dos_attribute (conn=0x7f1b79d36390, smb_fname=0x7f1b79d7e6e0, 
    pattr=0x7fffd37a0abc) at smbd/dosmode.c:234
#35699 0x00007f1b79071905 in dos_mode (conn=0x7f1b79d36390, smb_fname=0x7f1b79d7e6e0)
    at smbd/dosmode.c:649
#35700 0x00007f1b7905eee9 in smbd_do_qfilepathinfo (conn=0x7f1b79d36390, mem_ctx=0x7f1b79d7e490, 
    info_level=512, fsp=0x0, smb_fname=0x7f1b79d7e6e0, delete_pending=false, write_time_ts=..., 
    ea_list=0x0, lock_data_count=0, lock_data=0x0, flags2=49217, max_data_bytes=4000, 
    ppdata=0x7f1b79de60b0, pdata_size=0x7fffd37a0e88) at smbd/trans2.c:4225


Samba was performing a getxattr("user.DOSATTRIB") on a recursive symlink which triggered the crash.

Comment 6 Anand Avati 2013-09-05 12:27:13 UTC
COMMIT: http://review.gluster.org/5768 committed in master by Vijay Bellur (vbellur) 
------
commit 2a1f445d30f2d57951a4d5582c1d9b1f675ad704
Author: Anand Avati <avati>
Date:   Tue Sep 3 16:01:41 2013 -0700

    gfapi: apply an upper bound on nested symlink resolution
    
    In case of nested symlink resolution, implement an upper
    bound on the number of such nested levels the resolver will
    descend. This limit is arbitrary, and set to 2048 nested
    levels.
    
    Change-Id: I264e5bd60d317eda97f4e6f49bd5d8694f8664a9
    BUG: 1004100
    Signed-off-by: Anand Avati <avati>
    Reviewed-on: http://review.gluster.org/5768
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Raghavendra Talur <rtalur>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 7 Anand Avati 2013-09-10 00:51:41 UTC
REVIEW: http://review.gluster.org/5874 (gfapi: apply an upper bound on nested symlink resolution) posted (#1) for review on release-3.4 by Anand Avati (avati)

Comment 8 Anand Avati 2013-09-10 01:05:09 UTC
REVIEW: http://review.gluster.org/5874 (gfapi: apply an upper bound on nested symlink resolution) posted (#2) for review on release-3.4 by Anand Avati (avati)

Comment 9 Anand Avati 2013-09-10 08:16:56 UTC
COMMIT: http://review.gluster.org/5874 committed in release-3.4 by Vijay Bellur (vbellur) 
------
commit 255b0dc5201b012b0e8f02ea665ba08ab0a4a0d1
Author: Anand Avati <avati>
Date:   Tue Sep 3 16:01:41 2013 -0700

    gfapi: apply an upper bound on nested symlink resolution
    
    In case of nested symlink resolution, implement an upper
    bound on the number of such nested levels the resolver will
    descend. This limit is arbitrary, and set to 2048 nested
    levels.
    
    Change-Id: I264e5bd60d317eda97f4e6f49bd5d8694f8664a9
    BUG: 1004100
    Signed-off-by: Anand Avati <avati>
    Reviewed-on: http://review.gluster.org/5874
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Vijay Bellur <vbellur>

Comment 10 Niels de Vos 2014-04-17 11:47:15 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.0, please reopen this bug report.

glusterfs-3.5.0 has been announced on the Gluster Developers mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/6137
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.