Bug 763110 (GLUSTER-1378)

Summary: Deep directory creation crashes gnfs
Product: [Community] GlusterFS Reporter: Shehjar Tikoo <shehjart>
Component: nfsAssignee: Shehjar Tikoo <shehjart>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: low    
Version: 3.1-alphaCC: gluster-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTP Mount Type: nfs
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
NFS log none

Description Shehjar Tikoo 2010-08-16 08:05:58 UTC
Created attachment 286

Comment 1 Shehjar Tikoo 2010-08-16 09:19:41 UTC
First problem is simply that creation of a directory with hash count larger than 21 results in a file handle greater than 64 bytes. This is shown by:
[2010-08-16 16:25:16.166191] T [nfs-fops.c:697:nfs_fop_mkdir] nfs: Mkdir: /1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22
[2010-08-16 16:25:16.166206] T [nfs-fops.c:129:nfs_create_frame] nfs: uid: 0, gid 0, gids: 1
[2010-08-16 16:25:16.166221] T [nfs-fops.c:131:nfs_create_frame] nfs: gid: 0
[2010-08-16 16:25:16.166252] T [access-control.c:205:ac_test_access] access-control: Root has access
[2010-08-16 16:25:16.166362] D [nfs3-helpers.c:2412:nfs3_log_newfh_res] nfs-nfsv3: XID: 8db37b8c, MKDIR: NFS: 0(Call completed successfully.), POSIX: 0(Success), FH: hashcount 22, xlid 0, gen 5505959064846204950, ino 172970
[2010-08-16 16:25:16.166382] E [nfs3.c:274:nfs3_serialize_reply] nfs-nfsv3: Failed to encode message
[2010-08-16 16:25:16.166398] E [nfs3.c:303:nfs3svc_submit_reply] nfs-nfsv3: Failed to serialize reply


Fix to this is simple but a more worrying is the cs->fd that is pointed to a corrupt memory. This code path gets executed on every mkdir request so it is strange that the corrupted address is found only when the transmission of the mkdir request fails.

Comment 2 Shehjar Tikoo 2010-08-16 11:04:48 UTC
In qa5, creating a 30 level deep directory crashes gnfs.

The command to reproduce over a simple posix export:

$ mkdir -p /mnt/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30

Backtrace:
#0  0x00007f83ca3f0bf1 in nfs3_call_state_wipe (cs=0x7f83c6e93020) at nfs3.c:202
202	                gf_log (GF_NFS3, GF_LOG_TRACE, "fd ref: %d", cs->fd->refcount);
(gdb) p cs
$1 = (nfs3_call_state_t *) 0x7f83c6e93020
(gdb) p cs->fd
$2 = (fd_t *) 0xbce7
(gdb) p *cs->fd
Cannot access memory at address 0xbce7
(gdb) bt
#0  0x00007f83ca3f0bf1 in nfs3_call_state_wipe (cs=0x7f83c6e93020) at nfs3.c:202
#1  0x00007f83ca3f6e51 in nfs3svc_mkdir_cbk (frame=0x7f83cb5041b8, cookie=0x234a0e0, this=0x23490b0, op_ret=0, op_errno=0, inode=0x7f83c87b1c80, buf=0x7f83cd144fa0, 
    preparent=0x7f83cd144f30, postparent=0x7f83cd144ec0) at nfs3.c:2366
#2  0x00007f83ca3eab38 in nfs_inode_mkdir_cbk (frame=0x7f83cb5041b8, cookie=0x234a0e0, this=0x23490b0, op_ret=0, op_errno=0, inode=0x7f83c87b1c80, buf=0x7f83cd144fa0, 
    preparent=0x7f83cd144f30, postparent=0x7f83cd144ec0) at nfs-inodes.c:143
#3  0x00007f83ca3e64e9 in nfs_fop_mkdir_cbk (frame=0x7f83cb5041b8, cookie=0x234a0e0, this=0x23490b0, op_ret=0, op_errno=0, inode=0x7f83c87b1c80, buf=0x7f83cd144fa0, 
    preparent=0x7f83cd144f30, postparent=0x7f83cd144ec0) at nfs-fops.c:679
#4  0x00007f83ccd648bf in default_mkdir_cbk (frame=0x7f83cb7251a0, cookie=0x7f83cb725220, this=0x23490b0, op_ret=0, op_errno=0, inode=0x7f83c87b1c80, buf=0x7f83cd144fa0, 
    preparent=0x7f83cd144f30, postparent=0x7f83cd144ec0) at defaults.c:281
#5  0x00007f83ccd648bf in default_mkdir_cbk (frame=0x7f83cb725220, cookie=0x7f83cb725320, this=0x2347e50, op_ret=0, op_errno=0, inode=0x7f83c87b1c80, buf=0x7f83cd144fa0, 
    preparent=0x7f83cd144f30, postparent=0x7f83cd144ec0) at defaults.c:281
#6  0x00007f83caa57aef in posix_mkdir (frame=0x7f83cb725320, this=0x2346b70, loc=0x7f83cb46d058, mode=493) at posix.c:1434
#7  0x00007f83ca843903 in ac_mkdir_resume (frame=0x7f83cb725220, this=0x2347e50, loc=0x7f83cb46d058, mode=493) at access-control.c:632
#8  0x00007f83ccd7a317 in call_resume_wind (stub=0x7f83cb46d020) at call-stub.c:2165
#9  0x00007f83ccd80cd4 in call_resume (stub=0x7f83cb46d020) at call-stub.c:3862
#10 0x00007f83ca8439d1 in ac_mkdir_stat_cbk (frame=0x7f83cb725220, cookie=0x7f83cb7252a0, this=0x2347e50, op_ret=0, op_errno=0, buf=0x7f83cd145270) at access-control.c:660
#11 0x00007f83caa5480a in posix_stat (frame=0x7f83cb7252a0, this=0x2346b70, loc=0x7f83cd145370) at posix.c:580
#12 0x00007f83ca843e6a in ac_mkdir (frame=0x7f83cb725220, this=0x2347e50, loc=0x7f83c6e93408, mode=493) at access-control.c:693
#13 0x00007f83ccd64af7 in default_mkdir (frame=0x7f83cb7251a0, this=0x23490b0, loc=0x7f83c6e93408, mode=493) at defaults.c:297
#14 0x00007f83ca3e68c4 in nfs_fop_mkdir (nfsx=0x234a0e0, xl=0x23490b0, nfu=0x7f83cd1455f0, pathloc=0x7f83c6e93408, mode=493, cbk=0x7f83ca3eaa4d <nfs_inode_mkdir_cbk>, 
    local=0x7f83cd025020) at nfs-fops.c:702
#15 0x00007f83ca3eac81 in nfs_inode_mkdir (nfsx=0x234a0e0, xl=0x23490b0, nfu=0x7f83cd1455f0, pathloc=0x7f83c6e93408, mode=493, cbk=0x7f83ca3f6b70 <nfs3svc_mkdir_cbk>, 
    local=0x7f83c6e93020) at nfs-inodes.c:163
#16 0x00007f83ca3ec35f in nfs_mkdir (nfsx=0x234a0e0, xl=0x23490b0, nfu=0x7f83cd1455f0, pathloc=0x7f83c6e93408, mode=493, cbk=0x7f83ca3f6b70 <nfs3svc_mkdir_cbk>, 
    local=0x7f83c6e93020) at nfs-generics.c:122
#17 0x00007f83ca3f6f7c in nfs3_mkdir_resume (carg=0x7f83c6e93020) at nfs3.c:2390
#18 0x00007f83ca406073 in nfs3_fh_resolve_entry_lookup_cbk (frame=0x7f83cb5040a8, cookie=0x234a0e0, this=0x23490b0, op_ret=-1, op_errno=2, inode=0x7f83c87b1c80, buf=0x7f83cd145990, 
    xattr=0x0, postparent=0x7f83cd145920) at nfs3-helpers.c:2560
#19 0x00007f83ca3e3988 in nfs_fop_lookup_cbk (frame=0x7f83cb5040a8, cookie=0x234a0e0, this=0x23490b0, op_ret=-1, op_errno=2, inode=0x7f83c87b1c80, buf=0x7f83cd145990, xattr=0x0, 
    postparent=0x7f83cd145920) at nfs-fops.c:260
#20 0x00007f83ca633f3a in pl_lookup_cbk (frame=0x7f83cb725020, cookie=0x7f83cb7250a0, this=0x23490b0, op_ret=-1, op_errno=2, inode=0x7f83c87b1c80, buf=0x7f83cd145990, dict=0x0, 
    postparent=0x7f83cd145920) at posix.c:1123
#21 0x00007f83ccd62fa9 in default_lookup_cbk (frame=0x7f83cb7250a0, cookie=0x7f83cb725120, this=0x2347e50, op_ret=-1, op_errno=2, inode=0x7f83c87b1c80, buf=0x7f83cd145990, 
    dict=0x0, postparent=0x7f83cd145920) at defaults.c:47
#22 0x00007f83caa5436f in posix_lookup (frame=0x7f83cb725120, this=0x2346b70, loc=0x7f83c6e93408, xattr_req=0x0) at posix.c:534
#23 0x00007f83ccd631e1 in default_lookup (frame=0x7f83cb7250a0, this=0x2347e50, loc=0x7f83c6e93408, xattr_req=0x0) at defaults.c:63
#24 0x00007f83ca634368 in pl_lookup (frame=0x7f83cb725020, this=0x23490b0, loc=0x7f83c6e93408, xattr_req=0x0) at posix.c:1165
#25 0x00007f83ca3e3d5c in nfs_fop_lookup (nfsx=0x234a0e0, xl=0x23490b0, nfu=0x7f83cd145c90, loc=0x7f83c6e93408, cbk=0x7f83ca405f40 <nfs3_fh_resolve_entry_lookup_cbk>, 
    local=0x7f83c6e93020) at nfs-fops.c:284
#26 0x00007f83ca3ec1f9 in nfs_lookup (nfsx=0x234a0e0, xl=0x23490b0, nfu=0x7f83cd145c90, pathloc=0x7f83c6e93408, cbk=0x7f83ca405f40 <nfs3_fh_resolve_entry_lookup_cbk>, 
    local=0x7f83c6e93020) at nfs-generics.c:86
#27 0x00007f83ca40751e in nfs3_fh_resolve_entry_hard (cs=0x7f83c6e93020) at nfs3-helpers.c:2949
#28 0x00007f83ca4076ca in nfs3_fh_resolve_entry (cs=0x7f83c6e93020) at nfs3-helpers.c:2995
#29 0x00007f83ca407814 in nfs3_fh_resolve_and_resume (cs=0x7f83c6e93020, fh=0x7f83cd145e50, entry=0x7f83cd145ea0 "22", resum_fn=0x7f83ca3f6e60 <nfs3_mkdir_resume>)
    at nfs3-helpers.c:3029
#30 0x00007f83ca3f73b4 in nfs3_mkdir (req=0x2356bb0, dirfh=0x7f83cd145e50, name=0x7f83cd145ea0 "22", sattr=0x7f83cd145e08) at nfs3.c:2439
#31 0x00007f83ca3f7548 in nfs3svc_mkdir (req=0x2356bb0) at nfs3.c:2473
#32 0x00007f83ca40e66c in nfs_rpcsvc_handle_rpc_call (conn=0x234b610) at ../../../../xlators/nfs/lib/src//rpcsvc.c:1901
#33 0x00007f83ca40f425 in nfs_rpcsvc_record_update_state (conn=0x234b610, dataread=0) at ../../../../xlators/nfs/lib/src//rpcsvc.c:2383
#34 0x00007f83ca40f582 in nfs_rpcsvc_conn_data_poll_in (conn=0x234b610) at ../../../../xlators/nfs/lib/src//rpcsvc.c:2426
#35 0x00007f83ca40f98a in nfs_rpcsvc_conn_data_handler (fd=5, idx=3, data=0x234b610, poll_in=1, poll_out=0, poll_err=0) at ../../../../xlators/nfs/lib/src//rpcsvc.c:2555
#36 0x00007f83ccd847e6 in event_dispatch_epoll_handler (event_pool=0x234b370, events=0x234bb60, i=0) at event.c:812
#37 0x00007f83ccd849da in event_dispatch_epoll (event_pool=0x234b370) at event.c:876
#38 0x00007f83ccd84cb6 in event_dispatch (event_pool=0x234b370) at event.c:984
#39 0x00007f83ca40abc6 in nfs_rpcsvc_stage_proc (arg=0x2342a90) at ../../../../xlators/nfs/lib/src//rpcsvc.c:64
#40 0x00007f83cc4fca04 in start_thread (arg=<value optimized out>) at pthread_create.c:300
#41 0x00007f83cc26680d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#42 0x0000000000000000 in ?? ()
(gdb) p *cs
$3 = {req = 0x2356bb0, vol = 0x23490b0, resume_fn = 0x7f83ca3f6e60 <nfs3_mkdir_resume>, nfsx = 0x234a0e0, nfs3state = 0x7f83c000b5a0, openwait_q = {next = 0x7f83c6e93048, 
    prev = 0x7f83c6e93048}, parent = {ident = ":O", hashcount = 21, xlatorid = 0, gen = 5505959064846204949, ino = 172969, entryhash = {257, 38107, 38104, 38105, 40150, 40151, 
      40148, 40149, 38098, 38099, 38096, 38097, 44270, 44271, 44268, 44269, 46314, 46315, 46312, 46313, 48358}}, fh = {ident = ":O", hashcount = 22, xlatorid = 0, 
    gen = 5505959064846204950, ino = 172970, entryhash = {257, 38107, 38104, 38105, 40150, 40151, 40148, 40149, 38098, 38099, 38096, 38097, 44270, 44271, 44268, 44269, 46314, 
      46315, 46312, 46313, 48358}}, fd = 0xbce7, accessbits = 0, operrno = 22, dircount = 0, maxcount = 0, fsstat = {f_bsize = 0, f_frsize = 0, f_blocks = 0, f_bfree = 0, 
    f_bavail = 0, f_files = 0, f_ffree = 0, f_favail = 0, f_fsid = 0, f_flag = 0, f_namemax = 0, __f_spare = {0, 0, 0, 0, 0, 0}}, entries = {{list = {next = 0x7f83c6e93160, 
        prev = 0x7f83c6e93160}, {next = 0x7f83c6e93160, prev = 0x7f83c6e93160}}, d_ino = 0, d_off = 0, d_len = 0, d_type = 0, d_stat = {ia_ino = 0, ia_gen = 0, ia_dev = 0, 
      ia_type = IA_INVAL, ia_prot = {suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, group = {read = 0 '\000', 
          write = 0 '\000', exec = 0 '\000'}, other = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}}, ia_nlink = 0, ia_uid = 0, ia_gid = 0, ia_rdev = 0, ia_size = 0, 
      ia_blksize = 0, ia_blocks = 0, ia_atime = 0, ia_atime_nsec = 0, ia_mtime = 0, ia_mtime_nsec = 0, ia_ctime = 0, ia_ctime_nsec = 0}, d_name = 0x7f83c6e931f0 ""}, stbuf = {
    ia_ino = 0, ia_gen = 0, ia_dev = 0, ia_type = IA_INVAL, ia_prot = {suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 0 '\000', write = 0 '\000', 
        exec = 0 '\000'}, group = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, other = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}}, ia_nlink = 0, ia_uid = 0, 
    ia_gid = 0, ia_rdev = 0, ia_size = 0, ia_blksize = 0, ia_blocks = 0, ia_atime = 0, ia_atime_nsec = 0, ia_mtime = 0, ia_mtime_nsec = 0, ia_ctime = 0, ia_ctime_nsec = 0}, 
  preparent = {ia_ino = 0, ia_gen = 0, ia_dev = 0, ia_type = IA_INVAL, ia_prot = {suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 0 '\000', write = 0 '\000', 
        exec = 0 '\000'}, group = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, other = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}}, ia_nlink = 0, ia_uid = 0, 
    ia_gid = 0, ia_rdev = 0, ia_size = 0, ia_blksize = 0, ia_blocks = 0, ia_atime = 0, ia_atime_nsec = 0, ia_mtime = 0, ia_mtime_nsec = 0, ia_ctime = 0, ia_ctime_nsec = 0}, 
  postparent = {ia_ino = 0, ia_gen = 0, ia_dev = 0, ia_type = IA_INVAL, ia_prot = {suid = 0 '\000', sgid = 0 '\000', sticky = 0 '\000', owner = {read = 0 '\000', write = 0 '\000', 
        exec = 0 '\000'}, group = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}, other = {read = 0 '\000', write = 0 '\000', exec = 0 '\000'}}, ia_nlink = 0, ia_uid = 0, 
    ia_gid = 0, ia_rdev = 0, ia_size = 0, ia_blksize = 0, ia_blocks = 0, ia_atime = 0, ia_atime_nsec = 0, ia_mtime = 0, ia_mtime_nsec = 0, ia_ctime = 0, ia_ctime_nsec = 0}, 
  setattr_valid = 0, timestamp = {seconds = 0, nseconds = 0}, oploc = {path = 0x0, name = 0x0, ino = 0, inode = 0x0, parent = 0x0}, writetype = 0, datacount = 0, dataoffset = 0, 
  iob = 0x0, createmode = UNCHECKED, cookieverf = 0, sattrguardcheck = 0, pathname = 0x0, mknodtype = 0, devnums = {specdata1 = 0, specdata2 = 0}, cookie = 0, datavec = {
    iov_base = 0x0, iov_len = 0}, mode = 493, resolvefh = {ident = ":O", hashcount = 21, xlatorid = 0, gen = 5505959064846204949, ino = 172969, entryhash = {257, 38107, 38104, 
      38105, 40150, 40151, 40148, 40149, 38098, 38099, 38096, 38097, 44270, 44271, 44268, 44269, 46314, 46315, 46312, 46313, 48358}}, resolvedloc = {
    path = 0x23dc770 "/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22", name = 0x23dc7a7 "22", ino = 0, inode = 0x7f83c87b1c80, parent = 0x7f83c87b1bf0}, 
  resolve_ret = -1, resolve_errno = 2, hashidx = 0, resolve_dir_fd = 0x0, resolventry = 0x23dc830 "22"}

Comment 3 Shehjar Tikoo 2010-08-17 07:53:52 UTC
(In reply to comment #2)

> Fix to this is simple but a more worrying is the cs->fd that is pointed to a
> corrupt memory. This code path gets executed on every mkdir request so it is
> strange that the corrupted address is found only when the transmission of the
> mkdir request fails.


The fd in nfs3_call_state_t is corrupted because when the file handle size is computed to be greater than struct nfs3_fh, it ends up clobbering the fd member that follows the file handle in call state:

        /* Per-NFSv3 Op state */
        struct nfs3_fh          parent;
        struct nfs3_fh          fh;
        fd_t                    *fd;

Comment 4 Vijay Bellur 2010-08-31 11:44:31 UTC
PATCH: http://patches.gluster.com/patch/4419 in master (nfs3: Support hashcounts larger than hash array size)

Comment 5 Vijay Bellur 2010-08-31 11:44:36 UTC
PATCH: http://patches.gluster.com/patch/4420 in master (nfs3: Logging and comments made more accurate)

Comment 6 Vijay Bellur 2010-08-31 11:44:41 UTC
PATCH: http://patches.gluster.com/patch/4421 in master (nfs3: Return ESTALE when going beyond fh-hashcount or max-hashes)

Comment 7 Shehjar Tikoo 2010-09-01 02:40:05 UTC
Regression test:

1. Export a simple posix config through gnfs.

2. Mount at the client using the command:

mount <server>:/posix /mnt

3. Run the following command:
$ mkdir -p
/mnt/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30

4. If the glusterfs daemon crashes, there is a regression.