Bug 762676 (GLUSTER-944)

Summary: vmware unmount crashes nfsx
Product: [Community] GlusterFS Reporter: Shehjar Tikoo <shehjart>
Component: nfsAssignee: Shehjar Tikoo <shehjart>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: nfs-alphaCC: gluster-bugs, lakshmipathi
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTP Mount Type: nfs
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Shehjar Tikoo 2010-05-24 12:33:43 UTC 
Unmount of a posix volume exported through nfsx causes nfsx to crash in the unmount code path.

*** glibc detected *** /usr/local/sbin/glusterfs: free(): invalid pointer: 0x000000000067d4d8 ***
======= Backtrace: =========
/lib/libc.so.6[0x7faf0f4e7dd6]
/lib/libc.so.6(cfree+0x6c)[0x7faf0f4ec74c]
/usr/local/lib/glusterfs/3.0.0git/xlator/nfs/server.so(__mnt3svc_umount+0x15f)[0x7faf0e4087db]
/usr/local/lib/glusterfs/3.0.0git/xlator/nfs/server.so(mnt3svc_umount+0x5e)[0x7faf0e408856]
/usr/local/lib/glusterfs/3.0.0git/xlator/nfs/server.so(mnt3svc_umnt+0x294)[0x7faf0e408b02]
/usr/local/lib/libglrpcsvc.so.0(rpcsvc_handle_rpc_call+0xac)[0x7faf0e1e2ef4]
/usr/local/lib/libglrpcsvc.so.0(rpcsvc_record_update_state+0x231)[0x7faf0e1e3db0]
/usr/local/lib/libglrpcsvc.so.0(rpcsvc_conn_data_poll_in+0xf8)[0x7faf0e1e3f1b]
/usr/local/lib/libglrpcsvc.so.0(rpcsvc_conn_data_handler+0x8c)[0x7faf0e1e435b]
/usr/local/lib/libglusterfs.so.0[0x7faf0fc43736]
/usr/local/lib/libglusterfs.so.0[0x7faf0fc43928]
/usr/local/lib/libglusterfs.so.0(event_dispatch+0x74)[0x7faf0fc43c47]
/usr/local/lib/libglrpcsvc.so.0(rpcsvc_stage_proc+0x32)[0x7faf0e1def62]
/lib/libpthread.so.0[0x7faf0f7e7a04]
/lib/libc.so.6(clone+0x6d)[0x7faf0f55180d]
======= Memory map: ========
00400000-0040a000 r-xp 00000000 08:01 4270081                            /usr/local/sbin/glusterfsd
0060a000-0060b000 r--p 0000a000 08:01 4270081                            /usr/local/sbin/glusterfsd
0060b000-0060c000 rw-p 0000b000 08:01 4270081                            /usr/local/sbin/glusterfsd
00672000-00760000 rw-p 00000000 00:00 0                                  [heap]
7faf08000000-7faf08021000 rw-p 00000000 00:00 0 
7faf08021000-7faf0c000000 ---p 00000000 00:00 0 
7faf0c1b4000-7faf0c1ca000 r-xp 00000000 08:01 2293781                    /lib/libgcc_s.so.1
7faf0c1ca000-7faf0c3c9000 ---p 00016000 08:01 2293781                    /lib/libgcc_s.so.1
7faf0c3c9000-7faf0c3ca000 r--p 00015000 08:01 2293781                    /lib/libgcc_s.so.1
7faf0c3ca000-7faf0c3cb000 rw-p 00016000 08:01 2293781                    /lib/libgcc_s.so.1
7faf0c3cb000-7faf0c3cd000 r-xp 00000000 08:01 2293863                    /lib/libnss_mdns4.so.2
7faf0c3cd000-7faf0c5cc000 ---p 00002000 08:01 2293863                    /lib/libnss_mdns4.so.2
7faf0c5cc000-7faf0c5cd000 r--p 00001000 08:01 2293863                    /lib/libnss_mdns4.so.2
7faf0c5cd000-7faf0c5ce000 rw-p 00002000 08:01 2293863                    /lib/libnss_mdns4.so.2
7faf0c5ce000-7faf0c5e4000 r-xp 00000000 08:01 2293984                    /lib/libresolv-2.10.1.so
7faf0c5e4000-7faf0c7e3000 ---p 00016000 08:01 2293984                    /lib/libresolv-2.10.1.so
7faf0c7e3000-7faf0c7e4000 r--p 00015000 08:01 2293984                    /lib/libresolv-2.10.1.so
7faf0c7e4000-7faf0c7e5000 rw-p 00016000 08:01 2293984                    /lib/libresolv-2.10.1.so
7faf0c7e5000-7faf0c7e7000 rw-p 00000000 00:00 0 
7faf0c7e7000-7faf0c7ec000 r-xp 00000000 08:01 2293977                    /lib/libnss_dns-2.10.1.so
7faf0c7ec000-7faf0c9eb000 ---p 00005000 08:01 2293977                    /lib/libnss_dns-2.10.1.so
7faf0c9eb000-7faf0c9ec000 r--p 00004000 08:01 2293977                    /lib/libnss_dns-2.10.1.so
7faf0c9ec000-7faf0c9ed000 rw-p 00005000 08:01 2293977                    /lib/libnss_dns-2.10.1.so
7faf0c9ed000-7faf0c9ef000 r-xp 00000000 08:01 2293864                    /lib/libnss_mdns4_minimal.so.2
7faf0c9ef000-7faf0cbee000 ---p 00002000 08:01 2293864                    /lib/libnss_mdns4_minimal.so.2
7faf0cbee000-7faf0cbef000 r--p 00001000 08:01 2293864                    /lib/libnss_mdns4_minimal.so.2
7faf0cbef000-7faf0cbf0000 rw-p 00002000 08:01 2293864                    /lib/libnss_mdns4_minimal.so.2
7faf0cbf0000-7faf0cbfc000 r-xp 00000000 08:01 2293978                    /lib/libnss_files-2.10.1.so
7faf0cbfc000-7faf0cdfb000 ---p 0000c000 08:01 2293978                    /lib/libnss_files-2.10.1.so
7faf0cdfb000-7faf0cdfc000 r--p 0000b000 08:01 2293978                    /lib/libnss_files-2.10.1.so
7faf0cdfc000-7faf0cdfd000 rw-p 0000c000 08:01 2293978                    /lib/libnss_files-2.10.1.so
7faf0ce99000-7faf0d15f000 rw-p 00000000 00:00 0 
7faf0d15f000-7faf0d160000 ---p 00000000 00:00 0 
7faf0d160000-7faf0d9ce000 rw-p 00000000 00:00 0 
7faf0d9ce000-7faf0d9cf000 ---p 00000000 00:00 0 
7faf0d9cf000-7faf0e1cf000 rw-p 00000000 00:00 0 
7faf0e1cf000-7faf0e1ec000 r-xp 00000000 08:01 4269581                    /usr/local/lib/libglrpcsvc.so.0.0.0
7faf0e1ec000-7faf0e3eb000 ---p 0001d000 08:01 4269581                    /usr/local/lib/libglrpcsvc.so.0.0.0
7faf0e3eb000-7faf0e3ec000 r--p 0001c000 08:01 4269581                    /usr/local/lib/libglrpcsvc.so.0.0.0
7faf0e3ec000-7faf0e3ed000 rw-p 0001d000 08:01 4269581                    /usr/local/lib/libglrpcsvc.so.0.0.0
7faf0e3ed000-7faf0e42b000 r-xp 00000000 08:01 106896                     /usr/local/lib/glusterfs/3.0.0git/xlator/nfs/server.so.0.0.0
7faf0e42b000-7faf0e62a000 ---p 0003e000 08:01 106896                     /usr/local/lib/glusterfs/3.0.0git/xlator/nfs/server.so.0.0.0
7faf0e62a000-7faf0e62b000 r--p 0003d000 08:01 106896                     /usr/local/lib/glusterfs/3.0.0git/xlator/nfs/server.so.0.0.0
7faf0e62b000-7faf0e630000 rw-p 0003e000 08:01 106896                     /usr/local/lib/glusterfs/3.0.0git/xlator/nfs/server.so.0.0.0
7faf0e630000-7faf0e645000 r-xp 00000000 08:01 108875                     /usr/local/lib/glusterfs/3.0.0git/xlator/features/locks.so.0.0.0
7faf0e645000-7faf0e844000 ---p 00015000 08:01 108875                     /usr/local/lib/glusterfs/3.0.0git/xlator/features/locks.so.0.0.0
7faf0e844000-7faf0e845000 r--p 00014000 08:01 108875                     /usr/local/lib/glusterfs/3.0.0git/xlator/features/locks.so.0.0.0
7faf0e845000-7faf0e846000 rw-p 00015000 08:01 108875                     /usr/local/lib/glusterfs/3.0.0git/xlator/features/locks.so.0.0.0
7faf0e846000-7faf0e854000 r-xp 00000000 08:01 108868                     /usr/local/lib/glusterfs/3.0.0git/xlator/features/access-control.so.0.0.0





The log says:
[2010-05-24 18:00:04] T [rpcsvc-auth.c:247:rpcsvc_auth_request_init] rpc-service: Auth handler: AUTH_UNIX
[2010-05-24 18:00:04] D [rpcsvc.c:1817:rpcsvc_request_create] rpc-service: RPC XID: 54ef1931, Ver: 2, Program: 100005, ProgVers: 3, Proc: 3
[2010-05-24 18:00:04] T [auth-unix.c:66:auth_unix_authenticate] rpc-service: Auth Info: machine name: localhost.localdomain, uid: 0, gid: 0
[2010-05-24 18:00:04] D [rpcsvc.c:1266:rpcsvc_program_actor] rpc-service: Actor found: MOUNT3 - UMNT
[2010-05-24 18:00:09] D [mount3.c:593:mnt3svc_umnt] nfs-mount: dirpath: /posix, hostname: 192.168.1.181
[2010-05-24 18:00:09] D [mount3.c:527:__mnt3svc_umount] nfs-mount: Unmounting: dir , host: 
pending frames:



Looks like an empty hostname string causes the crash. Investigating.
Comment 1 Shehjar Tikoo 2010-05-25 05:32:52 UTC 
nfsx provides users to mount nfs exports using just volume names also. For eg, if there is a subvolume exported through nfsx called posix, then the traditional way to mount it is using the /posix export name with the mount command.

nfsx, in its attempt to be a little fancy, allows users to exclude the / in front of volume names. Unfortunately the mount code path supports such export names, i.e without a / but the unmount code path does not.

The way to reproduce is to mount using "posix" style and the unmount. Just doing that crashes it. The tests till now have all used the /posix as the export name so it never showed up.
Comment 2 Shehjar Tikoo 2010-05-25 05:49:34 UTC 
Ah! I know why it never crashed earlier when I tested it myself. Linux nfs client does not even send the unmount request when it sees that an export has been mounted without a / in front of the export name. So Linux NFS client never hit this code path, even when I tested it myself during development.
Comment 3 Anand Avati 2010-06-01 04:23:57 UTC 
PATCH: http://patches.gluster.com/patch/3356 in master (mount3: Handle <volname> as well for unmount dirpath)
Comment 4 Shehjar Tikoo 2010-06-02 05:56:49 UTC 
Regression Test
See Comment 1 for details.

Test Case
1. Create a posix+nfsx vol file and start glusterfsd.

2. Start vsphere client tool on a windows machine and connect to the ESX server.

3. In vSphere, create a NFS datastore through the Configuration->Storage tab.

4. Suppose the exported volume is called "posix", the traditional way of mounting will be /posix. But, while creating the data store, give the export name only as "posix" instead of /posix.

5. Start up the datastore.

6. Next, right click on the NFS data store and click unmount.

The test is a success if nfsx does not crash at the end of step 6.
Comment 5 Lakshmipathi G 2010-07-29 03:41:04 UTC 
Verified with nfs-beta-rc10.