Bug 763364 (GLUSTER-1632)

Summary: dbench fails in 3 dht setup
Product: [Community] GlusterFS Reporter: Lakshmipathi G <lakshmipathi>
Component: nfsAssignee: Shehjar Tikoo <shehjart>
Status: CLOSED NOTABUG QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: 3.1-alphaCC: gluster-bugs, vijay
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTP Mount Type: nfs
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:

Description Lakshmipathi G 2010-09-17 06:55:52 EDT
in 3 dht setup,running dbench (100 process) with 3.1.0qa25 ,shows 

 100       770     9.84 MB/sec  warmup 175 sec   
 100       777     9.81 MB/sec  warmup 176 sec   
 100       787     9.83 MB/sec  warmup 177 sec   
 100       797     9.80 MB/sec  warmup 178 sec   
 100       803     9.78 MB/sec  warmup 179 sec   
 100       809     9.76 MB/sec  warmup 180 sec   
 100       818     9.78 MB/sec  warmup 181 sec   
 100       825     9.77 MB/sec  warmup 182 sec   

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Oops: 0000 [#18]

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: SMP 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: CPU:    0

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: EIP:    0061:[<c1081d9b>]    Not tainted VLI

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: EFLAGS: 00210202   (2.6.21.7-2.fc8xen #1)

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: EIP is at iput+0x21/0x62

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: eax: 706e6920   ebx: ecdc0b64   ecx: e20f88f4   edx: ee21e3dc

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: esi: ecdc0a34   edi: ecdc0b64   ebp: ecdc0af4   esp: e35f0c64

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Process dbench (pid: 26166, ti=e35f0000 task=e01a2810 task.ti=e35f0000)

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Stack: c2f31de0 ee1ebf68 0000007f ecdc0aec c2f31dec c2f31dec 0000076c eccf1380 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel:        00000080 000200d2 c1059fc8 ec51c5a0 00000000 dab7b3ec 00076c00 00000000 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel:        00076c00 00062619 00000200 00000000 c12dd5b4 0000000c 00000000 00000080

========
dmesg output
--
=======================
Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d 
EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:ed11dce8
BUG: unable to handle kernel paging request at virtual address 223d658c
 printing eip:
c1081d9b
20d6b000 -> *pde = 00000002:104b2001
22d62000 -> *pme = 00000000:00000000
Oops: 0000 [#14]
SMP 
last sysfs file: /class/misc/fuse/dev
Modules linked in: nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ipv6(U) fuse(U) dm_mirror(U) dm_multipath(U) dm_mod(U) pcspkr(U) ext3(U) jbd(U) mbcache(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) xenblk(U) xennet(U)
CPU:    0
EIP:    0061:[<c1081d9b>]    Not tainted VLI
EFLAGS: 00210206   (2.6.21.7-2.fc8xen #1)
EIP is at iput+0x21/0x62
eax: 223d6578   ebx: eacebe24   ecx: ee1ed451   edx: eacebe24
esi: eacebe24   edi: 00000000   ebp: 0000007f   esp: e1d72cac
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069
Process dbench (pid: 26169, ti=e1d72000 task=ebe27310 task.ti=e1d72000)
Stack: e28bc114 c108029e ec511df8 c1080cba e28bc114 e28bc114 c1080dde c1328a3c 
       e28bc114 c1080ed9 0001255c c1c61a40 0000008c 000201d2 c1080f2d c1059fc8 
       00000000 00000000 c1058b2a 01255c00 00000000 01255c00 00061eec 00000200 
Call Trace:
 [<c108029e>] dentry_iput+0x74/0x93
 [<c1080cba>] dput+0x31/0xf7
 [<c1080dde>] prune_one_dentry+0x3f/0x60
 [<c1328a3c>] alloc_large_system_hash+0xea/0x1b4
 [<c1080ed9>] prune_dcache+0xda/0x118
 [<c1080f2d>] shrink_dcache_memory+0x16/0x2d
 [<c1059fc8>] shrink_slab+0xd9/0x13e
 [<c1058b2a>] invalidate_inode_pages2_range+0x262/0x27a
 [<c105a5b7>] try_to_free_pages+0x16d/0x23e
 [<c10566b4>] __alloc_pages+0x1a8/0x2aa
 [<c105420c>] read_cache_page+0x5a/0x142
 [<ee1ebfda>] nfs_readdir_filler+0x0/0x165 [nfs]
 [<ee1ec426>] nfs_readdir+0x29f/0x806 [nfs]
 [<c107c48d>] filldir+0x0/0xb9
 [<ee1ef5db>] nfs_open+0xe6/0xfc [nfs]
 [<c10e48a8>] copy_to_user+0x3c/0x50
 [<c107478f>] cp_new_stat64+0xfc/0x10e
 [<c10bc266>] file_has_perm+0x89/0x91
 [<ee1f94f3>] nfs3_decode_dirent+0x0/0x15f [nfs]
 [<c107c5a9>] vfs_readdir+0x63/0x8d
 [<c107c48d>] filldir+0x0/0xb9
 [<c107c807>] sys_getdents+0x5f/0x9c
 [<c1005688>] syscall_call+0x7/0xb
 [<c1200000>] __sched_text_start+0x268/0x83f
 =======================
Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d 
EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:e1d72cac
BUG: unable to handle kernel paging request at virtual address 223d658c
 printing eip:
c1081d9b
20e22000 -> *pde = 00000003:7c803001
221bb000 -> *pme = 00000000:00000000
Oops: 0000 [#15]
SMP 
last sysfs file: /class/misc/fuse/dev
Modules linked in: nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ipv6(U) fuse(U) dm_mirror(U) dm_multipath(U) dm_mod(U) pcspkr(U) ext3(U) jbd(U) mbcache(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) xenblk(U) xennet(U)
CPU:    0
EIP:    0061:[<c1081d9b>]    Not tainted VLI
EFLAGS: 00210206   (2.6.21.7-2.fc8xen #1)
EIP is at iput+0x21/0x62
eax: 223d6578   ebx: eb4016e4   ecx: ee1ed451   edx: eb4016e4
esi: eb4016e4   edi: 00000000   ebp: 00000080   esp: e2babc50
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069
Process dbench (pid: 26208, ti=e2bab000 task=e01ab5b0 task.ti=e2bab000)
Stack: ebbd22a0 c108029e ee1c43e0 ec5527f4 ebbd22a0 ebbd22a0 c1080dde c1328a3c 
       ebbd22a0 c1080ed9 0001255c c1c61a40 0000008c 000200d2 c1080f2d c1059fc8 
       ec51c5a0 00000000 ecf979ac 01255c00 00000000 01255c00 00061eb0 00000200 
Call Trace:
 [<c108029e>] dentry_iput+0x74/0x93
 [<ee1c43e0>] xs_sendpages+0x194/0x1bf [sunrpc]
 [<c1080dde>] prune_one_dentry+0x3f/0x60
 [<c1328a3c>] alloc_large_system_hash+0xea/0x1b4
 [<c1080ed9>] prune_dcache+0xda/0x118
 [<c1080f2d>] shrink_dcache_memory+0x16/0x2d
 [<c1059fc8>] shrink_slab+0xd9/0x13e
 [<c105a5b7>] try_to_free_pages+0x16d/0x23e
 [<c10566b4>] __alloc_pages+0x1a8/0x2aa
 [<c1053609>] generic_file_buffered_write+0x1af/0x61e
 [<c1058188>] pagevec_lookup_tag+0x24/0x2b
 [<c10570e1>] generic_writepages+0x275/0x2bd
 [<c1053f59>] __generic_file_aio_write_nolock+0x4e1/0x55a
 [<c1054027>] generic_file_aio_write+0x55/0xb3
 [<c10bb153>] avc_has_perm+0x4e/0x58
 [<ee1ede25>] nfs_file_write+0x125/0x185 [nfs]
 [<c10bc1d5>] inode_has_perm+0x66/0x6e
 [<c1071bb1>] do_sync_write+0xc7/0x10a
 [<c10342a9>] autoremove_wake_function+0x0/0x35
 [<c1071aea>] do_sync_write+0x0/0x10a
 [<c10723f6>] vfs_write+0xa8/0x154
 [<c1072ad2>] sys_pwrite64+0x48/0x5f
 [<c1005688>] syscall_call+0x7/0xb
 =======================
Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d 
EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:e2babc50
BUG: unable to handle kernel paging request at virtual address 223d658c
 printing eip:
c1081d9b
22559000 -> *pde = 00000002:15ed6001
223bf000 -> *pme = 00000000:00000000
Oops: 0000 [#16]
SMP 
last sysfs file: /class/misc/fuse/dev
Modules linked in: nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ipv6(U) fuse(U) dm_mirror(U) dm_multipath(U) dm_mod(U) pcspkr(U) ext3(U) jbd(U) mbcache(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) xenblk(U) xennet(U)
CPU:    0
EIP:    0061:[<c1081d9b>]    Not tainted VLI
EFLAGS: 00210206   (2.6.21.7-2.fc8xen #1)
EIP is at iput+0x21/0x62
eax: 223d6578   ebx: eb8a48e4   ecx: ee1ed451   edx: eb8a48e4
esi: eb8a48e4   edi: 00000000   ebp: 00000080   esp: e23e7c50
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069
Process dbench (pid: 26188, ti=e23e7000 task=e107d970 task.ti=e23e7000)
Stack: ebfd4f00 c108029e ee1c43e0 ec55245c ebfd4f00 ebfd4f00 c1080dde c1328a3c 
       ebfd4f00 c1080ed9 0001255c c1c61a40 0000008c 000200d2 c1080f2d c1059fc8 
       ec51c5a0 00000000 ec4f9e6c 01255c00 00000000 01255c00 00061ea6 00000200 
Call Trace:
 [<c108029e>] dentry_iput+0x74/0x93
 [<ee1c43e0>] xs_sendpages+0x194/0x1bf [sunrpc]
 [<c1080dde>] prune_one_dentry+0x3f/0x60
 [<c1328a3c>] alloc_large_system_hash+0xea/0x1b4
 [<c1080ed9>] prune_dcache+0xda/0x118
 [<c1080f2d>] shrink_dcache_memory+0x16/0x2d
 [<c1059fc8>] shrink_slab+0xd9/0x13e
 [<c105a5b7>] try_to_free_pages+0x16d/0x23e
 [<c10566b4>] __alloc_pages+0x1a8/0x2aa
 [<c1053609>] generic_file_buffered_write+0x1af/0x61e
 [<c1058188>] pagevec_lookup_tag+0x24/0x2b
 [<c1004334>] __switch_to+0x3e2/0x4ac
 [<c1053f59>] __generic_file_aio_write_nolock+0x4e1/0x55a
 [<c1054027>] generic_file_aio_write+0x55/0xb3
 [<c10bb153>] avc_has_perm+0x4e/0x58
 [<ee1ede25>] nfs_file_write+0x125/0x185 [nfs]
 [<c10bc1d5>] inode_has_perm+0x66/0x6e
 [<c1071bb1>] do_sync_write+0xc7/0x10a
 [<c10342a9>] autoremove_wake_function+0x0/0x35
 [<c1071aea>] do_sync_write+0x0/0x10a
 [<c10723f6>] vfs_write+0xa8/0x154
 [<c1072ad2>] sys_pwrite64+0x48/0x5f
 [<c1005688>] syscall_call+0x7/0xb
 =======================
Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d 
EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:e23e7c50
BUG: unable to handle kernel paging request at virtual address 223d658c
 printing eip:
c1081d9b
22559000 -> *pde = 00000002:15ed6001
223bf000 -> *pme = 00000000:00000000
Oops: 0000 [#17]
SMP 
last sysfs file: /class/misc/fuse/dev
Modules linked in: nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ipv6(U) fuse(U) dm_mirror(U) dm_multipath(U) dm_mod(U) pcspkr(U) ext3(U) jbd(U) mbcache(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) xenblk(U) xennet(U)
CPU:    0
EIP:    0061:[<c1081d9b>]    Not tainted VLI
EFLAGS: 00210206   (2.6.21.7-2.fc8xen #1)
EIP is at iput+0x21/0x62
eax: 223d6578   ebx: df4031a4   ecx: ee1ed451   edx: df4031a4
esi: df4031a4   edi: 00000000   ebp: 0000007e   esp: e3403ce8
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069
Process rsyslogd (pid: 27069, ti=e3403000 task=ebe3b490 task.ti=e3403000)
Stack: dfbb2744 c108029e ec511df8 c1080cba dfbb2744 dfbb2744 c1080dde c1328a3c 
       dfbb2744 c1080ed9 0001255c c1c61a40 0000008c 000200d2 c1080f2d c1059fc8 
       00000000 00000001 ebe3b490 01255c00 00000000 01255c00 00061ea7 00000200 
Call Trace:
 [<c108029e>] dentry_iput+0x74/0x93
 [<c1080cba>] dput+0x31/0xf7
 [<c1080dde>] prune_one_dentry+0x3f/0x60
 [<c1328a3c>] alloc_large_system_hash+0xea/0x1b4
 [<c1080ed9>] prune_dcache+0xda/0x118
 [<c1080f2d>] shrink_dcache_memory+0x16/0x2d
 [<c1059fc8>] shrink_slab+0xd9/0x13e
 [<c105a5b7>] try_to_free_pages+0x16d/0x23e
 [<c10566b4>] __alloc_pages+0x1a8/0x2aa
 [<c101b71e>] xen_invlpg+0x28/0x36
 [<c105caa6>] do_wp_page+0x2cf/0x640
 [<c105222f>] find_get_page+0x18/0x54
 [<c1054628>] filemap_nopage+0x18b/0x319
 [<c1060b2f>] __handle_mm_fault+0x107d/0x1146
 [<c12038ae>] do_page_fault+0x7a1/0xc24
 [<c120383a>] do_page_fault+0x72d/0xc24
 [<c120310d>] do_page_fault+0x0/0xc24
 [<c1201eb5>] error_code+0x35/0x3c
 [<c120310d>] do_page_fault+0x0/0xc24
 [<c1201eb5>] error_code+0x35/0x3c
 [<c1200000>] __sched_text_start+0x268/0x83f
 =======================
Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d 
EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:e3403ce8
[root@ip-10-245-210-193 ~]# 
Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Oops: 0000 [#18]

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: SMP 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: CPU:    0

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: EIP:    0061:[<c1081d9b>]    Not tainted VLI

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: EFLAGS: 00210202   (2.6.21.7-2.fc8xen #1)

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: EIP is at iput+0x21/0x62

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: eax: 706e6920   ebx: ecdc0b64   ecx: e20f88f4   edx: ee21e3dc

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: esi: ecdc0a34   edi: ecdc0b64   ebp: ecdc0af4   esp: e35f0c64

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Process dbench (pid: 26166, ti=e35f0000 task=e01a2810 task.ti=e35f0000)

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Stack: c2f31de0 ee1ebf68 0000007f ecdc0aec c2f31dec c2f31dec 0000076c eccf1380 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel:        00000080 000200d2 c1059fc8 ec51c5a0 00000000 dab7b3ec 00076c00 00000000 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel:        00076c00 00062619 00000200 00000000 c12dd5b4 0000000c 00000000 00000080 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Call Trace
Comment 1 Shehjar Tikoo 2010-09-28 04:02:44 EDT
The crash is in the kernel probably due to a buggy kernel or the client's memory allocation requests failing. Nothing to do in nfs.