Bug 763364 - (GLUSTER-1632) dbench fails in 3 dht setup
dbench fails in 3 dht setup
Status: CLOSED NOTABUG
Product: GlusterFS
Classification: Community
Component: nfs (Show other bugs)
3.1-alpha
All Linux
low Severity medium
: ---
: ---
Assigned To: Shehjar Tikoo
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-09-17 06:55 EDT by Lakshmipathi G
Modified: 2015-12-01 11:45 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: ---
Regression: RTP
Mount Type: nfs
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Lakshmipathi G 2010-09-17 06:55:52 EDT
in 3 dht setup,running dbench (100 process) with 3.1.0qa25 ,shows 

 100       770     9.84 MB/sec  warmup 175 sec   
 100       777     9.81 MB/sec  warmup 176 sec   
 100       787     9.83 MB/sec  warmup 177 sec   
 100       797     9.80 MB/sec  warmup 178 sec   
 100       803     9.78 MB/sec  warmup 179 sec   
 100       809     9.76 MB/sec  warmup 180 sec   
 100       818     9.78 MB/sec  warmup 181 sec   
 100       825     9.77 MB/sec  warmup 182 sec   

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Oops: 0000 [#18]

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: SMP 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: CPU:    0

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: EIP:    0061:[<c1081d9b>]    Not tainted VLI

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: EFLAGS: 00210202   (2.6.21.7-2.fc8xen #1)

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: EIP is at iput+0x21/0x62

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: eax: 706e6920   ebx: ecdc0b64   ecx: e20f88f4   edx: ee21e3dc

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: esi: ecdc0a34   edi: ecdc0b64   ebp: ecdc0af4   esp: e35f0c64

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Process dbench (pid: 26166, ti=e35f0000 task=e01a2810 task.ti=e35f0000)

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Stack: c2f31de0 ee1ebf68 0000007f ecdc0aec c2f31dec c2f31dec 0000076c eccf1380 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel:        00000080 000200d2 c1059fc8 ec51c5a0 00000000 dab7b3ec 00076c00 00000000 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel:        00076c00 00062619 00000200 00000000 c12dd5b4 0000000c 00000000 00000080

========
dmesg output
--
=======================
Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d 
EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:ed11dce8
BUG: unable to handle kernel paging request at virtual address 223d658c
 printing eip:
c1081d9b
20d6b000 -> *pde = 00000002:104b2001
22d62000 -> *pme = 00000000:00000000
Oops: 0000 [#14]
SMP 
last sysfs file: /class/misc/fuse/dev
Modules linked in: nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ipv6(U) fuse(U) dm_mirror(U) dm_multipath(U) dm_mod(U) pcspkr(U) ext3(U) jbd(U) mbcache(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) xenblk(U) xennet(U)
CPU:    0
EIP:    0061:[<c1081d9b>]    Not tainted VLI
EFLAGS: 00210206   (2.6.21.7-2.fc8xen #1)
EIP is at iput+0x21/0x62
eax: 223d6578   ebx: eacebe24   ecx: ee1ed451   edx: eacebe24
esi: eacebe24   edi: 00000000   ebp: 0000007f   esp: e1d72cac
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069
Process dbench (pid: 26169, ti=e1d72000 task=ebe27310 task.ti=e1d72000)
Stack: e28bc114 c108029e ec511df8 c1080cba e28bc114 e28bc114 c1080dde c1328a3c 
       e28bc114 c1080ed9 0001255c c1c61a40 0000008c 000201d2 c1080f2d c1059fc8 
       00000000 00000000 c1058b2a 01255c00 00000000 01255c00 00061eec 00000200 
Call Trace:
 [<c108029e>] dentry_iput+0x74/0x93
 [<c1080cba>] dput+0x31/0xf7
 [<c1080dde>] prune_one_dentry+0x3f/0x60
 [<c1328a3c>] alloc_large_system_hash+0xea/0x1b4
 [<c1080ed9>] prune_dcache+0xda/0x118
 [<c1080f2d>] shrink_dcache_memory+0x16/0x2d
 [<c1059fc8>] shrink_slab+0xd9/0x13e
 [<c1058b2a>] invalidate_inode_pages2_range+0x262/0x27a
 [<c105a5b7>] try_to_free_pages+0x16d/0x23e
 [<c10566b4>] __alloc_pages+0x1a8/0x2aa
 [<c105420c>] read_cache_page+0x5a/0x142
 [<ee1ebfda>] nfs_readdir_filler+0x0/0x165 [nfs]
 [<ee1ec426>] nfs_readdir+0x29f/0x806 [nfs]
 [<c107c48d>] filldir+0x0/0xb9
 [<ee1ef5db>] nfs_open+0xe6/0xfc [nfs]
 [<c10e48a8>] copy_to_user+0x3c/0x50
 [<c107478f>] cp_new_stat64+0xfc/0x10e
 [<c10bc266>] file_has_perm+0x89/0x91
 [<ee1f94f3>] nfs3_decode_dirent+0x0/0x15f [nfs]
 [<c107c5a9>] vfs_readdir+0x63/0x8d
 [<c107c48d>] filldir+0x0/0xb9
 [<c107c807>] sys_getdents+0x5f/0x9c
 [<c1005688>] syscall_call+0x7/0xb
 [<c1200000>] __sched_text_start+0x268/0x83f
 =======================
Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d 
EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:e1d72cac
BUG: unable to handle kernel paging request at virtual address 223d658c
 printing eip:
c1081d9b
20e22000 -> *pde = 00000003:7c803001
221bb000 -> *pme = 00000000:00000000
Oops: 0000 [#15]
SMP 
last sysfs file: /class/misc/fuse/dev
Modules linked in: nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ipv6(U) fuse(U) dm_mirror(U) dm_multipath(U) dm_mod(U) pcspkr(U) ext3(U) jbd(U) mbcache(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) xenblk(U) xennet(U)
CPU:    0
EIP:    0061:[<c1081d9b>]    Not tainted VLI
EFLAGS: 00210206   (2.6.21.7-2.fc8xen #1)
EIP is at iput+0x21/0x62
eax: 223d6578   ebx: eb4016e4   ecx: ee1ed451   edx: eb4016e4
esi: eb4016e4   edi: 00000000   ebp: 00000080   esp: e2babc50
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069
Process dbench (pid: 26208, ti=e2bab000 task=e01ab5b0 task.ti=e2bab000)
Stack: ebbd22a0 c108029e ee1c43e0 ec5527f4 ebbd22a0 ebbd22a0 c1080dde c1328a3c 
       ebbd22a0 c1080ed9 0001255c c1c61a40 0000008c 000200d2 c1080f2d c1059fc8 
       ec51c5a0 00000000 ecf979ac 01255c00 00000000 01255c00 00061eb0 00000200 
Call Trace:
 [<c108029e>] dentry_iput+0x74/0x93
 [<ee1c43e0>] xs_sendpages+0x194/0x1bf [sunrpc]
 [<c1080dde>] prune_one_dentry+0x3f/0x60
 [<c1328a3c>] alloc_large_system_hash+0xea/0x1b4
 [<c1080ed9>] prune_dcache+0xda/0x118
 [<c1080f2d>] shrink_dcache_memory+0x16/0x2d
 [<c1059fc8>] shrink_slab+0xd9/0x13e
 [<c105a5b7>] try_to_free_pages+0x16d/0x23e
 [<c10566b4>] __alloc_pages+0x1a8/0x2aa
 [<c1053609>] generic_file_buffered_write+0x1af/0x61e
 [<c1058188>] pagevec_lookup_tag+0x24/0x2b
 [<c10570e1>] generic_writepages+0x275/0x2bd
 [<c1053f59>] __generic_file_aio_write_nolock+0x4e1/0x55a
 [<c1054027>] generic_file_aio_write+0x55/0xb3
 [<c10bb153>] avc_has_perm+0x4e/0x58
 [<ee1ede25>] nfs_file_write+0x125/0x185 [nfs]
 [<c10bc1d5>] inode_has_perm+0x66/0x6e
 [<c1071bb1>] do_sync_write+0xc7/0x10a
 [<c10342a9>] autoremove_wake_function+0x0/0x35
 [<c1071aea>] do_sync_write+0x0/0x10a
 [<c10723f6>] vfs_write+0xa8/0x154
 [<c1072ad2>] sys_pwrite64+0x48/0x5f
 [<c1005688>] syscall_call+0x7/0xb
 =======================
Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d 
EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:e2babc50
BUG: unable to handle kernel paging request at virtual address 223d658c
 printing eip:
c1081d9b
22559000 -> *pde = 00000002:15ed6001
223bf000 -> *pme = 00000000:00000000
Oops: 0000 [#16]
SMP 
last sysfs file: /class/misc/fuse/dev
Modules linked in: nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ipv6(U) fuse(U) dm_mirror(U) dm_multipath(U) dm_mod(U) pcspkr(U) ext3(U) jbd(U) mbcache(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) xenblk(U) xennet(U)
CPU:    0
EIP:    0061:[<c1081d9b>]    Not tainted VLI
EFLAGS: 00210206   (2.6.21.7-2.fc8xen #1)
EIP is at iput+0x21/0x62
eax: 223d6578   ebx: eb8a48e4   ecx: ee1ed451   edx: eb8a48e4
esi: eb8a48e4   edi: 00000000   ebp: 00000080   esp: e23e7c50
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069
Process dbench (pid: 26188, ti=e23e7000 task=e107d970 task.ti=e23e7000)
Stack: ebfd4f00 c108029e ee1c43e0 ec55245c ebfd4f00 ebfd4f00 c1080dde c1328a3c 
       ebfd4f00 c1080ed9 0001255c c1c61a40 0000008c 000200d2 c1080f2d c1059fc8 
       ec51c5a0 00000000 ec4f9e6c 01255c00 00000000 01255c00 00061ea6 00000200 
Call Trace:
 [<c108029e>] dentry_iput+0x74/0x93
 [<ee1c43e0>] xs_sendpages+0x194/0x1bf [sunrpc]
 [<c1080dde>] prune_one_dentry+0x3f/0x60
 [<c1328a3c>] alloc_large_system_hash+0xea/0x1b4
 [<c1080ed9>] prune_dcache+0xda/0x118
 [<c1080f2d>] shrink_dcache_memory+0x16/0x2d
 [<c1059fc8>] shrink_slab+0xd9/0x13e
 [<c105a5b7>] try_to_free_pages+0x16d/0x23e
 [<c10566b4>] __alloc_pages+0x1a8/0x2aa
 [<c1053609>] generic_file_buffered_write+0x1af/0x61e
 [<c1058188>] pagevec_lookup_tag+0x24/0x2b
 [<c1004334>] __switch_to+0x3e2/0x4ac
 [<c1053f59>] __generic_file_aio_write_nolock+0x4e1/0x55a
 [<c1054027>] generic_file_aio_write+0x55/0xb3
 [<c10bb153>] avc_has_perm+0x4e/0x58
 [<ee1ede25>] nfs_file_write+0x125/0x185 [nfs]
 [<c10bc1d5>] inode_has_perm+0x66/0x6e
 [<c1071bb1>] do_sync_write+0xc7/0x10a
 [<c10342a9>] autoremove_wake_function+0x0/0x35
 [<c1071aea>] do_sync_write+0x0/0x10a
 [<c10723f6>] vfs_write+0xa8/0x154
 [<c1072ad2>] sys_pwrite64+0x48/0x5f
 [<c1005688>] syscall_call+0x7/0xb
 =======================
Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d 
EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:e23e7c50
BUG: unable to handle kernel paging request at virtual address 223d658c
 printing eip:
c1081d9b
22559000 -> *pde = 00000002:15ed6001
223bf000 -> *pme = 00000000:00000000
Oops: 0000 [#17]
SMP 
last sysfs file: /class/misc/fuse/dev
Modules linked in: nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ipv6(U) fuse(U) dm_mirror(U) dm_multipath(U) dm_mod(U) pcspkr(U) ext3(U) jbd(U) mbcache(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) xenblk(U) xennet(U)
CPU:    0
EIP:    0061:[<c1081d9b>]    Not tainted VLI
EFLAGS: 00210206   (2.6.21.7-2.fc8xen #1)
EIP is at iput+0x21/0x62
eax: 223d6578   ebx: df4031a4   ecx: ee1ed451   edx: df4031a4
esi: df4031a4   edi: 00000000   ebp: 0000007e   esp: e3403ce8
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069
Process rsyslogd (pid: 27069, ti=e3403000 task=ebe3b490 task.ti=e3403000)
Stack: dfbb2744 c108029e ec511df8 c1080cba dfbb2744 dfbb2744 c1080dde c1328a3c 
       dfbb2744 c1080ed9 0001255c c1c61a40 0000008c 000200d2 c1080f2d c1059fc8 
       00000000 00000001 ebe3b490 01255c00 00000000 01255c00 00061ea7 00000200 
Call Trace:
 [<c108029e>] dentry_iput+0x74/0x93
 [<c1080cba>] dput+0x31/0xf7
 [<c1080dde>] prune_one_dentry+0x3f/0x60
 [<c1328a3c>] alloc_large_system_hash+0xea/0x1b4
 [<c1080ed9>] prune_dcache+0xda/0x118
 [<c1080f2d>] shrink_dcache_memory+0x16/0x2d
 [<c1059fc8>] shrink_slab+0xd9/0x13e
 [<c105a5b7>] try_to_free_pages+0x16d/0x23e
 [<c10566b4>] __alloc_pages+0x1a8/0x2aa
 [<c101b71e>] xen_invlpg+0x28/0x36
 [<c105caa6>] do_wp_page+0x2cf/0x640
 [<c105222f>] find_get_page+0x18/0x54
 [<c1054628>] filemap_nopage+0x18b/0x319
 [<c1060b2f>] __handle_mm_fault+0x107d/0x1146
 [<c12038ae>] do_page_fault+0x7a1/0xc24
 [<c120383a>] do_page_fault+0x72d/0xc24
 [<c120310d>] do_page_fault+0x0/0xc24
 [<c1201eb5>] error_code+0x35/0x3c
 [<c120310d>] do_page_fault+0x0/0xc24
 [<c1201eb5>] error_code+0x35/0x3c
 [<c1200000>] __sched_text_start+0x268/0x83f
 =======================
Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d 
EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:e3403ce8
[root@ip-10-245-210-193 ~]# 
Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Oops: 0000 [#18]

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: SMP 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: CPU:    0

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: EIP:    0061:[<c1081d9b>]    Not tainted VLI

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: EFLAGS: 00210202   (2.6.21.7-2.fc8xen #1)

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: EIP is at iput+0x21/0x62

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: eax: 706e6920   ebx: ecdc0b64   ecx: e20f88f4   edx: ee21e3dc

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: esi: ecdc0a34   edi: ecdc0b64   ebp: ecdc0af4   esp: e35f0c64

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0069

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Process dbench (pid: 26166, ti=e35f0000 task=e01a2810 task.ti=e35f0000)

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Stack: c2f31de0 ee1ebf68 0000007f ecdc0aec c2f31dec c2f31dec 0000076c eccf1380 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel:        00000080 000200d2 c1059fc8 ec51c5a0 00000000 dab7b3ec 00076c00 00000000 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel:        00076c00 00062619 00000200 00000000 c12dd5b4 0000000c 00000000 00000080 

Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ...
 kernel: Call Trace
Comment 1 Shehjar Tikoo 2010-09-28 04:02:44 EDT
The crash is in the kernel probably due to a buggy kernel or the client's memory allocation requests failing. Nothing to do in nfs.

Note You need to log in before you can comment on or make changes to this bug.