Hide Forgot
in 3 dht setup,running dbench (100 process) with 3.1.0qa25 ,shows 100 770 9.84 MB/sec warmup 175 sec 100 777 9.81 MB/sec warmup 176 sec 100 787 9.83 MB/sec warmup 177 sec 100 797 9.80 MB/sec warmup 178 sec 100 803 9.78 MB/sec warmup 179 sec 100 809 9.76 MB/sec warmup 180 sec 100 818 9.78 MB/sec warmup 181 sec 100 825 9.77 MB/sec warmup 182 sec Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: Oops: 0000 [#18] Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: SMP Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: CPU: 0 Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: EIP: 0061:[<c1081d9b>] Not tainted VLI Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: EFLAGS: 00210202 (2.6.21.7-2.fc8xen #1) Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: EIP is at iput+0x21/0x62 Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: eax: 706e6920 ebx: ecdc0b64 ecx: e20f88f4 edx: ee21e3dc Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: esi: ecdc0a34 edi: ecdc0b64 ebp: ecdc0af4 esp: e35f0c64 Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0069 Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: Process dbench (pid: 26166, ti=e35f0000 task=e01a2810 task.ti=e35f0000) Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: Stack: c2f31de0 ee1ebf68 0000007f ecdc0aec c2f31dec c2f31dec 0000076c eccf1380 Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: 00000080 000200d2 c1059fc8 ec51c5a0 00000000 dab7b3ec 00076c00 00000000 Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: 00076c00 00062619 00000200 00000000 c12dd5b4 0000000c 00000000 00000080 ======== dmesg output -- ======================= Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:ed11dce8 BUG: unable to handle kernel paging request at virtual address 223d658c printing eip: c1081d9b 20d6b000 -> *pde = 00000002:104b2001 22d62000 -> *pme = 00000000:00000000 Oops: 0000 [#14] SMP last sysfs file: /class/misc/fuse/dev Modules linked in: nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ipv6(U) fuse(U) dm_mirror(U) dm_multipath(U) dm_mod(U) pcspkr(U) ext3(U) jbd(U) mbcache(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) xenblk(U) xennet(U) CPU: 0 EIP: 0061:[<c1081d9b>] Not tainted VLI EFLAGS: 00210206 (2.6.21.7-2.fc8xen #1) EIP is at iput+0x21/0x62 eax: 223d6578 ebx: eacebe24 ecx: ee1ed451 edx: eacebe24 esi: eacebe24 edi: 00000000 ebp: 0000007f esp: e1d72cac ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0069 Process dbench (pid: 26169, ti=e1d72000 task=ebe27310 task.ti=e1d72000) Stack: e28bc114 c108029e ec511df8 c1080cba e28bc114 e28bc114 c1080dde c1328a3c e28bc114 c1080ed9 0001255c c1c61a40 0000008c 000201d2 c1080f2d c1059fc8 00000000 00000000 c1058b2a 01255c00 00000000 01255c00 00061eec 00000200 Call Trace: [<c108029e>] dentry_iput+0x74/0x93 [<c1080cba>] dput+0x31/0xf7 [<c1080dde>] prune_one_dentry+0x3f/0x60 [<c1328a3c>] alloc_large_system_hash+0xea/0x1b4 [<c1080ed9>] prune_dcache+0xda/0x118 [<c1080f2d>] shrink_dcache_memory+0x16/0x2d [<c1059fc8>] shrink_slab+0xd9/0x13e [<c1058b2a>] invalidate_inode_pages2_range+0x262/0x27a [<c105a5b7>] try_to_free_pages+0x16d/0x23e [<c10566b4>] __alloc_pages+0x1a8/0x2aa [<c105420c>] read_cache_page+0x5a/0x142 [<ee1ebfda>] nfs_readdir_filler+0x0/0x165 [nfs] [<ee1ec426>] nfs_readdir+0x29f/0x806 [nfs] [<c107c48d>] filldir+0x0/0xb9 [<ee1ef5db>] nfs_open+0xe6/0xfc [nfs] [<c10e48a8>] copy_to_user+0x3c/0x50 [<c107478f>] cp_new_stat64+0xfc/0x10e [<c10bc266>] file_has_perm+0x89/0x91 [<ee1f94f3>] nfs3_decode_dirent+0x0/0x15f [nfs] [<c107c5a9>] vfs_readdir+0x63/0x8d [<c107c48d>] filldir+0x0/0xb9 [<c107c807>] sys_getdents+0x5f/0x9c [<c1005688>] syscall_call+0x7/0xb [<c1200000>] __sched_text_start+0x268/0x83f ======================= Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:e1d72cac BUG: unable to handle kernel paging request at virtual address 223d658c printing eip: c1081d9b 20e22000 -> *pde = 00000003:7c803001 221bb000 -> *pme = 00000000:00000000 Oops: 0000 [#15] SMP last sysfs file: /class/misc/fuse/dev Modules linked in: nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ipv6(U) fuse(U) dm_mirror(U) dm_multipath(U) dm_mod(U) pcspkr(U) ext3(U) jbd(U) mbcache(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) xenblk(U) xennet(U) CPU: 0 EIP: 0061:[<c1081d9b>] Not tainted VLI EFLAGS: 00210206 (2.6.21.7-2.fc8xen #1) EIP is at iput+0x21/0x62 eax: 223d6578 ebx: eb4016e4 ecx: ee1ed451 edx: eb4016e4 esi: eb4016e4 edi: 00000000 ebp: 00000080 esp: e2babc50 ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0069 Process dbench (pid: 26208, ti=e2bab000 task=e01ab5b0 task.ti=e2bab000) Stack: ebbd22a0 c108029e ee1c43e0 ec5527f4 ebbd22a0 ebbd22a0 c1080dde c1328a3c ebbd22a0 c1080ed9 0001255c c1c61a40 0000008c 000200d2 c1080f2d c1059fc8 ec51c5a0 00000000 ecf979ac 01255c00 00000000 01255c00 00061eb0 00000200 Call Trace: [<c108029e>] dentry_iput+0x74/0x93 [<ee1c43e0>] xs_sendpages+0x194/0x1bf [sunrpc] [<c1080dde>] prune_one_dentry+0x3f/0x60 [<c1328a3c>] alloc_large_system_hash+0xea/0x1b4 [<c1080ed9>] prune_dcache+0xda/0x118 [<c1080f2d>] shrink_dcache_memory+0x16/0x2d [<c1059fc8>] shrink_slab+0xd9/0x13e [<c105a5b7>] try_to_free_pages+0x16d/0x23e [<c10566b4>] __alloc_pages+0x1a8/0x2aa [<c1053609>] generic_file_buffered_write+0x1af/0x61e [<c1058188>] pagevec_lookup_tag+0x24/0x2b [<c10570e1>] generic_writepages+0x275/0x2bd [<c1053f59>] __generic_file_aio_write_nolock+0x4e1/0x55a [<c1054027>] generic_file_aio_write+0x55/0xb3 [<c10bb153>] avc_has_perm+0x4e/0x58 [<ee1ede25>] nfs_file_write+0x125/0x185 [nfs] [<c10bc1d5>] inode_has_perm+0x66/0x6e [<c1071bb1>] do_sync_write+0xc7/0x10a [<c10342a9>] autoremove_wake_function+0x0/0x35 [<c1071aea>] do_sync_write+0x0/0x10a [<c10723f6>] vfs_write+0xa8/0x154 [<c1072ad2>] sys_pwrite64+0x48/0x5f [<c1005688>] syscall_call+0x7/0xb ======================= Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:e2babc50 BUG: unable to handle kernel paging request at virtual address 223d658c printing eip: c1081d9b 22559000 -> *pde = 00000002:15ed6001 223bf000 -> *pme = 00000000:00000000 Oops: 0000 [#16] SMP last sysfs file: /class/misc/fuse/dev Modules linked in: nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ipv6(U) fuse(U) dm_mirror(U) dm_multipath(U) dm_mod(U) pcspkr(U) ext3(U) jbd(U) mbcache(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) xenblk(U) xennet(U) CPU: 0 EIP: 0061:[<c1081d9b>] Not tainted VLI EFLAGS: 00210206 (2.6.21.7-2.fc8xen #1) EIP is at iput+0x21/0x62 eax: 223d6578 ebx: eb8a48e4 ecx: ee1ed451 edx: eb8a48e4 esi: eb8a48e4 edi: 00000000 ebp: 00000080 esp: e23e7c50 ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0069 Process dbench (pid: 26188, ti=e23e7000 task=e107d970 task.ti=e23e7000) Stack: ebfd4f00 c108029e ee1c43e0 ec55245c ebfd4f00 ebfd4f00 c1080dde c1328a3c ebfd4f00 c1080ed9 0001255c c1c61a40 0000008c 000200d2 c1080f2d c1059fc8 ec51c5a0 00000000 ec4f9e6c 01255c00 00000000 01255c00 00061ea6 00000200 Call Trace: [<c108029e>] dentry_iput+0x74/0x93 [<ee1c43e0>] xs_sendpages+0x194/0x1bf [sunrpc] [<c1080dde>] prune_one_dentry+0x3f/0x60 [<c1328a3c>] alloc_large_system_hash+0xea/0x1b4 [<c1080ed9>] prune_dcache+0xda/0x118 [<c1080f2d>] shrink_dcache_memory+0x16/0x2d [<c1059fc8>] shrink_slab+0xd9/0x13e [<c105a5b7>] try_to_free_pages+0x16d/0x23e [<c10566b4>] __alloc_pages+0x1a8/0x2aa [<c1053609>] generic_file_buffered_write+0x1af/0x61e [<c1058188>] pagevec_lookup_tag+0x24/0x2b [<c1004334>] __switch_to+0x3e2/0x4ac [<c1053f59>] __generic_file_aio_write_nolock+0x4e1/0x55a [<c1054027>] generic_file_aio_write+0x55/0xb3 [<c10bb153>] avc_has_perm+0x4e/0x58 [<ee1ede25>] nfs_file_write+0x125/0x185 [nfs] [<c10bc1d5>] inode_has_perm+0x66/0x6e [<c1071bb1>] do_sync_write+0xc7/0x10a [<c10342a9>] autoremove_wake_function+0x0/0x35 [<c1071aea>] do_sync_write+0x0/0x10a [<c10723f6>] vfs_write+0xa8/0x154 [<c1072ad2>] sys_pwrite64+0x48/0x5f [<c1005688>] syscall_call+0x7/0xb ======================= Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:e23e7c50 BUG: unable to handle kernel paging request at virtual address 223d658c printing eip: c1081d9b 22559000 -> *pde = 00000002:15ed6001 223bf000 -> *pme = 00000000:00000000 Oops: 0000 [#17] SMP last sysfs file: /class/misc/fuse/dev Modules linked in: nfs(U) lockd(U) nfs_acl(U) sunrpc(U) ipv6(U) fuse(U) dm_mirror(U) dm_multipath(U) dm_mod(U) pcspkr(U) ext3(U) jbd(U) mbcache(U) uhci_hcd(U) ohci_hcd(U) ehci_hcd(U) xenblk(U) xennet(U) CPU: 0 EIP: 0061:[<c1081d9b>] Not tainted VLI EFLAGS: 00210206 (2.6.21.7-2.fc8xen #1) EIP is at iput+0x21/0x62 eax: 223d6578 ebx: df4031a4 ecx: ee1ed451 edx: df4031a4 esi: df4031a4 edi: 00000000 ebp: 0000007e esp: e3403ce8 ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0069 Process rsyslogd (pid: 27069, ti=e3403000 task=ebe3b490 task.ti=e3403000) Stack: dfbb2744 c108029e ec511df8 c1080cba dfbb2744 dfbb2744 c1080dde c1328a3c dfbb2744 c1080ed9 0001255c c1c61a40 0000008c 000200d2 c1080f2d c1059fc8 00000000 00000001 ebe3b490 01255c00 00000000 01255c00 00061ea7 00000200 Call Trace: [<c108029e>] dentry_iput+0x74/0x93 [<c1080cba>] dput+0x31/0xf7 [<c1080dde>] prune_one_dentry+0x3f/0x60 [<c1328a3c>] alloc_large_system_hash+0xea/0x1b4 [<c1080ed9>] prune_dcache+0xda/0x118 [<c1080f2d>] shrink_dcache_memory+0x16/0x2d [<c1059fc8>] shrink_slab+0xd9/0x13e [<c105a5b7>] try_to_free_pages+0x16d/0x23e [<c10566b4>] __alloc_pages+0x1a8/0x2aa [<c101b71e>] xen_invlpg+0x28/0x36 [<c105caa6>] do_wp_page+0x2cf/0x640 [<c105222f>] find_get_page+0x18/0x54 [<c1054628>] filemap_nopage+0x18b/0x319 [<c1060b2f>] __handle_mm_fault+0x107d/0x1146 [<c12038ae>] do_page_fault+0x7a1/0xc24 [<c120383a>] do_page_fault+0x72d/0xc24 [<c120310d>] do_page_fault+0x0/0xc24 [<c1201eb5>] error_code+0x35/0x3c [<c120310d>] do_page_fault+0x0/0xc24 [<c1201eb5>] error_code+0x35/0x3c [<c1200000>] __sched_text_start+0x268/0x83f ======================= Code: ff ff e8 21 e0 17 00 31 c0 c3 85 c0 53 89 c3 74 59 8b 80 9c 00 00 00 83 bb 38 01 00 00 20 8b 40 20 75 04 0f 0b eb fe 85 c0 74 0b <8b> 50 14 85 d2 74 04 89 d8 ff d2 8d 43 24 ba 00 dd 2d c1 e8 6d EIP: [<c1081d9b>] iput+0x21/0x62 SS:ESP 0069:e3403ce8 [root@ip-10-245-210-193 ~]# Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: Oops: 0000 [#18] Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: SMP Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: CPU: 0 Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: EIP: 0061:[<c1081d9b>] Not tainted VLI Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: EFLAGS: 00210202 (2.6.21.7-2.fc8xen #1) Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: EIP is at iput+0x21/0x62 Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: eax: 706e6920 ebx: ecdc0b64 ecx: e20f88f4 edx: ee21e3dc Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: esi: ecdc0a34 edi: ecdc0b64 ebp: ecdc0af4 esp: e35f0c64 Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0069 Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: Process dbench (pid: 26166, ti=e35f0000 task=e01a2810 task.ti=e35f0000) Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: Stack: c2f31de0 ee1ebf68 0000007f ecdc0aec c2f31dec c2f31dec 0000076c eccf1380 Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: 00000080 000200d2 c1059fc8 ec51c5a0 00000000 dab7b3ec 00076c00 00000000 Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: 00076c00 00062619 00000200 00000000 c12dd5b4 0000000c 00000000 00000080 Message from syslogd@ip-10-245-210-193 at Sep 17 06:49:44 ... kernel: Call Trace
The crash is in the kernel probably due to a buggy kernel or the client's memory allocation requests failing. Nothing to do in nfs.