Bug 54700
| Summary: | Oops in kernel-2.4.9-0.18enterprise. | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Retired] Red Hat Linux | Reporter: | Need Real Name <robert_macaulay> | ||||||
| Component: | kernel | Assignee: | Stephen Tweedie <sct> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Brock Organ <borgan> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | medium | ||||||||
| Version: | 7.3 | CC: | michael_e_brown | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | i386 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2001-11-01 16:04:17 UTC | Type: | --- | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
Need Real Name
2001-10-16 16:40:33 UTC
Created attachment 34201 [details]
vmstat, /proc/meminfo prior to oops
Created attachment 34202 [details]
The sysrq+* snapshots and the sysrqCrash
Can you test this with a 2.4.9-4.x kernel? That will have a working kksymoops.. (or run the oops through ksymoops) I saw some of the symbols in the dump, and thought that some how, the kernel
did this for me. Here is the dump. We're going to swap out its memory today,
since this kernel runs fine on 2 out of 3 machines perfectly, and all the
machines are close to identical besides the swap.
Error (expand_objects): cannot stat(/lib/megaraid.o) for megaraid
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
ksymoops: No such file or directory
Warning (compare_maps): mismatch on symbol partition_name , ksyms_base says
c01dfbf0, System.map says c0174810. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol
socket_file_ops_R__ver_socket_file_ops not found in System.map. Ignoring
ksyms_base entry
Unable to handle kernel NULL pointer dereference at virtual address 00000018
c013f587
*pde = 2468d001
Oops: 0000
CPU: 0
EIP: 0010:[<c013f587>] Tainted: P
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010217
eax: 00000000 ebx: c4f2de7c ecx: c0292964 edx: c33c434c
esi: c4f2de60 edi: c0292964 ebp: 000a7602 esp: efd75f08
ds: 0018 es: 0018 ss: 0018
Process agent (pid: 20718, stackpage=efd75000)
Stack: 00000040 0000093d 00000001 00000001 000000f0 00000000 000000f0 c0140201
000000f0 00000000 efd74000 00000001 c0140388 000000f0 00000001 efd74000
c0141131 000000f0 00000001 000000f0 0000000b 00000000 00000002 c01411ef
Call Trace: [<c0140201>] deactivate_page_Rsmp_f16413ac [] 0x2261
[<c0140388>] deactivate_page_Rsmp_f16413ac [] 0x23e8
[<c0141131>] _alloc_pages_Rsmp_686e8739 [] 0x1e1
[<c01411ef>] __alloc_pages_Rsmp_867830d7 [] 0xf
[<c0141290>] __get_free_pages_Rsmp_5b3b8f78 [] 0x10
[<c015d90a>] sys_poll_Rsmp_0cb659f1 [] 0x13a
[<c010777b>] sys_sigaltstack_Rsmp_ab65536b [] 0xfcb
Code: f7 40 18 06 00 00 00 75 f0 8b 40 28 39 d0 75 f0 31 d2 85 d2
>>EIP; c013f586 <page_launder+206/a10> <=====
Trace; c0140200 <do_try_to_free_pages+10/50>
Trace; c0140388 <try_to_free_pages+28/40>
Trace; c0141130 <_wrapped_alloc_pages+1c0/270>
Trace; c01411ee <__alloc_pages+e/a0>
Trace; c0141290 <__get_free_pages+10/20>
Trace; c015d90a <sys_poll+13a/380>
Trace; c010777a <system_call+32/38>
Code; c013f586 <page_launder+206/a10>
00000000 <_EIP>:
Code; c013f586 <page_launder+206/a10> <=====
0: f7 40 18 06 00 00 00 testl $0x6,0x18(%eax) <=====
Code; c013f58c <page_launder+20c/a10>
7: 75 f0 jne fffffff9 <_EIP+0xfffffff9> c013f57e
<page_launder+1fe/a10>
Code; c013f58e <page_launder+20e/a10>
9: 8b 40 28 mov 0x28(%eax),%eax
Code; c013f592 <page_launder+212/a10>
c: 39 d0 cmp %edx,%eax
Code; c013f594 <page_launder+214/a10>
e: 75 f0 jne 0 <_EIP>
Code; c013f596 <page_launder+216/a10>
10: 31 d2 xor %edx,%edx
Code; c013f598 <page_launder+218/a10>
12: 85 d2 test %edx,%edx
2 warnings and 3 errors issued. Results may not be reliable.
One followup. I've upgraded the test box that doesn't crash to 2.4.9- 6enterprise. Since the only real differences between the box that runs well and the one that dies is swap space, I've allocated more swap space to the good box. in the dmesg, I get this error. Adding Swap: 2097128k swap-space (priority -2) Adding Swap: 2097128k swap-space (priority -3) Adding Swap: 2096472k swap-space (priority -4) attempt to access beyond end of device 08:44: rw=0, want=4, limit=1 Unable to find swap-space signature Adding Swap: 2096440k swap-space (priority -5) Adding Swap: 2096440k swap-space (priority -6) Adding Swap: 2096440k swap-space (priority -7) Adding Swap: 755012k swap-space (priority -8) The priority -4 device from swapon -s is /dev/sde3, -5 is /dev/sde5 I assumed the 08:44 is major/minor, so brw-rw---- 1 root disk 8, 44 Mar 23 2001 /dev/sdc12 brw-rw---- 1 root disk 8, 67 Mar 23 2001 /dev/sde3 brw-rw---- 1 root disk 8, 69 Mar 23 2001 /dev/sde5 [root@kerneltest root]# fdisk -l /dev/sdc Disk /dev/sdc: 64 heads, 32 sectors, 16936 cylinders Units = cylinders of 2048 * 512 bytes Device Boot Start End Blocks Id System /dev/sdc1 1 2101 2151408 83 Linux /dev/sdc2 2102 4202 2151424 83 Linux /dev/sdc3 4203 6303 2151424 83 Linux /dev/sdc4 6304 16936 10888192 5 Extended /dev/sdc5 6304 8404 2151408 83 Linux /dev/sdc6 8405 10505 2151408 83 Linux /dev/sdc7 10506 12606 2151408 83 Linux /dev/sdc8 12607 14707 2151408 83 Linux /dev/sdc9 14708 16808 2151408 83 Linux There isn't even a /dev/sdc12 for it to read past the end of. The message about the couldn't find swap signature is random. If I swapoff the drive, and swapon it, there will be no message. I checked my history, and I did in fact run a mkswap on the device that generated the message. Is this related in any way? The box that crashes doesn't generate these messages. On the 2.4.9-6enterprise kernel test, I received an oops while running the ltp and
our app at the same time. The oops is below. This was with the 15GB swap. I'm
leaving it running now with the original 2GB.
ksymoops 2.4.3 on i686 2.4.9-6enterprise. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.9-6enterprise/ (default)
-m /boot/System.map-2.4.9-6enterprise (specified)
Error (expand_objects): cannot stat(/lib/megaraid.o) for megaraid
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
Error (pclose_local): find_objects pclose failed 0x100
Warning (compare_maps): mismatch on symbol partition_name , ksyms_base
says c01c3b90, System.map says c01626f0. Ignoring ksyms_base entry
kernel BUG at highmem.c:155!
invalid operand: 0000
CPU: 7
EIP: 0010:[<c013c58f>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010296
eax: 0000001d ebx: fe051000 ecx: c02fe164 edx: 000068f2
esi: f478d4e0 edi: 00000340 ebp: c8571d94 esp: eb49deac
ds: 0018 es: 0018 ss: 0018
Process writev01 (pid: 24866, stackpage=eb49d000)
Stack: c023b9eb 0000009b 00000000 c0131384 eb49dee8 00000356
00000000 00000340
00000016 fffffff2 00000000 00000340 00000000 d25b7ef8 d25b7e40
00000000
00000026 40018025 00000286 40018026 00000000 00000016
eb49df7c 00000000
Call Trace: [<c023b9eb>] Unused_offset [kernel] 0xbfc
[<c0131384>] generic_file_write [kernel] 0x5b4
[<c013e9a6>] do_readv_writev [kernel] 0x1e6
[<c0130dd0>] generic_file_write [kernel] 0x0
[<c01741c2>] tty_write [kernel] 0x1d2
[<c013e2f0>] generic_file_llseek [kernel] 0x0
[<c013eac3>] sys_writev [kernel] 0x43
[<c010719b>] system_call [kernel] 0x33
Code: 0f 0b 58 5a b9 01 00 00 00 ba 03 00 00 00 b8 84 08 30 c0 e8
>>EIP; c013c58e <kunmap_high+5e/80> <=====
Trace; c023b9ea <Unused_offset+bfc/5712>
Trace; c0131384 <generic_file_write+5b4/660>
Trace; c013e9a6 <do_readv_writev+1e6/260>
Trace; c0130dd0 <generic_file_write+0/660>
Trace; c01741c2 <tty_write+1d2/270>
Trace; c013e2f0 <generic_file_llseek+0/b0>
Trace; c013eac2 <sys_writev+42/60>
Trace; c010719a <system_call+32/38>
Code; c013c58e <kunmap_high+5e/80>
00000000 <_EIP>:
Code; c013c58e <kunmap_high+5e/80> <=====
0: 0f 0b ud2a <=====
Code; c013c590 <kunmap_high+60/80>
2: 58 pop %eax
Code; c013c590 <kunmap_high+60/80>
3: 5a pop %edx
Code; c013c592 <kunmap_high+62/80>
4: b9 01 00 00 00 mov $0x1,%ecx
Code; c013c596 <kunmap_high+66/80>
9: ba 03 00 00 00 mov $0x3,%edx
Code; c013c59c <kunmap_high+6c/80>
e: b8 84 08 30 c0 mov $0xc0300884,%eax
Code; c013c5a0 <kunmap_high+70/80>
13: e8 00 00 00 00 call 18 <_EIP+0x18> c013c5a6
<kunmap_high+76/80>
1 warning and 4 errors issued. Results may not be reliable.
This happened again, this time without swap. I believe this is memory related. The oops generated the second time is identical to the first. We have swapped the memory between the machines, and the problem migrated with the RAM. Unless the machine w/ the good ram fails in the next few days, I believe this is a hardware related bug. If nothing happens in the next week, I will close this bug. Sorry about the false alarm. I visited the box, and swapped memory boards and ram sticks around. The second Oops I posted(BUG in highmem.c:155) is reproducable by the linux test project, specifically the writev01 test in the syscalls section. This doesn't oops in 2.4.13pre5. The first oops is still probably hardware related, but it will take a while running on different DIMM modules to verify. Once again, I'll post more here as I find it. The kunmap_high() BUG seems to be a side effect of a change in the 2.4 mainline kernels, backported to our tree via the ext3 patches. The problem seems to be doing a write() from an illegal user address. That goes down the fail_write: label in generic_file_write(), which used to have to do a kunmap() because the normal kunmap in a_ops->commit_write would be skipped in that case. The new code calls a_ops->commit_write unconditionally, even if an error occurs, so the kunmap() gets done for us and the second kunmap() in generic_file_write causes the BUG trap. The quick-and-dirty fix is to remove that second kunmap(). The _right_ fix is probably to remove the kmap and kunmap pairs from prepare_write and commit_write entirely, and to have the callers call kmap beforehand. That will make it _far_ easier to verify the correctness of the code, as well as removing extra redundant code from every filesystem with its own *_write methods. Fix is in CVS. Do you want us to build a snapshot of the tree with the fix applied for your own tests? That would be good. We hit an oops after running 2.4.9-0.18 for about 14 days. The questionable
memory has been swapped out with known good memory. We have not upgraded to
2.4.9-6 yet. I have also captured the sysrq info if its needed/helpful.
ksymoops 2.4.3 on i686 2.4.9-0.18enterprise. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.9-0.18enterprise/ (default)
-m /boot/System.map-2.4.9-0.18enterprise (specified)
Error (expand_objects): cannot stat(/lib/qla2x00.o) for qla2x00
Error (expand_objects): cannot stat(/lib/megaraid.o) for megaraid
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
Error (pclose_local): find_objects pclose failed 0x100
Warning (compare_maps): mismatch on symbol partition_name , ksyms_base says
c01dfbf0, System.map says c0174
810. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol
socket_file_ops_R__ver_socket_file_ops not found in System.map. I
gnoring ksyms_base entry
Oops: 0000
CPU: 4
EIP: 0010:[<c013f587>] Tainted: P
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010207
eax: 00000000 ebx: c4a6f974 ecx: c0292964 edx: db191eb0
esi: c4a6f958 edi: c0292964 ebp: 000564ae esp: d0c27d90
ds: 0018 es: 0018 ss: 0018
Process db-unload (pid: 3662, stackpage=d0c27000)
Stack: 00000007 000000cf 00000001 00000001 000000d2 00000000 000000d2 c0140201
000000d2 00000000 d0c26000 00000001 c0140388 000000d2 00000001 d0c26000
c0141131 000000d2 00000001 000000d2 d056f450 00000000 c7e66c30 c01411ef
Call Trace: [<c0140201>] deactivate_page_Rsmp_f16413ac [] 0x2261
[<c0140388>] deactivate_page_Rsmp_f16413ac [] 0x23e8
[<c0141131>] _alloc_pages_Rsmp_686e8739 [] 0x1e1
[<c01411ef>] __alloc_pages_Rsmp_867830d7 [] 0xf
[<c01198e0>] do_BUG_Rsmp_577f4bff [] 0x110
[<c0136e9e>] filemap_nopage_Rsmp_ecbf8c86 [] 0x12e
[<c01574fc>] path_walk_Rsmp_931832bf [] 0xafc
[<c01198e0>] do_BUG_Rsmp_577f4bff [] 0x110
[<c0131a34>] vmtruncate_Rsmp_4f09e0fd [] 0x9e4
[<c01198e0>] do_BUG_Rsmp_577f4bff [] 0x110
[<c0131c0b>] vmtruncate_Rsmp_4f09e0fd [] 0xbbb
[<c015b01c>] vfs_follow_link_Rsmp_98ff6cea [] 0x17c
[<c0160de5>] dput_Rsmp_9946dfc7 [] 0x35
[<c01198e0>] do_BUG_Rsmp_577f4bff [] 0x110
[<c0119b16>] do_BUG_Rsmp_577f4bff [] 0x346
[<c0136867>] do_generic_file_read_Rsmp_051d766f [] 0x867
[<c016565a>] update_atime_Rsmp_82f08dd4 [] 0x4a
[<c013678f>] do_generic_file_read_Rsmp_051d766f [] 0x78f
[<c0160de5>] dput_Rsmp_9946dfc7 [] 0x35
[<c014a7a7>] fput_Rsmp_25a50b3a [] 0x77
[<c01291c2>] del_timer_sync_Rsmp_a0201047 [] 0xb02
[<c0114f69>] smp_call_function_Rsmp_0014bfd1 [] 0x929
[<c01198e0>] do_BUG_Rsmp_577f4bff [] 0x110
[<c0107878>] sys_sigaltstack_Rsmp_ab65536b [] 0x10c8
Code: f7 40 18 06 00 00 00 75 f0 8b 40 28 39 d0 75 f0 31 d2 85 d2
>>EIP; c013f586 <page_launder+206/a10> <=====
Trace; c0140200 <do_try_to_free_pages+10/50>
Trace; c0140388 <try_to_free_pages+28/40>
Trace; c0141130 <_wrapped_alloc_pages+1c0/270>
Trace; c01411ee <__alloc_pages+e/a0>
Trace; c01198e0 <do_page_fault+0/5d0>
Trace; c0136e9e <filemap_nopage+12e/570>
Trace; c01574fc <path_walk+afc/bf0>
Trace; c01198e0 <do_page_fault+0/5d0>
Trace; c0131a34 <do_no_page+c4/1d0>
Trace; c01198e0 <do_page_fault+0/5d0>
Trace; c0131c0a <handle_mm_fault+ca/1c0>
Trace; c015b01c <vfs_follow_link+17c/1f0>
Trace; c0160de4 <dput+34/230>
Trace; c01198e0 <do_page_fault+0/5d0>
Trace; c0119b16 <do_page_fault+236/5d0>
Trace; c0136866 <file_read_actor+c6/100>
Trace; c016565a <update_atime+4a/50>
Trace; c013678e <do_generic_file_read+78e/7a0>
Trace; c0160de4 <dput+34/230>
Trace; c014a7a6 <fput+76/140>
Trace; c01291c2 <run_local_timers+a2/1a0>
Trace; c0114f68 <smp_apic_timer_interrupt+118/130>
Trace; c01198e0 <do_page_fault+0/5d0>
Trace; c0107878 <error_code+38/40>
Code; c013f586 <page_launder+206/a10>
00000000 <_EIP>:
Code; c013f586 <page_launder+206/a10> <=====
0: f7 40 18 06 00 00 00 testl $0x6,0x18(%eax) <=====
Code; c013f58c <page_launder+20c/a10>
7: 75 f0 jne fffffff9 <_EIP+0xfffffff9> c013f57e
<page_launder+1fe/a10>
Code; c013f58e <page_launder+20e/a10>
9: 8b 40 28 mov 0x28(%eax),%eax
Code; c013f592 <page_launder+212/a10>
c: 39 d0 cmp %edx,%eax
Code; c013f594 <page_launder+214/a10>
e: 75 f0 jne 0 <_EIP>
Code; c013f596 <page_launder+216/a10>
10: 31 d2 xor %edx,%edx
Code; c013f598 <page_launder+218/a10>
12: 85 d2 test %edx,%edx
2 warnings and 5 errors issued. Results may not be reliable.
We are building a new kernel for testing and will make it available for you tomorrow. The kunmap() BUG should be fixed, but the page_launder oops you just added looks like a totally different footprint: could you please add that as a new bugzilla entry to let us track it once we've closed the kunmap() bug? We have just pushed out a 2.4.9-13 build as an errata. It contains a security fix which we did not want to preannounce, hence the silence about this build before today! I have been able to reproduce the "BUG in highmem.c:155" problem, and it should be fixed in this errata. |