Bug 54700 - Oops in kernel-2.4.9-0.18enterprise.
Oops in kernel-2.4.9-0.18enterprise.
Status: CLOSED ERRATA
Product: Red Hat Linux
Classification: Retired
Component: kernel (Show other bugs)
7.3
i386 Linux
medium Severity medium
: ---
: ---
Assigned To: Stephen Tweedie
Brock Organ
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2001-10-16 12:40 EDT by Need Real Name
Modified: 2007-04-18 12:37 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2001-11-01 11:04:17 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vmstat, /proc/meminfo prior to oops (553.46 KB, text/plain)
2001-10-16 12:42 EDT, Need Real Name
no flags Details
The sysrq+* snapshots and the sysrqCrash (72.75 KB, application/octet-stream)
2001-10-16 12:45 EDT, Need Real Name
no flags Details

  None (edit)
Description Need Real Name 2001-10-16 12:40:33 EDT
This is a spin off from bug 52340 since this seems to be a different issue 
now.

Description of Problem:
kernel 2.4.9-0.18 enterprise oopses

Version-Release number of selected component (if applicable):
2.4.9-0.18enterprise

How Reproducible:
Takes 3-4 days to reproduce

Steps to Reproduce:
1. Run internal program on box nightly
2. Wait 3 nights
3. 

Actual Results:
Oopses during a large sort operation on 3rd night.

The box is a 8-way highmem machine, 8GB ram, 9GB swap. The same process 
works fine on a 8-way, 8GB RAM, 2GB swap

The oops follows. I have also attached a file showing the vmstat 
and /proc/meminfo right before the crash. Also attached inside the tar.gz 
file is 2 snapshots of sysrq+t, 2 snapshots of sysrq+p, and one snapshot 
of sysrq+m. The sysrq+m caused the system to stop responding to any more 
sysrq commands, and eventually started spitting out the information 
contained in sysrqCrash

Unable to handle kernel NULL pointer dereference at virtual address 
00000018
 printing eip:
c013f587
*pde = 2468d001
*pte = 00000000
Oops: 0000
CPU:    0
EIP:    0010:[<c013f587>]    Tainted: P 
EFLAGS: 00010217
eax: 00000000   ebx: c4f2de7c   ecx: c0292964   edx: c33c434c
esi: c4f2de60   edi: c0292964   ebp: 000a7602   esp: efd75f08
ds: 0018   es: 0018   ss: 0018
Process agent (pid: 20718, stackpage=efd75000)
Stack: 00000040 0000093d 00000001 00000001 000000f0 00000000 000000f0 
c0140201 
       000000f0 00000000 efd74000 00000001 c0140388 000000f0 00000001 
efd74000 
       c0141131 000000f0 00000001 000000f0 0000000b 00000000 00000002 
c01411ef 
Call Trace: [<c0140201>] deactivate_page_Rsmp_f16413ac [] 0x2261 
[<c0140388>] deactivate_page_Rsmp_f16413ac [] 0x23e8 
[<c0141131>] _alloc_pages_Rsmp_686e8739 [] 0x1e1 
[<c01411ef>] __alloc_pages_Rsmp_867830d7 [] 0xf 
[<c0141290>] __get_free_pages_Rsmp_5b3b8f78 [] 0x10 
[<c015d90a>] sys_poll_Rsmp_0cb659f1 [] 0x13a 
[<c010777b>] sys_sigaltstack_Rsmp_ab65536b [] 0xfcb 


Code: f7 40 18 06 00 00 00 75 f0 8b 40 28 39 d0 75 f0 31 d2 85 d2
Comment 1 Need Real Name 2001-10-16 12:42:02 EDT
Created attachment 34201 [details]
vmstat, /proc/meminfo prior to oops
Comment 2 Need Real Name 2001-10-16 12:45:24 EDT
Created attachment 34202 [details]
The sysrq+* snapshots and the sysrqCrash
Comment 3 Arjan van de Ven 2001-10-17 06:50:10 EDT
Can you test this with a 2.4.9-4.x kernel? That will have a working kksymoops..
(or run the oops through ksymoops)
Comment 4 Need Real Name 2001-10-17 09:53:31 EDT
I saw some of the symbols in the dump, and thought that some how, the kernel 
did this for me. Here is the dump. We're going to swap out its memory today, 
since this kernel runs fine on 2 out of 3 machines perfectly, and all the 
machines are close to identical besides the swap.


Error (expand_objects): cannot stat(/lib/megaraid.o) for megaraid
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
ksymoops: No such file or directory
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
ksymoops: No such file or directory
Warning (compare_maps): mismatch on symbol partition_name  , ksyms_base says 
c01dfbf0, System.map says c0174810.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol 
socket_file_ops_R__ver_socket_file_ops not found in System.map.  Ignoring 
ksyms_base entry
Unable to handle kernel NULL pointer dereference at virtual address 00000018
c013f587
*pde = 2468d001
Oops: 0000
CPU:    0
EIP:    0010:[<c013f587>]    Tainted: P
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010217
eax: 00000000   ebx: c4f2de7c   ecx: c0292964   edx: c33c434c
esi: c4f2de60   edi: c0292964   ebp: 000a7602   esp: efd75f08
ds: 0018   es: 0018   ss: 0018
Process agent (pid: 20718, stackpage=efd75000)
Stack: 00000040 0000093d 00000001 00000001 000000f0 00000000 000000f0 c0140201
       000000f0 00000000 efd74000 00000001 c0140388 000000f0 00000001 efd74000
       c0141131 000000f0 00000001 000000f0 0000000b 00000000 00000002 c01411ef
Call Trace: [<c0140201>] deactivate_page_Rsmp_f16413ac [] 0x2261
[<c0140388>] deactivate_page_Rsmp_f16413ac [] 0x23e8
[<c0141131>] _alloc_pages_Rsmp_686e8739 [] 0x1e1
[<c01411ef>] __alloc_pages_Rsmp_867830d7 [] 0xf
[<c0141290>] __get_free_pages_Rsmp_5b3b8f78 [] 0x10
[<c015d90a>] sys_poll_Rsmp_0cb659f1 [] 0x13a
[<c010777b>] sys_sigaltstack_Rsmp_ab65536b [] 0xfcb
Code: f7 40 18 06 00 00 00 75 f0 8b 40 28 39 d0 75 f0 31 d2 85 d2

>>EIP; c013f586 <page_launder+206/a10>   <=====
Trace; c0140200 <do_try_to_free_pages+10/50>
Trace; c0140388 <try_to_free_pages+28/40>
Trace; c0141130 <_wrapped_alloc_pages+1c0/270>
Trace; c01411ee <__alloc_pages+e/a0>
Trace; c0141290 <__get_free_pages+10/20>
Trace; c015d90a <sys_poll+13a/380>
Trace; c010777a <system_call+32/38>
Code;  c013f586 <page_launder+206/a10>
00000000 <_EIP>:
Code;  c013f586 <page_launder+206/a10>   <=====
   0:   f7 40 18 06 00 00 00      testl  $0x6,0x18(%eax)   <=====
Code;  c013f58c <page_launder+20c/a10>
   7:   75 f0                     jne    fffffff9 <_EIP+0xfffffff9> c013f57e 
<page_launder+1fe/a10>
Code;  c013f58e <page_launder+20e/a10>
   9:   8b 40 28                  mov    0x28(%eax),%eax
Code;  c013f592 <page_launder+212/a10>
   c:   39 d0                     cmp    %edx,%eax
Code;  c013f594 <page_launder+214/a10>
   e:   75 f0                     jne    0 <_EIP>
Code;  c013f596 <page_launder+216/a10>
  10:   31 d2 xor    %edx,%edx
Code;  c013f598 <page_launder+218/a10>
  12:   85 d2                     test   %edx,%edx


2 warnings and 3 errors issued.  Results may not be reliable.


Comment 5 Need Real Name 2001-10-19 17:36:49 EDT
One followup. I've upgraded the test box that doesn't crash to 2.4.9-
6enterprise.

Since the only real differences between the box that runs well and the one that 
dies is swap space, I've allocated more swap space to the good box. in the 
dmesg, I get this error.

Adding Swap: 2097128k swap-space (priority -2)
Adding Swap: 2097128k swap-space (priority -3)
Adding Swap: 2096472k swap-space (priority -4)
attempt to access beyond end of device
08:44: rw=0, want=4, limit=1
Unable to find swap-space signature
Adding Swap: 2096440k swap-space (priority -5)
Adding Swap: 2096440k swap-space (priority -6)
Adding Swap: 2096440k swap-space (priority -7)
Adding Swap: 755012k swap-space (priority -8)

The priority -4 device from swapon -s is /dev/sde3, -5 is /dev/sde5
I assumed the 08:44 is major/minor, so

brw-rw----    1 root     disk       8,  44 Mar 23  2001 /dev/sdc12
brw-rw----    1 root     disk       8,  67 Mar 23  2001 /dev/sde3
brw-rw----    1 root     disk       8,  69 Mar 23  2001 /dev/sde5

[root@kerneltest root]# fdisk -l /dev/sdc

Disk /dev/sdc: 64 heads, 32 sectors, 16936 cylinders
Units = cylinders of 2048 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sdc1             1      2101   2151408   83  Linux
/dev/sdc2          2102      4202   2151424   83  Linux
/dev/sdc3          4203      6303   2151424   83  Linux
/dev/sdc4          6304     16936  10888192    5  Extended
/dev/sdc5          6304      8404   2151408   83  Linux
/dev/sdc6          8405     10505   2151408   83  Linux
/dev/sdc7         10506     12606   2151408   83  Linux
/dev/sdc8         12607     14707   2151408   83  Linux
/dev/sdc9         14708     16808   2151408   83  Linux

There isn't even a /dev/sdc12 for it to read past the end of. The message about 
the couldn't find swap signature is random. If I swapoff the drive, and swapon 
it, there will be no message. I checked my history, and I did in fact run a 
mkswap on the device that generated the message.

Is this related in any way? The box that crashes doesn't generate these 
messages.
Comment 6 Need Real Name 2001-10-19 19:07:25 EDT
On the 2.4.9-6enterprise kernel test, I received an oops while running the ltp and 
our app at the same time. The oops is below. This was with the 15GB swap. I'm 
leaving it running now with the original 2GB.

ksymoops 2.4.3 on i686 2.4.9-6enterprise.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.9-6enterprise/ (default)
     -m /boot/System.map-2.4.9-6enterprise (specified)

Error (expand_objects): cannot stat(/lib/megaraid.o) for megaraid
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
Error (pclose_local): find_objects pclose failed 0x100
Warning (compare_maps): mismatch on symbol partition_name  , ksyms_base 
says c01c3b90, System.map says c01626f0.  Ignoring ksyms_base entry
kernel BUG at highmem.c:155!
invalid operand: 0000
CPU:    7
EIP:    0010:[<c013c58f>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010296
eax: 0000001d   ebx: fe051000   ecx: c02fe164   edx: 000068f2
esi: f478d4e0   edi: 00000340   ebp: c8571d94   esp: eb49deac
ds: 0018   es: 0018   ss: 0018
Process writev01 (pid: 24866, stackpage=eb49d000)
Stack: c023b9eb 0000009b 00000000 c0131384 eb49dee8 00000356 
00000000 00000340
       00000016 fffffff2 00000000 00000340 00000000 d25b7ef8 d25b7e40 
00000000
       00000026 40018025 00000286 40018026 00000000 00000016 
eb49df7c 00000000
Call Trace: [<c023b9eb>] Unused_offset [kernel] 0xbfc
[<c0131384>] generic_file_write [kernel] 0x5b4
[<c013e9a6>] do_readv_writev [kernel] 0x1e6
[<c0130dd0>] generic_file_write [kernel] 0x0
[<c01741c2>] tty_write [kernel] 0x1d2
[<c013e2f0>] generic_file_llseek [kernel] 0x0
[<c013eac3>] sys_writev [kernel] 0x43
[<c010719b>] system_call [kernel] 0x33
Code: 0f 0b 58 5a b9 01 00 00 00 ba 03 00 00 00 b8 84 08 30 c0 e8

>>EIP; c013c58e <kunmap_high+5e/80>   <=====
Trace; c023b9ea <Unused_offset+bfc/5712>
Trace; c0131384 <generic_file_write+5b4/660>
Trace; c013e9a6 <do_readv_writev+1e6/260>
Trace; c0130dd0 <generic_file_write+0/660>
Trace; c01741c2 <tty_write+1d2/270>
Trace; c013e2f0 <generic_file_llseek+0/b0>
Trace; c013eac2 <sys_writev+42/60>
Trace; c010719a <system_call+32/38>
Code;  c013c58e <kunmap_high+5e/80>
00000000 <_EIP>:
Code;  c013c58e <kunmap_high+5e/80>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c013c590 <kunmap_high+60/80>
   2:   58                        pop    %eax
Code;  c013c590 <kunmap_high+60/80>
   3:   5a                        pop    %edx
Code;  c013c592 <kunmap_high+62/80>
   4:   b9 01 00 00 00            mov    $0x1,%ecx
Code;  c013c596 <kunmap_high+66/80>
   9:   ba 03 00 00 00            mov    $0x3,%edx
Code;  c013c59c <kunmap_high+6c/80>
   e:   b8 84 08 30 c0            mov    $0xc0300884,%eax
Code;  c013c5a0 <kunmap_high+70/80>
  13:   e8 00 00 00 00            call   18 <_EIP+0x18> c013c5a6 
<kunmap_high+76/80>


1 warning and 4 errors issued.  Results may not be reliable.


Comment 7 Need Real Name 2001-10-20 22:53:17 EDT
This happened again, this time without swap. I believe this is memory related. 
The oops generated the second time is identical to the first. We have swapped 
the memory between the machines, and the problem migrated with the RAM. Unless 
the machine w/ the good ram fails in the next few days, I believe this is a 
hardware related bug.

If nothing happens in the next week, I will close this bug. Sorry about the 
false alarm.
Comment 8 Need Real Name 2001-10-21 18:15:20 EDT
I visited the box, and swapped memory boards and ram sticks around. The second 
Oops I posted(BUG in highmem.c:155) is reproducable by the linux test project, 
specifically the writev01 test in the syscalls section. This doesn't oops in 
2.4.13pre5.

The first oops is still probably hardware related, but it will take a while 
running on different DIMM modules to verify. Once again, I'll post more here as 
I find it.
Comment 9 Stephen Tweedie 2001-10-24 11:23:17 EDT
The kunmap_high() BUG seems to be a side effect of a change in the 2.4 mainline
kernels, backported to our tree via the ext3 patches.

The problem seems to be doing a write() from an illegal user address.  That goes
down the fail_write: label in generic_file_write(), which used to have to do a
kunmap() because the normal kunmap in a_ops->commit_write would be skipped in
that case.

The new code calls a_ops->commit_write unconditionally, even if an error occurs,
so the kunmap() gets done for us and the second kunmap() in generic_file_write
causes the BUG trap.

The quick-and-dirty fix is to remove that second kunmap().  The _right_ fix is
probably to remove the kmap and kunmap pairs from prepare_write and commit_write
entirely, and to have the callers call kmap beforehand.  That will make it _far_
easier to verify the correctness of the code, as well as removing extra
redundant code from every filesystem with its own *_write methods.
Comment 10 Stephen Tweedie 2001-10-24 14:10:42 EDT
Fix is in CVS.  Do you want us to build a snapshot of the tree with the fix
applied for your own tests?
Comment 11 Need Real Name 2001-10-24 14:22:36 EDT
That would be good.
Comment 12 Need Real Name 2001-11-01 10:07:03 EST
We hit an oops after running 2.4.9-0.18 for about 14 days. The questionable 
memory has been swapped out with known good memory. We have not upgraded to 
2.4.9-6 yet. I have also captured the sysrq info if its needed/helpful.

ksymoops 2.4.3 on i686 2.4.9-0.18enterprise.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.9-0.18enterprise/ (default)
     -m /boot/System.map-2.4.9-0.18enterprise (specified)

Error (expand_objects): cannot stat(/lib/qla2x00.o) for qla2x00
Error (expand_objects): cannot stat(/lib/megaraid.o) for megaraid
Error (expand_objects): cannot stat(/lib/sd_mod.o) for sd_mod
Error (expand_objects): cannot stat(/lib/scsi_mod.o) for scsi_mod
Error (pclose_local): find_objects pclose failed 0x100
Warning (compare_maps): mismatch on symbol partition_name  , ksyms_base says 
c01dfbf0, System.map says c0174
810.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol 
socket_file_ops_R__ver_socket_file_ops not found in System.map.  I
gnoring ksyms_base entry
Oops: 0000
CPU:    4
EIP:    0010:[<c013f587>]    Tainted: P
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010207
eax: 00000000   ebx: c4a6f974   ecx: c0292964   edx: db191eb0
esi: c4a6f958   edi: c0292964   ebp: 000564ae   esp: d0c27d90
ds: 0018   es: 0018   ss: 0018
Process db-unload (pid: 3662, stackpage=d0c27000)
Stack: 00000007 000000cf 00000001 00000001 000000d2 00000000 000000d2 c0140201
       000000d2 00000000 d0c26000 00000001 c0140388 000000d2 00000001 d0c26000
       c0141131 000000d2 00000001 000000d2 d056f450 00000000 c7e66c30 c01411ef
Call Trace: [<c0140201>] deactivate_page_Rsmp_f16413ac [] 0x2261
[<c0140388>] deactivate_page_Rsmp_f16413ac [] 0x23e8
[<c0141131>] _alloc_pages_Rsmp_686e8739 [] 0x1e1
[<c01411ef>] __alloc_pages_Rsmp_867830d7 [] 0xf
[<c01198e0>] do_BUG_Rsmp_577f4bff [] 0x110
[<c0136e9e>] filemap_nopage_Rsmp_ecbf8c86 [] 0x12e
[<c01574fc>] path_walk_Rsmp_931832bf [] 0xafc
[<c01198e0>] do_BUG_Rsmp_577f4bff [] 0x110
[<c0131a34>] vmtruncate_Rsmp_4f09e0fd [] 0x9e4
[<c01198e0>] do_BUG_Rsmp_577f4bff [] 0x110
[<c0131c0b>] vmtruncate_Rsmp_4f09e0fd [] 0xbbb
[<c015b01c>] vfs_follow_link_Rsmp_98ff6cea [] 0x17c
[<c0160de5>] dput_Rsmp_9946dfc7 [] 0x35
[<c01198e0>] do_BUG_Rsmp_577f4bff [] 0x110
[<c0119b16>] do_BUG_Rsmp_577f4bff [] 0x346
[<c0136867>] do_generic_file_read_Rsmp_051d766f [] 0x867
[<c016565a>] update_atime_Rsmp_82f08dd4 [] 0x4a
[<c013678f>] do_generic_file_read_Rsmp_051d766f [] 0x78f
[<c0160de5>] dput_Rsmp_9946dfc7 [] 0x35
[<c014a7a7>] fput_Rsmp_25a50b3a [] 0x77
[<c01291c2>] del_timer_sync_Rsmp_a0201047 [] 0xb02
[<c0114f69>] smp_call_function_Rsmp_0014bfd1 [] 0x929
[<c01198e0>] do_BUG_Rsmp_577f4bff [] 0x110
[<c0107878>] sys_sigaltstack_Rsmp_ab65536b [] 0x10c8
Code: f7 40 18 06 00 00 00 75 f0 8b 40 28 39 d0 75 f0 31 d2 85 d2


>>EIP; c013f586 <page_launder+206/a10>   <=====
Trace; c0140200 <do_try_to_free_pages+10/50>
Trace; c0140388 <try_to_free_pages+28/40>
Trace; c0141130 <_wrapped_alloc_pages+1c0/270>
Trace; c01411ee <__alloc_pages+e/a0>
Trace; c01198e0 <do_page_fault+0/5d0>
Trace; c0136e9e <filemap_nopage+12e/570>
Trace; c01574fc <path_walk+afc/bf0>
Trace; c01198e0 <do_page_fault+0/5d0>
Trace; c0131a34 <do_no_page+c4/1d0>
Trace; c01198e0 <do_page_fault+0/5d0>
Trace; c0131c0a <handle_mm_fault+ca/1c0>
Trace; c015b01c <vfs_follow_link+17c/1f0>
Trace; c0160de4 <dput+34/230>
Trace; c01198e0 <do_page_fault+0/5d0>
Trace; c0119b16 <do_page_fault+236/5d0>
Trace; c0136866 <file_read_actor+c6/100>
Trace; c016565a <update_atime+4a/50>
Trace; c013678e <do_generic_file_read+78e/7a0>
Trace; c0160de4 <dput+34/230>
Trace; c014a7a6 <fput+76/140>
Trace; c01291c2 <run_local_timers+a2/1a0>
Trace; c0114f68 <smp_apic_timer_interrupt+118/130>
Trace; c01198e0 <do_page_fault+0/5d0>
Trace; c0107878 <error_code+38/40>
Code;  c013f586 <page_launder+206/a10>
00000000 <_EIP>:
Code;  c013f586 <page_launder+206/a10>   <=====
   0:   f7 40 18 06 00 00 00      testl  $0x6,0x18(%eax)   <=====
Code;  c013f58c <page_launder+20c/a10>
   7:   75 f0                     jne    fffffff9 <_EIP+0xfffffff9> c013f57e 
<page_launder+1fe/a10>
Code;  c013f58e <page_launder+20e/a10>
   9:   8b 40 28                  mov    0x28(%eax),%eax
Code;  c013f592 <page_launder+212/a10>
   c:   39 d0                     cmp    %edx,%eax
Code;  c013f594 <page_launder+214/a10>
   e:   75 f0                     jne    0 <_EIP>
Code;  c013f596 <page_launder+216/a10>
  10:   31 d2                     xor    %edx,%edx
Code;  c013f598 <page_launder+218/a10>
  12:   85 d2                     test   %edx,%edx


2 warnings and 5 errors issued.  Results may not be reliable.
Comment 13 Stephen Tweedie 2001-11-01 11:04:11 EST
We are building a new kernel for testing and will make it available for you
tomorrow.  The kunmap() BUG should be fixed, but the page_launder oops you just
added looks like a totally different footprint: could you please add that as a
new bugzilla entry to let us track it once we've closed the kunmap() bug?
Comment 14 Stephen Tweedie 2001-11-02 14:29:16 EST
We have just pushed out a 2.4.9-13 build as an errata.  It contains a security
fix which we did not want to preannounce, hence the silence about this build
before today!

I have been able to reproduce the "BUG in highmem.c:155" problem, and it should
be fixed in this errata.


Note You need to log in before you can comment on or make changes to this bug.