Bug 143732 - RAID1 chokes under heavy IO
RAID1 chokes under heavy IO
Status: CLOSED NOTABUG
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
3
x86_64 Linux
medium Severity high
: ---
: ---
Assigned To: Dave Jones
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2004-12-25 22:28 EST by Karl Grindley
Modified: 2015-01-04 17:14 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-02-28 10:50:31 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Karl Grindley 2004-12-25 22:28:03 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5)
Gecko/20041107 Firefox/1.0

Description of problem:

Running a simple RAID1 configuration on 2 IDE devices.  When booted
into 2.6.9-1.681_FC3, any heavy IO, (although writing seems to cause
more problems) causes a kernel panic, then endless errors to console.

Version-Release number of selected component (if applicable):
2.6.9-1.681_FC3

How reproducible:
Always

Steps to Reproduce:
1. mdadm --create -l 1 -n 2 /dev/md0 /dev/hdc1 /dev/hdd1
2. mkfs.ext3/mount /dev/md0 /raid
3. cd /raid; tar -xvpf bigtarfile.tar 
    

Actual Results:  after a few mins, kernel dumps LOTS of data to
console (too fast to read) then endlessly dumps below error to the
console and keyboard lights flash.
"atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly."

Expected Results:  complete writing to the raid device without issue.

Additional info:

Worth noting, this happens even in single user mode (ie, no X running).

Have not tested to see if this is a problem on i386 build.

The above results occur about 90% of the time.  I've also encountered
complete lockups, and sometimes massive amounts of memeory being
consumed until the kernel starts killing off processes, and eventually
freezes.
Comment 1 Karl Grindley 2004-12-25 22:29:25 EST
not sure if this is obvious in the ticket or not,  but experencing
this problem on the x86_64 platform.
Comment 2 Karl Grindley 2004-12-27 23:02:58 EST
so, i've tracked this problem down to the libata kernel module.  I do
not have any SATA devices, but just with the module loaded in the
kernel, panics soon follow.

panics are less in 2.6.9-1.667, but still happen.  Definately a huge
problem in 2.6.9-1.681_FC3.
Comment 3 Dave Jones 2004-12-28 01:35:07 EST
any chance you can capture and paste one of the panic messages ?
Comment 4 Karl Grindley 2005-01-02 10:10:45 EST
Finally caught some kernel messages...  Seem to be catching mostly all
Oops messages.

login: Unable to handle kernel paging request at 000000fff40561d4 RIP:
<ffffffff80166f88>{free_pages_bulk+575}
PML4 0
Oops: 0002 [1]
CPU 0
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack i
p_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm s
nd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi
snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext
3 jbd raid1 sata_via libata sd_mod scsi_mod
Pid: 48, comm: kswapd0 Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80166f88>] <ffffffff80166f88>{free_pages_bulk+575}
RSP: 0018:0000010037c85b18  EFLAGS: 00010093
RAX: ffffffff80432fb8 RBX: ffffffff80432dc0 RCX: 0000000000000000
RDX: 0000010001e10680 RSI: ffffffff80432eb0 RDI: ffffffff80432dc0
RBP: 6db6db692245b567 R08: 0000000000000000 R09: 0000010037c85b90
R10: 000001002d532088 R11: ffffffffa00624f8 R12: ffffffff80432e88
R13: 000001000160c848 R14: ffffffff80432eb0 R15: 36db6db49122dab3
FS:  0000002a95577a20(0000) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 000000fff40561d4 CR3: 0000000000101000 CR4: 00000000000006e0
Process kswapd0 (pid: 48, threadinfo 0000010037c84000, task
0000010037c831d0)
Stack: 0000010037c85b78 ffffffffffffffff 0000000001788138 0000000000000000
       0000000900000056 0000010001038000 ffffffff80432eb0 000000003c6bd460
       ffffffff80432fe0 0000000631958898
Call Trace:<ffffffff8016764a>{free_hot_cold_page+188}
<ffffffff801676ac>{__pagevec_free+38}
       <ffffffff8016f41a>{release_pages+642}
<ffffffff80162659>{__remove_from_page_cache+25}
       <ffffffff8016f9f6>{__pagevec_release+23}
<ffffffff8016ff8c>{invalidate_mapping_pages+163}
       <ffffffff801adf60>{shrink_icache_memory+571}
<ffffffff8017027f>{shrink_slab+188}
       <ffffffff80172021>{balance_pgdat+504}
<ffffffff801721d8>{kswapd+224}
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffff8013530f>{autoremove_wake_function+0}
       <ffffffff80133068>{schedule_tail+11}
<ffffffff8011124f>{child_rip+8}
       <ffffffff80155343>{kthread+0} <ffffffff801720f8>{kswapd+0}
       <ffffffff80111247>{child_rip+0}

Code: 44 0f bb 3a 19 c0 85 c0 0f 84 a4 00 00 00 0f b6 4c 24 14 b8
RIP <ffffffff80166f88>{free_pages_bulk+575} RSP <0000010037c85b18>
CR2: 000000fff40561d4
 mm/page_alloc.c:259: spin_lock(mm/page_alloc.c:ffffffff80432dc0)
already locked by mm/page_alloc.c/259
Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80167885>{buffered_rmqueue+446}
PML4 c7f8067 PGD 30577067 PMD 0
Oops: 0002 [2]
CPU 0
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack i
p_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm s
nd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi
snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext
3 jbd raid1 sata_via libata sd_mod scsi_mod
Pid: 1655, comm: kjournald Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446}
RSP: 0018:000001003bc87908  EFLAGS: 00010006
RAX: 000001000175cfe0 RBX: ffffffff80432dc0 RCX: ffffffff80432fe0
RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0
RBP: ffffffff80432fd0 R08: 000000000000eddd R09: 000000000000eddd
R10: 0000010038f43bc0 R11: ffffffff8028b44c R12: ffffffff804332c0
R13: 000001000175cfb8 R14: 000001003bed3610 R15: 0000000000000000
FS:  0000002a95574b00(0000) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0
Process kjournald (pid: 1655, threadinfo 000001003bc86000, task
000001003bed3610)
Stack: 0000010037cf2c00 0000000000000246 0000000000000000 000002003bed3610
       0000000000000046 000001003bc87960 0000000000000056 0000000000000000
       0000000000000200 ffffffff804332c0
Call Trace:<ffffffff80167b85>{__alloc_pages+200}
<ffffffff80167e1c>{__get_free_pages+28}
       <ffffffff8016bc2a>{kmem_getpages+31}
<ffffffff8016c422>{cache_alloc_refill+1083}
       <ffffffff8016bf89>{kmem_cache_alloc+79}
<ffffffff80165f3c>{mempool_alloc+133} 
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffff8013530f>{autoremove_wake_function+0} 
       <ffffffff80284d5b>{submit_bio+219}
<ffffffff8019071c>{bio_alloc+28} 
       <ffffffff8018e015>{submit_bh+144}
<ffffffff8018e0fd>{ll_rw_block+93} 
       <ffffffffa004b2b6>{:jbd:journal_commit_transaction+2612} 
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffff8013530f>{autoremove_wake_function+0} 
       <ffffffffa004f9ee>{:jbd:kjournald+524}
<ffffffff8013530f>{autoremove_wake_function+0} 
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffffa004f7dc>{:jbd:commit_timeout+0} 
       <ffffffff80133068>{schedule_tail+11}
<ffffffff8011124f>{child_rip+8} 
       <ffffffffa004f7e2>{:jbd:kjournald+0}
<ffffffff80111247>{child_rip+0} 
       

Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01 
RIP <ffffffff80167885>{buffered_rmqueue+446} RSP <000001003bc87908>
CR2: 0000000000100108
 <1>Unable to handle kernel paging request at 0000000000100108 RIP: 
<ffffffff80167885>{buffered_rmqueue+446}
PML4 c7f8067 PGD 30577067 PMD 0 
Oops: 0002 [3] 
CPU 0 
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack i
p_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm s
nd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi
snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext
3 jbd raid1 sata_via libata sd_mod scsi_mod
Pid: 46, comm: pdflush Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446}
RSP: 0018:0000010037c0d7b8  EFLAGS: 00010006
RAX: 000001000175cfe0 RBX: ffffffff80432dc0 RCX: ffffffff80432fe0
RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0
RBP: ffffffff80432fd0 R08: 0000010037c0da50 R09: 00000000fffffffa
R10: 00000100013a7408 R11: 0000000000000000 R12: ffffffff804332c0
R13: 000001000175cfb8 R14: 0000010037c08920 R15: 0000000000000000
FS:  0000002a95574b00(0000) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0
Process pdflush (pid: 46, threadinfo 0000010037c0c000, task
0000010037c08920)
Stack: 0000010037c0d7e0 0000010037ce1c00 0000000000021a01
0000020000000220 
       0000000000000046 000001003a66cda0 0000000000000056
0000000000000000 
       0000000000000200 ffffffff804332c0 
Call Trace:<ffffffff80167b85>{__alloc_pages+200}
<ffffffff80167e1c>{__get_free_pages+28} 
       <ffffffff8016bc2a>{kmem_getpages+31}
<ffffffff8016c422>{cache_alloc_refill+1083} 
       <ffffffff8016bf89>{kmem_cache_alloc+79}
<ffffffff80165f3c>{mempool_alloc+133} 
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffff80165f3c>{mempool_alloc+133} 
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffff8019071c>{bio_alloc+28} 
       <ffffffff8019096e>{bio_clone+13}
<ffffffffa003f1fb>{:raid1:make_request+1303} 
       <ffffffff80284c6b>{generic_make_request+361}
<ffffffff8013530f>{autoremove_wake_function+0} 
       <ffffffff80284d5b>{submit_bio+219}
<ffffffff80190820>{bio_alloc+288} 
       <ffffffff8018e080>{submit_bh+251}
<ffffffff8018f520>{__block_write_full_page+432} 
       <ffffffff8019433b>{blkdev_get_block+0}
<ffffffff801ba791>{mpage_writepages+561} 
       <ffffffff801bad45>{mpage_writepages+2021}
<ffffffff80194412>{blkdev_writepage+0} 
       <ffffffff8010ebbc>{__switch_to+254}
<ffffffff803557da>{thread_return+41} 
       <ffffffff801b89bd>{__writeback_single_inode+662}
<ffffffff801b96e4>{sync_sb_inodes+470} 
       <ffffffff801b9d6e>{writeback_inodes+605}
<ffffffff80169055>{wb_kupdate+147} 
       <ffffffff803557da>{thread_return+41} <ffffffff8016a085>{pdflush+0} 
       <ffffffff8016a4a7>{pdflush+1058} <ffffffff80168fc2>{wb_kupdate+0} 
       <ffffffff801553ff>{kthread+188}
<ffffffff80155428>{keventd_create_kthread+0} 
       <ffffffff8011124f>{child_rip+8}
<ffffffff80155428>{keventd_create_kthread+0} 
       <ffffffff80155343>{kthread+0} <ffffffff80111247>{child_rip+0} 
       

Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01 
RIP <ffffffff80167885>{buffered_rmqueue+446} RSP <0000010037c0d7b8>
CR2: 0000000000100108
 <1>Unable to handle kernel paging request at 0000000000100108 RIP: 
<ffffffff80167885>{buffered_rmqueue+446}
PML4 c7f8067 PGD 30577067 PMD 0 
Oops: 0002 [4] 
CPU 0 
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack i
p_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm s
nd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi
snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii floppy ext
3 jbd raid1 sata_via libata sd_mod scsi_mod
Pid: 46, comm: pdflush Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446}
RSP: 0018:ffffffff8049fa68  EFLAGS: 00010006
RAX: 000001000175cfe0 RBX: ffffffff80432dc0 RCX: ffffffff80432fe0
RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0
RBP: ffffffff80432fd0 R08: 0000000000000000 R09: ffffffff8049fbb8
R10: 000000002640f942 R11: 000001003c245900 R12: ffffffff804332c0
R13: 000001000175cfb8 R14: 0000010037c08920 R15: 0000000000000000
FS:  0000002a95574b00(0000) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0
Process pdflush (pid: 46, threadinfo 0000010037c0c000, task
0000010037c08920)
Stack: 0000000000000206 ffffffff8030d164 000001003c54f708
000002200000000f 
       0550000000000046 ffffffff8049fbb0 0000000000000056
0000000000000000 
       0000000000000220 ffffffff804332c0 
Call Trace:<IRQ> <ffffffff8030d164>{__ip_route_output_key+1880}
<ffffffff80167b85>{__alloc_pages+200} 
       <ffffffff80167e1c>{__get_free_pages+28}
<ffffffff8016bc2a>{kmem_getpages+31} 
       <ffffffff8016c422>{cache_alloc_refill+1083}
<ffffffff8016bf89>{kmem_cache_alloc+79} 
       <ffffffff802e953a>{sk_alloc+42}
<ffffffff8032be09>{tcp_create_openreq_child+50} 
       <ffffffff803281d2>{tcp_v4_syn_recv_sock+85}
<ffffffffa0262001>{:ipv6:tcp_v6_syn_recv_sock+64} 
       <ffffffff80327b8f>{tcp_v4_send_synack+202}
<ffffffff8032c705>{tcp_check_req+526} 
       <ffffffff803020e9>{qdisc_restart+26}
<ffffffff802f12f7>{dev_queue_xmit+1223} 
       <ffffffff802f6e0c>{neigh_resolve_output+516}
<ffffffff802fb1f2>{nf_iterate+82} 
       <ffffffff803117ec>{ip_finish_output2+0}
<ffffffff802fb561>{nf_hook_slow+184} 
       <ffffffff80311b68>{ip_finish_output+473}
<ffffffff803114ac>{dst_output+0} 
       <ffffffff803114c2>{dst_output+22}
<ffffffff802fb561>{nf_hook_slow+184} 
       <ffffffff80311f62>{ip_queue_xmit+1007}
<ffffffffa01e5ebc>{:ip_conntrack:__ip_conntrack_find+13} 
       <ffffffffa01e5f8b>{:ip_conntrack:ip_conntrack_tuple_taken+33} 
       <ffffffffa01e5ebc>{:ip_conntrack:__ip_conntrack_find+13} 
       <ffffffff80122312>{dma_map_sg+622}
<ffffffff80132a95>{recalc_task_prio+337} 
       <ffffffff802ac382>{dma_timer_expiry+0}
<ffffffff802abfae>{ide_dma_intr+0} 
       <ffffffffa01f4388>{:iptable_nat:do_bindings+169}
<ffffffff803285f8>{tcp_v4_do_rcv+207} 
       <ffffffff80328edd>{tcp_v4_rcv+1712}
<ffffffff8030ea83>{ip_local_deliver_finish+236} 
       <ffffffff8030e997>{ip_local_deliver_finish+0}
<ffffffff802fb561>{nf_hook_slow+184} 
       <ffffffff8030ecec>{ip_local_deliver+468}
<ffffffff8030eecb>{ip_rcv_finish+472} 
       <ffffffff8030ecf3>{ip_rcv_finish+0}
<ffffffff802fb561>{nf_hook_slow+184} 
       <ffffffff8030f2fd>{ip_rcv+1008}
<ffffffff802f191f>{netif_receive_skb+572} 
       <ffffffffa009baaf>{:e100:e100_poll+710}
<ffffffff802f191f>{netif_receive_skb+572} 
       <ffffffff802f19ce>{process_backlog+125}
<ffffffff802f1ac1>{net_rx_action+112} 
       <ffffffff8013ed50>{__do_softirq+76}
<ffffffff8013edd7>{do_softirq+49} 
       <ffffffff801138a3>{do_IRQ+756} <ffffffff80110d6b>{ret_from_intr+0} 
        <EOI> <ffffffff80111da2>{oops_end+159}
<ffffffff80111d15>{oops_end+18} 
       <ffffffff80124595>{do_page_fault+1074}
<ffffffff80111099>{error_exit+0} 
       <ffffffff80167885>{buffered_rmqueue+446}
<ffffffff80167b85>{__alloc_pages+200} 
       <ffffffff80167e1c>{__get_free_pages+28}
<ffffffff8016bc2a>{kmem_getpages+31} 
       <ffffffff8016c422>{cache_alloc_refill+1083}
<ffffffff8016bf89>{kmem_cache_alloc+79} 
       <ffffffff80165f3c>{mempool_alloc+133}
<ffffffff8013530f>{autoremove_wake_function+0} 
       <ffffffff80165f3c>{mempool_alloc+133}
<ffffffff8013530f>{autoremove_wake_function+0} 
       <ffffffff8019071c>{bio_alloc+28} <ffffffff8019096e>{bio_clone+13} 
       <ffffffffa003f1fb>{:raid1:make_request+1303}
<ffffffff80284c6b>{generic_make_request+361} 
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffff80284d5b>{submit_bio+219} 
       <ffffffff80190820>{bio_alloc+288}
<ffffffff8018e080>{submit_bh+251} 
       <ffffffff8018f520>{__block_write_full_page+432}
<ffffffff8019433b>{blkdev_get_block+0} 
       <ffffffff801ba791audit(1104659019.304:0): initialized


Sometimes after a kernel paging request oops messages, also get lots
of these at random blocks:
__find_get_block_slow() failed. block=1715, b_blocknr=4448602
b_state=0x00000019, b_size=4096
device blocksize: 4096
__find_get_block_slow() failed. block=1715, b_blocknr=4448602
b_state=0x00000019, b_size=4096
device blocksize: 4096
__find_get_block_slow() failed. block=1715, b_blocknr=4448602
b_state=0x00000019, b_size=4096
device blocksize: 4096
__find_get_block_slow() failed. block=1715, b_blocknr=4448602
b_state=0x00000019, b_size=4096
device blocksize: 4096
__find_get_block_slow() failed. block=1715, b_blocknr=4448602
b_state=0x00000019, b_size=4096
device blocksize: 4096
__find_get_block_slow() failed. block=1715, b_blocknr=4448602
b_state=0x00000019, b_size=4096
device blocksize: 4096
__find_get_block_slow() failed. block=1715, b_blocknr=4448602
b_state=0x00000019, b_size=4096
device blocksize: 4096
__find_get_block_slow() failed. block=1715, b_blocknr=4448602
b_state=0x00000019, b_size=4096
device blocksize: 4096
__find_get_block_slow() failed. block=1715, b_blocknr=4448602
b_state=0x00000019, b_size=4096
device blocksize: 4096
Comment 5 Karl Grindley 2005-01-03 12:10:43 EST
got another one this morning.

[snip]
lockd: couldn't shutdown host module!
Bad page state at prep_new_page (in process 'rsync', page
0000010001a0b4f8)
flags:0x2000103c mapping:00000100230ff450 mapcount:0 count:2
Backtrace:
                                                                     
                                   
Call Trace:<ffffffff80166d1d>{bad_page+114}
<ffffffff80167a57>{buffered_rmqueue+912}
       <ffffffff80167b85>{__alloc_pages+200}
<ffffffff8016482b>{generic_file_buffered_write+320}
       <ffffffff8016503f>{generic_file_aio_write_nolock+735}
       <ffffffff801650ef>{generic_file_aio_write+125}
<ffffffffa0060388>{:ext3:ext3_file_write+22}
       <ffffffff8018a9d0>{do_sync_write+173}
<ffffffff8013530f>{autoremove_wake_function+0}
       <ffffffff801c3a5c>{compat_sys_select+1428}
<ffffffff8018aacb>{vfs_write+207}
       <ffffffff8018abb3>{sys_write+69}
<ffffffff80125dc3>{cstar_do_call+27}
                                                                     
                                   
Trying to fix it up, but a reboot is needed
Unable to handle kernel paging request at 0000000000200200 RIP:
<ffffffff80166e63>{free_pages_bulk+282}
PML4 5a5d067 PGD 28852067 PMD 0
Oops: 0002 [1]
CPU 0
Modules linked in: nfs lockd parport_pc lp parport md5 ipv6 autofs4
sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_c
onntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd
ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss
 snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi
snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii 
floppy ext3 jbd raid1 sata_via libata sd_mod scsi_mod
Pid: 27745, comm: rsync Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80166e63>] <ffffffff80166e63>{free_pages_bulk+282}
RSP: 0018:0000010007d336c8  EFLAGS: 00010017
RAX: 000001000149a408 RBX: ffffffff80432dc0 RCX: 00000100010b0e58
RDX: 0000000000200200 RSI: 0000010001a0b520 RDI: ffffffff80432dc0
RBP: 000000000002cea8 R08: 0000000000000000 R09: 0000000000000040
R10: 0000000000000040 R11: 0000000000040000 R12: 0000010001a0b4f8
R13: 0000010001a0b4f8 R14: ffffffff80432ec8 R15: 000000000000b3aa
FS:  0000002a95577a20(0000) GS:ffffffff80511980(005b)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 0000000000200200 CR3: 0000000000101000 CR4: 00000000000006e0
Process rsync (pid: 27745, threadinfo 0000010007d32000, task
0000010038cfb750)
Stack: 0000010000000005 fffffffffffffffe 000000013fb95400 0000000000000000
       000000018bab5fdf 0000010001038000 ffffffff80432eb0 0000000007d339d0
       ffffffff80432fe0 0000000e3fb95400
Call Trace:<ffffffff8016764a>{free_hot_cold_page+188}
<ffffffff801676ac>{__pagevec_free+38}
       <ffffffff8016f4af>{release_pages+791}
<ffffffff801635e4>{find_get_pages_tag+179}
       <ffffffff8016f9f6>{__pagevec_release+23}
<ffffffff801bad2d>{mpage_writepages+1997}
       <ffffffffa0048a6f>{:jbd:do_get_write_access+1776}
<ffffffffa006214a>{:ext3:ext3_ordered_writepage+0}
       <ffffffffa0063053>{:ext3:ext3_mark_iloc_dirty+802}
       <ffffffff801b89bd>{__writeback_single_inode+662}
<ffffffff801b96e4>{sync_sb_inodes+470}
       <ffffffff801b9d6e>{writeback_inodes+605}
<ffffffff80168e41>{balance_dirty_pages_ratelimited+197}
       <ffffffff80164aff>{generic_file_buffered_write+1044}
       <ffffffff8016503f>{generic_file_aio_write_nolock+735}
       <ffffffff801650ef>{generic_file_aio_write+125}
<ffffffffa0060388>{:ext3:ext3_file_write+22}
       <ffffffff8018a9d0>{do_sync_write+173}
<ffffffff8013530f>{autoremove_wake_function+0}
       <ffffffff801c3a5c>{compat_sys_select+1428}
<ffffffff8018aacb>{vfs_write+207}
       <ffffffff8018abb3>{sys_write+69}
<ffffffff80125dc3>{cstar_do_call+27}
                                                                     
                                   
                                                                     
                                   
Code: 48 89 02 48 c7 46 08 00 02 20 00 48 c7 06 00 01 10 00 8b 54
RIP <ffffffff80166e63>{free_pages_bulk+282} RSP <0000010007d336c8>
CR2: 0000000000200200
 <1>Unable to handle kernel paging request at 0000000000200200 RIP:
<ffffffff80167889>{buffered_rmqueue+450}
PML4 5a5d067 PGD 28852067 PMD 0
Oops: 0002 [2]
CPU 0
Modules linked in: nfs lockd parport_pc lp parport md5 ipv6 autofs4
sunrpc ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_c
onntrack ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd
ehci_hcd snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss
 snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi
snd_seq_device snd soundcore via_velocity crc_ccitt e100 mii 
floppy ext3 jbd raid1 sata_via libata sd_mod scsi_mod
Pid: 27745, comm: rsync Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80167889>] <ffffffff80167889>{buffered_rmqueue+450}
RSP: 0018:ffffffff8049fc88  EFLAGS: 00010006
RAX: 0000010001a0b520 RBX: ffffffff80432dc0 RCX: 0000000000200200
RDX: 000001000149a408 RSI: 0000000000000000 RDI: ffffffff80432dc0
RBP: ffffffff80432fd0 R08: 0101a8c0ed01a8c0 R09: 0000000000020000
R10: 0000000000000003 R11: 0000000000000000 R12: ffffffff804332c0
R13: 0000010001a0b4f8 R14: 0000010038cfb750 R15: 0000000000000000
FS:  0000002a95577a20(0000) GS:ffffffff80511980(005b)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 0000000000200200 CR3: 0000000000101000 CR4: 00000000000006e0
Process rsync (pid: 27745, threadinfo 0000010007d32000, task
0000010038cfb750)
Stack: 000001001f6d4800 000001001f6d4800 0000000000000000 0000022000001000
       00000100063ebf68 ffffffffa02620df 0000000000000056 0000000000000000
       0000000000000220 ffffffff804332c0
Call Trace:<IRQ> <ffffffffa02620df>{:ipv6:tcp_v6_syn_recv_sock+286}
       <ffffffff80167b85>{__alloc_pages+200}
<ffffffff80167e1c>{__get_free_pages+28}
       <ffffffff8016bc2a>{kmem_getpages+31}
<ffffffff8016c422>{cache_alloc_refill+1083}
       <ffffffff8016bf89>{kmem_cache_alloc+79}
<ffffffff80327c90>{tcp_v4_conn_request+170}
       <ffffffffa01e6ecb>{:ip_conntrack:ip_ct_refresh_acct+164}
       <ffffffff802f0dd9>{__skb_linearize+252}
<ffffffff803020e9>{qdisc_restart+26}
       <ffffffff802f12f7>{dev_queue_xmit+1223}
<ffffffff802fb561>{nf_hook_slow+184}
       <ffffffff80311b68>{ip_finish_output+473}
<ffffffffa00a93f3>{:via_velocity:velocity_intr+1738}
       <ffffffffa01e5ebc>{:ip_conntrack:__ip_conntrack_find+13}
       <ffffffffa01e5f8b>{:ip_conntrack:ip_conntrack_tuple_taken+33}
       <ffffffffa01e5ebc>{:ip_conntrack:__ip_conntrack_find+13}
       <ffffffffa01e5f8b>{:ip_conntrack:ip_conntrack_tuple_taken+33}
       <ffffffffa01e5ebc>{:ip_conntrack:__ip_conntrack_find+13}
       <ffffffffa01e6d28>{:ip_conntrack:ip_conntrack_alter_reply+121}
       <ffffffffa01f41cc>{:iptable_nat:ip_nat_setup_info+2282}
       <ffffffffa01f41df>{:iptable_nat:ip_nat_setup_info+2301}
       <ffffffff80113176>{handle_IRQ_event+40}
<ffffffff801137fb>{do_IRQ+588}
       <ffffffff8031f9b9>{tcp_rcv_state_process+114}
<ffffffff803287e6>{tcp_v4_do_rcv+701}
       <ffffffff80328edd>{tcp_v4_rcv+1712}
<ffffffff8030ea83>{ip_local_deliver_finish+236}
       <ffffffff8030e997>{ip_local_deliver_finish+0}
<ffffffff802fb561>{nf_hook_slow+184}
       <ffffffff8030ecec>{ip_local_deliver+468}
<ffffffff8030eecb>{ip_rcv_finish+472}
       <ffffffff8030ecf3>{ip_rcv_finish+0}
<ffffffff802fb561>{nf_hook_slow+184}
       <ffffffff8030f2fd>{ip_rcv+1008}
<ffffffff802f191f>{netif_receive_skb+572}
       <ffffffff802f19ce>{process_backlog+125}
<ffffffff802f1ac1>{net_rx_action+112}
       <ffffffff8013ed50>{__do_softirq+76}
<ffffffff8013edd7>{do_softirq+49}
       <ffffffff801138a3>{do_IRQ+756} <ffffffff80110d6b>{ret_from_intr+0}
        <EOI> <ffffffff80111da2>{oops_end+159}
<ffffffff80111d15>{oops_end+18}
       <ffffffff80124595>{do_page_fault+1074}
<ffffffff80165f3c>{mempool_alloc+133}
       <ffffffff80111099>{error_exit+0}
<ffffffff80166e63>{free_pages_bulk+282}
       <ffffffff80166fe5>{free_pages_bulk+668}
<ffffffff8016764a>{free_hot_cold_page+188}
       <ffffffff801676ac>{__pagevec_free+38}
<ffffffff8016f4af>{release_pages+791}
       <ffffffff801635e4>{find_get_pages_tag+179}
<ffffffff8016f9f6>{__pagevec_release+23}
       <ffffffff801bad2d>{mpage_writepages+1997}
<ffffffffa0048a6f>{:jbd:do_get_write_access+1776}
       <ffffffffa006214a>{:ext3:ext3_ordered_writepage+0}
       <ffffffffa0063053>{:ext3:ext3_mark_iloc_dirty+802}
[/snip]

then the machine auto-rebooted (configured to reboot after 30 seconds
on panic)  No find_get_block_slow messages this time.
Comment 6 Karl Grindley 2005-01-03 12:15:28 EST
all of the above oops/panic messages are in kernel 2.6.9-1.681_FC3
WITH sata_via/libata/sd_mod/scsi_mod unloaded from the kernel and see
if the symptoms change.
Comment 7 Karl Grindley 2005-01-04 09:39:01 EST
2 more reboots from oops/panics in the last 24 hours.  The "Spurious
ACK on isa0060/serio0." message is back.  These are with
2.6.9-1.681_FC3 and the sata_via/libata/sd_mod/scsi_mod kernel modules
NOT loaded...   When the modules are not loaded, it seems to change
the behaviour of the oops/panics.

Btw, let me know when you guys have had enough....  Otherwise i'll
just try various kernel/module combinations to see if i can get them
to stop.... and continue posting results.
Karl



Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80166e5f>{free_pages_bulk+278}
PML4 1dcff067 PGD 32592067 PMD 0
Oops: 0002 [1]
CPU 0
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack
ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer
snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd
soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1
Pid: 46, comm: pdflush Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80166e5f>] <ffffffff80166e5f>{free_pages_bulk+278}
RSP: 0018:0000010037c0d9f8  EFLAGS: 00010017
RAX: 0000000000100100 RBX: ffffffff80432dc0 RCX: 0000010001a811a8
RDX: 0000000000200200 RSI: 00000100015ba390 RDI: ffffffff80432dc0
RBP: 00000000000192e8 R08: 0000000000000000 R09: 0000000000000040
R10: 0000000000000040 R11: 0000000000040000 R12: 00000100015ba368
R13: 00000100015ba330 R14: ffffffff80432ee0 R15: 000000000000325d
FS:  0000002a95578460(0000) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0
Process pdflush (pid: 46, threadinfo 0000010037c0c000, task
0000010037c08920)
Stack: 000001003e8ac840 fffffffffffffffc 000000023e8ac840 0000000000000000
       0000000100000050 0000010001038000 ffffffff80432eb0 0000000037c0dd00
       ffffffff80432fe0 0000000e3fb91800
Call Trace:<ffffffff8016764a>{free_hot_cold_page+188}
<ffffffff801676ac>{__pagevec_free+38}
       <ffffffff8016f4af>{release_pages+791}
<ffffffff801635e4>{find_get_pages_tag+179}
       <ffffffff8016f9f6>{__pagevec_release+23}
<ffffffff801bad2d>{mpage_writepages+1997}
       <ffffffffa006214a>{:ext3:ext3_ordered_writepage+0}
       <ffffffff8010ebbc>{__switch_to+254}
<ffffffff803557da>{thread_return+41}
       <ffffffff803566d3>{schedule_timeout+295}
<ffffffff801b89bd>{__writeback_single_inode+662}
       <ffffffff801b96e4>{sync_sb_inodes+470}
<ffffffff801b9d6e>{writeback_inodes+605}
       <ffffffff8016a085>{pdflush+0}
<ffffffff80168f5e>{background_writeout+108}
       <ffffffff8016a4a7>{pdflush+1058}
<ffffffff80168ef2>{background_writeout+0}
       <ffffffff801553ff>{kthread+188}
<ffffffff80155428>{keventd_create_kthread+0}
       <ffffffff8011124f>{child_rip+8}
<ffffffff80155428>{keventd_create_kthread+0}
       <ffffffff80155343>{kthread+0} <ffffffff80111247>{child_rip+0}
                                                                     
                                   
                                                                     
                                   
Code: 48 89 50 08 48 89 02 48 c7 46 08 00 02 20 00 48 c7 06 00 01
RIP <ffffffff80166e5f>{free_pages_bulk+278} RSP <0000010037c0d9f8>
CR2: 0000000000100108
 mm/page_alloc.c:259: spin_lock(mm/page_alloc.c:ffffffff80432dc0)
already locked by mm/page_alloc.c/259
mm/slab.c:2204: spin_is_locked on uninitialized spinlock 0000010001a81218.
kernel/printk.c:551: spin_lock(kernel/printk.c:ffffffff8042ce40)
already locked by kernel/printk.c/551
kernel/printk.c:551: spin_lock(kernel/printk.c:ffffffff8042ce40)
already locked by kernel/printk.c/551
kernel/printk.c:551: spin_lock(kernel/printk.c:ffffffff8042ce40)
already locked by kernel/printk.c/551
kernel/printk.c:551: spin_lock(kernel/printk.c:ffffffff8042ce40)
already locked by kernel/printk.c/551
kernel/printk.c:551: spin_lock(kernel/printk.c:ffffffff8042ce40)
already locked by kernel/printk.c/551
Unable to handle kernel paging request at 00000000ffffffff RIP:
<ffffffff8021331b>{strnlen+12}
PML4 1dcff067 PGD 0
Oops: 0000 [2]
CPU 0
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack
ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer
snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd
soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1
Pid: 3, comm: events/0 Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff8021331b>] <ffffffff8021331b>{strnlen+12}
RSP: 0018:ffffffff804a0160  EFLAGS: 00010097
RAX: 00000000ffffffff RBX: ffffffff804a0238 RCX: 000000000000000a
RDX: ffffffff804a0270 RSI: fffffffffffffffe RDI: 00000000ffffffff
RBP: 00000000ffffffff R08: 00000000ffffffff R09: 0000000000000003
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff804ab67a
R13: 00000000ffffffff R14: 0000000000000000 R15: ffffffff804aba5f
FS:  0000002a95578460(0000) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00000000ffffffff CR3: 0000000000101000 CR4: 00000000000006e0
Process events/0 (pid: 3, threadinfo 0000010037e3e000, task
0000010002140030)
Stack: ffffffff80213bbd 0000000000000400 ffffffff804ab660 ffffffff8036951f
       0000000000000400 ffffffff8036950d 0000010001af9408 0000000000000000
       000001003d88c012 0000000000000000
Call Trace:<IRQ> <ffffffff80213bbd>{vsnprintf+848}
<ffffffff80213e3a>{vscnprintf+9}
       <ffffffff80138a46>{vprintk+421}
<ffffffff802ac713>{__ide_dma_read+154}
       <ffffffff802ac80e>{__ide_dma_begin+40}
<ffffffff802af525>{__ide_do_rw_disk+853}
       <ffffffff80138d8b>{printk+141}
<ffffffff802a5076>{ide_do_request+977}
       <ffffffff8016cef8>{cache_flusharray+138}
<ffffffff8016cbf5>{kfree+106}
       <ffffffff802ebbad>{kfree_skbmem+9}
<ffffffffa009bf46>{:e100:e100_poll+1885}
       <ffffffff802f1ac1>{net_rx_action+112}
<ffffffff8013ed50>{__do_softirq+76}
       <ffffffff8013edd7>{do_softirq+49} <ffffffff801138a3>{do_IRQ+756}
       <ffffffff80110d6b>{ret_from_intr+0}  <EOI>
<ffffffff801670d7>{free_pages_bulk+910}
       <ffffffff801671cf>{__free_pages_ok+229}
<ffffffff80134add>{free_task+18}
       <ffffffff803557da>{thread_return+41}
<ffffffffa02171c3>{:sunrpc:do_cache_clean+0}
       <ffffffff8014f1bd>{worker_thread+219}
<ffffffff801337a8>{default_wake_function+0}
       <ffffffff801337a8>{default_wake_function+0}
<ffffffff8014f0e2>{worker_thread+0}
       <ffffffff801553ff>{kthread+188} <ffffffff8011124f>{child_rip+8}
       <ffffffff801fdc0f>{selinux_d_instantiate+0}
<ffffffff80155343>{kthread+0}
       <ffffffff80111247>{child_rip+0}
                                                                     
                                                                     
                 
Code: 80 3f 00 74 11 48 ff ce 48 ff c0 48 83 fe ff 74 05 80 38 00
RIP <ffffffff8021331b>{strnlen+12} RSP <ffffffff804a0160>
CR2: 00000000ffffffff
 <3>Debug: sleeping function called from invalid context at
include/linux/rwsem.h:43
in_atomic():1[expected: 0], irqs_disabled():0
                                                                     
                                                                     
                 
Call Trace:<IRQ> <ffffffff80133173>{__might_sleep+173}
<ffffffff80139d4d>{profile_task_exit+33}
       <ffffffff8013b87a>{do_exit+34} <ffffffff80111da2>{oops_end+159}
       <ffffffff801245e6>{do_page_fault+1155}
<ffffffffa01f41cc>{:iptable_nat:ip_nat_setup_info+2282}
       <ffffffff803406cb>{fib_lookup+188} <ffffffff80111099>{error_exit+0}
       <ffffffff8021331b>{strnlen+12} <ffffffff80213bbd>{vsnprintf+848}
       <ffffffff80213e3a>{vscnprintf+9} <ffffffff80138a46>{vprintk+421}
       <ffffffff802ac713>{__ide_dma_read+154}
<ffffffff802ac80e>{__ide_dma_begin+40}
       <ffffffff802af525>{__ide_do_rw_disk+853}
<ffffffff80138d8b>{printk+141}
       <ffffffff802a5076>{ide_do_request+977}
<ffffffff8016cef8>{cache_flusharray+138}
       <ffffffff8016cbf5>{kfree+106} <ffffffff802ebbad>{kfree_skbmem+9}
       <ffffffffa009bf46>{:e100:e100_poll+1885}
<ffffffff802f1ac1>{net_rx_action+112}
       <ffffffff8013ed50>{__do_softirq+76}
<ffffffff8013edd7>{do_softirq+49}
       <ffffffff801138a3>{do_IRQ+756} <ffffffff80110d6b>{ret_from_intr+0}
        <EOI> <ffffffff801670d7>{free_pages_bulk+910}
<ffffffff801671cf>{__free_pages_ok+229}
       <ffffffff80134add>{free_task+18}
<ffffffff803557da>{thread_return+41}
       <ffffffffa02171c3>{:sunrpc:do_cache_clean+0}
<ffffffff8014f1bd>{worker_thread+219}
       <ffffffff801337a8>{default_wake_function+0}
<ffffffff801337a8>{default_wake_function+0}
       <ffffffff8014f0e2>{worker_thread+0} <ffffffff801553ff>{kthread+188}
       <ffffffff8011124f>{child_rip+8}
<ffffffff801fdc0f>{selinux_d_instantiate+0}
       <ffffffff80155343>{kthread+0} <ffffffff80111247>{child_rip+0}
                                                                     
                                                                     
                 
Kernel panic - not syncing: Aiee, killing interrupt handler!
 <0>Rebooting in 30 seconds..<4>atkbd.c: Spurious ACK on
isa0060/serio0. Some program, like XFree86, might be trying access
hardware directly.
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
Losing some ticks... checking if CPU frequency changed.
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
warning: many lost ticks.
Your time source seems to be instable or some driver is hogging interupts
rip serio_interrupt+0x18c/0x19a
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
[auto reboot]




next oops attack:
net/core/skbuff.c:1270:
spin_lock(include/linux/skbuff.h:000001003b26faf8) already locked by
net/core/skbuff.c/1195
net/core/skbuff.c:1270:
spin_lock(include/linux/skbuff.h:000001003b26faf8) already locked by
net/core/skbuff.c/1270
net/core/skbuff.c:1270:
spin_lock(include/linux/skbuff.h:000001003edb8138) already locked by
net/core/skbuff.c/1195
Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80166e5f>{free_pages_bulk+278}
PML4 34904067 PGD 4d22067 PMD 0
Oops: 0002 [1]
CPU 0
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack
ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer
snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd
soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1
Pid: 46, comm: pdflush Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80166e5f>] <ffffffff80166e5f>{free_pages_bulk+278}
RSP: 0018:0000010037c0d9e8  EFLAGS: 00010013
RAX: 0000000000100100 RBX: ffffffff80432dc0 RCX: 00000100016058a8
RDX: 0000000000200200 RSI: 00000100015539a0 RDI: ffffffff80432dc0
RBP: 0000000000017598 R08: 0000000000000000 R09: 0000000000000040
R10: 0000000000000040 R11: 0000000000040000 R12: 0000010001553978
R13: 0000010001553940 R14: ffffffff80432ee0 R15: 0000000000002eb3
FS:  0000002a95577a20(0000) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0
Process pdflush (pid: 46, threadinfo 0000010037c0c000, task
0000010037c08920)
Stack: 0000010009b9b988 fffffffffffffffc 000000020000002e 0000000000000000
       000000021deffb80 0000010001038000 ffffffff80432eb0 0000000037c0dcf0
       ffffffff80432fe0 0000000d3fbd2c00
Call Trace:<ffffffff8016764a>{free_hot_cold_page+188}
<ffffffff801676ac>{__pagevec_free+38}
       <ffffffff8016f4af>{release_pages+791}
<ffffffff801635e4>{find_get_pages_tag+179}
       <ffffffff8016f9f6>{__pagevec_release+23}
<ffffffff801bad2d>{mpage_writepages+1997}
       <ffffffff801bad45>{mpage_writepages+2021}
<ffffffffa006214a>{:ext3:ext3_ordered_writepage+0}
       <ffffffff8010ebbc>{__switch_to+254}
<ffffffff803557da>{thread_return+41}
       <ffffffff801b89bd>{__writeback_single_inode+662}
<ffffffff801b96e4>{sync_sb_inodes+470}
       <ffffffff801b9d6e>{writeback_inodes+605}
<ffffffff80169055>{wb_kupdate+147}
       <ffffffff803557da>{thread_return+41} <ffffffff8016a085>{pdflush+0}
       <ffffffff8016a4a7>{pdflush+1058} <ffffffff80168fc2>{wb_kupdate+0}
       <ffffffff801553ff>{kthread+188}
<ffffffff80155428>{keventd_create_kthread+0}
       <ffffffff8011124f>{child_rip+8}
<ffffffff80155428>{keventd_create_kthread+0}
       <ffffffff80155343>{kthread+0} <ffffffff80111247>{child_rip+0}
                                                                     
                                                                     
                 
                                                                     
                                                                     
                 
                                                                     
                                                                     
                 
Code: 48 89 50 08 48 89 02 48 c7 46 08 00 02 20 00 48 c7 06 00 01
RIP <ffffffff80166e5f>{free_pages_bulk+278} RSP <0000010037c0d9e8>
CR2: 0000000000100108
 mm/page_alloc.c:259: spin_lock(mm/page_alloc.c:ffffffff80432dc0)
already locked by mm/page_alloc.c/259
Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80167885>{buffered_rmqueue+446}
PML4 34904067 PGD 4d22067 PMD 0
Oops: 0002 [2]
CPU 0
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack
ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer
snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd
soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1
Pid: 214, comm: kjournald Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446}
RSP: 0018:000001003f22d9f8  EFLAGS: 00010006
RAX: 00000100015539a0 RBX: ffffffff80432dc0 RCX: 0000000000200200
RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0
RBP: ffffffff80432fd0 R08: 0000000000000008 R09: 0000000000000007
R10: ffffffff804d5d60 R11: 00000100197a9360 R12: ffffffff804332c0
R13: 0000010001553978 R14: 000001003fb541b0 R15: 0000000000000000
FS:  0000002a95574b00(0000) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0
Process kjournald (pid: 214, threadinfo 000001003f22c000, task
000001003fb541b0)
Stack: 0000000000000000 ffffffff80162f40 0000000000000007 0000005000000000
       0000010037ff4728 ffffffff8018c8ce 0000000000000256 0000000000000000
       0000000000000050 ffffffff804332c0
Call Trace:<ffffffff80162f40>{find_get_page+156}
<ffffffff8018c8ce>{__find_get_block_slow+52}
       <ffffffff80167b85>{__alloc_pages+200}
<ffffffff801633ae>{find_or_create_page+68}
       <ffffffff801901d5>{__getblk_slow+246}
<ffffffff801904d2>{__getblk+61}
       <ffffffffa0052c50>{:jbd:journal_get_descriptor_buffer+42}
       <ffffffffa004ba68>{:jbd:journal_commit_transaction+4582}
       <ffffffff803557da>{thread_return+41}
<ffffffffa004f9ee>{:jbd:kjournald+524}
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffff8013530f>{autoremove_wake_function+0}
       <ffffffffa004f7dc>{:jbd:commit_timeout+0}
<ffffffff80133068>{schedule_tail+11}
       <ffffffff8011124f>{child_rip+8}
<ffffffffa004f7e2>{:jbd:kjournald+0}
       <ffffffff80111247>{child_rip+0}
                                                                     
                                                                     
                 
Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01
RIP <ffffffff80167885>{buffered_rmqueue+446} RSP <000001003f22d9f8>
CR2: 0000000000100108
 <1>Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80166e5f>{free_pages_bulk+278}
PML4 34904067 PGD 4d22067 PMD 0
Oops: 0002 [3]
CPU 0
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack
ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer
snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd
soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1
Pid: 3, comm: events/0 Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80166e5f>] <ffffffff80166e5f>{free_pages_bulk+278}
RSP: 0018:0000010037e3fcf8  EFLAGS: 00010003
RAX: 0000000000100100 RBX: ffffffff80432dc0 RCX: ffffffff80432fe0
RDX: 0000000000200200 RSI: 00000100015539a0 RDI: ffffffff80432dc0
RBP: 0000010001b9c7c0 R08: 0000000000000000 R09: 0000000000000005
R10: 0000010002141110 R11: 000000000000087f R12: 0000010001553978
R13: 0000010035148000 R14: ffffffff804d4460 R15: 0000000000000000
FS:  0000002a95574b00(0000) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0
Process events/0 (pid: 3, threadinfo 0000010037e3e000, task
0000010002140030)
Stack: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
       0000000000000000 0000010001038000 ffffffff80432eb0 0000000000000001
       ffffffff80432fe0 0000000f01038000
Call Trace:<ffffffff8016764a>{free_hot_cold_page+188}
<ffffffff8016bdcf>{kmem_freepages+195}
       <ffffffff8016ccbb>{slab_destroy+114}
<ffffffff8016cda1>{free_block+194}
       <ffffffff8016ce46>{drain_array_locked+99}
<ffffffff8016e2b1>{cache_reap+256}
       <ffffffff8016e1b1>{cache_reap+0}
<ffffffff8014f359>{worker_thread+631}
       <ffffffff801337a8>{default_wake_function+0}
<ffffffff801337a8>{default_wake_function+0}
       <ffffffff8014f0e2>{worker_thread+0} <ffffffff801553ff>{kthread+188}
       <ffffffff8011124f>{child_rip+8}
<ffffffff801fdc0f>{selinux_d_instantiate+0}
       <ffffffff80155343>{kthread+0} <ffffffff80111247>{child_rip+0}
                                                                     
                                                                     
                 
                                                                     
                                                                     
                 
Code: 48 89 50 08 48 89 02 48 c7 46 08 00 02 20 00 48 c7 06 00 01
RIP <ffffffff80166e5f>{free_pages_bulk+278} RSP <0000010037e3fcf8>
CR2: 0000000000100108
 <1>Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80167885>{buffered_rmqueue+446}
PML4 34904067 PGD 4d22067 PMD 0
Oops: 0002 [4]
CPU 0
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack
ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer
snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd
soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1
Pid: 1656, comm: kjournald Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446}
RSP: 0018:000001003eb6d908  EFLAGS: 00010006
RAX: 00000100015539a0 RBX: ffffffff80432dc0 RCX: 0000000000200200
RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0
RBP: ffffffff80432fd0 R08: 00000000000195f9 R09: 00000000000195f9
R10: 0000010037e34810 R11: ffffffff8028b44c R12: ffffffff804332c0
R13: 0000010001553978 R14: 000001003ccb95d0 R15: 0000000000000000
FS:  0000002a95574b00(0000) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0
Process kjournald (pid: 1656, threadinfo 000001003eb6c000, task
000001003ccb95d0)
Stack: 0000010037cf2c00 0000000000000246 0000000000000000 000002003ccb95d0
       ffffffff8013530f 000001003eb6d960 0000000000000056 0000000000000000
       0000000000000200 ffffffff804332c0
Call Trace:<ffffffff8013530f>{autoremove_wake_function+0}
<ffffffff80167b85>{__alloc_pages+200}
       <ffffffff80167e1c>{__get_free_pages+28}
<ffffffff8016bc2a>{kmem_getpages+31}
       <ffffffff8016c422>{cache_alloc_refill+1083}
<ffffffff8016bf89>{kmem_cache_alloc+79}
       <ffffffff80165f3c>{mempool_alloc+133}
<ffffffff8013530f>{autoremove_wake_function+0}
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffff80284d5b>{submit_bio+219}
       <ffffffff8019071c>{bio_alloc+28} <ffffffff8018e015>{submit_bh+144}
       <ffffffff8018e0fd>{ll_rw_block+93}
<ffffffffa004b2b6>{:jbd:journal_commit_transaction+2612}
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffff8013530f>{autoremove_wake_function+0}
       <ffffffffa004f9ee>{:jbd:kjournald+524}
<ffffffff8013530f>{autoremove_wake_function+0}
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffffa004f7dc>{:jbd:commit_timeout+0}
       <ffffffff80133068>{schedule_tail+11}
<ffffffff8011124f>{child_rip+8}
       <ffffffffa004f7e2>{:jbd:kjournald+0}
<ffffffff80111247>{child_rip+0}
                                                                     
                                                                     
                 
Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01
RIP <ffffffff80167885>{buffered_rmqueue+446} RSP <000001003eb6d908>
CR2: 0000000000100108
 <1>Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80167885>{buffered_rmqueue+446}
PML4 34904067 PGD 4d22067 PMD 0
Oops: 0002 [5]
CPU 0
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack
ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer
snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd
soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1
Pid: 1655, comm: kjournald Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446}
RSP: 0018:000001003db3b7c8  EFLAGS: 00010006
RAX: 00000100015539a0 RBX: ffffffff80432dc0 RCX: 0000000000200200
RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0
RBP: ffffffff80432fd0 R08: 000001003db13ac0 R09: 0000000000000000
R10: 0000010031697360 R11: 0000010031697360 R12: ffffffff804332c0
R13: 0000010001553978 R14: 000001003eb30070 R15: 0000000000000000
FS:  0000002a95574b00(0000) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0
Process kjournald (pid: 1655, threadinfo 000001003db3a000, task
000001003eb30070)
Stack: 0000000000000000 0000000000000202 0000000000000001 000002003eb30070
       ffffffff801337a8 0000000000100100 0000000000000056 0000000000000000
       0000000000000200 ffffffff804332c0
Call Trace:<ffffffff801337a8>{default_wake_function+0}
<ffffffff80167b85>{__alloc_pages+200}
       <ffffffff80167e1c>{__get_free_pages+28}
<ffffffff8016bc2a>{kmem_getpages+31}
       <ffffffff8016c422>{cache_alloc_refill+1083}
<ffffffff8016bf89>{kmem_cache_alloc+79}
       <ffffffff80165f3c>{mempool_alloc+133}
<ffffffff8013530f>{autoremove_wake_function+0}
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffff8019071c>{bio_alloc+28}
       <ffffffff8019096e>{bio_clone+13}
<ffffffffa003f1fb>{:raid1:make_request+1303}
       <ffffffff8028b8b8>{cfq_dispatch_requests+104}
<ffffffff80284c6b>{generic_make_request+361}
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffff80282e28>{generic_unplug_device+181}
       <ffffffff80284d5b>{submit_bio+219}
<ffffffff80190820>{bio_alloc+288}
       <ffffffff8018e080>{submit_bh+251}
<ffffffff8018e0fd>{ll_rw_block+93}
       <ffffffffa004b2b6>{:jbd:journal_commit_transaction+2612}
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffff8013530f>{autoremove_wake_function+0}
       <ffffffffa004f9ee>{:jbd:kjournald+524}
<ffffffff8013530f>{autoremove_wake_function+0}
       <ffffffff8013530f>{autoremove_wake_function+0}
<ffffffffa004f7dc>{:jbd:commit_timeout+0}
       <ffffffff80133068>{schedule_tail+11}
<ffffffff8011124f>{child_rip+8}
       <ffffffffa004f7e2>{:jbd:kjournald+0}
<ffffffff80111247>{child_rip+0}
                                                                     
                                                                     
                 
                                                                     
                                                                     
                 
Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01
RIP <ffffffff80167885>{buffered_rmqueue+446} RSP <000001003db3b7c8>
CR2: 0000000000100108
 mm/page_alloc.c:259: spin_lock(mm/page_alloc.c:ffffffff80432dc0)
already locked by mm/page_alloc.c/259
Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80166e5f>{free_pages_bulk+278}
PML4 14cc2067 PGD 0
Oops: 0002 [6]
CPU 0
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack
ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer
snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd
soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1
Pid: 10799, comm: httpd Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80166e5f>] <ffffffff80166e5f>{free_pages_bulk+278}
RSP: 0018:0000010014ff1e18  EFLAGS: 00010003
RAX: 0000000000100100 RBX: ffffffff80432dc0 RCX: ffffffff80432fe0
RDX: 0000000000200200 RSI: 00000100015539a0 RDI: 0000000000000046
RBP: 00000100018be908 R08: 000000000000000d R09: 0000000000000002
R10: 0000000000000000 R11: 0000000000000000 R12: 0000010001553978
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000000
FS:  0000002a97383140(0000) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0
Process httpd (pid: 10799, threadinfo 0000010014ff0000, task
0000010014ffe370)
Stack: 00000000000000d0 0000000000000246 2f2f2f2f2f2f2f2f 0000000000000000
       00000000010b1c06 0000010001038000 ffffffff80432eb0 00000000010b1c06
       ffffffff80432fe0 0000000f803566d3
Call Trace:<ffffffff8016764a>{free_hot_cold_page+188}
<ffffffff801a2a73>{poll_freewait+64}
       <ffffffff801a38da>{sys_poll+783} <ffffffff801a2a79>{__pollwait+0}
       <ffffffff801107a2>{system_call+126}
                                                                     
                                                                     
                 
Code: 48 89 50 08 48 89 02 48 c7 46 08 00 02 20 00 48 c7 06 00 01
RIP <ffffffff80166e5f>{free_pages_bulk+278} RSP <0000010014ff1e18>
CR2: 0000000000100108
 mm/page_alloc.c:259: spin_lock(mm/page_alloc.c:ffffffff80432dc0)
already locked by mm/page_alloc.c/259
Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80166e5f>{free_pages_bulk+278}
PML4 3b0d4067 PGD 0
Oops: 0002 [7]
CPU 0
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack
ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer
snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd
soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1
Pid: 2333, comm: named Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80166e5f>] <ffffffff80166e5f>{free_pages_bulk+278}
RSP: 0018:000001003b187d08  EFLAGS: 00010003
RAX: 0000000000100100 RBX: ffffffff80432dc0 RCX: ffffffff80432fe0
RDX: 0000000000200200 RSI: 00000100015539a0 RDI: 0000000000000046
RBP: 00000100015d5b50 R08: 000000000000000d R09: 0000000000000002
R10: 0000000000000000 R11: 0000000000000000 R12: 0000010001553978
R13: 0000000000000001 R14: 000000000000001d R15: 000001003b187f18
FS:  0000000041e02960(005b) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0
Process named (pid: 2333, threadinfo 000001003b186000, task
000001003b180630)
Stack: 00000000000000d0 ffffffff804332c0 0000000000000000 0000000000000000
       00000000ffffffff 0000010001038000 ffffffff80432eb0 000000003b187f18
       ffffffff80432fe0 0000000f80356611
Call Trace:<ffffffff8016764a>{free_hot_cold_page+188}
<ffffffff801a2a73>{poll_freewait+64}
       <ffffffff801a3110>{do_select+1501} <ffffffff801a2a79>{__pollwait+0}
       <ffffffff8019ada3>{pipe_readv+132}
<ffffffff801a3469>{sys_select+820}
       <ffffffff801107a2>{system_call+126}
                                                                     
                                                                     
                 
Code: 48 89 50 08 48 89 02 48 c7 46 08 00 02 20 00 48 c7 06 00 01
RIP <ffffffff80166e5f>{free_pages_bulk+278} RSP <000001003b187d08>
CR2: 0000000000100108
 <1>Unable to handle kernel paging request at 0000000000100108 RIP:
<ffffffff80167885>{buffered_rmqueue+446}
PML4 3b0d4067 PGD 0
Oops: 0002 [8]
CPU 0
Modules linked in: parport_pc lp parport md5 ipv6 autofs4 sunrpc
ipt_REJECT iptable_filter ipt_MASQUERADE iptable_nat ip_conntrack
ip_tables dm_mod button battery ac ohci1394 ieee1394 uhci_hcd ehci_hcd
snd_via82xx snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer
snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd
soundcore via_velocity crc_ccitt e100 mii floppy ext3 jbd raid1
Pid: 2333, comm: named Not tainted 2.6.9-1.681_FC3
RIP: 0010:[<ffffffff80167885>] <ffffffff80167885>{buffered_rmqueue+446}
RSP: 0018:ffffffff8049ffe8  EFLAGS: 00010006
RAX: 00000100015539a0 RBX: ffffffff80432dc0 RCX: 0000000000200200
RDX: 0000000000100100 RSI: 0000000000000000 RDI: ffffffff80432dc0
RBP: ffffffff80432fd0 R08: 000001003cdbe838 R09: 000001003cdbe638
R10: 000001003cdbe5c0 R11: 0000000000000000 R12: ffffffff804332c0
R13: 0000010001553978 R14: 000001003b180630 R15: 0000000000000000
FS:  0000000041e02960(005b) GS:ffffffff80511980(0000)
knlGS:00000000f7fdf6c0
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000100108 CR3: 0000000000101000 CR4: 00000000000006e0
Process named (pid: 2333, threadinfo 000001003b186000, task
000001003b180630)
Stack: 0000000000000206 ffffffffa01e6d28 ffffffff804a012c 00000220804a00b8
       000001003d2c4240 ffffffff8033e8af 0000000000000056 0000000000000000
       0000000000000220 ffffffff804332c0
Call Trace:<IRQ>
<ffffffffa01e6d28>{:ip_conntrack:ip_conntrack_alter_reply+121}
       <ffffffff8033e8af>{fn_hash_lookup+157}
<ffffffff80167b85>{__alloc_pages+200}
       <ffffffff80167e1c>{__get_free_pages+28}
<ffffffff8016bc2a>{kmem_getpages+31}
       <ffffffff8016c422>{cache_alloc_refill+1083}
<ffffffff8016bf89>{kmem_cache_alloc+79}
       <ffffffff802f51a7>{dst_alloc+43}
<ffffffff8030c71a>{ip_route_input+2280}
       <ffffffff8030ed25>{ip_rcv_finish+50}
<ffffffff8030ecf3>{ip_rcv_finish+0}
       <ffffffff802fb561>{nf_hook_slow+184}
<ffffffff8030f2fd>{ip_rcv+1008}
       <ffffffff802f191f>{netif_receive_skb+572}
<ffffffffa009baaf>{:e100:e100_poll+710}
       <ffffffff802f191f>{netif_receive_skb+572}
<ffffffff802f19ce>{process_backlog+125}
       <ffffffff802f1ac1>{net_rx_action+112}
<ffffffff8013ed50>{__do_softirq+76}
       <ffffffff8013edd7>{do_softirq+49} <ffffffff801138a3>{do_IRQ+756}
       <ffffffff80110d6b>{ret_from_intr+0}  <EOI>
<ffffffff80111da2>{oops_end+159}
       <ffffffff80111d15>{oops_end+18}
<ffffffff80124595>{do_page_fault+1074}
       <ffffffff80138d8b>{printk+141} <ffffffff80111099>{error_exit+0}
       <ffffffff80166e5f>{free_pages_bulk+278}
<ffffffff8016764a>{free_hot_cold_page+188}
       <ffffffff801a2a73>{poll_freewait+64}
<ffffffff801a3110>{do_select+1501}
       <ffffffff801a2a79>{__pollwait+0} <ffffffff8019ada3>{pipe_readv+132}
       <ffffffff801a3469>{sys_select+820}
<ffffffff801107a2>{system_call+126}
                                                                     
                                                                     
                 
                                                                     
                                                                     
                 
Code: 48 89 4a 08 48 89 11 48 c7 40 08 00 02 20 00 48 c7 00 00 01
RIP <ffffffff80167885>{buffered_rmqueue+446} RSP <ffffffff8049ffe8>
CR2: 0000000000100108
 <3>Debug: sleeping function called from invalid context at
include/linux/rwsem.h:43
in_atomic():1[expected: 0], irqs_disabled():0
                                                                     
                                                                     
                 
Call Trace:<IRQ> <ffffffff80133173>{__might_sleep+173}
<ffffffff80139d4d>{profile_task_exit+33}
       <ffffffff8013b87a>{do_exit+34} <ffffffff80111da2>{oops_end+159}
       <ffffffff801245e6>{do_page_fault+1155}
<ffffffff802f12f7>{dev_queue_xmit+1223}
       <ffffffff802fb561>{nf_hook_slow+184}
<ffffffff80111099>{error_exit+0}
       <ffffffff80167885>{buffered_rmqueue+446}
<ffffffffa01e6d28>{:ip_conntrack:ip_conntrack_alter_reply+121}
       <ffffffff8033e8af>{fn_hash_lookup+157}
<ffffffff80167b85>{__alloc_pages+200}
       <ffffffff80167e1c>{__get_free_pages+28}
<ffffffff8016bc2a>{kmem_getpages+31}
       <ffffffff8016c422>{cache_alloc_refill+1083}
<ffffffff8016bf89>{kmem_cache_alloc+79}
       <ffffffff802f51a7>{dst_alloc+43}
<ffffffff8030c71a>{ip_route_input+2280}
       <ffffffff8030ed25>{ip_rcv_finish+50}
<ffffffff8030ecf3>{ip_rcv_finish+0}
       <ffffffff802fb561>{nf_hook_slow+184}
<ffffffff8030f2fd>{ip_rcv+1008}
       <ffffffff802f191f>{netif_receive_skb+572}
<ffffffffa009baaf>{:e100:e100_poll+710}
       <ffffffff802f191f>{netif_receive_skb+572}
<ffffffff802f19ce>{process_backlog+125}
       <ffffffff802f1ac1>{net_rx_action+112}
<ffffffff8013ed50>{__do_softirq+76}
       <ffffffff8013edd7>{do_softirq+49} <ffffffff801138a3>{do_IRQ+756}
       <ffffffff80110d6b>{ret_from_intr+0}  <EOI>
<ffffffff80111da2>{oops_end+159}
       <ffffffff80111d15>{oops_end+18}
<ffffffff80124595>{do_page_fault+1074}
       <ffffffff80138d8b>{printk+141} <ffffffff80111099>{error_exit+0}
       <ffffffff80166e5f>{free_pages_bulk+278}
<ffffffff8016764a>{free_hot_cold_page+188}
       <ffffffff801a2a73>{poll_freewait+64}
<ffffffff801a3110>{do_select+1501}
       <ffffffff801a2a79>{__pollwait+0} <ffffffff8019ada3>{pipe_readv+132}
       <ffffffff801a3469>{sys_select+820}
<ffffffff801107a2>{system_call+126}
                                                                     
                                                                     
                 
Kernel panic - not syncing: Aiee, killing interrupt handler!
 <0>Rebooting in 30 seconds..<4>atkbd.c: Spurious ACK on
isa0060/serio0. Some program, like XFree86, might be trying access
hardware directly.
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
atkbd.c: Spurious ACK on isa0060/serio0. Some program, like XFree86,
might be trying access hardware directly.
[snip]
[auto reboot]

Comment 8 Karl Grindley 2005-01-06 19:07:08 EST
Installed 2.6.9-1.724_FC3.  Still crashes.

rebooted with "noapic" kernel boot param... And it's been up for a day
strait so far....  few segfaults by applications tho (different
problem tho?)


zmc[22651]: segfault at 000000001fffd85c rip 0000000008059074 rsp
00000000ffffd810 error 6
zmc[3731]: segfault at 000000001fffd85c rip 0000000008059074 rsp
00000000ffffd810 error 6
zmc[4068]: segfault at 000000001fffd85c rip 0000000008059074 rsp
00000000ffffd810 error 6
php[5349]: segfault at fffffffff84c2040 rip fffffffff84c2040 rsp
0000007fbffff840 error 14
zmc[4190]: segfault at 000000001fffd85c rip 0000000008059074 rsp
00000000ffffd810 error 6
spamassassin[17993]: segfault at 0000003d52ca20e8 rip 0000003d52ca20e8
rsp 0000007fbfffe800 error 14
lease broken - owner pid = 2997
zmc[8425]: segfault at 000000001fffd85c rip 0000000008059074 rsp
00000000ffffd810 error 6
Comment 9 Karl Grindley 2005-02-28 10:51:48 EST
turned out to be CAS timing on the ram....  i believe this is a
hardware problem with the Abit AV8

Note You need to log in before you can comment on or make changes to this bug.