Bug 523299

Summary: KVM based VM kernel results ooops on high load
Product: [Fedora] Fedora Reporter: Mohammad Mateen <mateenaslam>
Component: kerneloopsAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: low    
Version: 10CC: cebbert
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-12-18 09:42:57 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mohammad Mateen 2009-09-14 19:51:50 UTC
Description of problem:
High Load chock KVM based VM network and results kernel oops 

Version-Release number of selected component (if applicable):


How reproducible:
produce heavy traffic on VM

Steps to Reproduce:
1. Rsync heavy traffic from different sites
2. produce heavy database traffic b/w VM's
3.
  
Actual results:


Expected results:


Additional info:

Sep 14 19:17:37 hilli kernel: swapper: page allocation failure. order:0, mode:0x20
Sep 14 19:17:37 hilli kernel: Pid: 0, comm: swapper Not tainted 2.6.27.30-170.2.82.fc10.x86_64 #1
Sep 14 19:17:37 hilli kernel:
Sep 14 19:17:37 hilli kernel: Call Trace:
Sep 14 19:17:37 hilli kernel: <IRQ>  [<ffffffff8109551c>] __alloc_pages_internal+0x436/0x456
Sep 14 19:17:37 hilli kernel: [<ffffffff810b2cf1>] alloc_pages_current+0xb9/0xc2
Sep 14 19:17:37 hilli kernel: [<ffffffffa003f2da>] try_fill_recv+0xcb/0x1c0 [virtio_net]
Sep 14 19:17:37 hilli kernel: [<ffffffffa003ffdf>] virtnet_poll+0x2bf/0x381 [virtio_net]
Sep 14 19:17:37 hilli kernel: [<ffffffff812a61b4>] net_rx_action+0xd4/0x1fd
Sep 14 19:17:37 hilli kernel: [<ffffffff81046c36>] __do_softirq+0x7e/0x10c
Sep 14 19:17:37 hilli kernel: [<ffffffff81011bfc>] call_softirq+0x1c/0x28
Sep 14 19:17:37 hilli kernel: [<ffffffff81012e02>] do_softirq+0x4d/0xb0
Sep 14 19:17:37 hilli kernel: [<ffffffff8104680b>] irq_exit+0x4e/0x9d
Sep 14 19:17:37 hilli kernel: [<ffffffff8101311e>] do_IRQ+0x147/0x169
Sep 14 19:17:37 hilli kernel: [<ffffffff81010963>] ret_from_intr+0x0/0x2e
Sep 14 19:17:37 hilli kernel: <EOI>  [<ffffffff8102570e>] ? native_safe_halt+0x6/0x8
Sep 14 19:17:37 hilli kernel: [<ffffffff810172ff>] ? need_resched+0x1e/0x28
Sep 14 19:17:37 hilli kernel: [<ffffffff810173e4>] ? default_idle+0x2a/0x4c
Sep 14 19:17:37 hilli kernel: [<ffffffff8100f2a7>] ? cpu_idle+0xb2/0x10b
Sep 14 19:17:37 hilli kernel: [<ffffffff8132151d>] ? rest_init+0x61/0x63
Sep 14 19:17:37 hilli kernel:
Sep 14 19:17:37 hilli kernel: Mem-Info:
Sep 14 19:17:37 hilli kernel: Node 0 DMA per-cpu:
Sep 14 19:17:37 hilli kernel: CPU    0: hi:    0, btch:   1 usd:   0
Sep 14 19:17:37 hilli kernel: Node 0 DMA32 per-cpu:
Sep 14 19:17:37 hilli kernel: CPU    0: hi:  186, btch:  31 usd: 164
Sep 14 19:17:37 hilli kernel: Active:165009 inactive:34853 dirty:0 writeback:9434 unstable:0
Sep 14 19:17:37 hilli kernel: free:1370 slab:4285 mapped:1743 pagetables:6060 bounce:0
Sep 14 19:17:37 hilli kernel: Node 0 DMA free:3980kB min:24kB low:28kB high:36kB active:3596kB inactive:552kB present:6976kB pages_scanned:0 all_unreclaimable? no
Sep 14 19:17:37 hilli kernel: lowmem_reserve[]: 0 994 994 994
Sep 14 19:17:37 hilli kernel: Node 0 DMA32 free:1500kB min:4020kB low:5024kB high:6028kB active:656440kB inactive:138860kB present:1018016kB pages_scanned:0 all_unreclaimable? no
Sep 14 19:17:37 hilli kernel: lowmem_reserve[]: 0 0 0 0
Sep 14 19:17:37 hilli kernel: Node 0 DMA: 9*4kB 11*8kB 7*16kB 3*32kB 7*64kB 9*128kB 2*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 3980kB
Sep 14 19:17:37 hilli kernel: Node 0 DMA32: 0*4kB 0*8kB 5*16kB 2*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1488kB
Sep 14 19:17:37 hilli kernel: 33683 total pagecache pages
Sep 14 19:17:37 hilli kernel: 28875 pages in swap cache
Sep 14 19:17:37 hilli kernel: Swap cache stats: add 1341014, delete 1312139, find 1389976/1405177
Sep 14 19:17:37 hilli kernel: Free swap  = 876144kB
Sep 14 19:17:37 hilli kernel: Total swap = 2096440kB
Sep 14 19:17:37 hilli kernel: 262128 pages RAM
Sep 14 19:17:37 hilli kernel: 5779 pages reserved
Sep 14 19:17:37 hilli kernel: 40621 pages shared
Sep 14 19:17:37 hilli kernel: 250534 pages non-shared


i am  facing the same problem on FC11 as well. See https://bugzilla.redhat.com/show_bug.cgi?id=520119 for more info

Now i got the same problem on FC10 VM's as well

Comment 1 Mohammad Mateen 2009-09-24 12:23:42 UTC
any update on the issue ?

Comment 2 Mohammad Mateen 2009-09-26 00:24:07 UTC
Some times i got this in logs

Sep 25 23:14:01 dasar kernel: rsyslogd: page allocation failure. order:0, mode:0x20
Sep 25 23:14:01 dasar kernel: Pid: 12479, comm: rsyslogd Not tainted 2.6.27.30-170.2.82.fc10.x86_64 #1
Sep 25 23:14:01 dasar kernel:
Sep 25 23:14:01 dasar kernel: Call Trace:
Sep 25 23:14:01 dasar kernel: <IRQ>  [<ffffffff8109551c>] __alloc_pages_internal+0x436/0x456
Sep 25 23:14:01 dasar kernel: [<ffffffff810b2cf1>] alloc_pages_current+0xb9/0xc2
Sep 25 23:14:01 dasar kernel: [<ffffffffa003f2da>] try_fill_recv+0xcb/0x1c0 [virtio_net]
Sep 25 23:14:01 dasar kernel: [<ffffffffa003ffdf>] virtnet_poll+0x2bf/0x381 [virtio_net]
Sep 25 23:14:01 dasar kernel: [<ffffffff812c3178>] ? nf_conntrack_destroy+0x16/0x18
Sep 25 23:14:01 dasar kernel: [<ffffffff812a61b4>] net_rx_action+0xd4/0x1fd
Sep 25 23:14:01 dasar kernel: [<ffffffff81046c36>] __do_softirq+0x7e/0x10c
Sep 25 23:14:01 dasar kernel: [<ffffffff81011bfc>] call_softirq+0x1c/0x28
Sep 25 23:14:01 dasar kernel: [<ffffffff81012e02>] do_softirq+0x4d/0xb0
Sep 25 23:14:01 dasar kernel: [<ffffffff8104680b>] irq_exit+0x4e/0x9d
Sep 25 23:14:01 dasar kernel: [<ffffffff81020a22>] smp_apic_timer_interrupt+0x8f/0xa8
Sep 25 23:14:01 dasar kernel: [<ffffffff81011408>] apic_timer_interrupt+0x88/0x90
Sep 25 23:14:01 dasar kernel: <EOI>
Sep 25 23:14:01 dasar kernel: Mem-Info:
Sep 25 23:14:01 dasar kernel: Node 0 DMA per-cpu:
Sep 25 23:14:01 dasar kernel: CPU    0: hi:    0, btch:   1 usd:   0
Sep 25 23:14:01 dasar kernel: Node 0 DMA32 per-cpu:
Sep 25 23:14:01 dasar kernel: CPU    0: hi:  186, btch:  31 usd: 157
Sep 25 23:14:01 dasar kernel: Active:143088 inactive:55352 dirty:0 writeback:6545 unstable:0
Sep 25 23:14:01 dasar kernel: free:1370 slab:7227 mapped:2123 pagetables:4560 bounce:0
Sep 25 23:14:01 dasar kernel: Node 0 DMA free:3988kB min:24kB low:28kB high:36kB active:2608kB inactive:396kB present:6976kB pages_scanned:0 all_unreclaimable? no
Sep 25 23:14:01 dasar kernel: lowmem_reserve[]: 0 994 994 994
Sep 25 23:14:01 dasar kernel: Node 0 DMA32 free:1492kB min:4020kB low:5024kB high:6028kB active:569744kB inactive:221012kB present:1018016kB pages_scanned:0 all_unreclaimable? no
Sep 25 23:14:01 dasar kernel: lowmem_reserve[]: 0 0 0 0
Sep 25 23:14:01 dasar kernel: Node 0 DMA: 125*4kB 0*8kB 0*16kB 7*32kB 5*64kB 3*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3988kB
Sep 25 23:14:01 dasar kernel: Node 0 DMA32: 1*4kB 1*8kB 3*16kB 2*32kB 5*64kB 6*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1468kB
Sep 25 23:14:01 dasar kernel: 11499 total pagecache pages
Sep 25 23:14:01 dasar kernel: 8391 pages in swap cache
Sep 25 23:14:01 dasar kernel: Swap cache stats: add 1627657, delete 1619266, find 2862602/2901666
Sep 25 23:14:01 dasar kernel: Free swap  = 2000528kB
Sep 25 23:14:01 dasar kernel: Total swap = 2096440kB
Sep 25 23:14:01 dasar kernel: 262128 pages RAM
Sep 25 23:14:01 dasar kernel: 5779 pages reserved
Sep 25 23:14:01 dasar kernel: 41721 pages shared
Sep 25 23:14:01 dasar kernel: 250276 pages non-shared

Comment 3 Bug Zapper 2009-11-18 10:06:02 UTC
This message is a reminder that Fedora 10 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 10.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '10'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 10's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 10 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 4 Bug Zapper 2009-12-18 09:42:57 UTC
Fedora 10 changed to end-of-life (EOL) status on 2009-12-17. Fedora 10 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.