Bug 734661

Summary: host kernel panic when guest network traffic increases
Product: Red Hat Enterprise Linux 6 Reporter: Xiaoqing Wei <xwei>
Component: kernelAssignee: Red Hat Kernel Manager <kernel-mgr>
Status: CLOSED DUPLICATE QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: high Docs Contact:
Priority: high    
Version: 6.2CC: juzhang, shuang
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-09-05 11:39:59 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
crash foreach bt > foreach-bt.txt
none
crash log none

Description Xiaoqing Wei 2011-08-31 05:35:25 UTC 
Description of problem:
when guest network traffic increase greatly,host kernel panic and kdump

Version-Release number of selected component (if applicable):
kernel-2.6.32-192.el6.x86_64

How reproducible:
4 / 8

Steps to Reproduce:
1.boot a rhel6.0-64 guest(virtio-blk,virtio-nic) :
qemu-kvm -name rhel60-64 -monitor stdio -chardev socket,id=serial_id_20110825-183504-Mlka,path=/tmp/serial-20110825-183504-Mlka,server,nowait -device isa-serial,chardev=serial_id_20110825-183504-Mlka -drive file=RHEL-Server-6.0-64-virtio.qcow2,index=0,if=none,id=drive-virtio-disk1,media=disk,cache=none,format=qcow2,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 \
\
-device virtio-net-pci,netdev=idMBchoX,mac=9a:6f:a2:f7:9f:27,id=ndev00idMBchoX,bus=pci.0,addr=0x3 \
-netdev tap,id=idMBchoX,vhost=on,ifname=t0-183504-Mlka,script=qemu-ifup-switch \
\
-m 2048 -smp 2,cores=1,threads=1,sockets=2 -cpu cpu64-rhel6,+sse2,+x2apic -vnc :0 -rtc base=utc,clock=host,driftfix=none -M rhel6.2.0 -boot order=cdn,once=c,menu=off -usbdevice tablet -no-kvm-pit-reinjection -enable-kvmby :

2. increase guest network by ping / netperf

1) ping $guest_ip -i 0.00001 -s 65507    (I run 30+ pings at one time)
2) sometimes ping is not enough to make host crash, setup guest as netserver,and host as netperf client, do tcp_stream test

3. wait till host crash
  
Actual results:
host kernel panic and kdump

Expected results:
both guest / host work healthy

Additional info:
qemu-kvm-0.12.1.2-2.179.el6.x86_64
bridge-utils-1.2-9.el6.x86_64
host mem:  12G

# brctl show
bridge name	bridge id		STP enabled	interfaces
switch		8000.00247e700452	no		eth0
							t0-183504-Mlka
Comment 1 Xiaoqing Wei 2011-08-31 05:38:17 UTC 
Code: 31 c9 49 c7 c1 00 36 45 a0 48 89 da be 03 00 00 00 bf 07 00 00 00 c7 04 24 00 00 00 80 e8 88 32 00 e1 83 f8 01 74 33 48 8b 43 20 <48> 8b 80 00 04 00 00 48 85 c0 74 19 48 8b 80 b8 00 00 00 4c 89
RIP  [<ffffffffa04537f1>] __br_deliver+0x61/0x100 [bridge]
 RSP <ffff88033441d708>
CR2: 0000000000000400


crash log output above, will attach as a txt file.
Comment 2 Xiaoqing Wei 2011-08-31 05:40:41 UTC 
Created attachment 520741 [details]
crash foreach bt > foreach-bt.txt
Comment 3 Xiaoqing Wei 2011-08-31 05:41:08 UTC 
Created attachment 520742 [details]
crash log
Comment 5 Xiaoqing Wei 2011-08-31 05:49:08 UTC 
host with kernel 2.6.32-178.el6.x86_64 , does not trigger .
 host# ifconfig  (network traffic increased by ping/netperf)
 [root@intel-w3520-12-1 ~]# ifconfig 
eth0      Link encap:Ethernet  HWaddr 00:24:7E:70:04:52  
          inet6 addr: fe80::224:7eff:fe70:452/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1981064 errors:0 dropped:5 overruns:0 frame:0
          TX packets:3562973 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:498199333 (475.1 MiB)  TX bytes:1569368218 (1.4 GiB)
          Interrupt:16 

switch    Link encap:Ethernet  HWaddr 00:24:7E:70:04:52  
          inet addr:10.66.85.217  Bcast:10.66.85.255  Mask:255.255.254.0
          inet6 addr: fe80::224:7eff:fe70:452/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:528588936 errors:0 dropped:0 overruns:0 frame:0
          TX packets:637420227 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:777058081280 (723.6 GiB)  TX bytes:969676296915 (903.0 GiB)

t0-183504-Mlka Link encap:Ethernet  HWaddr A6:79:73:09:69:80  
          inet6 addr: fe80::a479:73ff:fe09:6980/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:526836468 errors:0 dropped:0 overruns:0 frame:0
          TX packets:603497381 errors:0 dropped:0 overruns:539050671 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:784302170512 (730.4 GiB)  TX bytes:922781698720 (859.4 GiB)

guest ifconfig  (network traffic increased by ping/netperf)

[root@virtlab-66-85-153 ~]# ifconfig 
eth0      Link encap:Ethernet  HWaddr 9A:6F:A2:F7:9F:27  
          inet addr:10.66.85.240  Bcast:10.66.85.255  Mask:255.255.254.0
          inet6 addr: fe80::986f:a2ff:fef7:9f27/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:624826012 errors:0 dropped:0 overruns:0 frame:0
          TX packets:545844675 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:954574081790 (889.0 GiB)  TX bytes:812622236618 (756.8 GiB)
Comment 10 Xiaoqing Wei 2011-09-05 11:39:59 UTC 
RIP  [<ffffffffa045a7f1>] __br_deliver+0x61/0x100 [bridge]
 RSP <ffff88011a139708>
CR2: 0000000000000400


same crash output. duplicated

*** This bug has been marked as a duplicate of bug 730756 ***