Bug 643576

Summary: soft lockup during bonding test with rtl8139 nic
Product: Red Hat Enterprise Linux 6 Reporter: Amos Kong <akong>
Component: qemu-kvmAssignee: jason wang <jasowang>
Status: CLOSED WONTFIX QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.0CC: ailan, jasowang, jkurik, leiwang, mfuruta, michen, mkenneth, mst, plyons, rhod, syeghiay, tburke, virt-maint
Target Milestone: rcKeywords: Reopened
Target Release: 6.1   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 643858 672769 (view as bug list) Environment:
Last Closed: 2011-11-17 10:57:02 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 580953, 643858, 658636, 672769    

Description Amos Kong 2010-10-16 04:03:23 UTC
Description of problem:
I try to boot up a rhel6 guest with 4 rtl8139 nics, and setup bonding in guest. When I repeatedly down/up some interfaces, guest soft-lockup.

Version-Release number of selected component (if applicable):
host kernel: 2.6.32-71.3.1.el6_0.x86_64
guest kernel: 2.6.32-70.el6.x86_64
# rpm -qa |grep qemu
qemu-kvm-debuginfo-0.12.1.2-2.113.el6_0.1.x86_64
gpxe-roms-qemu-0.9.7-6.3.el6.noarch
qemu-img-0.12.1.2-2.113.el6_0.1.x86_64
qemu-kvm-tools-0.12.1.2-2.113.el6_0.1.x86_64
qemu-kvm-0.12.1.2-2.113.el6_0.1.x86_64

How reproducible:
always

Steps to Reproduce:
1. Start guest with four nic models.
2. Setup bond0 in guest by script bonding_setup.py.
3. Run test.sh in guest to repeatedly down/up interfaces.
4. ping guest from host

Actual results:
guest soft-lookup

Expected results:
the lost packet ratio of ping is 0.

Additional info:

1. scripts' content:

# cat bonding_setup.py
import os, re, commands
eth_nums = 0
for ename in ['eth0', 'eth1', 'eth2', 'eth3']:
    eth_config_file = "/etc/sysconfig/network-scripts/ifcfg-%s" % ename
    eth_config = """DEVICE=%s
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
""" % ename
    file(eth_config_file,'w').write(eth_config)
bonding_config_file = "/etc/sysconfig/network-scripts/ifcfg-bond0"
bond_config = """DEVICE=bond0
BOOTPROTO=dhcp
ONBOOT=yes
USERCTL=no
"""
file(bonding_config_file, "w").write(bond_config)
os.system("modprobe bonding")
os.system("service network restart")

# cat test.sh  
while true;do
ifconfig bond0 down;ifconfig bond0 up
ifconfig eth2 down;ifconfig eth2 up
ifconfig eth3 down;ifconfig eth3 up
ifconfig eth4 down;ifconfig eth4 up
done


2. qemu-kvm commandline:
# qemu-kvm -name 'vm1' -chardev socket,id=human_monitor_9oBS,path=/tmp/monitor-humanmonitor1-20101011-212704-VvdR,server,nowait -mon chardev=human_monitor_9oBS,mode=readline -chardev socket,id=serial_Pivx,path=/tmp/serial-20101011-212704-VvdR,server,nowait -device isa-serial,chardev=serial_Pivx -drive file='/home/devel/autotest-devel/client/tests/kvm/images/RHEL-Server-6.0-64-virtio.qcow2',index=0,if=none,id=drive-virtio-disk1,media=disk,cache=none,snapshot=on,boot=on,format=qcow2,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk1,id=virtio-disk1 -device rtl8139,netdev=idOCJBNn,id=ndev00idOCJBNn,mac='02:A9:7C:6C:1a:90',bus=pci.0,addr=0x3 -netdev tap,id=idOCJBNn,ifname='rtl8139_0_8000',script='/home/devel/autotest-devel/client/tests/kvm/scripts/qemu-ifup-switch',downscript='no' -device rtl8139,netdev=id0zt7sU,id=ndev00id0zt7sU,mac='02:A9:7C:6C:54:f6',bus=pci.0,addr=0x5 -netdev tap,id=id0zt7sU,ifname='rtl8139_1_8000',script='/home/devel/autotest-devel/client/tests/kvm/scripts/qemu-ifup-switch',downscript='no' -device rtl8139,netdev=idjrlnQd,id=ndev00idjrlnQd,mac='02:A9:7C:6C:d6:6b',bus=pci.0,addr=0x6 -netdev tap,id=idjrlnQd,ifname='rtl8139_2_8000',script='/home/devel/autotest-devel/client/tests/kvm/scripts/qemu-ifup-switch',downscript='no' -device rtl8139,netdev=idVEUn79,id=ndev00idVEUn79,mac='02:A9:7C:6C:c5:69',bus=pci.0,addr=0x7 -netdev tap,id=idVEUn79,ifname='rtl8139_3_8000',script='/home/devel/autotest-devel/client/tests/kvm/scripts/qemu-ifup-switch',downscript='no' -m 2048 -smp 2 -cpu cpu64-rhel6,+sse2,+x2apic -vnc :0 -spice port=8000,disable-ticketing -vga qxl -rtc base=utc,clock=host,driftfix=none -M rhel6.0.0 -usbdevice tablet -no-kvm-pit-reinjection -enable-kvm 

3. call trace:
BUG: soft lockup - CPU#0 stuck for 61s! [ifconfig:578]
Modules linked in: sit tunnel4 bonding virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport 8139too 8139cp mii i2c_piix4 i2c_core sg ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
CPU 0:
Modules linked in: sit tunnel4 bonding virtio_balloon ipv6 dm_mirror dm_region_hash dm_log ppdev parport_pc parport 8139too 8139cp mii i2c_piix4 i2c_core sg ext4 mbcache jbd2 virtio_blk sr_mod cdrom virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mod [last unloaded: speedstep_lib]
Pid: 578, comm: ifconfig Not tainted 2.6.32-70.el6.x86_64 #1 KVM
RIP: 0010:[<ffffffffa013be08>]  [<ffffffffa013be08>] cp_interrupt+0x38/0x510 [8139cp]
RSP: 0018:ffff880001e03d00  EFLAGS: 00010286
RAX: ffffc9000112c000 RBX: ffff880001e03d70 RCX: 0000000000000011
RDX: 0000000000000000 RSI: ffff880037b47000 RDI: 000000000000000a
RBP: ffffffff81013c93 R08: ffff880037214000 R09: ffff880001e03ed0
R10: 000000000000000c R11: 0000000000000000 R12: ffff880001e03c80
R13: 0000000000000001 R14: ffff880037b476c0 R15: ffffffff814cf9fc
FS:  00007f89746847a0(0000) GS:ffff880001e00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f896de65304 CR3: 000000007a594000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>  [<ffffffff8103be2c>] ? kvm_clock_read+0x1c/0x20
 [<ffffffff8102f52d>] ? lapic_next_event+0x1d/0x30
 [<ffffffff810d8740>] ? handle_IRQ_event+0x60/0x170
 [<ffffffff810325c9>] ? ack_apic_level+0x79/0x1b0
 [<ffffffff810daf39>] ? handle_fasteoi_irq+0x79/0xf0
 [<ffffffff81015fb9>] ? handle_irq+0x49/0xa0
 [<ffffffff814cf90c>] ? do_IRQ+0x6c/0xf0
 [<ffffffff81013ad3>] ? ret_from_intr+0x0/0x11
 [<ffffffff81073b8b>] ? __do_softirq+0x6b/0x1e0
 [<ffffffff8103540e>] ? native_apic_msr_read+0x2e/0x40
 [<ffffffff810353d7>] ? native_apic_msr_write+0x37/0x40
 [<ffffffff810142cc>] ? call_softirq+0x1c/0x30
 [<ffffffff81015f35>] ? do_softirq+0x65/0xa0
 [<ffffffff810739d5>] ? irq_exit+0x85/0x90
 [<ffffffff814cf915>] ? do_IRQ+0x75/0xf0
 [<ffffffff81013ad3>] ? ret_from_intr+0x0/0x11
 <EOI>  [<ffffffffa013aba3>] ? cp_init_hw+0x193/0x1d0 [8139cp]
 [<ffffffffa013aadf>] ? cp_init_hw+0xcf/0x1d0 [8139cp]
 [<ffffffffa013ca77>] ? cp_open+0xe7/0x220 [8139cp]
 [<ffffffff814115a1>] ? dev_open+0xa1/0x100
 [<ffffffff81410cd1>] ? dev_change_flags+0xa1/0x1d0
 [<ffffffff81470b2b>] ? devinet_ioctl+0x5eb/0x690
 [<ffffffff81471b78>] ? inet_ioctl+0x88/0xa0
 [<ffffffff813fcdaa>] ? sock_ioctl+0x7a/0x280
 [<ffffffff8117f182>] ? vfs_ioctl+0x22/0xa0
 [<ffffffff8117f324>] ? do_vfs_ioctl+0x84/0x580
 [<ffffffff8117f8a1>] ? sys_ioctl+0x81/0xa0
 [<ffffffff81013172>] ? system_call_fastpath+0x16/0x1b