Bug 1029743

Found call trace in host dmesg.

[11150.023832] ------------[ cut here ]------------
[11150.023843] WARNING: at net/core/dev.c:5018 rollback_registered_many+0x1e2/0x210()
[11150.023845] Modules linked in: tun igbvf vfio_pci vfio_iommu_type1 vfio xt_CHECKSUM nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ip_tables openvswitch vxlan ip_tunnel gre sg ipmi_devintf dcdbas kvm_amd kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd microcode serio_raw pcspkr amd64_edac_mod edac_mce_amd fam15h_power k10temp edac_core sp5100_tco i2c_piix4 igb
[11150.023888]  ipmi_si ptp bnx2 pps_core ipmi_msghandler dca shpchp acpi_power_meter mperf nfsd auth_rpcgss nfs_acl lockd sunrpc xfs libcrc32c sd_mod crc_t10dif crct10dif_common sr_mod cdrom ata_generic pata_acpi mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper ttm ahci drm mptsas pata_atiixp libahci scsi_transport_sas crc32c_intel mptscsih libata i2c_core mptbase dm_mirror dm_region_hash dm_log dm_mod
[11150.023918] CPU: 1 PID: 4947 Comm: qemu-kvm Not tainted 3.10.0-48.el7.x86_64 #1
[11150.023920] Hardware name: Dell Inc. PowerEdge R415/0YFWR8, BIOS 1.8.6 12/06/2011
[11150.023921]  0000000000000009 ffff8807616d3b38 ffffffff815abfff ffff8807616d3b70
[11150.023926]  ffffffff810589d1 ffff8808226b4000 ffff8807616d3bb8 ffff8807616d3bb8
[11150.023930]  ffff8808287006c0 0000000000000001 ffff8807616d3b80 ffffffff81058aaa
[11150.023933] Call Trace:
[11150.023940]  [<ffffffff815abfff>] dump_stack+0x19/0x1b
[11150.023946]  [<ffffffff810589d1>] warn_slowpath_common+0x61/0x80
[11150.023949]  [<ffffffff81058aaa>] warn_slowpath_null+0x1a/0x20
[11150.023951]  [<ffffffff8149eaf2>] rollback_registered_many+0x1e2/0x210
[11150.023954]  [<ffffffff8149eb51>] rollback_registered+0x31/0x40
[11150.023956]  [<ffffffff8149fc08>] unregister_netdevice_queue+0x48/0x90
[11150.023961]  [<ffffffffa065a31e>] __tun_detach+0x10e/0x2a0 [tun]
[11150.023964]  [<ffffffffa065a4dd>] tun_chr_close+0x2d/0x50 [tun]
[11150.023968]  [<ffffffff8119f3a9>] __fput+0xe9/0x270
[11150.023971]  [<ffffffff8119f65e>] ____fput+0xe/0x10
[11150.023987]  [<ffffffff8107b544>] task_work_run+0xc4/0xe0
[11150.024004]  [<ffffffff8105e68b>] do_exit+0x2bb/0xa30
[11150.024017]  [<ffffffff8105ee7f>] do_group_exit+0x3f/0xa0
[11150.024028]  [<ffffffff8106dab0>] get_signal_to_deliver+0x1d0/0x6e0
[11150.024033]  [<ffffffff81012408>] do_signal+0x48/0x5e0
[11150.024037]  [<ffffffff81012a11>] do_notify_resume+0x71/0xc0
[11150.024040]  [<ffffffff815bba52>] int_signal+0x12/0x17
[11150.024043] ---[ end trace c68788361608cdef ]---

Call trace in Comment 1 caused by openvswitch.

Test with 82576 VF.

Result:
After 333th hot plug, qemu-kvm core dump.

I don't think comment 1 is related to the problem described in comment 0.  Have you seen comment 1 consistently in relation to the core dump?  My guess is that we're leaking irq routes somewhere in qemu-kvm and after enough hotplugs we ask for too many, kvm says no, and due to the spectacular error handling in qemu-kvm, we give up with an assert.  Please confirm whether or not the host dmesg in comment 1 is repeatable in relation to the original problem.

Sorry, my mistake. As what I said in Comment 3, this call trace actually caused by some error of openvswitch, and was not related to this issue. So please ignore it.

Sorry, I skimmed right over comment 3.  Thanks.

Can QE provide access to a system that produces this?  I'm unable to reproduce in my environment.

Patch posted upstream:

http://lists.nongnu.org/archive/html/qemu-devel/2013-11/msg02767.html

Reproduced this bug.

Host:
qemu-kvm-rhev-1.5.3-18.el7.x86_64
kernel-3.10.0-61.el7.x86_64

Guest:
win2008r2-64

Cli:
gdb --args /usr/libexec/qemu-kvm -nodefaults -M pc -m 4G -cpu Opteron_G1 -smp 4,cores=2,threads=2,sockets=1 -boot menu=on -monitor stdio -rtc base=localtime,clock=host,driftfix=slew -qmp tcp:0:5555,server,nowait -vga qxl -drive file=/home/win2008r2-64.qcow2,format=qcow2,id=guest-img,if=none,werror=stop,rerror=stop -device virtio-blk-pci,scsi=off,drive=guest-img,id=os-disk -spice disable-ticketing,port=5900 -device virtio-balloon,id=balloon -monitor unix:/tmp/guest-sock,server,nowait

Steps:
Start qemu-kvm, do hotplug/hotunplug pf.
[root@dell-per415-03 ~]# cat hotplug.sh 
#!/bin/bash
for i in $(seq 500)
do
    echo "device_add vfio-pci,host=01:00.0,id=pf0" | nc -U /tmp/guest-sock
    sleep 1
    echo "device_del pf0" | nc -U /tmp/guest-sock
    sleep 5
done


Result:
Qemu-kvm will aborted.

qemu-kvm: /builddir/build/BUILD/qemu-1.5.3/kvm-all.c:967: kvm_irqchip_commit_routes: Assertion `ret == 0' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffeab07700 (LWP 17852)]
0x00007ffff30db979 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff30db979 in raise () from /lib64/libc.so.6
#1  0x00007ffff30dd088 in abort () from /lib64/libc.so.6
#2  0x00007ffff30d48e6 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff30d4992 in __assert_fail () from /lib64/libc.so.6
#4  0x00005555557c0c20 in kvm_irqchip_commit_routes ()
#5  0x00005555557c0ee1 in kvm_irqchip_add_msi_route ()
#6  0x00005555557ac5e9 in vfio_msix_vector_do_use ()
#7  0x00005555557ac78e in vfio_msix_vector_use ()
#8  0x00005555556ba27a in msix_set_vector_notifiers ()
#9  0x00005555557aff91 in vfio_pci_write_config ()
#10 0x00005555557c4142 in access_with_adjusted_size ()
#11 0x00005555557c5617 in memory_region_iorange_write ()
#12 0x00005555557c2ef5 in kvm_cpu_exec ()
#13 0x000055555576e6f5 in qemu_kvm_cpu_thread_fn ()
#14 0x00007ffff6055de3 in start_thread () from /lib64/libpthread.so.0
#15 0x00007ffff319c16d in clone () from /lib64/libc.so.6

Verified this bug with qemu-kvm-1.5.3-28.el7.x86_64

Host:
qemu-kvm-1.5.3-28.el7.x86_64
kernel-3.10.0-61.el7.x86_64

Guest:
win2008r2-64

Steps:
Hotplug/hotunplug PF 500 times, guest works well, no crash.

So this bug has been fixed.

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.