Red Hat Bugzilla – Bug 1460872
Aborted(core dumped) when booting guest with "-netdev tap....vhost=on,queues=32"
Last modified: 2018-04-10 10:32:19 EDT
Description of problem Can not boot up guest with netdev and set multi queues > 31, and core dumped created. Version-Release number of selected component (if applicable): kernel-3.10.0-671.el7.x86_64 qemu-kvm-1.5.3-140.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.boot guest with netdev -netdev tap,id=hostnet0,script=/etc/qemu-ifup1,downscript=no,vhost=on,queues=32 \ -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128 2. 3. Actual results: Booting guest failed and core dumped created. Expected results: Boot up successfully. Additional info: 1.Did not repoduced in qemu-kvm-rhev-2.9.0-6.el7.x86_64. 2.if set "queues=31", did not hit it. 3.qemu command line: /usr/libexec/qemu-kvm -name 2016-SVVP-INTEL -boot menu=on --nodefaults --nodefconfig -m 2G -smp 2 -cpu Westmere,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -uuid 7ef09466-6c67-4db7-87ba-a5b0d1a1c2d6 -smbios 'type=1,manufacturer=Red Hat,product=Red Hat Enterprise Linux Version 7.4,version=7Server-0.4,serial=44454C4C-5700-1058-804B-B8 C05 F483258_00:21:9b:58:2d:46,uuid=7ef09466-6c67-4db7-87ba-a5b0d1a1c2d6' -usb -device usb-tablet,id=tablet0 -rtc base=localtime,clock=host,driftfix=none -chardev socket,id=b111a,path=/tmp/monitor-win2016-intel-max,server,nowait -mon chardev=b111a,mode=readline -monitor stdio -vga std -vnc :0 \ -drive file=win2016-sut-intel.raw,if=none,id=drive-virtio0-0-0,format=raw,werror=stop,rerror=stop,cache=none,serial=number -device virtio-blk-pci,drive=drive-virtio0-0-0,id=virti0-0-0,bootindex=1 \ -cdrom en_windows_server_2016_x64_dvd_9718492.iso \ -device usb-ehci,id=ehci0 -drive file=usb-storage-intel-max.raw,if=none,id=drive-usb-2-0,media=disk,format=raw,cache=none,werror=stop,rerror=stop,aio=threads -device usb-storage,bus=ehci0.0,drive=drive-usb-2-0,id=usb-2-0,removable=on \ -netdev tap,id=hostnet0,script=/etc/qemu-ifup1,downscript=no,vhost=on,queues=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128 \ -drive file=virtio-win-1.9.1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
Can reproduced in qemu-kvm-1.5.3-126.el7.x86_64, so it's not a regression.
Could not reproduce on rhel7 guest. qemu-kvm-rhev-2.9.0-9.el7.x86_64 host & guest: 3.10.0-679.el7.x86_64 1. boot a guest /usr/libexec/qemu-kvm -name rhel7.4 -cpu IvyBridge -m 4096 -realtime mlock=off -smp 4 \ -drive file=/home/nfs_root/rhel7.4.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,snapshot=off -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0 \ -netdev tap,id=hostnet0,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown,queues=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a1:d0:5f,vectors=128,mq=on,host_mtu=65520 \ -monitor stdio -device qxl-vga,id=video0 -serial unix:/tmp/console,server,nowait -vnc :1 -spice port=5900,disable-ticketing 2. check eth0 in guest # ethtool -l eth0 Channel parameters for eth0: Pre-set maximums: RX: 0 TX: 0 Other: 0 Combined: 32 Current hardware settings: RX: 0 TX: 0 Other: 0 Combined: 4 3. ping external host from guest, sucessful 4. netperf in guest, sucessful # netperf -H 10.73.72.146 -l 30 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.73.72.146 () port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 30.00 17890.62
(In reply to xiywang from comment #5) > Could not reproduce on rhel7 guest. > qemu-kvm-rhev-2.9.0-9.el7.x86_64 Hi xiyue, I have mentioned in additional info. 1.Did not repoduced in qemu-kvm-rhev-2.9.0-6.el7.x86_64. So, maybe it's just qemu-kvm's issue. > host & guest: 3.10.0-679.el7.x86_64 > > 1. boot a guest > /usr/libexec/qemu-kvm -name rhel7.4 -cpu IvyBridge -m 4096 -realtime > mlock=off -smp 4 \ > -drive
(In reply to juzhang from comment #7) > Can we update the flag to RHEL7.5 if we do not plan to fix in the RHEL7.4? (In reply to xiagao from comment #6) > (In reply to xiywang from comment #5) > > Could not reproduce on rhel7 guest. > > qemu-kvm-rhev-2.9.0-9.el7.x86_64 > Hi xiyue, > > I have mentioned in additional info. > 1.Did not repoduced in qemu-kvm-rhev-2.9.0-6.el7.x86_64. > > So, maybe it's just qemu-kvm's issue. > > > host & guest: 3.10.0-679.el7.x86_64 > > > > 1. boot a guest > > /usr/libexec/qemu-kvm -name rhel7.4 -cpu IvyBridge -m 4096 -realtime > > mlock=off -smp 4 \ > > -drive Can we trigger the issue throught libvirt? Thanks
Hi xiywang, Could you try with libvirt according to https://bugzilla.redhat.com/show_bug.cgi?id=1460872#c8 ? Thanks Yu Wang
This issue can be triggered by libvirt. VM can not boot up with queues='32': <interface type='bridge'> <mac address='18:66:da:5f:dd:01'/> <source bridge='switch'/> <model type='virtio'/> <driver name='vhost' queues='32'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> # cat /var/log/libvirt/qemu/rhel7.4_nonrt.log ... 2017-06-14 07:30:24.426+0000: shutting down, reason=crashed
Fix included in qemu-kvm-1.5.3-154.el7
Reproduced with qemu-kvm-1.5.3-153.el7.x86_64 by: [root@localhost BZ1460827]# cat log # gdb --args /usr/libexec/qemu-kvm -netdev tap,id=hostnet0,downscript=no,vhost=on,queues=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128 Program received signal SIGABRT, Aborted. (gdb) (gdb) bt #0 0x00007fffed5fc1d7 in raise () at /lib64/libc.so.6 #1 0x00007fffed5fd8d0 in abort () at /lib64/libc.so.6 #2 0x0000555555771e28 in virtio_del_queue (vdev=vdev@entry=0x55555750e7e8, n=n@entry=64) at /usr/src/debug/qemu-1.5.3/hw/virtio/virtio.c:780 #3 0x0000555555768075 in virtio_net_set_features (multiqueue=0, n=0x55555750e7e8) at /usr/src/debug/qemu-1.5.3/hw/net/virtio-net.c:1255 #4 0x0000555555768075 in virtio_net_set_features (vdev=<optimized out>, features=32) at /usr/src/debug/qemu-1.5.3/hw/net/virtio-net.c:513 #5 0x00005555557721fe in virtio_set_features (vdev=0x55555750e7e8, val=32) at /usr/src/debug/qemu-1.5.3/hw/virtio/virtio.c:912 #6 0x0000555555776af3 in access_with_adjusted_size (addr=addr@entry=4, value=value@entry=0x7fffe3625a48, size=4, access_size_min=<optimized out>, access_size_max=<optimized out>, access=access@entry= 0x555555777010 <memory_region_write_accessor>, opaque=opaque@entry=0x55555750e6a8) at /usr/src/debug/qemu-1.5.3/memory.c:365 #7 0x0000555555778483 in memory_region_iorange_write (iorange=<optimized out>, offset=4, width=4, data=32) at /usr/src/debug/qemu-1.5.3/memory.c:471 #8 0x0000555555775ddc in kvm_cpu_exec (count=1, size=4, direction=1, data=<optimized out>, port=49156) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1525 #9 0x0000555555775ddc in kvm_cpu_exec (env=env@entry=0x555557272110) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1676 #10 0x00005555557281d5 in qemu_kvm_cpu_thread_fn (arg=0x555557272110) at /usr/src/debug/qemu-1.5.3/cpus.c:802 #11 0x00007ffff4f74dd5 in start_thread () at /lib64/libpthread.so.0 #12 0x00007fffed6c49bd in clone () at /lib64/libc.so.6 ------------------------------------------------------- Verified pass with qemu-kvm-1.5.3-154.el7.x86_64. # /usr/libexec/qemu-kvm -netdev tap,id=hostnet0,downscript=no,vhost=on,queues=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128 qemu-kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128: Invalid number of queues (= 32), must be a postive integer less than 31. qemu-kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128: Device initialization failed. qemu-kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128: Device initialization failed. qemu-kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128: Device 'virtio-net-pci' could not be initialized
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:0816