Bug 1460872 - Aborted(core dumped) when booting guest with "-netdev tap....vhost=on,queues=32"
Aborted(core dumped) when booting guest with "-netdev tap....vhost=on,queues=32"
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
7.4
Unspecified Unspecified
high Severity high
: rc
: ---
Assigned To: jason wang
xiywang
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-12 22:44 EDT by xiagao
Modified: 2018-04-10 10:32 EDT (History)
14 users (show)

See Also:
Fixed In Version: qemu-kvm-1.5.3-154.el7
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-04-10 10:32:19 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description xiagao 2017-06-12 22:44:30 EDT
Description of problem
Can not boot up guest with netdev and set multi queues > 31, and core dumped created.


Version-Release number of selected component (if applicable):
kernel-3.10.0-671.el7.x86_64
qemu-kvm-1.5.3-140.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.boot guest with netdev
-netdev tap,id=hostnet0,script=/etc/qemu-ifup1,downscript=no,vhost=on,queues=32 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128

2.
3.

Actual results:
Booting guest failed and core dumped created.

Expected results:
Boot up successfully.


Additional info:
1.Did not repoduced in qemu-kvm-rhev-2.9.0-6.el7.x86_64.

2.if set "queues=31", did not hit it.

3.qemu command line:
/usr/libexec/qemu-kvm -name 2016-SVVP-INTEL -boot menu=on --nodefaults --nodefconfig -m 2G -smp 2 -cpu Westmere,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -uuid 7ef09466-6c67-4db7-87ba-a5b0d1a1c2d6 -smbios 'type=1,manufacturer=Red Hat,product=Red Hat Enterprise Linux Version 7.4,version=7Server-0.4,serial=44454C4C-5700-1058-804B-B8 C05 F483258_00:21:9b:58:2d:46,uuid=7ef09466-6c67-4db7-87ba-a5b0d1a1c2d6' -usb -device usb-tablet,id=tablet0 -rtc base=localtime,clock=host,driftfix=none -chardev socket,id=b111a,path=/tmp/monitor-win2016-intel-max,server,nowait -mon chardev=b111a,mode=readline -monitor stdio -vga std -vnc :0 \
-drive file=win2016-sut-intel.raw,if=none,id=drive-virtio0-0-0,format=raw,werror=stop,rerror=stop,cache=none,serial=number -device virtio-blk-pci,drive=drive-virtio0-0-0,id=virti0-0-0,bootindex=1 \
-cdrom en_windows_server_2016_x64_dvd_9718492.iso \
-device usb-ehci,id=ehci0 -drive file=usb-storage-intel-max.raw,if=none,id=drive-usb-2-0,media=disk,format=raw,cache=none,werror=stop,rerror=stop,aio=threads -device usb-storage,bus=ehci0.0,drive=drive-usb-2-0,id=usb-2-0,removable=on \
-netdev tap,id=hostnet0,script=/etc/qemu-ifup1,downscript=no,vhost=on,queues=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128 \
-drive file=virtio-win-1.9.1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
Comment 4 xiagao 2017-06-12 23:37:30 EDT
Can reproduced in qemu-kvm-1.5.3-126.el7.x86_64, so it's not a regression.
Comment 5 xiywang 2017-06-12 23:53:14 EDT
Could not reproduce on rhel7 guest.
qemu-kvm-rhev-2.9.0-9.el7.x86_64
host & guest: 3.10.0-679.el7.x86_64

1. boot a guest
/usr/libexec/qemu-kvm -name rhel7.4 -cpu IvyBridge -m 4096 -realtime mlock=off -smp 4 \
-drive file=/home/nfs_root/rhel7.4.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,snapshot=off -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0 \
-netdev tap,id=hostnet0,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown,queues=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a1:d0:5f,vectors=128,mq=on,host_mtu=65520 \
-monitor stdio -device qxl-vga,id=video0 -serial unix:/tmp/console,server,nowait -vnc :1 -spice port=5900,disable-ticketing

2. check eth0 in guest
# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	32
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	4

3. ping external host from guest, sucessful

4. netperf in guest, sucessful
# netperf -H 10.73.72.146 -l 30
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.73.72.146 () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    30.00    17890.62
Comment 6 xiagao 2017-06-13 00:15:53 EDT
(In reply to xiywang from comment #5)
> Could not reproduce on rhel7 guest.
> qemu-kvm-rhev-2.9.0-9.el7.x86_64
Hi xiyue,

I have mentioned in additional info.
1.Did not repoduced in qemu-kvm-rhev-2.9.0-6.el7.x86_64.

So, maybe it's just qemu-kvm's issue.

> host & guest: 3.10.0-679.el7.x86_64
> 
> 1. boot a guest
> /usr/libexec/qemu-kvm -name rhel7.4 -cpu IvyBridge -m 4096 -realtime
> mlock=off -smp 4 \
> -drive
Comment 8 jason wang 2017-06-14 00:13:01 EDT
(In reply to juzhang from comment #7)
> Can we update the flag to RHEL7.5 if we do not plan to fix in the RHEL7.4?

(In reply to xiagao from comment #6)
> (In reply to xiywang from comment #5)
> > Could not reproduce on rhel7 guest.
> > qemu-kvm-rhev-2.9.0-9.el7.x86_64
> Hi xiyue,
> 
> I have mentioned in additional info.
> 1.Did not repoduced in qemu-kvm-rhev-2.9.0-6.el7.x86_64.
> 
> So, maybe it's just qemu-kvm's issue.
> 
> > host & guest: 3.10.0-679.el7.x86_64
> > 
> > 1. boot a guest
> > /usr/libexec/qemu-kvm -name rhel7.4 -cpu IvyBridge -m 4096 -realtime
> > mlock=off -smp 4 \
> > -drive

Can we trigger the issue throught libvirt?

Thanks
Comment 9 Yu Wang 2017-06-14 01:07:10 EDT
Hi xiywang,

Could you try with libvirt according to https://bugzilla.redhat.com/show_bug.cgi?id=1460872#c8 ?

Thanks
Yu Wang
Comment 10 Pei Zhang 2017-06-14 03:33:05 EDT
This issue can be triggered by libvirt.

VM can not boot up with queues='32':

    <interface type='bridge'>
      <mac address='18:66:da:5f:dd:01'/>
      <source bridge='switch'/>
      <model type='virtio'/>
      <driver name='vhost' queues='32'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>


# cat /var/log/libvirt/qemu/rhel7.4_nonrt.log
...
2017-06-14 07:30:24.426+0000: shutting down, reason=crashed
Comment 20 Miroslav Rezanina 2018-01-24 06:18:48 EST
Fix included in qemu-kvm-1.5.3-154.el7
Comment 22 Chao Yang 2018-01-25 01:14:04 EST
Reproduced with qemu-kvm-1.5.3-153.el7.x86_64 by:

[root@localhost BZ1460827]# cat log 
# gdb --args /usr/libexec/qemu-kvm -netdev tap,id=hostnet0,downscript=no,vhost=on,queues=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128

Program received signal SIGABRT, Aborted.
(gdb) 
(gdb) bt
#0  0x00007fffed5fc1d7 in raise () at /lib64/libc.so.6
#1  0x00007fffed5fd8d0 in abort () at /lib64/libc.so.6
#2  0x0000555555771e28 in virtio_del_queue (vdev=vdev@entry=0x55555750e7e8, n=n@entry=64)
    at /usr/src/debug/qemu-1.5.3/hw/virtio/virtio.c:780
#3  0x0000555555768075 in virtio_net_set_features (multiqueue=0, n=0x55555750e7e8)
    at /usr/src/debug/qemu-1.5.3/hw/net/virtio-net.c:1255
#4  0x0000555555768075 in virtio_net_set_features (vdev=<optimized out>, features=32)
    at /usr/src/debug/qemu-1.5.3/hw/net/virtio-net.c:513
#5  0x00005555557721fe in virtio_set_features (vdev=0x55555750e7e8, val=32) at /usr/src/debug/qemu-1.5.3/hw/virtio/virtio.c:912
#6  0x0000555555776af3 in access_with_adjusted_size (addr=addr@entry=4, value=value@entry=0x7fffe3625a48, size=4, access_size_min=<optimized out>, access_size_max=<optimized out>, access=access@entry=
    0x555555777010 <memory_region_write_accessor>, opaque=opaque@entry=0x55555750e6a8) at /usr/src/debug/qemu-1.5.3/memory.c:365
#7  0x0000555555778483 in memory_region_iorange_write (iorange=<optimized out>, offset=4, width=4, data=32)
    at /usr/src/debug/qemu-1.5.3/memory.c:471
#8  0x0000555555775ddc in kvm_cpu_exec (count=1, size=4, direction=1, data=<optimized out>, port=49156)
    at /usr/src/debug/qemu-1.5.3/kvm-all.c:1525
#9  0x0000555555775ddc in kvm_cpu_exec (env=env@entry=0x555557272110) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1676
#10 0x00005555557281d5 in qemu_kvm_cpu_thread_fn (arg=0x555557272110) at /usr/src/debug/qemu-1.5.3/cpus.c:802
#11 0x00007ffff4f74dd5 in start_thread () at /lib64/libpthread.so.0
#12 0x00007fffed6c49bd in clone () at /lib64/libc.so.6


-------------------------------------------------------


Verified pass with qemu-kvm-1.5.3-154.el7.x86_64. 

# /usr/libexec/qemu-kvm -netdev tap,id=hostnet0,downscript=no,vhost=on,queues=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128
qemu-kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128: Invalid number of queues (= 32), must be a postive integer less than 31.
qemu-kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128: Device initialization failed.
qemu-kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128: Device initialization failed.
qemu-kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128: Device 'virtio-net-pci' could not be initialized
Comment 26 errata-xmlrpc 2018-04-10 10:32:19 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0816

Note You need to log in before you can comment on or make changes to this bug.