1460872 – Aborted(core dumped) when booting guest with "-netdev tap....vhost=on,queues=32"

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1460872 - Aborted(core dumped) when booting guest with "-netdev tap....vhost=on,queues=32"

Summary: Aborted(core dumped) when booting guest with "-netdev tap....vhost=on,queues=32"

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	qemu-kvm
Sub Component:
Version:	7.4
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	jason wang
QA Contact:	xiywang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-06-13 02:44 UTC by xiagao
Modified:	2018-04-10 14:32 UTC (History)
CC List:	14 users (show)
Fixed In Version:	qemu-kvm-1.5.3-154.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-04-10 14:32:19 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2018:0816	0	normal	SHIPPED_LIVE	Low: qemu-kvm security, bug fix, and enhancement update	2018-04-10 12:47:03 UTC

Description xiagao 2017-06-13 02:44:30 UTC

Description of problem
Can not boot up guest with netdev and set multi queues > 31, and core dumped created.


Version-Release number of selected component (if applicable):
kernel-3.10.0-671.el7.x86_64
qemu-kvm-1.5.3-140.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.boot guest with netdev
-netdev tap,id=hostnet0,script=/etc/qemu-ifup1,downscript=no,vhost=on,queues=32 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128

2.
3.

Actual results:
Booting guest failed and core dumped created.

Expected results:
Boot up successfully.


Additional info:
1.Did not repoduced in qemu-kvm-rhev-2.9.0-6.el7.x86_64.

2.if set "queues=31", did not hit it.

3.qemu command line:
/usr/libexec/qemu-kvm -name 2016-SVVP-INTEL -boot menu=on --nodefaults --nodefconfig -m 2G -smp 2 -cpu Westmere,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -uuid 7ef09466-6c67-4db7-87ba-a5b0d1a1c2d6 -smbios 'type=1,manufacturer=Red Hat,product=Red Hat Enterprise Linux Version 7.4,version=7Server-0.4,serial=44454C4C-5700-1058-804B-B8 C05 F483258_00:21:9b:58:2d:46,uuid=7ef09466-6c67-4db7-87ba-a5b0d1a1c2d6' -usb -device usb-tablet,id=tablet0 -rtc base=localtime,clock=host,driftfix=none -chardev socket,id=b111a,path=/tmp/monitor-win2016-intel-max,server,nowait -mon chardev=b111a,mode=readline -monitor stdio -vga std -vnc :0 \
-drive file=win2016-sut-intel.raw,if=none,id=drive-virtio0-0-0,format=raw,werror=stop,rerror=stop,cache=none,serial=number -device virtio-blk-pci,drive=drive-virtio0-0-0,id=virti0-0-0,bootindex=1 \
-cdrom en_windows_server_2016_x64_dvd_9718492.iso \
-device usb-ehci,id=ehci0 -drive file=usb-storage-intel-max.raw,if=none,id=drive-usb-2-0,media=disk,format=raw,cache=none,werror=stop,rerror=stop,aio=threads -device usb-storage,bus=ehci0.0,drive=drive-usb-2-0,id=usb-2-0,removable=on \
-netdev tap,id=hostnet0,script=/etc/qemu-ifup1,downscript=no,vhost=on,queues=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128 \
-drive file=virtio-win-1.9.1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-1-0,id=ide0-1-0

Comment 4 xiagao 2017-06-13 03:37:30 UTC

Can reproduced in qemu-kvm-1.5.3-126.el7.x86_64, so it's not a regression.

Comment 5 xiywang 2017-06-13 03:53:14 UTC

Could not reproduce on rhel7 guest.
qemu-kvm-rhev-2.9.0-9.el7.x86_64
host & guest: 3.10.0-679.el7.x86_64

1. boot a guest
/usr/libexec/qemu-kvm -name rhel7.4 -cpu IvyBridge -m 4096 -realtime mlock=off -smp 4 \
-drive file=/home/nfs_root/rhel7.4.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,snapshot=off -device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0 \
-netdev tap,id=hostnet0,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown,queues=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a1:d0:5f,vectors=128,mq=on,host_mtu=65520 \
-monitor stdio -device qxl-vga,id=video0 -serial unix:/tmp/console,server,nowait -vnc :1 -spice port=5900,disable-ticketing

2. check eth0 in guest
# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX:		0
TX:		0
Other:		0
Combined:	32
Current hardware settings:
RX:		0
TX:		0
Other:		0
Combined:	4

3. ping external host from guest, sucessful

4. netperf in guest, sucessful
# netperf -H 10.73.72.146 -l 30
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.73.72.146 () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    30.00    17890.62

Comment 6 xiagao 2017-06-13 04:15:53 UTC

(In reply to xiywang from comment #5)
> Could not reproduce on rhel7 guest.
> qemu-kvm-rhev-2.9.0-9.el7.x86_64
Hi xiyue,

I have mentioned in additional info.
1.Did not repoduced in qemu-kvm-rhev-2.9.0-6.el7.x86_64.

So, maybe it's just qemu-kvm's issue.

> host & guest: 3.10.0-679.el7.x86_64
> 
> 1. boot a guest
> /usr/libexec/qemu-kvm -name rhel7.4 -cpu IvyBridge -m 4096 -realtime
> mlock=off -smp 4 \
> -drive

Comment 8 jason wang 2017-06-14 04:13:01 UTC

(In reply to juzhang from comment #7)
> Can we update the flag to RHEL7.5 if we do not plan to fix in the RHEL7.4?

(In reply to xiagao from comment #6)
> (In reply to xiywang from comment #5)
> > Could not reproduce on rhel7 guest.
> > qemu-kvm-rhev-2.9.0-9.el7.x86_64
> Hi xiyue,
> 
> I have mentioned in additional info.
> 1.Did not repoduced in qemu-kvm-rhev-2.9.0-6.el7.x86_64.
> 
> So, maybe it's just qemu-kvm's issue.
> 
> > host & guest: 3.10.0-679.el7.x86_64
> > 
> > 1. boot a guest
> > /usr/libexec/qemu-kvm -name rhel7.4 -cpu IvyBridge -m 4096 -realtime
> > mlock=off -smp 4 \
> > -drive

Can we trigger the issue throught libvirt?

Thanks

Comment 9 Yu Wang 2017-06-14 05:07:10 UTC

Hi xiywang,

Could you try with libvirt according to https://bugzilla.redhat.com/show_bug.cgi?id=1460872#c8 ?

Thanks
Yu Wang

Comment 10 Pei Zhang 2017-06-14 07:33:05 UTC

This issue can be triggered by libvirt.

VM can not boot up with queues='32':

    <interface type='bridge'>
      <mac address='18:66:da:5f:dd:01'/>
      <source bridge='switch'/>
      <model type='virtio'/>
      <driver name='vhost' queues='32'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>


# cat /var/log/libvirt/qemu/rhel7.4_nonrt.log
...
2017-06-14 07:30:24.426+0000: shutting down, reason=crashed

Comment 20 Miroslav Rezanina 2018-01-24 11:18:48 UTC

Fix included in qemu-kvm-1.5.3-154.el7

Comment 22 Chao Yang 2018-01-25 06:14:04 UTC

Reproduced with qemu-kvm-1.5.3-153.el7.x86_64 by:

[root@localhost BZ1460827]# cat log 
# gdb --args /usr/libexec/qemu-kvm -netdev tap,id=hostnet0,downscript=no,vhost=on,queues=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128

Program received signal SIGABRT, Aborted.
(gdb) 
(gdb) bt
#0  0x00007fffed5fc1d7 in raise () at /lib64/libc.so.6
#1  0x00007fffed5fd8d0 in abort () at /lib64/libc.so.6
#2  0x0000555555771e28 in virtio_del_queue (vdev=vdev@entry=0x55555750e7e8, n=n@entry=64)
    at /usr/src/debug/qemu-1.5.3/hw/virtio/virtio.c:780
#3  0x0000555555768075 in virtio_net_set_features (multiqueue=0, n=0x55555750e7e8)
    at /usr/src/debug/qemu-1.5.3/hw/net/virtio-net.c:1255
#4  0x0000555555768075 in virtio_net_set_features (vdev=<optimized out>, features=32)
    at /usr/src/debug/qemu-1.5.3/hw/net/virtio-net.c:513
#5  0x00005555557721fe in virtio_set_features (vdev=0x55555750e7e8, val=32) at /usr/src/debug/qemu-1.5.3/hw/virtio/virtio.c:912
#6  0x0000555555776af3 in access_with_adjusted_size (addr=addr@entry=4, value=value@entry=0x7fffe3625a48, size=4, access_size_min=<optimized out>, access_size_max=<optimized out>, access=access@entry=
    0x555555777010 <memory_region_write_accessor>, opaque=opaque@entry=0x55555750e6a8) at /usr/src/debug/qemu-1.5.3/memory.c:365
#7  0x0000555555778483 in memory_region_iorange_write (iorange=<optimized out>, offset=4, width=4, data=32)
    at /usr/src/debug/qemu-1.5.3/memory.c:471
#8  0x0000555555775ddc in kvm_cpu_exec (count=1, size=4, direction=1, data=<optimized out>, port=49156)
    at /usr/src/debug/qemu-1.5.3/kvm-all.c:1525
#9  0x0000555555775ddc in kvm_cpu_exec (env=env@entry=0x555557272110) at /usr/src/debug/qemu-1.5.3/kvm-all.c:1676
#10 0x00005555557281d5 in qemu_kvm_cpu_thread_fn (arg=0x555557272110) at /usr/src/debug/qemu-1.5.3/cpus.c:802
#11 0x00007ffff4f74dd5 in start_thread () at /lib64/libpthread.so.0
#12 0x00007fffed6c49bd in clone () at /lib64/libc.so.6


-------------------------------------------------------


Verified pass with qemu-kvm-1.5.3-154.el7.x86_64. 

# /usr/libexec/qemu-kvm -netdev tap,id=hostnet0,downscript=no,vhost=on,queues=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128
qemu-kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128: Invalid number of queues (= 32), must be a postive integer less than 31.
qemu-kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128: Device initialization failed.
qemu-kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128: Device initialization failed.
qemu-kvm: -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:e2:52:28:67:6a,mq=on,vectors=128: Device 'virtio-net-pci' could not be initialized

Comment 26 errata-xmlrpc 2018-04-10 14:32:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0816

Note You need to log in before you can comment on or make changes to this bug.