Bug 1607768

Hi, yafu

  I tested with 1000 iothreads, don't hit your issue.
  kernel version:3.10.0-918.el7.x86_64
  qemu-kvm-rhev version:qemu-kvm-rhev-2.12.0-8.el7.x86_64

  Need to confirm info bellow:
   1. your testing scenarios: iommu + iothreads? or only big iothreads?
   2. if your testing scenarios is iommu, do you configure iommu in both host and guest cmdline?

(In reply to aihua liang from comment #3)
> Hi, yafu
> 
>   I tested with 1000 iothreads, don't hit your issue.
>   kernel version:3.10.0-918.el7.x86_64
>   qemu-kvm-rhev version:qemu-kvm-rhev-2.12.0-8.el7.x86_64
> 
>   Need to confirm info bellow:
>    1. your testing scenarios: iommu + iothreads? or only big iothreads?
>    2. if your testing scenarios is iommu, do you configure iommu in both
> host and guest cmdline?

Iommu is not enabled. The issue can be reproduced with 10000 iothreads.

Yes, can reproduce it with 10000 iothreads.
 
Test env:
  kernel version: 3.10.0-926.el7.x86_64
  qemu-kvm-rhev version:qemu-kvm-rhev-2.12.0-8.el7.x86_64

Test steps:
 1.Start rhel7.6 guest with 10000 iothreads, cmds as bellow:
    /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20180730-052157-8R2ZuIGd,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20180730-052157-8R2ZuIGd,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idPX2yVp  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20180730-052157-8R2ZuIGd,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20180730-052157-8R2ZuIGd,path=/var/tmp/seabios-20180730-052157-8R2ZuIGd,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20180730-052157-8R2ZuIGd,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/rhel76-64-virtio.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bus=pci.0 \
    device virtio-net-pci,mac=9a:be:bf:c0:c1:c2,id=idZKQtxs,vectors=4,netdev=idvsIEDa,bus=pci.0  \
    -netdev tap,id=idvsIEDa,vhost=on \
    -m 4096  \
    -smp 2,maxcpus=2,cores=1,threads=1,sockets=2  \
    -cpu 'Penryn' \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=on,strict=off,order=cdn  \
    -no-shutdown \
    -enable-kvm \
    -monitor stdio \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -object iothread,id=iothread2 \
    -object iothread,id=iothread3 \
    -object iothread,id=iothread4 \
    -object iothread,id=iothread5 \
    ....
    -object iothread,id=iothread9999 \

Test reuslt:
 Qemu core dump:
   [root@intel-3323-24-2 home]# sh ta.txt
Failed to create epoll instance: Too many open filesqemu-kvm: util/qemu-thread-posix.c:131: qemu_cond_destroy: Assertion `cond->initialized' failed.
ta.txt: line 10031: 10845 Aborted                 (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox off ....


Attachment it the coredump file.

Created attachment 1472254 [details]
coredump file

The minimum iothreads loaded that will cause qemu core dump is 4085.

So when you start guest with 4085 iothreads, qemu will core dump.

I will backport the following fix:

commit 14a2d11825ddc37d6547a80704ae6450e9e376c7
Author: Marc-André Lureau <marcandre.lureau>
Date:   Tue Aug 21 12:07:16 2018 +0200

    iothread: fix crash with invalid properties

Fix included in qemu-kvm-rhev-2.12.0-20.el7

Tested on qemu-kvm-rhev-2.12.0-20.el7, not hit this issue. So set bug status to VERIFIED.


Versions:
Host:
kernel-3.10.0-957.el7.x86_64
qemu-kvm-rhev-2.12.0-20.el7

Guest:
kernel-3.10.0-957.el7.x86_64



# ulimit -n 102400

start guest with 10000 iothreads, it works well.

# sh bug_1607768.sh 
QEMU 2.12.0 monitor - type 'help' for more information
(qemu) c
(qemu) system_reset      
(qemu) system_powerdown

*** Bug 1622963 has been marked as a duplicate of this bug. ***

* my last comment is related to the slow train.
There will be a fast train update for 8.0.1. The branch is ready anyway.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2553