Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1609150

Summary: Qemu hang when quit vm with iothreads > 4
Product: Red Hat Enterprise Linux 7 Reporter: aihua liang <aliang>
Component: qemu-kvm-rhevAssignee: Stefan Hajnoczi <stefanha>
Status: CLOSED CURRENTRELEASE QA Contact: aihua liang <aliang>
Severity: medium Docs Contact:
Priority: high    
Version: 7.6CC: aliang, coli, ngu, stefanha, virt-maint, xuwei, yhong
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-rhev-2.12.0-18.el7_6.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-18 15:33:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
iothreads.log none

Description aihua liang 2018-07-27 07:04:31 UTC
Created attachment 1471000 [details]
iothreads.log

Description of problem:
  Qemu hang when waiting to quit vm with iothreads > 4

Version-Release number of selected component (if applicable):
  kernel version:3.10.0-918.el7.x86_64
  qemu-kvm-rhev version:qemu-kvm-rhev-2.12.0-8.el7.x86_64

How reproducible:
 100%

Steps to Reproduce:
1.Start guest with 5 iothreads, cmds as bellow:
   /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20180720-030354-ueFtK0UG,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20180720-030354-ueFtK0UG,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idbB13JZ  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20180720-030354-ueFtK0UG,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20180720-030354-ueFtK0UG,path=/var/tmp/seabios-20180720-030354-ueFtK0UG,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20180720-030354-ueFtK0UG,iobase=0x402 \
    -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -object iothread,id=iothread2 \
    -object iothread,id=iothread3 \
    -object iothread,id=iothread4 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=raw,file=/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.raw \
    -device virtio-blk-pci,id=image1,drive=drive_image1 \
    -device virtio-net-pci,mac=9a:06:07:08:09:0a,id=idOc8BkD,vectors=4,netdev=idb7ErEu,bus=pci.0,addr=0x5  \
    -netdev tap,id=idb7ErEu,vhost=on,vhostfd=19,fd=10 \
    -m 14336  \
    -smp 12,maxcpus=12,cores=6,threads=1,sockets=2  \
    -cpu 'Opteron_G5',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,strict=off,order=cdn,once=c \
    -enable-kvm \
    -monitor stdio \

2. Wait for some minutes, shutdown guest
      ---by quit or by system_powerdown in hmp

3. If quit fail, kill the qemu process, restart guest with 4 iothreads:
      .....
      -object iothread,id=iothread0 \
      -object iothread,id=iothread1 \
      -object iothread,id=iothread2 \
      -object iothread,id=iothread3 \
      ...

4. repeat step 2.

Actual results:
  In step2, qemu hang when waiting to quit vm, info as bellow:
    (host#)strace -T -tt -e trace=all -p 27845 -o iothreads.log
       ....
       02:56:50.810145 write(19, "\1\0\0\0\0\0\0\0", 8) = -1 EBADF (Bad file descriptor) <0.000018>
       02:56:50.810198 futex(0x7fc83c50a9d0, FUTEX_WAIT, 27851, NULL
       ...
       details, see attachment ---iothreads.log
 But in step4, when start guest with 4 iothreads, then quit, qemu can terminated successfully.

Expected results:
  Guest can shutdown successfully with iothreads > 4.

Comment 2 Ademar Reis 2018-12-10 22:10:54 UTC
We had several iothread-related fixes at the very end of RHEL-7.6 development. Can you please retest with the latest packages? Thanks.

Comment 3 aihua liang 2018-12-11 02:37:50 UTC
Hi, Ademar
 
   Test it on qemu-kvm-rhev-2.12.0-18.el7_6.2.x86_64, don't hit this issue any more.
 
   So please add some comments and set bug's status to "ON_QA", then i can set it to "Verified".


Thanks,
aliang

Comment 4 Ademar Reis 2018-12-11 14:17:12 UTC
This was likely fixed in late RHEL-7.6, with downstream patch series like these:

* [RHV-7.6 qemu-kvm-rhev PATCH 00/42] block: Drain and iothread fixes
* [RHV-7.6 qemu-kvm-rhev PATCH 0/3] block: Fix another drain hang

Comment 5 aihua liang 2018-12-12 02:21:43 UTC
According to the test result in #c3, change bug's status to "Verified".