RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1536461 - Libvirtd hangs when shutting down domains
Summary: Libvirtd hangs when shutting down domains
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: beta
: ---
Assignee: Michal Privoznik
QA Contact: chhu
URL:
Whiteboard:
: 1589603 (view as bug list)
Depends On:
Blocks: 1532277
TreeView+ depends on / blocked
 
Reported: 2018-01-19 13:23 UTC by Michal Privoznik
Modified: 2021-12-10 15:34 UTC (History)
10 users (show)

Fixed In Version: libvirt-3.9.0-9.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-10 11:04:21 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
stats.py (103 bytes, text/plain)
2018-01-19 13:23 UTC, Michal Privoznik 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1584950 0 medium CLOSED "End of file from qemu monitor" when shutdown guest 2021-02-22 00:41:40 UTC
Red Hat Knowledge Base (Solution) 4709321 0 None None None 2020-06-05 04:01:47 UTC
Red Hat Product Errata RHEA-2018:0704 0 None None None 2018-04-10 11:05:38 UTC

Internal Links: 1584950

Description Michal Privoznik 2018-01-19 13:23:09 UTC 
Created attachment 1383404 [details]
stats.py

Description of problem:

If there's one thread talking to qemu on the monitor while the other one is executing shut down of the qemu, libvirtd will hang because the first thread is waiting for a reply that never comes.

Version-Release number of selected component (if applicable):
libvirt-3.9.0-8.el7

How reproducible:
100%

Steps to Reproduce:
1. start half a dozen of machines
2. run the attached script in another terminal
3. shutdown the machines at once, e.g. via:

# for i in $(virsh list --name); do virsh shutdown $i &  done

Actual results:
The script hangs (because libvirtd hangs)

Expected results:
The script can continue gathering stats.


Additional info:
Comment 1 Michal Privoznik 2018-01-19 13:34:01 UTC 
Patch pushed upstream:

commit 72adaf2f10509c3682f2c65ffad4176e00e5a2fb
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Jan 19 13:53:57 2018 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Jan 19 14:31:03 2018 +0100

    Revert "qemu: monitor: do not report error on shutdown"
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1536461
    
    This reverts commit aeda1b8c56dc58b0a413acc61bbea938b40499e1.
    
    Problem is that we need mon->lastError to be set because it's
    used all over the place. Also, there's nothing wrong with
    reporting error if one occurred. I mean, if there's a thread
    executing an API and which currently is talking on monitor it
    definitely wants the error reported.
    
    Signed-off-by: Michal Privoznik <mprivozn>

v4.0.0-2-g72adaf2f1
Comment 2 Michal Privoznik 2018-01-19 13:36:03 UTC 
To POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2018-January/msg00397.html
Comment 4 chhu 2018-02-05 07:33:19 UTC 
Reproduced on packages:
libvirt-3.9.0-8.el7.x86_64
qemu-kvm-rhev-2.10.0-18.el7.x86_64

Verified on packages:
libvirt-3.9.0-10.el7.x86_64
qemu-kvm-rhev-2.10.0-18.el7.x86_64

Test steps:
1. Start 100 guests on host.
2. Run script below in one terminal A.
# cat stats.py
import libvirt
conn = libvirt.open("qemu:///system")
while True:
    print(conn.getAllDomainStats())

# python stats.py

3. On another terminal B, shutdown 100 machines at once:
# for i in $(virsh list --name); do virsh shutdown $i &  done

4. Check the stats.py script is not hang on terminal A.
   Check the 100 guests are shutdown.

5. Start/shutdown the 100 guests at the same time for 10 times,
check the states.py script is not hang.

# cat test.sh
#!/bin/bash

for t in {1..10}
do
echo "-------------$t-start-------------"
virsh list --all
for i in $(virsh list --name --all); do virsh start $i &  done
wait
sleep 180
echo "-------------$t-shutdown-------------"
virsh list --all
for i in $(virsh list --name); do virsh shutdown $i &  done
wait
sleep 120
done

# sh test.sh


According to the test results above, set the status to "VERIFIED".
Comment 8 errata-xmlrpc 2018-04-10 11:04:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0704
Comment 9 Xuesong Zhang 2018-09-06 03:47:41 UTC 
*** Bug 1589603 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.