Description of problem: Hit error when shutdown 100 VMs and run `virsh domstats` concurrently, "error: internal error: End of file from qemu monitor error: Unable to read from monitor: Connection reset by peer". Version-Release number of selected component (if applicable): libvirt-6.6.0-6.module+el8.3.0+8125+aefcf088.x86_64 qemu-kvm-5.1.0-9.module+el8.3.0+8182+ac9ced32.x86_64 kernel: 4.18.0-234.el8.x86_64 How reproducible: 100% Steps to Reproduce: 1. Start 100 VMs 2. Run domstats.sh to get all domain stats on one terminal # cat domstats.sh #!/bin/sh for i in {1..50} do virsh domstats 2>&1 >>stats.log sleep 2 done #sh domstats.sh 3. On another terminal, shutdown 100 VMs concurrently # for i in $(virsh list --name); do virsh shutdown $i & done 4. Get error in libvirtd.log and on the terminal which is running domstats.sh ------------------------------------------------------------------- # sh domstats.sh error: internal error: End of file from qemu monitor error: Unable to read from monitor: Connection reset by peer error: Unable to read from monitor: Connection reset by peer error: Unable to read from monitor: Connection reset by peer -------------------------------------------------------------------- Actual results: In Step4, Get error in libvirtd.log Expected results: In Step4, No error. Additional info: - libvirtd.log - qemu guest log - In step2 run `python3 domstats.py`, hit same error. # cat domstats.py import libvirt import time conn = libvirt.open("qemu:///system") for x in range(1,100): print(conn.getAllDomainStats()) time.sleep(2)
Created attachment 1717488 [details] libvirtd.log
Created attachment 1717489 [details] qemu-guest.log
Just for the record, I helped Chenli debug this and the problem is how libvirt deals with monitor disappearing during domstats. While we check whether monitor is available when getting stats for a domain, we do it upfront and don't take into account that it may disappear between two stat callbacks.
Bulk update: Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.