Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
When opening virt-manager I see a high iowait, on fairly idle system
with 8 KVM hosts (which are all idle). When closing virt-manager, to
IO-wait disappears (vmstat says 0 "wa")
Further analysis show, that the IO is caused by re-creating the XML
files (under /var/run/libvirt/qemu/*.xml) every 2 sec, which seems a
waist of time as the files don't change.
Notice, that the files are written by libvirtd (by creating a .new
files, and moving it). So, I expect this behavior is caused by a bad
interaction between virt-manager and libvirtd.
Version-Release number of selected component (if applicable):
# rpm -q libvirt virt-manager
libvirt-0.9.10-11.el6.x86_64
virt-manager-0.9.0-11.el6.x86_64
How reproducible: Always
Steps to Reproduce:
1. While running commands, observe IO-wait via 'vmstat 1' column "wa"
2. Run: strace -f -p `pgrep libvirtd` -s 512 -e trace=rename
3. Start virt-manager
4. Observe all the syscall rename calls from the strace
5. Stop virt-manager
6. Observe the rename syscalls stops
Actual results:
Starting virt-manager causes IO-waits, due to rewriting XML files repeatable
Expected results:
Starting virt-manager should not case IO-waits, and XML files should
only be written when something change in them.
Additional info:
Tracking this down was a long process, here are some hints that others might find usable.
Normally you can "just" look at processes in state "D" Uninterruptible sleep (usually IO) via the "ps" command:
ps ax | awk '$3 ~ /^D/ { print $0 }'
But it just returned/pointed at the process: "[jbd2/sda6-8]" from this info I could NOT connect it with "libvirtd"
I used blktrace and blkparse on the device /dev/sda6, to figure out the it was libvirtd that caused to IO.
As I had mounted /dev/sda6 on /var, I found an easier method of finding files, that was modified within the last minute:
find /var/ -mmin -1
This find command pointed to both /var/run/libvirt/qemu/*.xml and /var/lib/libvirt/images/*.img. Looking at the files via the stat command, it was clear that it was the XML files that changed too often (the img files were only updated every 30 sec).
The following command will help you spot the changes:
watch -d 'for f in /var/run/libvirt/qemu/*.xml /var/lib/libvirt/images/*.img; do echo -ne "File $f:\t"; stat $f | grep Modify ; done'
When running the watch command, like this:
watch -d 'for f in /var/run/libvirt/qemu/* /var/lib/libvirt/images/*; do echo -ne "File $f:\t"; stat $f | grep Modify ; done'
I realized that files named "*.xml.new" appeared, then I used strace to trace renaming of files via:
strace -f -p `pgrep libvirtd` -s 512 -e trace=rename
A work-around to mitigate this issue is to lower how often virt-manager updates its status.
Select menu: "Edit"
-> Choose: Preferences
(new window pops up)
-> Goto fan "Stats"
Lower: "Update status every"
pkgs:
libvirt-0.9.13-3.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.297.el6_3.x86_64
kernel-2.6.32-280.el6.x86_64
prepare a running domain
# virsh list
Id Name State
----------------------------------------------------
3 aaa running
steps:
1. run vmstat -1 in one terminal
2. run strace on second terminal
# strace -f -p `pidof libvirtd` -s 512 -e trace=rename
Process 25638 attached with 11 threads - interrupt to quit
3. start virt-manager
3.1
# virt-manager
3.2
Destroy domain aaa through virt-manager
what's show up in strace:
[pid 25648] rename("/var/run/libvirt/qemu/aaa.xml.new", "/var/run/libvirt/qemu/aaa.xml") = 0
At same time the wa value stay low.
3.3
Start domain aaa through virt-manager
what's show up in strace:
[pid 25645] --- SIGCHLD (Child exited) @ 0 (0) ---
[pid 25642] rename("/var/run/libvirt/qemu/aaa.xml.new", "/var/run/libvirt/qemu/aaa.xml") = 0
Process 26127 attached
Process 26128 attached
[pid 26124] --- SIGALRM (Alarm clock) @ 0 (0) ---
[pid 25642] rename("/var/run/libvirt/qemu/aaa.xml.new", "/var/run/libvirt/qemu/aaa.xml") = 0
Also, the wa value stay low.
Test with libvirt-0.9.10-21.el6.x86_64, the rename continuously pop up in strace.
So this is fixed now.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
http://rhn.redhat.com/errata/RHSA-2013-0276.html
Description of problem: When opening virt-manager I see a high iowait, on fairly idle system with 8 KVM hosts (which are all idle). When closing virt-manager, to IO-wait disappears (vmstat says 0 "wa") Further analysis show, that the IO is caused by re-creating the XML files (under /var/run/libvirt/qemu/*.xml) every 2 sec, which seems a waist of time as the files don't change. Notice, that the files are written by libvirtd (by creating a .new files, and moving it). So, I expect this behavior is caused by a bad interaction between virt-manager and libvirtd. Version-Release number of selected component (if applicable): # rpm -q libvirt virt-manager libvirt-0.9.10-11.el6.x86_64 virt-manager-0.9.0-11.el6.x86_64 How reproducible: Always Steps to Reproduce: 1. While running commands, observe IO-wait via 'vmstat 1' column "wa" 2. Run: strace -f -p `pgrep libvirtd` -s 512 -e trace=rename 3. Start virt-manager 4. Observe all the syscall rename calls from the strace 5. Stop virt-manager 6. Observe the rename syscalls stops Actual results: Starting virt-manager causes IO-waits, due to rewriting XML files repeatable Expected results: Starting virt-manager should not case IO-waits, and XML files should only be written when something change in them. Additional info: Tracking this down was a long process, here are some hints that others might find usable. Normally you can "just" look at processes in state "D" Uninterruptible sleep (usually IO) via the "ps" command: ps ax | awk '$3 ~ /^D/ { print $0 }' But it just returned/pointed at the process: "[jbd2/sda6-8]" from this info I could NOT connect it with "libvirtd" I used blktrace and blkparse on the device /dev/sda6, to figure out the it was libvirtd that caused to IO. As I had mounted /dev/sda6 on /var, I found an easier method of finding files, that was modified within the last minute: find /var/ -mmin -1 This find command pointed to both /var/run/libvirt/qemu/*.xml and /var/lib/libvirt/images/*.img. Looking at the files via the stat command, it was clear that it was the XML files that changed too often (the img files were only updated every 30 sec). The following command will help you spot the changes: watch -d 'for f in /var/run/libvirt/qemu/*.xml /var/lib/libvirt/images/*.img; do echo -ne "File $f:\t"; stat $f | grep Modify ; done' When running the watch command, like this: watch -d 'for f in /var/run/libvirt/qemu/* /var/lib/libvirt/images/*; do echo -ne "File $f:\t"; stat $f | grep Modify ; done' I realized that files named "*.xml.new" appeared, then I used strace to trace renaming of files via: strace -f -p `pgrep libvirtd` -s 512 -e trace=rename