Description of problem: Currently blktap doesn't export any statistics about reads, writes, oob, etc. This makes monitoring of file-backed guests difficult, since you can't really tell what is going on with the blktap module. Upstream has added some patches to enhance blktap to be like the blkback driver and export this information via sysfs: http://lists.xensource.com/archives/html/xen-changelog/2007-02/msg00278.html RHEL-5 should really have this patch so customers can monitor guests more effectively.
What tools should I use to see these stats? vmstat? dstat? ... (Beyond just cat-ting sysfs files I mean).
Richard, Currently, cat'ting the files in /sys is the only way. I started on a program called "virtstat" that would actually collect various statistics (block, network, cpu, memory) on the dom0 about the running domains by looking at these /sys files for you; however, it is very Xen specific. I have a feeling this is this sort of thing we want to include in libvirt; something like "getBlockStats(domain)" or something. Additionally, there is no easy way to get stats out of qemu (either for network or block), so work is needed there. The whole thing needs more thought, obviously. Chris Lalancette
I believe we will want xen-3.1-testing changesets 14040, 15078, 15389, 15390, 15402 to fix this. Chris Lalancette
This request was evaluated by Red Hat Product Management for inclusion, but this component is not scheduled to be updated in the current Red Hat Enterprise Linux release. If you would like this request to be reviewed for the next minor release, ask your support representative to set the next rhel-x.y flag to "?".
Just a note that it's possible to test this using the 'virsh' command (assuming you have libvirt 0.3.3, see bug 306931). # virsh dumpxml 1 | grep 'target' <target dev='vif1.0'/> <target dev='hda'/> # virsh domifstat 1 vif1.0 vif1.0 rx_bytes 0 vif1.0 rx_packets 0 vif1.0 rx_errs 0 vif1.0 rx_drop 103 vif1.0 tx_bytes 0 vif1.0 tx_packets 0 vif1.0 tx_errs 0 vif1.0 tx_drop 0 # virsh domblkstat 1 hda Network stats should work always. Block device (disk) stats are more problematic. They are most likely to work with a paravirt, partition-backed domain. They probably *don't* work for the following: (a) Fullvirt domains without PV drivers. (b) File-backed domains (because of lack of Xen support). (c) QEMU (because of lack of libvirt support).
In addition to the above mentioned c/s, we need 12333, 12334, 13325, and 14044. So, the full list is: 12333 12334 13325 14040 14044 15078 15389 15390 15402 With all of the above changesets in place, I was able to generate a decent patchset for the RHEL-5 kernel. I will attach the various patches that apply cleanly to a 2.6.18-77.el5 kernel. Note that this has only been compile tested so far on i686; it will obviously need compile and run-time testing on all arches before we can put it in. Chris Lalancette
Created attachment 293926 [details] Patch 1 for blktap statistics
Created attachment 293927 [details] Patch 2 for blktap statistics
Created attachment 293928 [details] Patch 3 for blktap statistics
Created attachment 293929 [details] Patch 4 for blktap statistics
Created attachment 293930 [details] Patch 5 for blktap statistics
Created attachment 293931 [details] Patch 6 for blktap statistics
Created attachment 293932 [details] Patch 7 for blktap statistics
Created attachment 293933 [details] Patch 8 for blktap statistics
Created attachment 293934 [details] Patch 9 for blktap statistics
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release.
The first 2 patches in this series to add barrier requests turned out to be controversial. Because they weren't actually integral to the patchset, I've removed these two and re-diffed the patches. I'll attach the 7 patches in this series here. Chris Lalancette
Created attachment 312441 [details] Patch 1 for blktap statistics v2
Created attachment 312442 [details] Patch 2 for blktap statistics v2
Created attachment 312443 [details] Patch 3 for blktap statistics v2
Created attachment 312444 [details] Patch 4 for blktap statistics v2
Created attachment 312445 [details] Patch 5 for blktap statistics v2
Created attachment 312446 [details] Patch 6 for blktap statistics v2
Created attachment 312447 [details] Patch 7 for blktap statistics v2
in kernel-2.6.18-100.el5 You can download this test kernel from http://people.redhat.com/dzickus/el5
This bug has been marked for inclusion in the Red Hat Enterprise Linux 5.3 Release Notes. To aid in the development of relevant and accurate release notes, please fill out the "Release Notes" field above with the following 4 pieces of information: Cause: What actions or circumstances cause this bug to present. Consequence: What happens when the bug presents. Fix: What was done to fix the bug. Result: What now happens when the actions or circumstances above occur. (NB: this is not the same as 'the bug doesn't present anymore')
Release note added. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: In previous versions of RHEL-5, it was not possible to monitor the transfer statistics of blktap backed Xen guests. These are guests that use files as their backend, represented by tap:aio in the /etc/xen configuration file. This set of patches introduces blktap statistics into RHEL-5. With these patches in place, you can correctly monitor the transfer statistics of blktap backed guests.
Release note updated. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,3 +1 @@ -In previous versions of RHEL-5, it was not possible to monitor the transfer statistics of blktap backed Xen guests. These are guests that use files as their backend, represented by tap:aio in the /etc/xen configuration file. +The blktap (blocktap) userspace toolkit has been updated providing the functionality to monitor the transfer statistics of blktap (block tap) backed virtualized guests.- -This set of patches introduces blktap statistics into RHEL-5. With these patches in place, you can correctly monitor the transfer statistics of blktap backed guests.
Release note updated. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1 @@ -The blktap (blocktap) userspace toolkit has been updated providing the functionality to monitor the transfer statistics of blktap (block tap) backed virtualized guests.+The blktap (blocktap) userspace toolkit has been updated providing the functionality to monitor the transfer statistics of blktap backed virtualized guests.
Release note updated. If any revisions are required, please set the "requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1 @@ -The blktap (blocktap) userspace toolkit has been updated providing the functionality to monitor the transfer statistics of blktap backed virtualized guests.+The blktap (blocktap) userspace toolkit has been updated, providing the functionality to monitor the transfer statistics of blktap backed virtualized guests.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2009-0225.html
Just clearing a needinfo request on me. Chris Lalancette