Bug 715019

Summary: (libvirt) Report disk latency (read and write) for each storage device
Product: Red Hat Enterprise Linux 6 Reporter: Andrew Cathrow <acathrow>
Component: libvirtAssignee: Osier Yang <jyang>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: high    
Version: 6.2CC: abaron, ajia, bsarathy, cpelland, dallan, danken, dyuan, eblake, gren, iheim, ilvovsky, mshao, mzhan, rbalakri, rjones, rwu, tburke, veillard, whuang, yupzhang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-0.9.9-1.el6 Doc Type: Bug Fix
Doc Text:
Cause: Management system wants to known the disk latency via libvirt. Consequense: None Fix: New API is introduced to allow the management system to query the disk lantency via libvirt. Result: Management system could get the latencies of each disk devices via the new API.
Story Points: ---
Clone Of:
: 740918 (view as bug list) Environment:
Last Closed: 2012-06-20 06:28:58 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 715017    
Bug Blocks: 740918, 747667, 756082, 773650, 773651, 773677, 773696    

Description Andrew Cathrow 2011-06-21 16:03:21 UTC
Measure disk latency (read and write) measured in ms for each virtual storage device in QEMU.
Expose through virDomainBlockStats API and domblkstat virsh command.


 Use case:

 Allow a management system to poll QEMU via libvirt to record the
 read
 and write disk latency.
 This management system could use this information to report on
 potential bottlenecks or I/O issues experienced by virtual
 machines.

 Later this information would be used by Red Hat tools as part of an
 "SLA" management application to set thresholds and guaranteed
 resources for virtual machines.

Comment 9 Eric Blake 2011-08-17 14:49:47 UTC
https://www.redhat.com/archives/libvir-list/2011-August/msg00496.html was a first cut at adding new fields, but it was not done correctly to avoid an API break.  I'll reply to that with how to properly extend information available to virDomainBlockStats.  Now the question is which fields to expose in a v2 of the struct.

Comment 21 Igor Lvovsky 2011-09-05 15:07:15 UTC
so, if everybody agree with virDomainQemuMonitorCommand for RHEL6.2 let's do it.

Add blocks for bug 720385

Comment 22 Osier Yang 2011-09-07 04:08:20 UTC
patches that introduces new API are in upstream. 

http://www.redhat.com/archives/libvir-list/2011-September/msg00276.html

Working on exposing the virDomainQemuMonitorCommand API to Python binding for RHEV use.

Comment 23 Osier Yang 2011-09-07 12:44:58 UTC
The initial thought is create a independant libvirt-qemu.py, and also libvirtmod-qemu.so, as the API is not support officially, working is still on process.

Comment 24 Osier Yang 2011-09-09 12:14:52 UTC
patchset v1 are posted to upstream.

https://www.redhat.com/archives/libvir-list/2011-September/msg00425.html

Comment 28 Dave Allan 2011-09-23 19:28:49 UTC
Hi Dan, 736040 tracks that; it's in VERIFIED, so you can try using the 0.9.4-12 build to make sure it works for you (it hasn't passed full QE yet).

Comment 30 Dave Allan 2011-11-10 20:11:29 UTC
Osier, can you provide the upstream commit details?

Comment 31 Osier Yang 2011-11-11 03:51:20 UTC
Patches are committed to upstream. They are available since "0.9.5: Sep 20 2011"

commit ffe28ab74b821c916ec4ba8efb5c992454e4bd24
Author: Osier Yang <jyang>
Date:   Mon Sep 5 16:24:51 2011 +0800

    latency: Update virsh command domblkstat to use new API
    
    The modified function fallbacks to use virDomainBlockStats if
    virDomainBlockStatsFlags is not supported by the hypervisor driver.
    If the new API is supported, it will be invoked instead of the
    old API.

commit f4c331a7444f46b12c9d12c3a3790dc4a28833b4
Author: Osier Yang <jyang>
Date:   Mon Sep 5 16:24:21 2011 +0800

    latency: Expose the new API for Python binding

commit ee0d8c3b5ca51c9525f926ee309beb2cbb721446
Author: Osier Yang <jyang>
Date:   Mon Sep 5 16:22:58 2011 +0800

    latency: Implemente internal API for qemu driver

commit 2f58ba89960fa7d63a3cae0c4580990b63137736
Author: Osier Yang <jyang>
Date:   Mon Sep 5 16:22:17 2011 +0800

    latency: Update monitor functions for new latency fields
    
    The mainly changes are:

    1) Update qemuMonitorGetBlockStatsInfo and it's children (Text/JSON)
       functions to return the value of new latency fields.
    2) Add new function qemuMonitorGetBlockStatsParamsNumber, which is
       to count how many parameters the underlying QEMU supports.
    3) Update virDomainBlockStats in src/qemu/qemu_driver.c to be
       compatible with the changes by 1).

commit efa7fc9f7507a5cdb9ce66078d25c0ef2906ad1d
Author: Osier Yang <jyang>
Date:   Mon Sep 5 16:20:03 2011 +0800

    latency: Wire up the remote protocol

commit 1f80c3eb86b6a2b319d57063221e80a93b578e7c
Author: Osier Yang <jyang>
Date:   Mon Sep 5 16:16:03 2011 +0800

    latency: Implemente the public API

commit 1c622a3d534f292aa7b0be7a393e31e76c37277f
Author: Osier Yang <jyang>
Date:   Mon Sep 5 16:15:14 2011 +0800

    latency: Define the internal driver callback

commit c843478ec805602c2236b58a3d8008fefbe4732d
Author: Osier Yang <jyang>
Date:   Mon Sep 5 16:14:29 2011 +0800

    latency: Define new public API and structure

Comment 33 Alex Jia 2012-01-11 08:53:43 UTC
The issue has been fixed on libvirt-0.9.9-1.el6.x86_64.

# virsh domblklist vr-rhel5u8-x86_64-kvm
Target     Source
------------------------------------------------
hda        /var/lib/libvirt/images/vr-rhel5u8-x86_64-kvm

# virsh domblkstat vr-rhel5u8-x86_64-kvm hda --human
Device: hda
 number of read operations:      5964
 number of bytes read:           29983744
 number of write operations:     10249
 number of bytes written:        1008155648
 number of flush operations:     366

# virsh domblkstat vr-rhel5u8-x86_64-kvm hda --human
Device: hda
 number of read operations:      5965
 number of bytes read:           29987840
 number of write operations:     10310
 number of bytes written:        1011235840
 number of flush operations:     366

Comment 34 Osier Yang 2012-05-04 10:02:50 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause: Management system wants to known the disk latency via libvirt.
Consequense: None
Fix: New API is introduced to allow the management system to query the disk lantency via libvirt.
Result: Management system could get the latencies of each disk devices via the new API.

Comment 36 errata-xmlrpc 2012-06-20 06:28:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0748.html