1113116 – [RFE] add API to query the stats of multiple VMs at once

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1113116 - [RFE] add API to query the stats of multiple VMs at once

Summary: [RFE] add API to query the stats of multiple VMs at once

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	7.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Peter Krempa
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1102147 1116365 1390171
TreeView+	depends on / blocked

Reported:	2014-06-25 13:00 UTC by Francesco Romani
Modified:	2016-10-31 12:05 UTC (History)
CC List:	9 users (show)
Fixed In Version:	libvirt-1.2.8-5.el7
Doc Type:	Enhancement
Doc Text:	Feature: Libvirt historically lacked a way to query statistics about a VM in a unified way. A user was then required to call multiple calls to gather statistics and possibly also parse the live domain XML. Reason: Management apps built on libvirt usually query a lot of statistics of the VMs they are managing. Having to do multiple calls per VM hinders performance. Result: The new API allows to query a lot of statistics for all the VMs in one call.
Clone Of:
Environment:
Last Closed:	2015-03-05 07:38:24 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2015:0323	0	normal	SHIPPED_LIVE	Low: libvirt security, bug fix, and enhancement update	2015-03-05 12:10:54 UTC

Description Francesco Romani 2014-06-25 13:00:34 UTC

Description of problem:
libvirt has APIs to query the statistics of a domain, for example virDomainGetInfo, virDomainBlockInfo.

All these API are synchronous. Thus, a client application, like VDSM, needs to spawn a thread per Domain/VM in order to run all the queries.

This leads to waste of resources like the one found in https://bugzilla.redhat.com/show_bug.cgi?id=1102147#c54

An API to query multiple Domain/VMs at once would be greatly beneficial.
Please note that this is both to save resources and to improve performance.

Comment 1 Francesco Romani 2014-06-25 13:08:43 UTC

Affected APIs as in vdsm 4.15 / oVirt 3.5.0:

virDomainBlockInfo
virDomainGetInfo
virDomainGetCPUStats
virDomainBlockStats
virDomainBlockStatsFlags
virDomainInterfaceStats
virDomainGetVcpusFlags
virDomainGetMetadata

and their python counterpart

Comment 3 Francesco Romani 2014-07-08 06:22:53 UTC

from the list on https://bugzilla.redhat.com/show_bug.cgi?id=1113116#c1

virDomainGetMetadata can be elided once https://bugzilla.redhat.com/show_bug.cgi?id=1114492 is fixed

Comment 4 Francesco Romani 2014-08-19 14:25:31 UTC

Updated list. Narrowed down to

virDomainBlockInfo
virDomainGetInfo
virDomainGetCPUStats
virDomainBlockStatsFlags
virDomainInterfaceStats
virDomainGetVcpusFlags

Comment 6 Peter Krempa 2014-10-01 09:19:07 UTC

The functionality should now be finished upstream. On top of libvirt-1.2.8 the following commits are required:

1f4831ee6ecc17d0f2008d7db15bfd9bc3b1d685
9ebbb8699eab6f9115b8ecbe171ee687a5dee198
2a9bd4a8736fcb90e231b9f7109aa66d9108b286
74c066df4d8d9d49eee6a4416752440cdfaa8005
c8e523722ecfced8e937f106b0a07be6c32febdb
290e3c6b07a273444f681de198d2416ccd2d4702
1db475650a0fb6ebf4ac62a5067f10905319e313
b7dd3d9b46d3f0cbc22a637e30d932f94dc44a11
a6500ee4f2f0db3fb8c408075b7dbcd77f7669d2
74cf34bc94c5147b217a86f7f0c2686dc86b62bb
96c0f57a82ef73ca924b7333d1001f05ecf5df86
bd2b0968c1bc5950685f61829ed711625e8e9850
8caded6b8ec567eadf5a339ba221d5de3b03a509
7557ddf8be79f9687bc8f69946aba71c4c29d7d5

Comment 9 Luyao Huang 2014-10-22 07:55:59 UTC

Verify this bug with libvirt-1.2.8-5.el7:

Steps:

1.Check man of virsh:

       domstats [--raw] [--enforce] [--state] [--cpu-total] [--balloon] [--vcpu] [--interface] [--block] [[--list-active] [--list-inactive] [--list-persistent] [--list-transient]
       [--list-running] [--list-paused] [--list-shutoff] [--list-other]] | [domain ...]
           Get statistics for multiple or all domains. Without any argument this command prints all available statistics for all domains.

           The list of domains to gather stats for can be either limited by listing the domains as a space separated list, or by specifying one of the filtering flags --list-*.
           (The approaches can't be combined.)

           By default some of the returned fields may be converted to more human friendly values by a set of pretty-printers. To suppress this behavior use the --raw flag.

           The individual statistics groups are selectable via specific flags. By default all supported statistics groups are returned. Supported statistics groups flags are:
           --state, --cpu-total, --balloon, --vcpu, --interface, --block.

           When selecting the --state group the following fields are returned: "state.state" - state of the VM, returned as number from virDomainState enum, "state.reason" - reason
           for entering given state, returned as int from, virDomain*Reason enum corresponding to given state.

           --cpu-total returns: "cpu.time" - total cpu time spent for this domain in nanoseconds, "cpu.user" - user cpu time spent in nanoseconds, "cpu.system" - system cpu time
           spent in nanoseconds

           --balloon returns: "balloon.current" - the memory in kiB currently used, "balloon.maximum" - the maximum memory in kiB allowed

           --vcpu returns: "vcpu.current" - current number of online virtual CPUs, "vcpu.maximum" - maximum number of online virtual CPUs, "vcpu.<num>.state" - state of the virtual
           CPU <num>, as number from virVcpuState enum, "vcpu.<num>.time" - virtual cpu time spent by virtual CPU <num>

           --interface returns: "net.count" - number of network interfaces on this domain, "net.<num>.name" - name of the interface <num>, "net.<num>.rx.bytes" - number of bytes
           received, "net.<num>.rx.pkts" - number of packets received, "net.<num>.rx.errs" - number of receive errors, "net.<num>.rx.drop" - number of receive packets dropped,
           "net.<num>.tx.bytes" - number of bytes transmitted, "net.<num>.tx.pkts" - number of packets transmitted, "net.<num>.tx.errs" - number of transmission errors,
           "net.<num>.tx.drop" - number of transmit packets dropped,

           block returns: "block.count" - number of block devices on this domain, "block.<num>.name" - name of the target of the block device <num>, "block.<num>.rd.reqs" - number
           of read requests, "block.<num>.rd.bytes" - number of read bytes, "block.<num>.rd.times" - total time (ns) spent on reads, "block.<num>.wr.reqs" - number of write
           requests, "block.<num>.wr.bytes" - number of written bytes, "block.<num>.wr.times" - total time (ns) spent on writes, "block.<num>.fl.reqs" - total flush requests,
           "block.<num>.fl.times" - total time (ns) spent on cache flushing, "block.<num>.errors" - Xen only: the 'oo_req' value,

           Selecting a specific statistics groups doesn't guarantee that the daemon supports the selected group of stats. Flag --enforce forces the command to fail if the daemon
           doesn't support the selected group.

There is a small issue block should be --block.But this issue is so small.

2.check document in src/libvirt.c 

21706  * The statistic groups are enabled using the @stats parameter which is a
21707  * binary-OR of enum virDomainStatsTypes. The stats groups are documented
21708  * in virConnectGetAllDomainStats.

21625  * Note that entire stats groups or individual stat fields may be missing from
21626  * the output in case they are not supported by the given hypervisor, are not
21627  * applicable for the current state of the guest domain, or their retrieval
21628  * was not successful.

3.test new options for domstats:

# virsh domstats --help
  NAME
    domstats - get statistics about one or multiple domains

  SYNOPSIS
    domstats [--state] [--cpu-total] [--balloon] [--vcpu] [--interface] [--block] [--list-active] [--list-inactive] [--list-persistent] [--list-transient] [--list-running] [--list-paused] [--list-shutoff] [--list-other] [--raw] [--enforce] [<domain>]...

  DESCRIPTION
    Gets statistics about one or more (or all) domains

  OPTIONS
    --state          report domain state
    --cpu-total      report domain physical cpu usage
    --balloon        report domain balloon statistics
    --vcpu           report domain virtual cpu information
    --interface      report domain network interface information
    --block          report domain block device statistics
    --list-active    list only active domains
    --list-inactive  list only inactive domains
    --list-persistent  list only persistent domains
    --list-transient  list only transient domains
    --list-running   list only running domains
    --list-paused    list only paused domains
    --list-shutoff   list only shutoff domains
    --list-other     list only domains in other states
    --raw            do not pretty-print the fields
    --enforce        enforce requested stats parameters
    <domain>         list of domains to get stats for

# virsh domstats
Domain: 'rhel6'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'test4'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=5
  vcpu.0.state=0

Domain: 'test3'
  state.state=1
  state.reason=1
  cpu.time=214671870344
  cpu.user=63850000000
  cpu.system=129580000000
  balloon.current=1048576
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=2
  vcpu.0.state=1
  vcpu.0.time=14970000000
  net.count=1
  net.0.name=vnet0
  net.0.rx.bytes=36807
  net.0.rx.pkts=696
  net.0.rx.errs=0
  net.0.rx.drop=0
  net.0.tx.bytes=1175
  net.0.tx.pkts=13
  net.0.tx.errs=0
  net.0.tx.drop=0
  block.count=2
  block.0.name=hda
  block.0.rd.reqs=45769
  block.0.rd.bytes=89470464
  block.0.rd.times=7828066472
  block.0.wr.reqs=387
  block.0.wr.bytes=3751936
  block.0.wr.times=108619545
  block.0.fl.reqs=52
  block.0.fl.times=812211991
  block.0.allocation=3221089792
  block.0.capacity=4294967296
  block.0.physical=4294967296
  block.1.name=vda
  block.1.rd.reqs=168
  block.1.rd.bytes=688128
  block.1.rd.times=7860246
  block.1.wr.reqs=0
  block.1.wr.bytes=0
  block.1.wr.times=0
  block.1.fl.reqs=0
  block.1.fl.times=0
  block.1.allocation=0
  block.1.capacity=2174156800
  block.1.physical=2174156800

Domain: 'win7'
  state.state=5
  state.reason=0
  balloon.maximum=2069504
  vcpu.current=2
  vcpu.maximum=4
  vcpu.0.state=0
  vcpu.1.state=0

Domain: 'test5'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=20
  vcpu.maximum=20
  vcpu.0.state=0
  vcpu.1.state=0
  vcpu.2.state=0
  vcpu.3.state=0
  vcpu.4.state=0
  vcpu.5.state=0
  vcpu.6.state=0
  vcpu.7.state=0
  vcpu.8.state=0
  vcpu.9.state=0
  vcpu.10.state=0
  vcpu.11.state=0
  vcpu.12.state=0
  vcpu.13.state=0
  vcpu.14.state=0
  vcpu.15.state=0
  vcpu.16.state=0
  vcpu.17.state=0
  vcpu.18.state=0
  vcpu.19.state=0

Domain: 'r6-qcow2'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'test3-clone'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=2
  vcpu.maximum=2
  vcpu.0.state=0
  vcpu.1.state=0

Domain: 'r6'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=2
  vcpu.maximum=2
  vcpu.0.state=0
  vcpu.1.state=0

Domain: 'test'
  state.state=5
  state.reason=0
  balloon.maximum=524288
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'etsd'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'r7'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=4
  vcpu.maximum=4
  vcpu.0.state=0
  vcpu.1.state=0
  vcpu.2.state=0
  vcpu.3.state=0

Domain: 'test34'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'r64'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'raw'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'test6'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

# virsh domstats test3
Domain: 'test3'
  state.state=1
  state.reason=1
  cpu.time=220121563309
  cpu.user=65540000000
  cpu.system=132960000000
  balloon.current=1048576
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=2
  vcpu.0.state=1
  vcpu.0.time=14990000000
  net.count=1
  net.0.name=vnet0
  net.0.rx.bytes=37830
  net.0.rx.pkts=715
  net.0.rx.errs=0
  net.0.rx.drop=0
  net.0.tx.bytes=1175
  net.0.tx.pkts=13
  net.0.tx.errs=0
  net.0.tx.drop=0
  block.count=2
  block.0.name=hda
  block.0.rd.reqs=45769
  block.0.rd.bytes=89470464
  block.0.rd.times=7828066472
  block.0.wr.reqs=388
  block.0.wr.bytes=3756032
  block.0.wr.times=109075851
  block.0.fl.reqs=52
  block.0.fl.times=812211991
  block.0.allocation=3221089792
  block.0.capacity=4294967296
  block.0.physical=4294967296
  block.1.name=vda
  block.1.rd.reqs=168
  block.1.rd.bytes=688128
  block.1.rd.times=7860246
  block.1.wr.reqs=0
  block.1.wr.bytes=0
  block.1.wr.times=0
  block.1.fl.reqs=0
  block.1.fl.times=0
  block.1.allocation=0
  block.1.capacity=2174156800
  block.1.physical=2174156800

# virsh domstats test3 test4
Domain: 'test3'
  state.state=1
  state.reason=1
  cpu.time=221465018271
  cpu.user=65970000000
  cpu.system=133760000000
  balloon.current=1048576
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=2
  vcpu.0.state=1
  vcpu.0.time=15000000000
  net.count=1
  net.0.name=vnet0
  net.0.rx.bytes=38090
  net.0.rx.pkts=720
  net.0.rx.errs=0
  net.0.rx.drop=0
  net.0.tx.bytes=1175
  net.0.tx.pkts=13
  net.0.tx.errs=0
  net.0.tx.drop=0
  block.count=2
  block.0.name=hda
  block.0.rd.reqs=45769
  block.0.rd.bytes=89470464
  block.0.rd.times=7828066472
  block.0.wr.reqs=391
  block.0.wr.bytes=3768320
  block.0.wr.times=110026388
  block.0.fl.reqs=54
  block.0.fl.times=843742540
  block.0.allocation=3221089792
  block.0.capacity=4294967296
  block.0.physical=4294967296
  block.1.name=vda
  block.1.rd.reqs=168
  block.1.rd.bytes=688128
  block.1.rd.times=7860246
  block.1.wr.reqs=0
  block.1.wr.bytes=0
  block.1.wr.times=0
  block.1.fl.reqs=0
  block.1.fl.times=0
  block.1.allocation=0
  block.1.capacity=2174156800
  block.1.physical=2174156800

Domain: 'test4'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=5
  vcpu.0.state=0


# virsh domstats test3 --cpu-total
Domain: 'test3'
  cpu.time=225936089727
  cpu.user=67320000000
  cpu.system=136540000000


# virsh domstats test3 --cpu-total --balloon
Domain: 'test3'
  cpu.time=226713396977
  cpu.user=67560000000
  cpu.system=137030000000
  balloon.current=1048576
  balloon.maximum=1048576

# virsh domstats test3 --cpu-total --balloon --vcpu
Domain: 'test3'
  cpu.time=233694024516
  cpu.user=69730000000
  cpu.system=141320000000
  balloon.current=1048576
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=2
  vcpu.0.state=1
  vcpu.0.time=15070000000

# virsh domstats test3 --cpu-total --balloon --vcpu --interface
Domain: 'test3'
  cpu.time=234827299812
  cpu.user=70110000000
  cpu.system=141990000000
  balloon.current=1048576
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=2
  vcpu.0.state=1
  vcpu.0.time=15080000000
  net.count=1
  net.0.name=vnet0
  net.0.rx.bytes=40908
  net.0.rx.pkts=769
  net.0.rx.errs=0
  net.0.rx.drop=0
  net.0.tx.bytes=1685
  net.0.tx.pkts=18
  net.0.tx.errs=0
  net.0.tx.drop=0

# virsh domstats test3 --cpu-total --balloon --vcpu --interface --block
Domain: 'test3'
  cpu.time=236034500913
  cpu.user=70480000000
  cpu.system=142750000000
  balloon.current=1048576
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=2
  vcpu.0.state=1
  vcpu.0.time=15080000000
  net.count=1
  net.0.name=vnet0
  net.0.rx.bytes=41116
  net.0.rx.pkts=773
  net.0.rx.errs=0
  net.0.rx.drop=0
  net.0.tx.bytes=1685
  net.0.tx.pkts=18
  net.0.tx.errs=0
  net.0.tx.drop=0
  block.count=2
  block.0.name=hda
  block.0.rd.reqs=45769
  block.0.rd.bytes=89470464
  block.0.rd.times=7828066472
  block.0.wr.reqs=402
  block.0.wr.bytes=3821568
  block.0.wr.times=113168223
  block.0.fl.reqs=58
  block.0.fl.times=916528178
  block.0.allocation=3221089792
  block.0.capacity=4294967296
  block.0.physical=4294967296
  block.1.name=vda
  block.1.rd.reqs=168
  block.1.rd.bytes=688128
  block.1.rd.times=7860246
  block.1.wr.reqs=0
  block.1.wr.bytes=0
  block.1.wr.times=0
  block.1.fl.reqs=0
  block.1.fl.times=0
  block.1.allocation=0
  block.1.capacity=2174156800
  block.1.physical=2174156800


# virsh domstats test3 --cpu-total --balloon --vcpu --interface --block --state
Domain: 'test3'
  state.state=1
  state.reason=1
  cpu.time=237455734124
  cpu.user=70920000000
  cpu.system=143620000000
  balloon.current=1048576
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=2
  vcpu.0.state=1
  vcpu.0.time=15090000000
  net.count=1
  net.0.name=vnet0
  net.0.rx.bytes=41376
  net.0.rx.pkts=778
  net.0.rx.errs=0
  net.0.rx.drop=0
  net.0.tx.bytes=1685
  net.0.tx.pkts=18
  net.0.tx.errs=0
  net.0.tx.drop=0
  block.count=2
  block.0.name=hda
  block.0.rd.reqs=45769
  block.0.rd.bytes=89470464
  block.0.rd.times=7828066472
  block.0.wr.reqs=402
  block.0.wr.bytes=3821568
  block.0.wr.times=113168223
  block.0.fl.reqs=58
  block.0.fl.times=916528178
  block.0.allocation=3221089792
  block.0.capacity=4294967296
  block.0.physical=4294967296
  block.1.name=vda
  block.1.rd.reqs=168
  block.1.rd.bytes=688128
  block.1.rd.times=7860246
  block.1.wr.reqs=0
  block.1.wr.bytes=0
  block.1.wr.times=0
  block.1.fl.reqs=0
  block.1.fl.times=0
  block.1.allocation=0
  block.1.capacity=2174156800
  block.1.physical=2174156800

# virsh detach-disk test3 vda
Disk detached successfully

# virsh domstats test3 --block
Domain: 'test3'
  block.count=1
  block.0.name=hda
  block.0.rd.reqs=45769
  block.0.rd.bytes=89470464
  block.0.rd.times=7828066472
  block.0.wr.reqs=409
  block.0.wr.bytes=3866624
  block.0.wr.times=115275935
  block.0.fl.reqs=62
  block.0.fl.times=972375118
  block.0.allocation=3221089792
  block.0.capacity=4294967296
  block.0.physical=4294967296

# virsh attach-interface test3 network default 
Interface attached successfully

# virsh domstats test3 --interface
Domain: 'test3'
  net.count=2
  net.0.name=vnet0
  net.0.rx.bytes=44808
  net.0.rx.pkts=844
  net.0.rx.errs=0
  net.0.rx.drop=0
  net.0.tx.bytes=1685
  net.0.tx.pkts=18
  net.0.tx.errs=0
  net.0.tx.drop=0
  net.1.name=vnet1
  net.1.rx.bytes=908
  net.1.rx.pkts=13
  net.1.rx.errs=0
  net.1.rx.drop=0
  net.1.tx.bytes=0
  net.1.tx.pkts=0
  net.1.tx.errs=0
  net.1.tx.drop=0

# virsh setvcpus test3 2

# virsh domstats test3 --vcpu
Domain: 'test3'
  vcpu.current=2
  vcpu.maximum=2
  vcpu.0.state=1
  vcpu.0.time=15360000000
  vcpu.1.state=1
  vcpu.1.time=0

# virsh domstats test3 test4 --vcpu --state
Domain: 'test3'
  state.state=1
  state.reason=1
  vcpu.current=2
  vcpu.maximum=2
  vcpu.0.state=1
  vcpu.0.time=15410000000
  vcpu.1.state=1
  vcpu.1.time=10000000

Domain: 'test4'
  state.state=5
  state.reason=0
  vcpu.current=1
  vcpu.maximum=5
  vcpu.0.state=0

# virsh domstats --list-other --vcpu --state
Domain: 'r6'
  state.state=7
  state.reason=0
  vcpu.current=2
  vcpu.maximum=2
  vcpu.0.state=1
  vcpu.0.time=42440000000
  vcpu.1.state=1
  vcpu.1.time=8080000000


# virsh domstats --list-shutoff --vcpu
Domain: 'test34'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'etsd'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'r6-qcow2'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'r7'
  vcpu.current=4
  vcpu.maximum=4
  vcpu.0.state=0
  vcpu.1.state=0
  vcpu.2.state=0
  vcpu.3.state=0

Domain: 'test6'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'raw'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'test4'
  vcpu.current=1
  vcpu.maximum=5
  vcpu.0.state=0

Domain: 'win7'
  vcpu.current=2
  vcpu.maximum=4
  vcpu.0.state=0
  vcpu.1.state=0

Domain: 'test'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'r64'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'test5'
  vcpu.current=20
  vcpu.maximum=20
  vcpu.0.state=0
  vcpu.1.state=0
  vcpu.2.state=0
  vcpu.3.state=0
  vcpu.4.state=0
  vcpu.5.state=0
  vcpu.6.state=0
  vcpu.7.state=0
  vcpu.8.state=0
  vcpu.9.state=0
  vcpu.10.state=0
  vcpu.11.state=0
  vcpu.12.state=0
  vcpu.13.state=0
  vcpu.14.state=0
  vcpu.15.state=0
  vcpu.16.state=0
  vcpu.17.state=0
  vcpu.18.state=0
  vcpu.19.state=0

Domain: 'test3-clone'
  vcpu.current=2
  vcpu.maximum=2
  vcpu.0.state=0
  vcpu.1.state=0

Domain: 'rhel6'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'test3'
  vcpu.current=1
  vcpu.maximum=2
  vcpu.0.state=0


# virsh dompmwakeup r6
Domain r6 successfully woken up

# virsh domstats --list-running --vcpu
Domain: 'r6'
  vcpu.current=2
  vcpu.maximum=2
  vcpu.0.state=1
  vcpu.0.time=42760000000
  vcpu.1.state=1
  vcpu.1.time=8410000000

# virsh domstats --list-persistent --interface
Domain: 'test34'

Domain: 'etsd'

Domain: 'r6-qcow2'

Domain: 'r7'

Domain: 'test6'

Domain: 'raw'

Domain: 'test4'

Domain: 'win7'

Domain: 'test'

Domain: 'r64'

Domain: 'test5'

Domain: 'test3-clone'

Domain: 'rhel6'

Domain: 'test3'

Domain: 'r6'
  net.count=2
  net.0.name=vnet0
  net.0.rx.bytes=16826
  net.0.rx.pkts=289
  net.0.rx.errs=0
  net.0.rx.drop=0
  net.0.tx.bytes=1152
  net.0.tx.pkts=8
  net.0.tx.errs=0
  net.0.tx.drop=0
  net.1.name=vnet1
  net.1.rx.bytes=17286
  net.1.rx.pkts=292
  net.1.rx.errs=0
  net.1.rx.drop=0
  net.1.tx.bytes=1842
  net.1.tx.pkts=14
  net.1.tx.errs=0
  net.1.tx.drop=0

# virsh domstats --list-inactive --vcpu
Domain: 'test34'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'etsd'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'r6-qcow2'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'r7'
  vcpu.current=4
  vcpu.maximum=4
  vcpu.0.state=0
  vcpu.1.state=0
  vcpu.2.state=0
  vcpu.3.state=0

Domain: 'test6'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'raw'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'test4'
  vcpu.current=1
  vcpu.maximum=5
  vcpu.0.state=0

Domain: 'win7'
  vcpu.current=2
  vcpu.maximum=4
  vcpu.0.state=0
  vcpu.1.state=0

Domain: 'test'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'r64'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'test5'
  vcpu.current=20
  vcpu.maximum=20
  vcpu.0.state=0
  vcpu.1.state=0
  vcpu.2.state=0
  vcpu.3.state=0
  vcpu.4.state=0
  vcpu.5.state=0
  vcpu.6.state=0
  vcpu.7.state=0
  vcpu.8.state=0
  vcpu.9.state=0
  vcpu.10.state=0
  vcpu.11.state=0
  vcpu.12.state=0
  vcpu.13.state=0
  vcpu.14.state=0
  vcpu.15.state=0
  vcpu.16.state=0
  vcpu.17.state=0
  vcpu.18.state=0
  vcpu.19.state=0

Domain: 'test3-clone'
  vcpu.current=2
  vcpu.maximum=2
  vcpu.0.state=0
  vcpu.1.state=0

Domain: 'rhel6'
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'test3'
  vcpu.current=1
  vcpu.maximum=2
  vcpu.0.state=0

Comment 10 Luyao Huang 2014-10-23 09:08:00 UTC

4.check memory :
# virsh domstats

Domain: 'test34'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'etsd'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'r6-qcow2'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'r7'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=4
  vcpu.maximum=4
  vcpu.0.state=0
  vcpu.1.state=0
  vcpu.2.state=0
  vcpu.3.state=0

Domain: 'test6'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'raw'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'test4'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=5
  vcpu.0.state=0

Domain: 'win7'
  state.state=5
  state.reason=2
  balloon.maximum=2069504
  vcpu.current=2
  vcpu.maximum=4
  vcpu.0.state=0
  vcpu.1.state=0

Domain: 'test'
  state.state=5
  state.reason=0
  balloon.maximum=524288
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'r64'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'test5'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=20
  vcpu.maximum=20
  vcpu.0.state=0
  vcpu.1.state=0
  vcpu.2.state=0
  vcpu.3.state=0
  vcpu.4.state=0
  vcpu.5.state=0
  vcpu.6.state=0
  vcpu.7.state=0
  vcpu.8.state=0
  vcpu.9.state=0
  vcpu.10.state=0
  vcpu.11.state=0
  vcpu.12.state=0
  vcpu.13.state=0
  vcpu.14.state=0
  vcpu.15.state=0
  vcpu.16.state=0
  vcpu.17.state=0
  vcpu.18.state=0
  vcpu.19.state=0

Domain: 'test3-clone'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=2
  vcpu.maximum=2
  vcpu.0.state=0
  vcpu.1.state=0

Domain: 'rhel6'
  state.state=5
  state.reason=0
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=1
  vcpu.0.state=0

Domain: 'test3'
  state.state=1
  state.reason=1
  cpu.time=1096447270349
  cpu.user=328510000000
  cpu.system=695060000000
  balloon.current=1048576
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=2
  vcpu.0.state=1
  vcpu.0.time=19110000000
  net.count=1
  net.0.name=vnet0
  net.0.rx.bytes=196827
  net.0.rx.pkts=3746
  net.0.rx.errs=0
  net.0.rx.drop=0
  net.0.tx.bytes=3795
  net.0.tx.pkts=39
  net.0.tx.errs=0
  net.0.tx.drop=0
  block.count=1
  block.0.name=hda
  block.0.rd.reqs=45898
  block.0.rd.bytes=88999424
  block.0.rd.times=8177028688
  block.0.wr.reqs=779
  block.0.wr.bytes=4816896
  block.0.wr.times=1541997742
  block.0.fl.reqs=272
  block.0.fl.times=3521448382
  block.0.allocation=3221089792
  block.0.capacity=4294967296
  block.0.physical=4294967296

Domain: 'r6'
  state.state=5
  state.reason=2
  balloon.maximum=1048576
  vcpu.current=2
  vcpu.maximum=2
  vcpu.0.state=0
  vcpu.1.state=0


==30515== 
==30515== HEAP SUMMARY:
==30515==     in use at exit: 109,021 bytes in 1,093 blocks
==30515==   total heap usage: 5,680 allocs, 4,587 frees, 1,484,320 bytes allocated
==30515== 
==30515== Searching for pointers to 1,093 not-freed blocks
==30515== Checked 1,487,152 bytes
==30515== 
==30515== 40 bytes in 1 blocks are possibly lost in loss record 118 of 189
==30515==    at 0x4C29BBD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==30515==    by 0xBA21FDB: _PR_Getfd (prfdcach.c:112)
==30515==    by 0xBA3BCB6: pt_SetMethods.isra.11 (ptio.c:3303)
==30515==    by 0xBA3C2A4: PR_OpenFile (ptio.c:3581)
==30515==    by 0xD951989: blapi_SHVerifyFile (shvfy.c:355)
==30515==    by 0xD951CB0: blapi_SHVerify (shvfy.c:289)
==30515==    by 0xD927AE9: freebl_fipsSoftwareIntegrityTest (fipsfreebl.c:1541)
==30515==    by 0xD927AE9: bl_startup_tests (fipsfreebl.c:1732)
==30515==    by 0x400F502: call_init (dl-init.c:82)
==30515==    by 0x400F502: _dl_init (dl-init.c:131)
==30515==    by 0x4001459: ??? (in /usr/lib64/ld-2.17.so)
==30515==    by 0x1: ???
==30515==    by 0xFFF0002EE: ???
==30515==    by 0xFFF0002F4: ???
==30515== 
==30515== 48 bytes in 1 blocks are possibly lost in loss record 123 of 189
==30515==    at 0x4C29BBD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==30515==    by 0xBA21FC9: _PR_Getfd (prfdcach.c:109)
==30515==    by 0xBA3BCB6: pt_SetMethods.isra.11 (ptio.c:3303)
==30515==    by 0xBA3C2A4: PR_OpenFile (ptio.c:3581)
==30515==    by 0xD951989: blapi_SHVerifyFile (shvfy.c:355)
==30515==    by 0xD951CB0: blapi_SHVerify (shvfy.c:289)
==30515==    by 0xD927AE9: freebl_fipsSoftwareIntegrityTest (fipsfreebl.c:1541)
==30515==    by 0xD927AE9: bl_startup_tests (fipsfreebl.c:1732)
==30515==    by 0x400F502: call_init (dl-init.c:82)
==30515==    by 0x400F502: _dl_init (dl-init.c:131)
==30515==    by 0x4001459: ??? (in /usr/lib64/ld-2.17.so)
==30515==    by 0x1: ???
==30515==    by 0xFFF0002EE: ???
==30515==    by 0xFFF0002F4: ???
==30515== 
==30515== LEAK SUMMARY:
==30515==    definitely lost: 0 bytes in 0 blocks
==30515==    indirectly lost: 0 bytes in 0 blocks
==30515==      possibly lost: 88 bytes in 2 blocks
==30515==    still reachable: 108,933 bytes in 1,091 blocks
==30515==         suppressed: 0 bytes in 0 blocks
==30515== Reachable blocks (those to which a pointer was found) are not shown.
==30515== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30515== 
==30515== Use --track-origins=yes to see where uninitialised values come from
==30515== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 2 from 2)
==30515== 
==30515== 1 errors in context 1 of 3:
==30515== Conditional jump or move depends on uninitialised value(s)
==30515==    at 0x85E9402: xdr_array (xdr_array.c:88)
==30515==    by 0x4FD8FC9: xdr_remote_connect_get_all_domain_stats_args (remote_protocol.c:6473)
==30515==    by 0x4FE72F2: virNetMessageEncodePayload (virnetmessage.c:350)
==30515==    by 0x4FDD21C: virNetClientProgramCall (virnetclientprogram.c:326)
==30515==    by 0x4FB4D01: callFull.isra.2 (remote_driver.c:6667)
==30515==    by 0x4FCBD45: call (remote_driver.c:6689)
==30515==    by 0x4FCBD45: remoteConnectGetAllDomainStats (remote_driver.c:7793)
==30515==    by 0x4FA0E75: virConnectGetAllDomainStats (libvirt.c:21678)
==30515==    by 0x147FD1: cmdDomstats (virsh-domain-monitor.c:2148)
==30515==    by 0x13006B: vshCommandRun (virsh.c:1915)
==30515==    by 0x12A9E1: main (virsh.c:3699)
==30515== 
--30515-- 
--30515-- used_suppression:      2 glibc-2.5.x-on-SUSE-10.2-(PPC)-2a /usr/lib64/valgrind/default.supp:1296
==30515== 
==30515== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 2 from 2)

Seems there is some problem.

Comment 11 Luyao Huang 2014-10-24 08:38:08 UTC

5.use gdb or check code for commit:
96c0f57a82ef73ca924b7333d1001f05ecf5df86
bd2b0968c1bc5950685f61829ed711625e8e9850
8caded6b8ec567eadf5a339ba221d5de3b03a509
7557ddf8be79f9687bc8f69946aba71c4c29d7d5
58699b77eee9e45cbf8c9f08291fc298aae37eea

And i will file a bug for the issue in comment 10

Verify this bug now.

Comment 13 errata-xmlrpc 2015-03-05 07:38:24 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0323.html

Note You need to log in before you can comment on or make changes to this bug.