Bug 947974

Steps to Reproduce:
1. Run POWER-ON 500 VM's action, paralleled via PythonSDK

Impact on user:
None

this seems to be purely libvirt/qemu issue.
Please see related bug 947821 which oints to some libvirt issues during the same scenario like
libvirtError: internal error No such domain <C1><BA>8<E5><84><E2>NקDZ<CA>^B\<9D>^?

Hi, vvyazmin

I want to reproduce this bug with RHEVM+libvirt 
but I do not know how to start 500 vms via pythonSDK 
Can you give me some more details about pythonSDK to start vms in RHEVM ?
thanks very much .

Wenlong

From the original bug report I understand that the issue is that the error log is filled with messages about non-exting domains while they are mass-shutdowned by vdsm or a higher level management. This is not a libvirt issue as asking about a non-existing domain is valid and error is returned correctly. To avoid such messages the management needs to be adapted not to ask about guests that were already removed.

The damaged error message issue pointed out in libvirt was caused by printing the raw UUID buffer instead of a formatted string. The UUID buffer was fortunately always padded with zeroes, so a invalid read issue was not possible. The issue was fixed by:

commit 54a99ba86791beefd39732f07b03099cb3d85696
Author: Peter Krempa <pkrempa>
Date:   Wed Apr 10 14:04:45 2013 +0200

    qemu: Refactor lookup of domain object
    
    Use the helper to lookup the domain object in the remaining places.
    
    This patch also fixes error reporting when the domain was not found in several
    functions that were printing the raw UUID buffer instead of the formatted
    string. The offending functions were:
    
    qemuDomainGetInterfaceParameters
    qemuDomainSetInterfaceParameters
    qemuGetSchedulerParametersFlags
    qemuSetSchedulerParametersFlags
    qemuDomainGetNumaParameters
    qemuDomainSetNumaParameters
    qemuDomainGetMemoryParameters
    qemuDomainSetMemoryParameters
    qemuDomainGetBlkioParameters
    qemuDomainSetBlkioParameters
    qemuDomainGetCPUStats

v1.0.4-149-g54a99ba

As there is bug https://bugzilla.redhat.com/show_bug.cgi?id=947821 tracking the same root issue in VDSM I'm changing the summary to reflect the libvirt problem described above and moving this to POST as this issue is fixed already.

Reproducing the libvirt issue is easy:

1) Start a transient guest, configuration is not important:

$ virsh create migr.xml 
Domain migr created from migr.xml

2) Run a python interpreter and use the following code:

import libvirt
conn = libvirt.open("")
dom = conn.lookupByName("migr")
dom.getCPUStats(0, 0)
dom.destroy()
dom.getCPUStats(0, 0)

The output should look like this:
$ python
Python 2.6.6 (r266:84292, Oct 12 2012, 14:23:48) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>> conn = libvirt.open("")
>>> dom = conn.lookupByName("migr")
>>> dom.getCPUStats(0, 0)
[{'vcpu_time': 616121970L, 'cpu_time': 647032859L}, {'vcpu_time': 4314806L, 'cpu_time': 33909433L}]
>>> dom.destroy()
0
>>> dom.getCPUStats(0, 0)
libvir: QEMU Driver error : internal error No such domain x�V�"&��(#c��s
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1849, in getCPUStats
    if ret is None: raise libvirtError ('virDomainGetCPUStats() failed', dom=self)
libvirt.libvirtError: internal error No such domain x�V�"&��(#c��s

Thank you very much Peter!
I can reproduce this bug as comment 8 steps 

]# virsh list --transient
 Id    Name                           State
----------------------------------------------------
 25    whuang                         running

[root@mig2 ~]# python
Python 2.6.6 (r266:84292, Oct 12 2012, 14:23:48) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>> conn = libvirt.open("")
>>> dom = conn.lookupByName("whuang")
>>> dom.getCPUStats(0, 0)
[{'vcpu_time': 7747341023L, 'cpu_time': 9001077424L}, {'vcpu_time': 2742925675L, 'cpu_time': 2903694329L}, {'vcpu_time': 618646347L, 'cpu_time': 704083346L}, {'vcpu_time': 765533779L, 'cpu_time': 765582454L}, {'vcpu_time': 3145758379L, 'cpu_time': 3488304815L}, {'vcpu_time': 1185157464L, 'cpu_time': 1292114073L}, {'vcpu_time': 621729785L, 'cpu_time': 649608964L}, {'vcpu_time': 4771901L, 'cpu_time': 7750826L}]
>>> dom.destroy()
0
>>> dom.getCPUStats(0, 0)
libvir: QEMU Driver error : internal error No such domain ��C'vQ
�����J*
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1849, in getCPUStats
    if ret is None: raise libvirtError ('virDomainGetCPUStats() failed', dom=self)
libvirt.libvirtError: internal error No such domain ��C'vQ
�����J*

I reproduce this bug so cancel the needinfo

We decided not to rebase libvirt in RHEL 6.5 to avoid stability issues
we faced in 6.4. This bug has already been fixed upstream but it is
considered unsuitable for backporting to RHEL 6.5 because at least one
of the following conditions is met:

- this bug requires new API(s), which we cannot introduce without
  rebasing libvirt
- the patches required to address this bug are complex or invasive
  causing the backport to be too risky
- this bug is not important enough to justify backporting non-trivial
  patches for it

Thus I'm pushing this bug to RHEL 6.6 (and setting Upstream keyword to
indicate we have patches upstream) for now. If you don't agree with
this resolution, please, give us reasons which you think are strong
enough for us to reevaluate the decision not to backport patches for
this bug.

I can reproduce it with -32 version, but can not reproduce it on -34 version, libvirt can return a correct error message for all below functions:
    qemuDomainGetInterfaceParameters
    qemuDomainSetInterfaceParameters
    qemuGetSchedulerParametersFlags
    qemuSetSchedulerParametersFlags
    qemuDomainGetNumaParameters
    qemuDomainSetNumaParameters
    qemuDomainGetMemoryParameters
    qemuDomainSetMemoryParameters
    qemuDomainGetBlkioParameters
    qemuDomainSetBlkioParameters
    qemuDomainGetCPUStats

Version:
libvirt-0.10.2-34.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.425.el6.x86_64
kernel-2.6.32-459.el6.x86_64

[root@ibm-x3650m4-03 ~]# virsh create test.xml 
Domain test created from test.xml

[root@ibm-x3650m4-03 ~]# virsh list --transient
 Id    Name                           State
----------------------------------------------------
 1     test                           running

[root@ibm-x3650m4-03 ~]# python
Python 2.6.6 (r266:84292, Sep  4 2013, 07:46:00) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>> conn = libvirt.open("")
>>> dom = conn.lookupByName("test")
>>> dom.getCPUStats(0, 0)
[{'vcpu_time': 16731731776L, 'cpu_time': 17389491755L}, {'vcpu_time': 310741782L, 'cpu_time': 633758349L}, {'vcpu_time': 174519974L, 'cpu_time': 1458780282L}, {'vcpu_time': 3069430L, 'cpu_time': 3213549L}, {'vcpu_time': 704659458L, 'cpu_time': 704659458L}, {'vcpu_time': 2366210L, 'cpu_time': 2366210L}, {'vcpu_time': 1577631684L, 'cpu_time': 1725847573L}, {'vcpu_time': 5794185991L, 'cpu_time': 5824658462L}, {'vcpu_time': 2864552L, 'cpu_time': 2864552L}, {'vcpu_time': 459060L, 'cpu_time': 3040813L}, {'vcpu_time': 394120L, 'cpu_time': 430647L}, {'vcpu_time': 49926L, 'cpu_time': 2660931L}, {'vcpu_time': 2355289634L, 'cpu_time': 4530518564L}, {'vcpu_time': 0L, 'cpu_time': 20532116L}, {'vcpu_time': 0L, 'cpu_time': 19437608L}, {'vcpu_time': 0L, 'cpu_time': 0L}, {'vcpu_time': 0L, 'cpu_time': 0L}, {'vcpu_time': 0L, 'cpu_time': 0L}, {'vcpu_time': 69038908L, 'cpu_time': 76507472L}, {'vcpu_time': 97689872L, 'cpu_time': 472938691L}, {'vcpu_time': 0L, 'cpu_time': 0L}, {'vcpu_time': 0L, 'cpu_time': 0L}, {'vcpu_time': 0L, 'cpu_time': 0L}, {'vcpu_time': 0L, 'cpu_time': 0L}]
>>> dom.memoryParameters(0)
{'swap_hard_limit': 9007199254740991L, 'hard_limit': 9007199254740991L, 'soft_limit': 9007199254740991L}
>>> params  = {'hard_limit': 5007199254740991L, 'soft_limit': 5007199254740991L, 'swap_hard_limit': 5007199254740991L}
>>> dom.setMemoryParameters(params,0)
0
>>> dom.memoryParameters(0)
{'swap_hard_limit': 5007199254740992L, 'hard_limit': 5007199254740992L, 'soft_limit': 5007199254740992L}
>>> dom.blkioParameters(0)
{'device_weight': '', 'weight': 500}
>>> params = {'device_weight': '', 'weight': 200}
>>> dom.setBlkioParameters(params,0)
0
>>> dom.blkioParameters(0)
{'device_weight': '', 'weight': 200}
>>> dom.numaParameters(0)
{'numa_nodeset': '0-1', 'numa_mode': 0}
>>> params  = {'numa_nodeset': '0', 'numa_mode': 0}
>>> dom.setNumaParameters(params,0)
0
>>> dom.numaParameters(0)
{'numa_nodeset': '0', 'numa_mode': 0}
>>> dom.schedulerParametersFlags(0)
{'vcpu_quota': -1L, 'vcpu_period': 100000L, 'emulator_period': 100000L, 'emulator_quota': -1L, 'cpu_shares': 1024L}
>>> params  ={'vcpu_quota': -1L, 'vcpu_period': 100000L, 'emulator_period': 100000L, 'emulator_quota': -1L, 'cpu_shares': 2048L}
>>> dom.setSchedulerParametersFlags(params,0)
0
>>> dom.schedulerParametersFlags(0)
{'vcpu_quota': -1L, 'vcpu_period': 100000L, 'emulator_period': 100000L, 'emulator_quota': -1L, 'cpu_shares': 2048L}
>>> dom.interfaceParameters("vnet0",0)
{'outbound.peak': 0, 'inbound.peak': 0, 'inbound.burst': 0, 'inbound.average': 0, 'outbound.average': 0, 'outbound.burst': 0}
>>> params  = {'outbound.peak': 0, 'inbound.peak': 0, 'inbound.burst': 0, 'inbound.average': 0, 'outbound.average': 128, 'outbound.burst': 0}
>>> dom.setInterfaceParameters("vnet0",params,0)
0
>>> dom.interfaceParameters("vnet0",0)
{'outbound.peak': 0, 'inbound.peak': 0, 'inbound.burst': 0, 'inbound.average': 0, 'outbound.average': 128, 'outbound.burst': 0}
>>> dom.destroy()
0

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

>>> dom.getCPUStats(0, 0)
libvirt: QEMU Driver error : Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1879, in getCPUStats
    if ret is None: raise libvirtError ('virDomainGetCPUStats() failed', dom=self)
libvirt.libvirtError: Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

>>> dom.interfaceParameters("vnet0",0)
libvirt: QEMU Driver error : Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1891, in interfaceParameters
    if ret is None: raise libvirtError ('virDomainGetInterfaceParameters() failed', dom=self)
libvirt.libvirtError: Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
>>> params  = {'outbound.peak': 0, 'inbound.peak': 0, 'inbound.burst': 0, 'inbound.average': 0, 'outbound.average': 128, 'outbound.burst': 0}
>>> dom.setInterfaceParameters("vnet0",params,0)
libvirt: QEMU Driver error : Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1989, in setInterfaceParameters
    if ret == -1: raise libvirtError ('virDomainSetInterfaceParameters() failed', dom=self)
libvirt.libvirtError: Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

>>> dom.schedulerParametersFlags(0)
libvirt: QEMU Driver error : Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1965, in schedulerParametersFlags
    if ret is None: raise libvirtError ('virDomainGetSchedulerParametersFlags() failed', dom=self)
libvirt.libvirtError: Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
>>> params  ={'vcpu_quota': -1L, 'vcpu_period': 100000L, 'emulator_period': 100000L, 'emulator_quota': -1L, 'cpu_shares': 2048L}
>>> dom.setSchedulerParametersFlags(params,0)
libvirt: QEMU Driver error : Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2013, in setSchedulerParametersFlags
    if ret == -1: raise libvirtError ('virDomainSetSchedulerParametersFlags() failed', dom=self)
libvirt.libvirtError: Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

>>> dom.numaParameters(0)
libvirt: QEMU Driver error : Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1933, in numaParameters
    if ret is None: raise libvirtError ('virDomainGetNumaParameters() failed', dom=self)
libvirt.libvirtError: Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
>>> params  = {'numa_nodeset': '0', 'numa_mode': 0}
>>> dom.setNumaParameters(params,0)
libvirt: QEMU Driver error : Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2001, in setNumaParameters
    if ret == -1: raise libvirtError ('virDomainSetNumaParameters() failed', dom=self)
libvirt.libvirtError: Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

>>> dom.memoryParameters(0)
libvirt: QEMU Driver error : Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1909, in memoryParameters
    if ret is None: raise libvirtError ('virDomainGetMemoryParameters() failed', dom=self)
libvirt.libvirtError: Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
>>> params  = {'hard_limit': 5007199254740991L, 'soft_limit': 5007199254740991L, 'swap_hard_limit': 5007199254740991L}
>>> dom.setMemoryParameters(params,0)
libvirt: QEMU Driver error : Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1995, in setMemoryParameters
    if ret == -1: raise libvirtError ('virDomainSetMemoryParameters() failed', dom=self)
libvirt.libvirtError: Domain not found: no domain with matching uuid '827e2e30-43cb-b86a-3e57-de42b2de1e4b'
>>> 

We can get expected results, change to Verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1374.html