Description of problem: Customer found that after updating RHEV-H and migrating VMs to the updated host, logs were flooded with messages like: Jan 29 05:17:39 rhevh04 vdsm vm.Vm ERROR vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x25fa650>#012Traceback (most recent call last):#012 File "/usr/share/vdsm/sampling.py", line 351, in collect#012 File "/usr/share/vdsm/sampling.py", line 226, in __call__#012 File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012 File "/usr/share/vdsm/vm.py", line 2316, in extendDrivesIfNeeded#012 File "/usr/share/vdsm/vm.py", line 842, in f#012 File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper#012 File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1814, in blockInfo#012libvirtError: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain The VMs otherwise work. The problem is because the newer vdsm version configures libvirt to use paths of the form: /rhev/data-center/mnt/<mountpoint>/<sd uuid>/images/<disk uuid>/<img uuid> ^^^^^^^^^^^^^^^^ whereas the older vdsm used the form /rhev/data-center/<sp uuid>/<sd uuid>/images/<disk uuid>/<img uuid> ^^^^^^^^^ For VMs started under the new vdsm this is fine, but for VMs migrated from an older hypervisor, the libvirt XML, including the old-style paths is carried along. That means that the vdsm and libvirt paths are out of sync, causing the error above when vdsm attempts to query libvirt for information about the disk. Version-Release number of selected component (if applicable): vdsm 4.13.2-0.6 uses the new path format How reproducible: 100% Steps to Reproduce: 1. Have a RHEV setup with pre-vdsm-4.13.2 hypervisors 2. Update one hypervisor to the RHEL 6.5 based image with vdsm-4.13.2 3. Migrate a VM from the old to the new hypervisor Actual results: VM works, but logs are spammed with libvirt and vdsm errors. Expected results: VM works, without extraneous errors. Additional info: This can be worked around by restarting the VMs under the new hypervisor, but obviously customers prefer not to do that.
In fact, fixing only for 3.4.0 could actually be *worse* than not fixing it at all. Because the problem is a mismatch in behaviour between hypervisor versions, just reverting the behaviour in 3.4 would cause this problem to occur again for all those people who've switched to the current behaviour in the interim, and restarted their VMs to work around the errors. A real fix would need to check for this situation during migrate, and adjust the libvirt XML to match whatever form the destination vdsm is using.
@Ulhas, Which workaround from the KCS? Restarting the VM, or migrating back to the old vdsm?
David, It's restart the VM. 2) Otherwise if the VM can be shutdown then shut it down on the vdsm 4.13.2-0.6 hypervisor and start it again. Note it must be shutdown completely and not simply rebooted.
*** Bug 1055437 has been marked as a duplicate of this bug. ***
Tested with two RHEVH nodes and 3.4 management. First RHEVH - RHEL 6.4 with vdsm-4.10.2-25.1.el6ev Second RHEVH - RHEL 6.5 with vdsm-4.13.2-0.6.el6ev. VM migrated from the first rhevh to the second one and vise versa. No error appear in vdsm.log.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-0504.html