Created attachment 934459 [details] Sample domain XML Description of problem: When creating a VM, vdsm uses the <min_guarantee/> parameter in the XML to pass through the minimum amount of memory that should be always available for the VM. MOM uses this value in its memory ballooning algorithm. This field is not supported by qemu but in the past libvirt has allowed it to be set anyway. Recently, libvirt is now reporting an error since qemu won't honor it. The proper long-term solution is to use the metadata feature of libvirt to attach this information as arbitrary metadata. We need to explore backwards compatibility and upgrade issues for oVirt when making this change. Here is the error returned by vdsm: Thread-35147::ERROR::2014-09-04 09:09:15,331::vm::2336::vm.Vm::(_startUnderlyingVm) vmId=`e167e221-9dff-40b1-b911-d83ea74bbb20`::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 2276, in _startUnderlyingVm self._run() File "/usr/share/vdsm/virt/vm.py", line 3367, in _run self._connection.createXML(domxml, flags), File "/usr/lib64/python2.7/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3424, in createXML if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self) libvirtError: unsupported configuration: Parameter 'min_guarantee' not supported by QEMU. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Ensure libvirt >= libvirt-1.2.8-1 is installed on the host 2. Start a VM Actual results: VM creation fails due to above error Expected results: VM creation succeeds Additional info: Reproducible on el7 with the following package versions: libvirt-1.2.8-1.el7.x86_64 qemu-kvm-rhev-2.1.0-3.el7.x86_64
Please note that using an invalid VM XML will also affect migrations as the target of the migration will refuse to start the VM with invalid configuration.
I believe it is bad decision on the libvirt side as they are forcing their users to store the logically and semantically identical value differently depending on the hypervisor. Unfortunately for us, they seem to be pretty adamant about it. We will also have to revert the fix if qemu starts supporting it in the future.
If libvirt does not agree to ignore this, we have to backport the fix all the way to ovirt-3.3, when we introduced its usage (http://gerrit.ovirt.org/15799 ). VMs started by ovirt-3.3.0 cannot migrate to destinations with new libvirt; but as long as we support ovirt-3.3 we must make sure that that ovirt-3.3.5 would be able to start migratable machines. For people with long-living VMs started back in 3.3.0, we must supply a libvirt hook that strips <min_guarantee> on the destination. Peter, could you help us write one?
Does the hook need to convert the used value to anything in the custom metadata?
A different option I'm now investigating is that we might be able to implement the min_guarantee element in a way that would work for both libvirt (and the documented meaning of the element) and oVirt where it shouldn't have impact on the current code. I'm looking into the feasibility of this. In that case, only the upstream 1.2.8 release of libvirt wouldn't work.
(In reply to Peter Krempa from comment #5) > A different option I'm now investigating is that we might be able to > implement the min_guarantee element in a way that would work for both > libvirt (and the documented meaning of the element) and oVirt where it > shouldn't have impact on the current code. > > I'm looking into the feasibility of this. In that case, only the upstream > 1.2.8 release of libvirt wouldn't work. Unfortunately, the semantics of the min_guarantee field is to guarantee a certain amount of ram (not including swap) to be available to a single guest. Currently libvirt isn't able to do such a guarantee as cgroups don't expose such mechanism. Thus we won't be able to provide a suitable implementation.
If I remember correctly we were talking about ballooning limits or qemu. cgroups have nothing to do with this as they are used for the upper limits only. Is the integration to balloon limits really something that would violate the documented semantics?
Dan, 3.3.5?
We need to backport this to all supported stable versions. 3.3.z is not one of them; changing to 3.4.4.
oVirt 3.4.4 has been released.
Created attachment 970915 [details] before_vm_start hook to remove min_guarantee Running oVirt Engine 3.4.4 (on Fedora 19) with Fedora 21 hosts, I had to add a before_vm_start hook to remove the min_guarantee element from the domain XML. The attached python script works in our environment.
which vdsm version did you use on your hosts?
The host in question is running Fedora 21, with vdsm 4.14.8.1-1.fc21. The engine (version 3.4.4) is running on Fedora 19. Most of our hosts are currently running Fedora 19, but under the assumption that F19 will soon reach end of support now that F21 has been released, I'm starting to test F21 on the hosts. It took a workaround outlined by Adam Litke in Bugzilla 1138807 to get oVirt Engine 3.4.4 to recognize the F21 host. Once that step was done, I could not start a VM on the F21 host or migrate a running VM to it. That was the point at which I stumbled on this ticket. We have a schedule for updating oVirt Engine to 3.5, but we're for the time being we're stuck with 3.4.4.