Bug 1128631

Please attach VDSM's log too

Created attachment 925712 [details]
vdsm log

From the attached log I see the following important entries.  Live merge was called twice in rapid succession by engine.  Once called, vdsm properly handled both calls.  The host does not have a capable version of libvirt so vdsm returns an error code.  This smells like some sort of engine side race condition to me.  Please attach the engine.log as well.


Thread-65::DEBUG::2014-08-11 11:13:44,187::BindingXMLRPC::1127::vds::(wrapper) client [10.35.161.54]::call merge with ('42e19c61-73f6-4b8b-8478-07c0d0b6265a', {'domainID': 'ab1a30d2-6b93-4302-a028-a1f506f3b1da', 'volumeID': 'fd306b46-52b4-4715-a73f-a946d996adbf', 'poolID': 'f603339e-c4aa-474c-bb83-df768af662c8', 'imageID': '969ec9dc-a95e-4129-97f1-8465a1b79804'}, 'df6bc58a-105c-4964-9a56-caa268ebce5d', 'fd306b46-52b4-4715-a73f-a946d996adbf', '0', '0a93e33a-a4cc-4a0f-884f-233e85050d4b') {} flowID [80d09bd]


Thread-58::DEBUG::2014-08-11 11:13:44,212::BindingXMLRPC::1127::vds::(wrapper) client [10.35.161.54]::call merge with ('42e19c61-73f6-4b8b-8478-07c0d0b6265a', {'domainID': '80f91f5c-f5cc-4ae2-953e-b2ef2f812a7e', 'volumeID': 'fd890dce-4d33-48f4-bb51-4f05ece61825', 'poolID': 'f603339e-c4aa-474c-bb83-df768af662c8', 'imageID': '8773e3b2-fe5a-4388-a8e0-a8f9d5217d67'}, 'f30d23dc-4ac5-45ec-b39d-64fb35f4772c', 'fd890dce-4d33-48f4-bb51-4f05ece61825', '0', 'ea9ef3dd-ce79-4cc1-9396-2d6ba3c92f32') {} flowID [37537b98]


Thread-65::DEBUG::2014-08-11 11:13:44,342::vm::5566::vm.Vm::(merge) vmId=`42e19c61-73f6-4b8b-8478-07c0d0b6265a`::Starting merge with jobUUID='0a93e33a-a4cc-4a0f-884f-233e85050d4b'
Thread-65::ERROR::2014-08-11 11:13:44,342::vm::5575::vm.Vm::(merge) vmId=`42e19c61-73f6-4b8b-8478-07c0d0b6265a`::Libvirt missing VIR_DOMAIN_BLOCK_COMMIT_RELATIVE. Unable to perform live merge.
Thread-65::DEBUG::2014-08-11 11:13:44,343::BindingXMLRPC::1134::vds::(wrapper) return merge with {'status': {'message': 'Merge failed', 'code': 52}}


Thread-58::DEBUG::2014-08-11 11:13:44,398::vm::5566::vm.Vm::(merge) vmId=`42e19c61-73f6-4b8b-8478-07c0d0b6265a`::Starting merge with jobUUID='ea9ef3dd-ce79-4cc1-9396-2d6ba3c92f32'
Thread-58::ERROR::2014-08-11 11:13:44,399::vm::5575::vm.Vm::(merge) vmId=`42e19c61-73f6-4b8b-8478-07c0d0b6265a`::Libvirt missing VIR_DOMAIN_BLOCK_COMMIT_RELATIVE. Unable to perform live merge.
Thread-58::DEBUG::2014-08-11 11:13:44,399::BindingXMLRPC::1134::vds::(wrapper) return merge with {'status': {'message': 'Merge failed', 'code': 52}}

This looks just like what I experienced with builds not containing a commit [1] merged on August 7, which fixes the locked snapshot issue.  Without it, Live Merge attempts won't converge, and will leave the disks illegal and snapshots locked.  For an error case like this (libvirt not capable), it would leave things in an inoperable state.

Can you retry with a build containing this commit?

[1] 209ec823a03dd5838eed3d711fd821d2a1aba9dd core: Live Merge command hangs

Hi Adam,
engine.log attached

(In reply to Greg Padgett from comment #4)
> This looks just like what I experienced with builds not containing a commit
> [1] merged on August 7, which fixes the locked snapshot issue.  Without it,
> Live Merge attempts won't converge, and will leave the disks illegal and
> snapshots locked.  For an error case like this (libvirt not capable), it
> would leave things in an inoperable state.
> 
> Can you retry with a build containing this commit?
> 
> [1] 209ec823a03dd5838eed3d711fd821d2a1aba9dd core: Live Merge command hangs
Moving to MODIFIED based on this comment - there is no such build publicly available yet.

Raz - do you have the resources to give this a quick pre-integ run?

Hi Allon,
This operation is blocked right now.
Is this the correct behaviour ?

(In reply to ratamir from comment #7)
> Hi Allon,
> This operation is blocked right now.
> Is this the correct behaviour ?
Depends on the host - can you share the versions of qemu/libvirt/vdsm please?

- vdsm-4.16.2-1.gite8cba75.el6.x86_64

- qemu-img-rhev-0.12.1.2-2.415.el6_5.14.x86_64
- qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64
- qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.14.x86_64

- libvirt-0.10.2-29.el6_5.10.x86_64

These versions of libvirt and qemu lack support for live merge so in this case it's working as designed.  See http://www.ovirt.org/Features/Live_Merge#IMPORTANT:_Special_environment_setup for information on testing the feature with Fedora 20 hosts.

oVirt 3.5 has been released and should include the fix for this issue.