Bug 997910 - Cannot get snapshots collection via API after live snapshot creation
Cannot get snapshots collection via API after live snapshot creation
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-restapi (Show other bugs)
3.3.0
x86_64 Linux
unspecified Severity urgent
: ---
: 3.3.0
Assigned To: Omer Frenkel
Pavel Novotny
virt
: Regression, TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-16 09:29 EDT by Jakub Libosvar
Modified: 2015-09-22 09 EDT (History)
14 users (show)

See Also:
Fixed In Version: is14
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vdsm and engine logs (366.08 KB, application/gzip)
2013-08-16 09:29 EDT, Jakub Libosvar
no flags Details
engine vdsm logs (232.71 KB, application/gzip)
2013-08-19 09:01 EDT, Jakub Libosvar
no flags Details

  None (edit)
Description Jakub Libosvar 2013-08-16 09:29:28 EDT
Created attachment 787262 [details]
vdsm and engine logs

Description of problem:
After Live storage migration is started, user cannot obtain vms' snapshots collection

<fault><reason>Operation Failed</reason><detail>element type "openstack" must be followed by either attribute specifications, ">" or "/>".</detail></fault>

2013-08-16 15:19:48,860 ERROR [org.ovirt.engine.core.bll.GetVmConfigurationBySnapshotQuery] (ajp-/127.0.0.1:8702-9) Query GetVmConfigurationBySnapshotQuery failed. Exception message is Element type "openstack" must be followed by either attribute specifications, ">" or "/>".
2013-08-16 15:19:48,864 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (ajp-/127.0.0.1:8702-9) Operation Failed: element type "openstack" must be followed by either attribute specifications, ">" or "/>".
2013-08-16 15:20:09,272 ERROR [org.ovirt.engine.core.bll.GetVmConfigurationBySnapshotQuery] (ajp-/127.0.0.1:8702-9) Query GetVmConfigurationBySnapshotQuery failed. Exception message is Element type "openstack" must be followed by either attribute specifications, ">" or "/>".


Version-Release number of selected component (if applicable):
rhevm-3.3.0-0.15.master.el6ev.noarch
vdsm-4.12.0-61.git8178ec2.el6ev.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Perform LSM
2. go to /api/vms/<vm:id>/snapsnots
3.

Actual results:
Exception message is Element type "openstack" must be followed by either attribute specifications, ">" or "/>"

Expected results:
Snapshots collection

Additional info:
Logs attached, blocks live storage migration test
Comment 3 Jakub Libosvar 2013-08-19 09:01:16 EDT
Created attachment 788034 [details]
engine vdsm logs

Ooops, I took the logs from wrong directory. Sorry and thanks for noticing.
Comment 5 Liron Aravot 2013-08-20 10:37:57 EDT
It seems like the vmPayload isn't being serialized as it should which causes to error when trying to read the vm configuration - that's the saved specParams in the snapshots ovf, therefore moving to virt.

<SpecParams><vmPayload><file><openstack/latest/meta_data.json>ewogICJsYXVuY2hfaW5kZXgiIDogIjAiLAogICJhdmFpbGFiaWxpdHlfem9uZSIgOiAibm92YSIs
                 | CiAgInV1aWQiIDogIjQyYTAyNWI0LTViMDctNDkzYy04NjNmLTYwZWU3YmMzMGRhNCIsCiAgIm1l
                 | dGEiIDogewogICAgImVzc2VudGlhbCIgOiAiZmFsc2UiLAogICAgInJvbGUiIDogInNlcnZlciIs
                 | CiAgICAiZHNtb2RlIiA6ICJsb2NhbCIKICB9Cn0=
                 | </openstack/latest/meta_data.json><openstack/latest/user_data>b3V0cHV0OgogIGFsbDogJz4+IC92YXIvbG9nL2Nsb3VkLWluaXQtb3V0cHV0LmxvZycKdXNlcjog
                 | cm9vdApydW5jbWQ6Ci0gJ3NlZCAtaSAnJy9eZGF0YXNvdXJjZV9saXN0OiAvZCcnIC9ldGMvY2xv
                 | dWQvY2xvdWQuY2ZnOyBlY2hvICcnZGF0YXNvdXJjZV9saXN0OgogIFsiTm9DbG91ZCIsICJDb25m
                 | aWdEcml2ZSJdJycgPj4gL2V0Yy9jbG91ZC9jbG91ZC5jZmcnCg==
                 | </openstack/latest/user_data></file><volId>config-2</volId></vmPayload></SpecParams>
Comment 6 Jakub Libosvar 2013-08-20 10:56:45 EDT
Exact steps to reproduce are:
 1. Have a running vm that has Operating System set to "Red Hat Enterprise Linux 6.x"
 2. Create a live snapshot of running vm
 3. Check api/vms/{vm:id}/snapshots
Comment 7 Omer Frenkel 2013-08-25 10:36:09 EDT
looks like there is a problem in the way cd spec params are saved to the snpashot ovf, which doesnt make a valid xml file.

i think this currently happen only if the vm was run once with cloud init and then live snapshot was taken, so i dont see how this is a regression or a test blocker.
is there other way to reproduce this? (i tried the steps from comment 6 , it didn't work if the vm wasnt started with cloud init)
Comment 8 Jakub Libosvar 2013-08-25 11:03:02 EDT
(In reply to Omer Frenkel from comment #7)
> looks like there is a problem in the way cd spec params are saved to the
> snpashot ovf, which doesnt make a valid xml file.
> 
> i think this currently happen only if the vm was run once with cloud init
> and then live snapshot was taken, so i dont see how this is a regression or
> a test blocker.
It's failing live snapshot sanity and live storage migration sanity tests. And it was not failing on is9.
> is there other way to reproduce this? (i tried the steps from comment 6 , it
> didn't work if the vm wasnt started with cloud init)
I didn't find any other way. I don't think we use cloud-init. Vm was started by:
url:/api/vms/47f22238-5832-48ae-b3bc-2a02380c2655/start
body:<action>
    <async>false</async>
    <grace_period>
        <expiry>10</expiry>
    </grace_period>
    <vm>
        <os>
            <boot dev="hd"/>
            <boot dev="network"/>
        </os>
    </vm>
</action>
Comment 9 Omer Frenkel 2013-08-25 11:44:10 EDT
(In reply to Jakub Libosvar from comment #8)
> I didn't find any other way. I don't think we use cloud-init. Vm was started
> by:
> url:/api/vms/47f22238-5832-48ae-b3bc-2a02380c2655/start
> body:<action>
>     <async>false</async>
>     <grace_period>
>         <expiry>10</expiry>
>     </grace_period>
>     <vm>
>         <os>
>             <boot dev="hd"/>
>             <boot dev="network"/>
>         </os>
>     </vm>
> </action>

couldn't reproduce the issue using this.
can you please verify it happens always? (without cloud init),
if so, can you please check if some payload defined for the vm?
thanks!
Comment 10 Omer Frenkel 2013-08-26 07:13:20 EDT
Update: this happens when vm has cloud-init payload: when creating live snapshot, all devices spec params are written to the ovf, withing cloud init has the files, and the file paths are inserted as elements, which is wrong because its not a valid xml value.

and cloud init is used when running first time un-initialized linux vm (same as sysprep for windows) which is new behavior.
Comment 11 Pavel Novotny 2013-09-27 11:27:29 EDT
Verified in rhevm-3.3.0-0.23.master.el6ev (is16).

Verified either with a clean Linux VM and with Run Once VM with cloud-init enabled. In both cases the live snapshots were successfully created and correctly listed via REST API.
Comment 12 Itamar Heim 2014-01-21 17:28:36 EST
Closing - RHEV 3.3 Released
Comment 13 Itamar Heim 2014-01-21 17:28:38 EST
Closing - RHEV 3.3 Released
Comment 14 Itamar Heim 2014-01-21 17:31:33 EST
Closing - RHEV 3.3 Released

Note You need to log in before you can comment on or make changes to this bug.