Bug 863265

its look like the problem is in the syntax...
we remove the vm_prefix from the api, so now we use <payloads> and <payload> instead of <vm_payloads> and <vm_payload>, unfortunately we didn't update the ovrit wiki page: 
http://wiki.ovirt.org/wiki/Features/VMPayload#API_Design
now its updated with the right syntax,

also tested with the following curl command:

curl -X PUT -H "Accept: application/xml" -H "Content-Type: application/xml" -u user:pass -d "<vm> <payloads> <payload type='cdrom'> <file name='test.txt'> <content>my content</content> </file> </payload> </payloads> </vm>" http://shaharh:8080/api/vms/82f8c82a-2f36-4a32-a120-d18cb93df5e0

Joe please check if that is working for you as well.

(In reply to comment #1)
> its look like the problem is in the syntax...
> we remove the vm_prefix from the api, so now we use <payloads> and <payload>
> instead of <vm_payloads> and <vm_payload>, unfortunately we didn't update
> the ovrit wiki page: 
> http://wiki.ovirt.org/wiki/Features/VMPayload#API_Design
> now its updated with the right syntax,
> 
> also tested with the following curl command:
> 
> curl -X PUT -H "Accept: application/xml" -H "Content-Type: application/xml"
> -u user:pass -d "<vm> <payloads> <payload type='cdrom'> <file
> name='test.txt'> <content>my content</content> </file> </payload>
> </payloads> </vm>"
> http://shaharh:8080/api/vms/82f8c82a-2f36-4a32-a120-d18cb93df5e0
> 
> Joe please check if that is working for you as well.

Thank you Shahar!

That correct syntax seems to work correctly.

The output from the "curl -X PUT..." does show the payload and
a query of the VM also does. So that is good progress!

However the VM fails to start with the error.

I had created the VM using the UI, then issued the curl command
you provided from the command line. Then I tried to use the UI
to start the VM the following errors were listed in Events:

Failed to run VM joev_vm_pl2 (User: xyz@dom)
Failed to run VM joev_vm_pl2 on Host XYZ
VM joev_vm_pl2 is down. Wxit message: [Errno 2] No such file or directory: '/var/run/vdsm/payload/<vm ID>.<some other ID>.img'.


So it seems like we are making progress now that we have the correct
syntax but it still seems the vm_payload injection is not working
correctly.

Sorry for the typo; The Events errors should read: "Exit message:" instead
of "Wxit message:"

i.e.:

Failed to run VM joev_vm_pl2 (User: xyz@dom)
Failed to run VM joev_vm_pl2 on Host XYZ
VM joev_vm_pl2 is down. Exit message: [Errno 2] No such file or directory: '/var/run/vdsm/payload/<vm ID>.<some other ID>.img'.

I just thought of something else: When we had been using the wrong
syntax why didn't we get an error from the PUT? Shouldn't incorrect
input produce an error?

(In reply to comment #4)
> I just thought of something else: When we had been using the wrong
> syntax why didn't we get an error from the PUT? Shouldn't incorrect
> input produce an error?

its look like it does related to that patch:
http://gerrit.ovirt.org/#/c/8092/
the path shouldn't be empty, it should contain the vmid which is missing here:
'/var/run/vdsm/payload/<vm ID>.<some other ID>.img'.

I will check it and modify you Joe.

(In reply to comment #4)
> I just thought of something else: When we had been using the wrong
> syntax why didn't we get an error from the PUT? Shouldn't incorrect
> input produce an error?

no you shouldn't, - this is by spec, non-existent xml elements just
get ignored.

Hi,
I just tested a VM with the following payload
     <payloads>
         <payload type='cdrom'>
             <file name='my.txt'>
                <content>some content</content>
             </file>
         </payload>
     </payloads>

It works fine (upstream), running with no problem and I mounted the /dev/cdrom and saw the file my.txt with its content.

Joe, please try to create a new VM and test if its work for you.

Using the latest bits in RHEVm3.1 (not oVirt)

I am seeing the same error. The image I am launching is RHEL6.

I created a new VM using the UI.
Started it fine.
I stopped it .
Then using curl added a vm_payload.
After which the VM will not start with the same error I reported earlier.



The RHEVm version from the UI "About" box is:
Red Hat Enterprise Virtualization Manager Version: '3.1.0-20.el6ev'


My understanding is that this is the version QA has for testing.

can you please follow comment #7, i.e. use "payload">

(In reply to comment #9)
> can you please follow comment #7, i.e. use "payload">

Yes Michal that is what I did use to set the payload.

I tried to describe that in my comment 2 but perhaps it wasn't fully clear.

Here is an example of it:
% curl -X PUT -H "Accept: application/xml" -u [UN:PW] --cacert [cert] -H "Content-Type: application/xml" -d "<vm> <payloads> <payload type='cdrom'> <file name='test.txt'> <content>my content</content> </file> </payload> </payloads> </vm>" https://[RHEVm Host]/api/vms/[UUID]

This does seem to work in that after I issue it the returned XML does
show the payload and future GET to: https://[RHEVm Host]/api/vms/[UUID] 
do too. However the instance fails to start with the error I reported
in comments 2 and 3

The problem was old VDSM version, 
after running in Joes environment and updating to latest version,
was: vdsm-4.9.6-36.0.el6_3.x86_64
and now: vdsm-4.9.6-37.0.el6_3.x86_64

Joe please verify.

Shahar,

Thanks for looking into this but something is still not right.

I am still seeing the same error under the VM's Events.

I also tried creating a new VM from scratch.

I see the vdsm update to version 4.9.6-37.0.el6_3

# yum list 2>&1 | grep -i vdsm
vdsm.x86_64                     4.9.6-37.0.el6_3   @ovirt20                     
vdsm-bootstrap.noarch           4.9.6-37.0.el6_3   @ovirt20                     
vdsm-cli.noarch                 4.9.6-37.0.el6_3   @ovirt20                     
vdsm-python.x86_64              4.9.6-37.0.el6_3   @ovirt20                     
vdsm-debug-plugin.noarch        4.9.6-37.0.el6_3   ovirt20                      
vdsm-gluster.noarch             4.9.6-37.0.el6_3   ovirt20                      
vdsm-hook-faqemu.noarch         4.9.6-37.0.el6_3   ovirt20                      
vdsm-hook-vhostmd.noarch        4.9.6-37.0.el6_3   ovirt20                      
vdsm-reg.noarch                 4.9.6-37.0.el6_3   ovirt20    


Here is part of the tailing of the vdsm.log file with the traceback:

% tail -f /var/log/vdsm/vdsm.log
...
MainProcess|Thread-11952::ERROR::2012-10-09 20:43:09,593::supervdsmServer::66::SuperVdsm.ServerCallback::(wrapper) Error in mkIsoFs
Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer.py", line 64, in wrapper
    return func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer.py", line 227, in mkIsoFs
    return mkimage.mkIsoFs(vmId, files)
  File "/usr/share/vdsm/mkimage.py", line 104, in mkIsoFs
    _commonCleanFs(dirname, isopath)
  File "/usr/share/vdsm/mkimage.py", line 54, in _commonCleanFs
    resolveGid(DISKIMAGE_GROUP))
OSError: [Errno 2] No such file or directory: '/var/run/vdsm/payload/f3ec6cd4-da70-49b8-8c8b-6185a7b4eed5.374196c7b0b9d960449b0b862c06753d.img'
Thread-11952::DEBUG::2012-10-09 20:43:09,594::vm::589::vm.Vm::(_startUnderlyingVm) vmId=`f3ec6cd4-da70-49b8-8c8b-6185a7b4eed5`::_ongoingCreations released
Thread-11952::ERROR::2012-10-09 20:43:09,594::vm::613::vm.Vm::(_startUnderlyingVm) vmId=`f3ec6cd4-da70-49b8-8c8b-6185a7b4eed5`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 579, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/libvirtvm.py", line 1318, in _run
    self.preparePaths(devices[vm.DISK_DEVICES])
  File "/usr/share/vdsm/vm.py", line 625, in preparePaths
    drive['path'] = self.cif.prepareVolumePath(drive, self.id)
  File "/usr/share/vdsm/clientIF.py", line 221, in prepareVolumePath
    volPath = supervdsm.getProxy().mkIsoFs(vmId, files)
  File "/usr/share/vdsm/supervdsm.py", line 59, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 57, in <lambda>
    callMethod = lambda : getattr(self._supervdsmProxy._svdsm, self._funcName)(*args, **kwargs)
  File "<string>", line 2, in mkIsoFs
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
OSError: [Errno 2] No such file or directory: '/var/run/vdsm/payload/f3ec6cd4-da70-49b8-8c8b-6185a7b4eed5.374196c7b0b9d960449b0b862c06753d.img'

I will check it on a QE environment, and if I cannot reproduce it we will see why it happened only at your environment.

Ok, I think I found the problem....
Joe, please check if you have the file /bin/mkisofs if not please install it via:
yum install mkisofs

I will send a vdsm fix with the appropriate error.

Requires: genisoimage  has already added upstream (thanks, Zhou)

http://gerrit.ovirt.org/7232

a downstream port is

https://gerrit.eng.lab.tlv.redhat.com/2529

(In reply to comment #14)
> Ok, I think I found the problem....
> Joe, please check if you have the file /bin/mkisofs if not please install it
> via:
> yum install mkisofs


After installing mkisofs I was able to:
- create a VM using the UI.
- inject the payload using API
- start the VM using UI
- access the payload via /dev/cdrom

So that is successful.

> I will send a vdsm fix with the appropriate error.

I am not sure what you mean by that ^^



The reason I marked this BZ "needs more information" is because
it seems there were three issues here. How will each of these
be tracked?

Issue 1.
   A documentation issue describing the correct syntax
   <vm_payload> to <payload>

   I see that has been fixed in the wiki here:
   http://wiki.ovirt.org/wiki/Features/VMPayload#API_Design

   Is there more documentation that needs to be updated?
   Is a "doc" BZ required?

Issue 2.
   vdsm must be at least version 4.9.6-37.0

   Is a BZ required to track the fixing of this packaging?

Issue 3.

   mkisofs must be install.

   Is a BZ required to track the fixing of this packaging?

issue 2:
    I don't think so.

issue 3:
    this bug is the fix with the patch that Dan sent at: https://gerrit.eng.lab.tlv.redhat.com/2529

Thank you Shahar for all the help! I greatly appreciate it.

Also:

If using the API to create the VM, instead of the UI, the VM will
not start. The event error is:
	
VM joev_pl_5 is down. Exit message: local variable 'isopath' referenced before assignment.

I believe this is because of:
http://gerrit.ovirt.org/#/c/8092/

Please confirm this is correct and advise how this should be tracked.

(In reply to comment #18)
> Thank you Shahar for all the help! I greatly appreciate it.
> 
> Also:
> 
> If using the API to create the VM, instead of the UI, the VM will
> not start. The event error is:
> 	
> VM joev_pl_5 is down. Exit message: local variable 'isopath' referenced
> before assignment.
> 
> I believe this is because of:
> http://gerrit.ovirt.org/#/c/8092/
> 
> Please confirm this is correct and advise how this should be tracked.
its not related to this bug...
Federico, do you know if that is related to your fix: http://gerrit.ovirt.org/#/c/8092/

Do we need a doc bug to ensure the correct syntax as described in comment #1
is reflected in all the proper docs?

It is already updated on the wiki:
http://wiki.ovirt.org/wiki/Features/VMPayload#API_Design

but... are there other docs that need to be updated? Is a BZ needed to
track that doc work?


sgrinber If you are the wrong person to ask please direct me
to the correct person.

(In reply to comment #20)
> but... are there other docs that need to be updated? Is a BZ needed to
> track that doc work?
> 
> 
> sgrinber If you are the wrong person to ask please direct me
> to the correct person.

I've set the requires_doc_text? request to get doc team attention, no need for another bug ATM, they will clone if needed. 

The effected guides should be the CLI guide and the developer guide.

(In reply to comment #19)
> (In reply to comment #18)
> > Thank you Shahar for all the help! I greatly appreciate it.
> > 
> > Also:
> > 
> > If using the API to create the VM, instead of the UI, the VM will
> > not start. The event error is:
> > 	
> > VM joev_pl_5 is down. Exit message: local variable 'isopath' referenced
> > before assignment.
> > 
> > I believe this is because of:
> > http://gerrit.ovirt.org/#/c/8092/
> > 
> > Please confirm this is correct and advise how this should be tracked.
> 
> its not related to this bug...
> Federico, do you know if that is related to your fix:
> http://gerrit.ovirt.org/#/c/8092/

I don't know where you saw that error but it looks more a vdsm issue, related to:

23985ce with exceptions some variables could be uninitialised

Which could eventually be in some way mitigated by my engine fix but probably 23985ce is a better fix for this specific situation.

Moving to verified per comment32

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-1508.html