Bug 871514

Created attachment 635648 [details]
rhevm webui cloning a template

the cloning feature works when initiated via webui

michael - still relevant in 3.1?

Giulio,

To make it work, you need to copy template's disk/s to desired SD first,
please recheck and let me know if you still see any problem.

also please take a look on Bug 866985

(In reply to comment #3)
> Giulio,
> 
> To make it work, you need to copy template's disk/s to desired SD first,
> please recheck and let me know if you still see any problem.

Michael, if this is the case then clone would have failed from the GUI as well. 
Could it be that what SDK declares as clone is note what GUI declares as clone?

In the GUI, Clone ==: Copy the template disks while creating the VM. Meaning a copy of the template is not required on the target storage domain. 

Is there a different method on doing this via the API?

(In reply to comment #5)
> (In reply to comment #3)
> > Giulio,
> > 
> > To make it work, you need to copy template's disk/s to desired SD first,
> > please recheck and let me know if you still see any problem.
> 
> Michael, if this is the case then clone would have failed from the GUI as
> well. 
> Could it be that what SDK declares as clone is note what GUI declares as
> clone?

he doing this via api and not sdk, also afaiu he cloned vm on same SD in 
GUI.

> 
> In the GUI, Clone ==: Copy the template disks while creating the VM. Meaning
> a copy of the template is not required on the target storage domain. 

note: Giulio wants template's disks to be cloned to a different SD,
so disks should reside on a destination SD.

> 
> Is there a different method on doing this via the API?

no, and afaik this use-case was tested by QE.

(In reply to comment #6)
> (In reply to comment #5)
> > (In reply to comment #3)
> > > Giulio,
> > > 
> > > To make it work, you need to copy template's disk/s to desired SD first,
> > > please recheck and let me know if you still see any problem.
> > 
> > Michael, if this is the case then clone would have failed from the GUI as
> > well. 
> > Could it be that what SDK declares as clone is note what GUI declares as
> > clone?
> 
> he doing this via api and not sdk, also afaiu he cloned vm on same SD in 
> GUI.
> 
> > 
> > In the GUI, Clone ==: Copy the template disks while creating the VM. Meaning
> > a copy of the template is not required on the target storage domain. 
> 
> note: Giulio wants template's disks to be cloned to a different SD,
> so disks should reside on a destination SD.

clone VM from template copies all the data from the template disk to a new disk that belong to the VM so the template disk doesn't need to reside on the destination domain at all...

> 
> > 
> > Is there a different method on doing this via the API?
> 
> no, and afaik this use-case was tested by QE.

> clone VM from template copies all the data from the template disk to a new
> disk that belong to the VM so the template disk doesn't need to reside on
> the destination domain at all...

exactly to the point

this is the expected behaviour, it is what happens when cloning a template via webui (at VM creation time) and, as per bug report, this is what is _not_ happening when cloning via rest api

(In reply to comment #7)
> (In reply to comment #6)
> > (In reply to comment #5)
> > > (In reply to comment #3)
> > > > Giulio,
> > > > 
> > > > To make it work, you need to copy template's disk/s to desired SD first,
> > > > please recheck and let me know if you still see any problem.
> > > 
> > > Michael, if this is the case then clone would have failed from the GUI as
> > > well. 
> > > Could it be that what SDK declares as clone is note what GUI declares as
> > > clone?
> > 
> > he doing this via api and not sdk, also afaiu he cloned vm on same SD in 
> > GUI.
> > 
> > > 
> > > In the GUI, Clone ==: Copy the template disks while creating the VM. Meaning
> > > a copy of the template is not required on the target storage domain. 
> > 
> > note: Giulio wants template's disks to be cloned to a different SD,
> > so disks should reside on a destination SD.
> 
> clone VM from template copies all the data from the template disk to a new
> disk that belong to the VM so the template disk doesn't need to reside on
> the destination domain at all...

you saying that if template's disks on domain A and someone wants creating
vm on domain B (in this case entire disk should be cloned), disks copied at runtime?

(In reply to comment #9)
> (In reply to comment #7)
> > (In reply to comment #6)
> > > (In reply to comment #5)
> > > > (In reply to comment #3)
> > > > > Giulio,
> > > > > 
> > > > > To make it work, you need to copy template's disk/s to desired SD first,
> > > > > please recheck and let me know if you still see any problem.
> > > > 
> > > > Michael, if this is the case then clone would have failed from the GUI as
> > > > well. 
> > > > Could it be that what SDK declares as clone is note what GUI declares as
> > > > clone?
> > > 
> > > he doing this via api and not sdk, also afaiu he cloned vm on same SD in 
> > > GUI.
> > > 
> > > > 
> > > > In the GUI, Clone ==: Copy the template disks while creating the VM. Meaning
> > > > a copy of the template is not required on the target storage domain. 
> > > 
> > > note: Giulio wants template's disks to be cloned to a different SD,
> > > so disks should reside on a destination SD.
> > 
> > clone VM from template copies all the data from the template disk to a new
> > disk that belong to the VM so the template disk doesn't need to reside on
> > the destination domain at all...
> 
> you saying that if template's disks on domain A and someone wants creating
> vm on domain B (in this case entire disk should be cloned), disks copied at
> runtime?

don't forget that: copying disks between storage domains (over the wire) != local disk copy

> > clone VM from template copies all the data from the template disk to a new
> > disk that belong to the VM so the template disk doesn't need to reside on
> > the destination domain at all...
> 
> you saying that if template's disks on domain A and someone wants creating
> vm on domain B (in this case entire disk should be cloned), disks copied at
> runtime?

not at runtime, but at VM creation time

note: *this is perfectly working* when you choose to create a new VM from a template and set the 'Clone' option in the resources allocation page (as per screenshot) in the webui.

also: this is also working when the destination storage domain chosen is _not_ the one where the template is stored

the bug is about the rest api: such a functionality is _not_ working when the <clone>true</clone> tag is set in the POST request (see comment #1); in that case the VMs are created always in the same storage domain where the template is stored, even if one specifies a different storage domain id in the request

(In reply to comment #11)
> > > clone VM from template copies all the data from the template disk to a new
> > > disk that belong to the VM so the template disk doesn't need to reside on
> > > the destination domain at all...
> > 
> > you saying that if template's disks on domain A and someone wants creating
> > vm on domain B (in this case entire disk should be cloned), disks copied at
> > runtime?
> 
> not at runtime, but at VM creation time

creation time = run time

> 
> note: *this is perfectly working* when you choose to create a new VM from a
> template and set the 'Clone' option in the resources allocation page (as per
> screenshot) in the webui.
> 
> also: this is also working when the destination storage domain chosen is
> _not_ the one where the template is stored

i guess this can be true only if template already has disk/s on destination SD,
cause copying disk of N Gb over the wire to another SD at vm creation would 
take hours ...

also only now looking at your screenshot i realised that you're using
3.0 rhevm (missed that from your Comment 1), as latest GUI only has thin/clone options in "resource allocation" without ability to specify the SD, i assumed that you've cloned your disk on same SD,

anyway just check 3.0 api and it has bug, it ignores the SD indeed.

(In reply to comment #12)
> i guess this can be true only if template already has disk/s on destination
> SD, cause copying disk of N Gb over the wire to another SD at vm creation would 
> take hours ...

I'm not sure I understand your point here, it would take time but what should the destination SD already have?
  
> also only now looking at your screenshot i realised that you're using
> 3.0 rhevm (missed that from your Comment 1), as latest GUI only has
> thin/clone options in "resource allocation" without ability to specify the
> SD, i assumed that you've cloned your disk on same SD,

the GUI in 3.0 does allow for SD selection, it's the first drop down menu in the resources allocation tab (as per screenshot)
 
> anyway just check 3.0 api and it has bug, it ignores the SD indeed.

great, that's the actual issue we're facing!

(In reply to comment #13)
> (In reply to comment #12)
> > i guess this can be true only if template already has disk/s on destination
> > SD, cause copying disk of N Gb over the wire to another SD at vm creation would 
> > take hours ...
> 
> I'm not sure I understand your point here, it would take time but what
> should the destination SD already have?
>   

the template disk/s you would like to clone vm from, cause cloning existent
disk (on same storage device) is much more quicker than coping it over the
network (from another storage device).

basically that's the reason for template having disk/s on different storage
domains.

(In reply to comment #14)
> (In reply to comment #13)
> > (In reply to comment #12)
> > > i guess this can be true only if template already has disk/s on destination
> > > SD, cause copying disk of N Gb over the wire to another SD at vm creation would 
> > > take hours ...
> > 
> > I'm not sure I understand your point here, it would take time but what
> > should the destination SD already have?
> >   
> 
> the template disk/s you would like to clone vm from, cause cloning existent
> disk (on same storage device) is much more quicker than coping it over the
> network (from another storage device).
> 
> basically that's the reason for template having disk/s on different storage
> domains.

We're not on the same page here so let me try to explain myself again.
There are 2 options for creating a VM from a template:
1. thin provision - this requires template disks to reside on same storage domain as disks of the VM we want to create (because the VM disks are cow over the template disks).
2. clone - this copies all the data from the template into new disks and once copy is finished the VM is totally independent of the template.

Number 1 above is the *only* reason we need to copy template disks around.
This bug is about scenario number 2.

Wrt how the copy is performed, what you're describing is totally wrong since currently copy is host side copy which means that host always reads the bytes from the source disk (wherever it is) and writes to target disk.  To get the *optimization* you are referring to we'd need xcopy or another way of offloading the copy to the storage array.

hi Ayal,

thanks for contributing but, to be more clear, I'm not asking for any sort of optimization. Copying data from the source storage domain (where the template is stored) into the destination storage domain (where the VM will be allocated) is perfectly fine.

This is the exact requirement from our customer.

This is a bug request against the REST API, which is _not_ doing what it is supposed to do, as it is _not_ copying the data onto the second storage domain. The VM is always allocated withing the same storage domain where also the template is stored, regardless of the ID passed in the POST request.

As per commment #11, cloning via webui is working, cloning via REST API is not.

I hope this finally clarifies what the bugzilla is about, because there have been speculation about performance issues and the like, but it is a bug that I'm reporting, not an RFE.

(In reply to comment #16)
> hi Ayal,
> 
> thanks for contributing but, to be more clear, I'm not asking for any sort
> of optimization. Copying data from the source storage domain (where the
> template is stored) into the destination storage domain (where the VM will
> be allocated) is perfectly fine.
> 
> This is the exact requirement from our customer.

Hi Giulio, I know and that is exactly the point.

> 
> This is a bug request against the REST API, which is _not_ doing what it is
> supposed to do, as it is _not_ copying the data onto the second storage
> domain. The VM is always allocated withing the same storage domain where
> also the template is stored, regardless of the ID passed in the POST request.

Ack.
 
> As per commment #11, cloning via webui is working, cloning via REST API is
> not.
> 
> I hope this finally clarifies what the bugzilla is about, because there have
> been speculation about performance issues and the like, but it is a bug that
> I'm reporting, not an RFE.

Giulio,

(In reply to comment #16)
> hi Ayal,
> 
> I hope this finally clarifies what the bugzilla is about, because there have
> been speculation about performance issues and the like, but it is a bug that
> I'm reporting, not an RFE.

Obviously none of the clone design related comments is related to the bug
you've reported (no worries it will be addressed), this discussion for general education in this area and for others benefit.

Ayal,

> 
> We're not on the same page here so let me try to explain myself again.
> There are 2 options for creating a VM from a template:
> 1. thin provision - this requires template disks to reside on same storage
> domain as disks of the VM we want to create (because the VM disks are cow
> over the template disks).
> 2. clone - this copies all the data from the template into new disks and
> once copy is finished the VM is totally independent of the template.
> 
> Number 1 above is the *only* reason we need to copy template disks around.
> This bug is about scenario number 2.
> 
> Wrt how the copy is performed, what you're describing is totally wrong since
> currently copy is host side copy which means that host always reads the
> bytes from the source disk (wherever it is) and writes to target disk.  To
> get the *optimization* you are referring to we'd need xcopy or another way
> of offloading the copy to the storage array.

Thanks for the clarifications, of course i'm aware of host-side-copy, and i'm not talking about xcopy (what is reasonable for this use-case btw), just copying data from remote storage device [1] to local (when it could be done from local to local) does not seems logical to me.

[1] remote storage device = device physically located on remote site.

(In reply to comment #19)
> Ayal,
> 
> > 
> > We're not on the same page here so let me try to explain myself again.
> > There are 2 options for creating a VM from a template:
> > 1. thin provision - this requires template disks to reside on same storage
> > domain as disks of the VM we want to create (because the VM disks are cow
> > over the template disks).
> > 2. clone - this copies all the data from the template into new disks and
> > once copy is finished the VM is totally independent of the template.
> > 
> > Number 1 above is the *only* reason we need to copy template disks around.
> > This bug is about scenario number 2.
> > 
> > Wrt how the copy is performed, what you're describing is totally wrong since
> > currently copy is host side copy which means that host always reads the
> > bytes from the source disk (wherever it is) and writes to target disk.  To
> > get the *optimization* you are referring to we'd need xcopy or another way
> > of offloading the copy to the storage array.
> 
> Thanks for the clarifications, of course i'm aware of host-side-copy, and
> i'm not talking about xcopy (what is reasonable for this use-case btw), just
> copying data from remote storage device [1] to local (when it could be done
> from local to local) does not seems logical to me.
> 
> [1] remote storage device = device physically located on remote site.

You have no way of differentiating between remote and local.  In addition, even if it is remote the performance can still be better reading from remote.
All this doesn't matter because the bug is:
User asked to clone VM from tepmlate on Domain A to Domain B but VM was created on Domain A.
This is a bug any way you look at it.

(In reply to comment #19)
> Thanks for the clarifications, of course i'm aware of host-side-copy, and
> i'm not talking about xcopy (what is reasonable for this use-case btw), just
> copying data from remote storage device [1] to local (when it could be done
> from local to local) does not seems logical to me.

to add some context, the customer has some storage replication in place only for the master data domain

the templates are pushed on the master data domains by cloudforms and the critical VMs are deployed there

they prefer to copy and run their test and development VMs on the secondary storage, slower and for which there is is no replication in place

it has also to be noticed that they would like to use the cloning feature _also_ when instantiating new VMs on the same data domain where the template is stored (apparently to maintain some control over the storage overbooking)

regarding the bug, I just realized I made a big mistake not to clarify in the subject that this is affecting the restapi only :(

Changes the summary line to indicate the issue better. 
Need to make sure we keep all 3 options that exists in the UI 

1. Create thin -> VM disks are a snapshot out of the template disk
2. Create Clone: Templates Disks are copied onto any selected storage domain in the DC
   2.1 Thin -> VM Disks are copied into a thin format based on the storage type 
   2.2 Pre-allocated -> VM disks are copied while converting into RAW format 

Ayal, and as I remember you are blocked from converting pre-allocated disk templates to Thin.

Verified rhevm-3.1.0-28.el6ev.noarch


<vm>
   <name>esil933</name>
   <cluster>
      <name>iscsi</name>
   </cluster>
   <template id=\"53cbb087-c0d2-493c-9de5-ab9dcc64a169\"/>
   <disks>
      <clone>true</clone>
      <disk id=\"44dd071a-c068-4e9c-9bb8-8fe73096e765\">
         <storage_domains>
            <storage_domain id=\"7e15311a-9154-496c-82d6-53c2df72dfff\"/>
         </storage_domains>
         <format>RAW</format>
         <sparse>false</sparse>
      </disk>
   </disks>
   <memory>1073741824</memory>
   <cpu><topology cores=\"1\" sockets=\"1\"/></cpu>
   <os><boot dev=\"hd\"/></os>
   <high_availability>
      <enabled>true</enabled>
   </high_availability>
   <display>
      <type>vnc</type>
   </display>
</vm>

where storage_domain id is different than template's disk's storage domain id