Created attachment 1843027 [details]
engine.log from automation run of TC 16868

Another issue we are seeing on with this issue is in our automation flow: tier2 TestCase16968

version: 
ovirt-engine-4.4.9.5-0.1.el8ev.noarch
vdsm-4.40.90.4-1.el8ev.x86_64

Test scenario:
- Create VM with disk format "QCOW2" from a template (which has copies on NFS/Gluster/ISCSI/FCP) as clone on block-based domain.

Expected results:
- Disk virtual size should be larger than the disk actual size.

Actual results:

The actual size is larger than the virtual size.



Description:

In the UI I see the Virtual Size == Actual Size == 10 GB. 

In API I can see:

<provisioned_size>10737418240</provisioned_size>
<total_size>10871635968</total_size>
<actual_size>10871635968</actual_size>
<format>cow</format>
<qcow_version>qcow2_v3</qcow_version>



In vdsm I can see: (using vdsm-client StorageDomain dump sd_id=<id> | grep -A 16 <disk_id>
            "image": "357a5b94-84fc-424f-a037-3e704879c6f3",
            "legality": "LEGAL",
            "mdslot": 5,
            "parent": "00000000-0000-0000-0000-000000000000",
            "status": "OK",
            "truesize": 10871635968,
            "type": "SPARSE",
            "voltype": "LEAF"
        },
        "41c55e59-c5e4-456b-9edd-b119a1caf498": {
            "apparentsize": 134217728,
            "capacity": 134217728,
            "ctime": 1637258391,
            "description": "{\"Updated\":true,\"Size\":35840,\"Last Updated\":\"Mon Nov 22 16:00:18 IST 2021\",\"Storage Domains\":[{\"uuid\":\"55adc111-f75b-4831-a200-23166811d98e\"}],\"Disk Description\":\"OVF_STORE\"}",
            "disktype": "OVFS",
            "format": "RAW",
            "generation": 0,



In addition I can see (using lvs -o lv_name,lv_size,devices,tags --config='devices/filter = ["a|.*|"]' <sd_id>)
  LV                                   LSize   Devices                                            LV Tags                                                                             
  14fe070f-a93f-46f7-a253-ce6d18bc7ebc  10.12g /dev/mapper/3600a098038304479363f4c4870454f75(200) IU_357a5b94-84fc-424f-a037-3e704879c6f3,MD_5,PU_00000000-0000-0000-0000-000000000000




I attached in the previous comment the engine log from this run.
Relevant sd - <iscsi_2> id: 
Relevant disk id: 357a5b94-84fc-424f-a037-3e704879c6f3

Nir is this expected behavior or a bug? 
My team needs to change our automation in accordance. 

Thanks

(In reply to Amit Sharir from comment #0)
...
> Steps to Reproduce:
> 1. Create a new disk on VM via the UI with the following parameters:
> - storage domain: iSCSI/FC
> - allocation policy: preallocated

This means the system will allocate virtual size bytes for the image.

> - enable incremental backup: true

This means system will create qcow2 image.

> Actual results:
> Virtual size == actual size.

Yes, this is expected.

> Expected results:
> Virtual size should not be the same as actual size.

Why? what should be the actual size?
 
> Additional info:
> 
> I created a disk on ISCSI storage, incremental backup enabled, with a
> preallocated size of 12GB. In the UI I see the virtual size is 12GB. 
> 
> When getting the disk information (using : GET
> <engine_fqdn>/ovirt-engine/api/disks/<disk_id>) via API I can see that:
> 
> <provisioned_size>12884901888</provisioned_size> ==    
> <total_size>12884901888</total_size> ==    
> <actual_size>12884901888</actual_size> which is not as expected.

It is.

> Looking at the logical volume (using on vdsm: "lvs -o
> lv_name,lv_size,devices,tags --config='devices/filter = ["a|.*|"]' <sd-id>")
> 
> 35707e37-cbc0-4bb9-984f-d2c2eb0696d2  12.00g
> /dev/mapper/3600a098038304479363f4c487045506f(346)
> IU_c4fd533f-7125-413f-9915-4f32f05eeaf8,MD_9,PU_00000000-0000-0000-0000-
> 000000000000

Looks right.

> Looking at the logical volume (using on vdsm: vdsm-client StorageDomain dump
> sd_id=<id> | grep -A 16 <disk_id>)
> 
> "image": "c4fd533f-7125-413f-9915-4f32f05eeaf8",
>             "legality": "LEGAL",
>             "mdslot": 9,
>             "parent": "00000000-0000-0000-0000-000000000000",
>             "status": "OK",
>             "truesize": 12884901888,
>             "type": "PREALLOCATED",
>             "voltype": "LEAF"

No capacity or apparentsize?
did you change the output of the command?

This is the way we create disks with "enable incremental backup" since this feature was
introduced in 4.4. Not sure why this an issue for your tests now.

Hi Nir,

Thanks for the detailed response. 
Regarding what you asked on the "capacity" and "apparent size" - it does exist as the output of "vdsm-client StorageDomain dump> sd_id=<id> | grep -A 16 <disk_id>", I omitted it from the bug. 



Can you please also go over the automation failure I elaborated on in comment #2.
The relevant logs are also attached in comment #1. 

Please note that we see this failure for a long time in our automation. 
The test case expects that the disk virtual size should be larger than the disk actual size. But we are seeing the opposite in our execution analysis.
We are suspecting that the system behavior changed and that we need to change our automation in accordance (in other words, we think we might need to change our automation to expect that the disk's actual size should be bigger than the disk's virtual size). Before we refactor our automation we just want to check with you that indeed this is not a bug. 

Thanks.

Changed to assigned to reflect that it's investigated

(In reply to Amit Sharir from comment #2)
> Another issue we are seeing on with this issue is in our automation flow:
> tier2 TestCase16968

This indeed looks like another issue, so it should not be discussed in this
bug. Bug should focus on single issue, not on all case that have unexpected
actual size.

> version: 
> ovirt-engine-4.4.9.5-0.1.el8ev.noarch
> vdsm-4.40.90.4-1.el8ev.x86_64
> 
> Test scenario:
> - Create VM with disk format "QCOW2" from a template (which has copies on
> NFS/Gluster/ISCSI/FCP) as clone on block-based domain.

It does not matter that we have copies on NFS/Gluster/ISCSI/FCP. When we 
clone the image, we clone the disk we copy the template from the same
storage domain.

What matters is the format of the template disk. When we create a template
we can choose the disk format - which format did you choose?

I created two templates in my system, one as "Raw" and the other as "QCOW2".

Here are the details about these templates:

Template created using RAW format:

<disk href="/ovirt-engine/api/disks/da0ac715-d511-4468-9675-bf3871b2be5b" id="da0ac715-d511-4468-9675-bf3871b2be5b">
  <format>raw</format>
  <provisioned_size>6442450944</provisioned_size>
  <sparse>false</sparse>
  ...
</disk>
 
Template create using QCOW2 format:

<disk href="/ovirt-engine/api/disks/d06b0a1e-aef6-4011-9252-fa6c4c5f2b61" id="d06b0a1e-aef6-4011-9252-fa6c4c5f2b61">
  <format>cow</format>
  <provisioned_size>6442450944</provisioned_size>
  <qcow_version>qcow2_v3</qcow_version>
  <sparse>true</sparse>
  ...
</disk>

Which format is your template?

> Expected results:
> - Disk virtual size should be larger than the disk actual size.
> 
> Actual results:
> 
> The actual size is larger than the virtual size.

These exception depends on the template format and the type of the storage
domain.

> Description:
> 
> In the UI I see the Virtual Size == Actual Size == 10 GB. 
> 
> In API I can see:
> 
> <provisioned_size>10737418240</provisioned_size>
> <total_size>10871635968</total_size>
> <actual_size>10871635968</actual_size>
> <format>cow</format>
> <qcow_version>qcow2_v3</qcow_version>

So you got a qcow2 image as expected.
 
> In vdsm I can see: (using vdsm-client StorageDomain dump sd_id=<id> | grep
> -A 16 <disk_id>
>             "image": "357a5b94-84fc-424f-a037-3e704879c6f3",
>             "legality": "LEGAL",
>             "mdslot": 5,
>             "parent": "00000000-0000-0000-0000-000000000000",
>             "status": "OK",
>             "truesize": 10871635968,
>             "type": "SPARSE",
>             "voltype": "LEAF"
>         },

I guess this is the vm disk?

>         "41c55e59-c5e4-456b-9edd-b119a1caf498": {
>             "apparentsize": 134217728,
>             "capacity": 134217728,
>             "ctime": 1637258391,
>             "description": "{\"Updated\":true,\"Size\":35840,\"Last
> Updated\":\"Mon Nov 22 16:00:18 IST 2021\",\"Storage
> Domains\":[{\"uuid\":\"55adc111-f75b-4831-a200-23166811d98e\"}],\"Disk
> Description\":\"OVF_STORE\"}",
>             "disktype": "OVFS",
>             "format": "RAW",
>             "generation": 0,

This is OVF storage disk, how is it related to the vm disk?

> In addition I can see (using lvs -o lv_name,lv_size,devices,tags
> --config='devices/filter = ["a|.*|"]' <sd_id>)
>   LV                                   LSize   Devices                      
> LV Tags                                                                     
> 
>   14fe070f-a93f-46f7-a253-ce6d18bc7ebc  10.12g
> /dev/mapper/3600a098038304479363f4c4870454f75(200)
> IU_357a5b94-84fc-424f-a037-3e704879c6f3,MD_5,PU_00000000-0000-0000-0000-
> 000000000000

So the logical volume size is 10.12g.

I created a new vm from my templates, and got this:


Clone from qcow2 template:

Virtual Size: 6 GiB
Actual Size: 1 GiB
disk id: 1f90d45b-fefe-4dd4-83b1-de651adeff6d

# lvs -o vg_name,lv_name,size,tags | grep 1f90d45b-fefe-4dd4-83b1-de651adeff6d
  feab3738-c158-4d48-8a41-b5a95c057a50 e4145b1c-0e65-419d-996d-1c987096194b  <1.88g IU_1f90d45b-fefe-4dd4-83b1-de651adeff6d,MD_12,PU_00000000-0000-0000-0000-000000000000

# lvchange -ay feab3738-c158-4d48-8a41-b5a95c057a50/e4145b1c-0e65-419d-996d-1c987096194b

# qemu-img info /dev/feab3738-c158-4d48-8a41-b5a95c057a50/e4145b1c-0e65-419d-996d-1c987096194b
image: /dev/feab3738-c158-4d48-8a41-b5a95c057a50/e4145b1c-0e65-419d-996d-1c987096194b
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 0 B
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

# lvchange -an feab3738-c158-4d48-8a41-b5a95c057a50/e4145b1c-0e65-419d-996d-1c987096194b


Clone from raw template:

Virtual Size: 6 GiB
Actual Size: 6 GiB
disk id: d3c98805-b3c2-4562-9d50-29e46314075d

# lvs -o vg_name,lv_name,size,tags | grep d3c98805-b3c2-4562-9d50-29e46314075d

# lvs -o vg_name,lv_name,size,tags | grep d3c98805-b3c2-4562-9d50-29e46314075d
  feab3738-c158-4d48-8a41-b5a95c057a50 d29f8f4a-4536-4b37-b89d-41739d561343   6.62g IU_d3c98805-b3c2-4562-9d50-29e46314075d,MD_13,PU_00000000-0000-0000-0000-000000000000

# lvchange -ay feab3738-c158-4d48-8a41-b5a95c057a50/d29f8f4a-4536-4b37-b89d-41739d561343

# qemu-img info /dev/feab3738-c158-4d48-8a41-b5a95c057a50/d29f8f4a-4536-4b37-b89d-41739d561343
image: /dev/feab3738-c158-4d48-8a41-b5a95c057a50/d29f8f4a-4536-4b37-b89d-41739d561343
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 0 B
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

# lvchange -an feab3738-c158-4d48-8a41-b5a95c057a50/d29f8f4a-4536-4b37-b89d-41739d561343


When we clone vm from qcow2 template on block storage, we create a small
logical volume.

When we clone vm from raw template on block storage, we create logical
volume in the maximum size (virtual size * 1.1).

Why? when we copy a disk, we measure the required size for the disk using
qemu-img measure.

Measuring the template disks:

QCOW2 template:

disk id: d06b0a1e-aef6-4011-9252-fa6c4c5f2b61

# lvs -o vg_name,lv_name,size,tags | grep d06b0a1e-aef6-4011-9252-fa6c4c5f2b61
  feab3738-c158-4d48-8a41-b5a95c057a50 7fea7783-cb4b-419a-a0aa-95999c6b1686  <7.38g IU_d06b0a1e-aef6-4011-9252-fa6c4c5f2b61,MD_11,PU_00000000-0000-0000-0000-000000000000

# lvchange -ay c158-4d48-8a41-b5a95c057a50/7fea7783-cb4b-419a-a0aa-95999c6b1686

# lvchange -ay feab3738-c158-4d48-8a41-b5a95c057a50/7fea7783-cb4b-419a-a0aa-95999c6b1686
[root@host4 virt-v2v]# qemu-img measure -O qcow2 /dev/feab3738-c158-4d48-8a41-b5a95c057a50/7fea7783-cb4b-419a-a0aa-95999c6b1686
required size: 1796866048
fully allocated size: 6443696128
bitmaps size: 0


Raw template:

disk id: da0ac715-d511-4468-9675-bf3871b2be5b

# lvs -o vg_name,lv_name,size,tags | grep da0ac715-d511-4468-9675-bf3871b2be5b
  feab3738-c158-4d48-8a41-b5a95c057a50 be385bd3-3777-4240-95f5-147d56f66c48   6.00g IU_da0ac715-d511-4468-9675-bf3871b2be5b,MD_10,PU_00000000-0000-0000-0000-000000000000

# lvchange -ay feab3738-c158-4d48-8a41-b5a95c057a50/be385bd3-3777-4240-95f5-147d56f66c48

# qemu-img measure -O qcow2 /dev/feab3738-c158-4d48-8a41-b5a95c057a50/be385bd3-3777-4240-95f5-147d56f66c48
required size: 6443696128
fully allocated size: 6443696128


Since raw template does not have any metadata, qemu-img measure cannot tell
which areas are allocated and which are not, so must report that we need 
the fully allocated size (6443696128 bytes, 6.001 GiB).

Engine sends this value to vdsm, which always allocate 1.1 * virtual size
for block based volumes. Allocating 10% more is not needed in this case, but
removing this may break old engine that assume that vdsm allocates more.
So we allocate 6.60 GiB, aligned to 6.62 by lvm.

Since you did not specify the format of the template, I cannot tell if the
behavior is expected or now, but you have here all the info needed to 
check your flow.

The vm cloned from the raw template can be reduced to optimal size using:

$ ./reduce_disk.py -c engine-dev d3c98805-b3c2-4562-9d50-29e46314075d
[   0.0 ] Connecting...
[   0.1 ] Reducing disk...
[   2.0 ] Disk was reduced successfully

After the reduce the logical volume as reduced to:

# lvs -o vg_name,lv_name,size,tags | grep d3c98805-b3c2-4562-9d50-29e46314075d
  feab3738-c158-4d48-8a41-b5a95c057a50 d29f8f4a-4536-4b37-b89d-41739d561343   2.75g IU_d3c98805-b3c2-4562-9d50-29e46314075d,MD_13,PU_00000000-0000-0000-0000-000000000000

When we reduce we leave 1 GiB free space at the end of the  disk.
The actual disk allocation is ~1.75g.

Please file RFE to reduce vm disks created from raw template on block
storage.

Following the correspondence above, it was decided to close this bug and open new (and more specific bugs) for each issues mentioned.

Opened a new bug regarding "Cloning a VM from a QCOW based template on a block-based storage domain, results with a VM that has a disk with the same actual and virtual size ": https://bugzilla.redhat.com/show_bug.cgi?id=2034531

Closed this bug as a duplicate of 2034531: https://bugzilla.redhat.com/show_bug.cgi?id=1932794

In addition, opened an RFE regarding "reduce size of VM disks that are created from a RAW based template on a block-based storage domain": https://bugzilla.redhat.com/show_bug.cgi?id=2034542

*** This bug has been marked as a duplicate of bug 2034531 ***