Bug 1977232 - Create template broken with block storage
Summary: Create template broken with block storage
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.4.7
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ovirt-4.4.9
: ---
Assignee: Nir Soffer
QA Contact: Evelina Shames
URL:
Whiteboard:
Depends On: 1973345
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-29 09:20 UTC by Eyal Shenitzky
Modified: 2021-12-23 16:31 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of: 1973345
Environment:
Last Closed: 2021-11-16 14:46:53 UTC
oVirt Team: Storage
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:4626 0 None None None 2021-11-16 14:47:14 UTC
oVirt gerrit 115303 0 None MERGED image: Fix measure of block volume in copyCollpase 2021-09-13 12:23:27 UTC

Description Eyal Shenitzky 2021-06-29 09:20:10 UTC
+++ This bug was initially created as a clone of Bug #1973345 +++

Description of problem:

Creating a template from vm when source disk is on block storage, and the
volume does not have a parent fails with:

engine.log:

VDSM host3 command HSMGetAllTasksStatusesVDS failed: value=low level Image copy failed: ('Destination volume 0195a9f2-51f6-49c8-8520-9c630cbdfd59 error: Command [\'/usr/bin/qemu-img\', \'measure\', \'--output\', \'json\', \'-O\', \'qcow2\', \'json:{"file": {"driver": "file", "filename": "/rhev/data-center/mnt/blockSD/feab3738-c158-4d48-8a41-b5a95c057a50/images/3e8214f8-7a70-4e6b-a2fe-84c7103cd8ae/2eda0099-4dbc-4602-a418-fbd1ccee3357"}, "driver": "qcow2"}\'] failed with rc=1 out=b\'\' err=b\'qemu-img: Could not open \\\'json:{"file": {"driver": "file", "filename": "/rhev/data-center/mnt/blockSD/feab3738-c158-4d48-8a41-b5a95c057a50/images/3e8214f8-7a70-4e6b-a2fe-84c7103cd8ae/2eda0099-4dbc-4602-a418-fbd1ccee3357"}, "driver": "qcow2"}\\\': \\\'file\\\' driver requires \\\'/rhev/data-center/mnt/blockSD/feab3738-c158-4d48-8a41-b5a95c057a50/images/3e8214f8-7a70-4e6b-a2fe-84c7103cd8ae/2eda0099-4dbc-4602-a418-fbd1ccee3357\\\' to be a regular file\\n\'',) abortedcode=261

vdsm.log:

2021-06-17 03:14:17,571+0300 INFO  (tasks/1) [storage.ThreadPool.WorkerThread] START task 62624777-82b1-4d25-8db7-1bffbf34f
f90 (cmd=<bound method Task.commit of <vdsm.storage.task.Task object at 0x7fa62aee1f60>>, args=None) (threadPool:146)
2021-06-17 03:14:17,925+0300 INFO  (tasks/1) [storage.Image] sdUUID=feab3738-c158-4d48-8a41-b5a95c057a50 vmUUID= srcImgUUID
=3e8214f8-7a70-4e6b-a2fe-84c7103cd8ae srcVolUUID=2eda0099-4dbc-4602-a418-fbd1ccee3357 dstImgUUID=77eec8ed-7dce-4250-8575-42
5d661e64e6 dstVolUUID=0195a9f2-51f6-49c8-8520-9c630cbdfd59 dstSdUUID=feab3738-c158-4d48-8a41-b5a95c057a50 volType=6 volForm
at=COW preallocate=SPARSE force=False postZero=False discard=True (image:635)
2021-06-17 03:14:17,926+0300 INFO  (tasks/1) [storage.VolumeManifest] Volume: preparing volume feab3738-c158-4d48-8a41-b5a9
5c057a50/2eda0099-4dbc-4602-a418-fbd1ccee3357 (volume:599)
2021-06-17 03:14:17,934+0300 INFO  (tasks/1) [storage.LVM] Activating lvs: vg=feab3738-c158-4d48-8a41-b5a95c057a50 lvs=['2e
da0099-4dbc-4602-a418-fbd1ccee3357'] (lvm:1755)
2021-06-17 03:14:18,298+0300 ERROR (tasks/1) [storage.Image] Unexpected error (image:729)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/storage/image.py", line 673, in copyCollapsed
    sdUUID, volParams, dstSdUUID, dstVolFormat)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/image.py", line 850, in calculate_vol_alloc
    return self.estimate_qcow2_size(src_vol_params, dst_sd_id)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/image.py", line 117, in estimate_qcow2_size
    output_format=qemuimg.FORMAT.QCOW2)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/qemuimg.py", line 156, in measure
    out = _run_cmd(cmd)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/qemuimg.py", line 578, in _run_cmd
    raise cmdutils.Error(cmd, rc, out, err)
vdsm.common.cmdutils.Error: Command ['/usr/bin/qemu-img', 'measure', '--output', 'json', '-O', 'qcow2', 'json:{"file": {"driver": "file", "filename": "/rhev/data-center/mnt/blockSD/feab3738-c158-4d48-8a41-b5a95c057a50/images/3e8214f8-7a70-4e6b-a2fe-84c7103cd8ae/2eda0099-4dbc-4602-a418-fbd1ccee3357"}, "driver": "qcow2"}'] failed with rc=1 out=b'' err=b'qemu-img: Could not open \'json:{"file": {"driver": "file", "filename": "/rhev/data-center/mnt/blockSD/feab3738-c158-4d48-8a41-b5a95c057a50/images/3e8214f8-7a70-4e6b-a2fe-84c7103cd8ae/2eda0099-4dbc-4602-a418-fbd1ccee3357"}, "driver": "qcow2"}\': \'file\' driver requires \'/rhev/data-center/mnt/blockSD/feab3738-c158-4d48-8a41-b5a95c057a50/images/3e8214f8-7a70-4e6b-a2fe-84c7103cd8ae/2eda0099-4dbc-4602-a418-fbd1ccee3357\' to be a regular file\n'

The issues is using:

    'json:{"file": {"driver": "file", "filename": "/path"}, "driver": "qcow2"}'

When /path is block storage. It should be:

    'json:{"file": {"driver": "host_device", "filename": "/path"}, "driver": "qcow2"}'

The root cause is not calling qemuimg.measure() without is_block=True.

This is a regression introduce in:
$ git describe 1718b8784e841405574c44abe2357997e3235723
v4.40.32-4-g1718b8784

Before this change, qemu-img measure was called with a path instead of json: uri,
and qemu-img detected the right driver.

Version-Release number of selected component (if applicable):
vdsm-4.40.70.4

How reproducible:
100$

Steps to Reproduce:
1. Create think vm on block storage (iSCSI/FC)
   The vm disk must not have a parent.
2. Create template from the VM using qcow2 format

The operation fails.

--- Additional comment from RHEL Program Management on 2021-06-18 11:27:44 UTC ---

This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

--- Additional comment from Nir Soffer on 2021-06-19 15:43:38 UTC ---

Should be available in vdsm 4.40.70.5.

--- Additional comment from Lukas Svaty on 2021-06-21 10:43:12 UTC ---

Based on Infro from mavital adding blocker+

--- Additional comment from  on 2021-06-28 09:06:17 UTC ---

Could't reproduce it on older version (VDSM 4.40.7.4) according to these steps mentioned above.

1) created a VM from Blank (no template)
2) Attached new disk that I created with thin provision on iscsi storage domain
3) From virtual machines -> choose the new VM -> Make Template using qcow2 format.

The process passes successfully and the template and can be seen in the template section under compute.


Nir, Can you please add additional steps if something is missing here?

--- Additional comment from  on 2021-06-28 09:07:57 UTC ---

correcting : VDSM 4.40.70.4

--- Additional comment from Nir Soffer on 2021-06-28 11:53:37 UTC ---

(In reply to sshmulev from comment #4)
This isssue can be reproduced only with qemu-6.0.0 available in RHEL 8.5 nightly repos
or on Centos Stream.

On RHEL 8.4 we have qemu 5.2.0:

[root@storage-ge12-vdsm3 ~]# rpm -q qemu-img
qemu-img-5.2.0-16.module+el8.4.0+10806+b7d97207.x86_64

With this version trying to measure block device with incorrect json uri does
not fail, it only logs a warning:

# qemu-img info /dev/0bc333f0-815d-4613-87a4-341dbde9413e/eb83cee8-424d-4edc-9695-ca1570dea42c
image: /dev/0bc333f0-815d-4613-87a4-341dbde9413e/eb83cee8-424d-4edc-9695-ca1570dea42c
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 0 B
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

# qemu-img measure -O qcow2 'json:{"driver": "qcow2", "file":{"driver": "file", "filename": "/dev/0bc333f0-815d-4613-87a4-341dbde9413e/eb83cee8-424d-4edc-9695-ca1570dea42c"}}'; echo $?
qemu-img: warning: Opening a block device as a file using the 'file' driver is deprecated
required size: 1900544
fully allocated size: 10739318784
bitmaps size: 0
0

So this bug affects oVirt 4.4.7 *now* on Centos Stream, but it will affect RHV only when
RHEL 8.5 will be released.

To test this properly, you need to upgrade qemu to version 6.0.0. This requires also 
upgrading libvirt. It can be done by using RHEL 8.5.0 nightly repos instead of the 8.4.0
nightly repos.

Comment 2 Michal Skrivanek 2021-06-30 07:15:56 UTC
4.4.8 likely won't use 8.5.
TBH I don't think we need to track this, verification in Stream is enough, and if it comes back it will be detected by automation right away

Comment 3 Eyal Shenitzky 2021-07-05 06:14:02 UTC
(In reply to Michal Skrivanek from comment #2)
> 4.4.8 likely won't use 8.5.
> TBH I don't think we need to track this, verification in Stream is enough,
> and if it comes back it will be detected by automation right away

Right, this was my opinion here too, but QE requested to track and test this bug manually using 8.5 hosts on a later version.

Comment 4 Marina Kalinin 2021-08-12 17:28:36 UTC
(In reply to Michal Skrivanek from comment #2)
> 4.4.8 likely won't use 8.5.
> TBH I don't think we need to track this, verification in Stream is enough,
> and if it comes back it will be detected by automation right away

So, this is not relevant to 8.4?
And we assume that once 8.5 is out, customers are expected to upgrade altogether and there is no way they will hit this?
However, if anyone upgrades individual packages (unsupported), this may happen.

Comment 11 Avihai 2021-10-11 11:03:46 UTC
(In reply to Evelina Shames from comment #8)
> The mentioned flow:
> 1) created a VM from Blank (no template)
> 2) Attached new disk that I created with thin provision on iscsi storage
> domain
> 3) From virtual machines -> choose the new VM -> Make Template using qcow2
> format.
> 
> works in 4.4.9 with rhel8.5:
> qemu-img-6.0.0-30.module+el8.5.0+12586+476da3e1.x86_64
> vdsm-4.40.90.1-1.el8ev.x86_64
> ovirt-engine-4.4.9-0.6.el8ev.noarch
> 
> Will move this to VERIFIED once it moves to ON_QA.

As comment #8 suggests this was already tested and verified.
Moving to VERIFIED

Comment 15 errata-xmlrpc 2021-11-16 14:46:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: RHV Manager (ovirt-engine) security update [ovirt-4.4.9]), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4626


Note You need to log in before you can comment on or make changes to this bug.