Bug 1589763 - [downstream clone] Error changing CD for a running VM when ISO image is on a block domain
Summary: [downstream clone] Error changing CD for a running VM when ISO image is on a ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.4.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-4.4.6
: 4.4.6
Assignee: Vojtech Juranek
QA Contact: Evelina Shames
URL:
Whiteboard:
: 1588400 1722474 1725910 1735778 (view as bug list)
Depends On: 1588400
Blocks: 1660199 gss_rhv_4_3_4 1736852 1789090 1835640
TreeView+ depends on / blocked
 
Reported: 2018-06-11 11:00 UTC by Pawan kumar Vilayatkar
Modified: 2024-10-01 16:08 UTC (History)
44 users (show)

Fixed In Version: ovirt-engine-4.4.6.6
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1588400
: 1660199 (view as bug list)
Environment:
Last Closed: 2021-06-01 13:22:09 UTC
oVirt Team: Storage
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Logs (795.76 KB, application/zip)
2021-05-06 08:40 UTC, Evelina Shames 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3626821 0 None None None 2018-09-27 02:51:04 UTC
Red Hat Knowledge Base (Solution) 4237861 0 None None None 2021-05-11 02:03:28 UTC
Red Hat Product Errata RHSA-2021:2179 0 None None None 2021-06-01 13:23:11 UTC
oVirt gerrit 111547 0 master ABANDONED tests: add CDROM tests with real storage 2021-02-16 00:52:34 UTC
oVirt gerrit 111569 0 master MERGED tests: move tests related to CD into separate module 2021-02-16 00:52:34 UTC
oVirt gerrit 111733 0 master ABANDONED vm: add cdrom metadata helper methods 2021-02-16 00:52:35 UTC
oVirt gerrit 111780 0 master MERGED vm: refactor Vm._changeBlockDev() 2021-02-16 00:52:35 UTC
oVirt gerrit 111804 0 master ABANDONED virt: add method for removing device metadata from VM xml 2021-02-16 00:52:35 UTC
oVirt gerrit 111831 0 master ABANDONED vm: add exact attribute to device context manager 2021-02-16 00:52:36 UTC
oVirt gerrit 111908 0 master ABANDONED vm: add helper method for obtaining cdrom metadata 2021-02-16 00:52:36 UTC
oVirt gerrit 111948 0 master ABANDONED vm: add new private change_cd method 2021-02-16 00:52:36 UTC
oVirt gerrit 112203 0 master ABANDONED vm: add helper method for obtaining CDROM attributes 2021-02-16 00:52:36 UTC
oVirt gerrit 112277 0 master MERGED virt: add `change` key into metadata 2021-02-16 00:52:36 UTC
oVirt gerrit 112334 0 master MERGED vm: add helper methods for updating CDROM metadata 2021-02-16 00:52:37 UTC
oVirt gerrit 112342 0 master MERGED vm: add new private change_cd method 2021-02-16 00:52:37 UTC
oVirt gerrit 112552 0 master MERGED tests: add test change loaded CD 2021-02-16 00:52:37 UTC
oVirt gerrit 112553 0 master MERGED vm: refactor xml creation from Vm._changeBlockDev() 2021-02-16 00:52:37 UTC
oVirt gerrit 112716 0 master MERGED vm: always return empty list from changeCD 2021-02-16 00:52:37 UTC
oVirt gerrit 113079 0 master MERGED vm: use new _change_cd() when engine sends CD spec as PDIV 2021-02-16 00:52:38 UTC
oVirt gerrit 113324 0 master ABANDONED api: use correct type for change CD 2021-02-16 00:52:39 UTC
oVirt gerrit 113325 0 master MERGED api: add DriveSpecVolumeCdrom CD volume specification 2021-02-16 00:52:39 UTC
oVirt gerrit 113326 0 master MERGED caps: add capability for specifying CD via PDIV 2021-02-16 00:52:38 UTC
oVirt gerrit 113830 0 master MERGED core: always set CD index and iface 2021-03-16 13:27:28 UTC
oVirt gerrit 113831 0 master MERGED core: add cd_change_pdiv field to VDS 2021-03-16 13:27:31 UTC
oVirt gerrit 113832 0 master MERGED core: adjust command for changing disk to use PDIV 2021-03-16 13:27:37 UTC
oVirt gerrit 114030 0 master MERGED virt: add helper for cd recovery 2021-04-26 10:02:57 UTC
oVirt gerrit 114031 0 master MERGED virt: recover CD during VM recovery 2021-04-26 10:03:01 UTC

Description Pawan kumar Vilayatkar 2018-06-11 11:00:34 UTC 
+++ This bug was initially created as a clone of Bug #1588400 +++

Description of problem:

I have a CentOS 7 VM. I correctly uploaded an iSO image to the disks of a block based data storage domain (iSCSI).
Now in web admin portal I select the VM, then the 3 dots in top right, then change cd
I'm proposed with the [Eject] line and into the dropdown I see the two iso images I have (theoretically) uploaded up to now.
I select an image and then OK

I get a window with title "Operation canceled" and content
Error while executing action Change CD: Drive image file could not be found

Version-Release number of selected component (if applicable):
4.2.3.7-1.el7

How reproducible:
always

Steps to Reproduce:
1. select storage domain and upload iso
2. wait for completion and then you will see the iso listed between disks of the storage domain
3. try to attach the iso to a vm. You see the iso in dropdown list and select it

Actual results:
Error while executing action Change CD: Drive image file could not be found

Expected results:
attaching the iso to the guest and using it

Additional info:

I'm going to provide logs.
I have tried the same on the same version of oVirt but with NFS based storage domain where I upload the iso and in that environment it works as expected.
So it seems related only to block based data domains, where probably it should be created a sort of a link to an LVM structure?

--- Additional comment from Gianluca Cecchi on 2018-06-07 04:38:30 EDT ---

The error in engine.log is

2018-05-30 11:58:58,772+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ChangeDiskVDSCommand] (default task-9) [d9aae17f-2a49-4d70-a909-6395d61d3ab1] Failed in 'ChangeDiskVDS' method
2018-05-30 11:58:58,775+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-9) [d9aae17f-2a49-4d70-a909-6395d61d3ab1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM ov200 command ChangeDiskVDS failed: Drive image file could not be found

See engine.log attached

--- Additional comment from Gianluca Cecchi on 2018-06-07 04:40 EDT ---

You can view at 11:58:57 the time I try to attach the iso

The upload seemed to have completed ok
2018-05-30 11:58:34,679+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirect
or] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [2b74b21b-7a68-45ae-9b09-1e6cc13b63ce] EVENT
_ID: TRANSFER_IMAGE_SUCCEEDED(1,032), Image Upload with disk win7_32b_eng_sp1.iso succeeded.

but probably something is wrong. See my further comment on this

--- Additional comment from Gianluca Cecchi on 2018-06-07 04:45 EDT ---

You can see apparently ok:
2018-05-30 11:58:19,820 INFO    (Thread-12) [images] Writing 40026112 bytes at offset 524288000 flush True to /rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/71a84a1c-0c53-4bb9-9474-deb92419e955/5404add1-cac4-4129-b8f5-2e7b2fc0da86 for ticket dcacf739-241d-472b-87b7-e0653424d3e5

But if I go on the host (that is the SPM right now) I see this:

directory exists and contains a link:
[root@ov200 tmp]# ll /rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/71a84a1c-0c53-4bb9-9474-deb92419e955/
total 0
lrwxrwxrwx. 1 vdsm kvm 78 May 30 11:58 5404add1-cac4-4129-b8f5-2e7b2fc0da86 -> /dev/fa33df49-b09d-4f86-9719-ede649542c21/5404add1-cac4-4129-b8f5-2e7b2fc0da86
[root@ov200 tmp]# 

But hte link points to a not existent device file:
[root@ov200 tmp]# ll /dev/fa33df49-b09d-4f86-9719-ede649542c21/5404add1-cac4-4129-b8f5-2e7b2fc0da86
ls: cannot access /dev/fa33df49-b09d-4f86-9719-ede649542c21/5404add1-cac4-4129-b8f5-2e7b2fc0da86: No such file or directory
[root@ov200 tmp]# 

So I have a link that actually point to a non existent device file...

--- Additional comment from Gianluca Cecchi on 2018-06-07 04:48:26 EDT ---

Inside the directory pointed by the link I have this:

[root@ov200 tmp]# ll /dev/fa33df49-b09d-4f86-9719-ede649542c21/
total 0
lrwxrwxrwx. 1 root root 8 May 28 14:34 04548ff7-863d-4959-9f84-abf18cf38584 -> ../dm-19
lrwxrwxrwx. 1 root root 8 May 24 16:03 177ac2a5-49d4-4d01-a39a-b2e4b984f57c -> ../dm-16
lrwxrwxrwx. 1 root root 8 May 24 16:03 20daa039-9648-4998-b082-0e94dce0001b -> ../dm-15
lrwxrwxrwx. 1 root root 8 May 28 14:34 252baa4d-54b7-4f36-88e9-3a59e97cec7c -> ../dm-22
lrwxrwxrwx. 1 root root 8 May 24 16:03 3569d3ae-a366-4f33-8667-036e6874b873 -> ../dm-17
lrwxrwxrwx. 1 root root 8 May 28 14:34 58001deb-908e-4fd2-a1cf-f00b84ac600b -> ../dm-24
lrwxrwxrwx. 1 root root 8 May 28 14:34 62e3f367-545b-4608-bc8b-45ac08944acf -> ../dm-13
lrwxrwxrwx. 1 root root 8 May 28 14:34 634cbebb-0303-410f-8331-f51274d38e29 -> ../dm-23
lrwxrwxrwx. 1 root root 8 May 28 14:34 779e2720-cb5b-447a-ac7d-6264a559b26e -> ../dm-12
lrwxrwxrwx. 1 root root 8 May 28 14:34 8c44b67c-2408-454b-af43-352cdcc2f6e7 -> ../dm-20
lrwxrwxrwx. 1 root root 8 May 28 15:19 9af3574d-dc83-485f-b906-0970ad09b660 -> ../dm-25
lrwxrwxrwx. 1 root root 8 May 28 14:34 9e5a64b4-e3cc-4d4e-aa4f-35a9ae3f2f6f -> ../dm-21
lrwxrwxrwx. 1 root root 8 May 24 15:57 a3f315ab-a5c4-4366-bbe7-2fee006fb4ad -> ../dm-10
lrwxrwxrwx. 1 root root 8 May 28 15:13 aeb2e4bf-54df-4324-b1ee-283890afe4cf -> ../dm-27
lrwxrwxrwx. 1 root root 8 May 28 14:34 b36fbe5b-56e0-4f23-a9b7-cd80f7f5c0ab -> ../dm-18
lrwxrwxrwx. 1 root root 8 May 28 14:34 c0414aed-1459-48d7-ae22-5b0215c7e7de -> ../dm-14
lrwxrwxrwx. 1 root root 8 May 24 15:57 d83f5aaf-b4ae-4c20-9f05-bf9162c6cbf8 -> ../dm-11
lrwxrwxrwx. 1 root root 8 May 28 15:13 e6d44b9c-a9ae-426b-8c16-2c0f43e0faed -> ../dm-28
lrwxrwxrwx. 1 root root 8 May 28 15:13 ec1f2904-4263-4322-8ae8-35c91759f88d -> ../dm-26
lrwxrwxrwx. 1 root root 8 May 28 15:13 f772414d-a1b9-45fc-a8e9-640f62592333 -> ../dm-29
lrwxrwxrwx. 1 root root 7 May 24 15:55 ids -> ../dm-4
lrwxrwxrwx. 1 root root 7 May 24 17:16 inbox -> ../dm-8
lrwxrwxrwx. 1 root root 7 Jun  7 10:47 leases -> ../dm-5
lrwxrwxrwx. 1 root root 7 May 30 13:09 master -> ../dm-9
lrwxrwxrwx. 1 root root 7 Jun  5 16:16 metadata -> ../dm-3
lrwxrwxrwx. 1 root root 7 May 30 13:09 outbox -> ../dm-6
lrwxrwxrwx. 1 root root 7 May 24 15:55 xleases -> ../dm-7
[root@ov200 tmp]# 


[root@ov200 tmp]# ll -L /dev/fa33df49-b09d-4f86-9719-ede649542c21/
total 0
brw-rw----. 1 vdsm qemu    253, 19 May 28 14:34 04548ff7-863d-4959-9f84-abf18cf38584
brw-rw----. 1 vdsm qemu    253, 16 May 24 16:03 177ac2a5-49d4-4d01-a39a-b2e4b984f57c
brw-rw----. 1 vdsm qemu    253, 15 May 24 16:03 20daa039-9648-4998-b082-0e94dce0001b
brw-rw----. 1 vdsm qemu    253, 22 May 28 14:34 252baa4d-54b7-4f36-88e9-3a59e97cec7c
brw-rw----. 1 vdsm qemu    253, 17 May 24 16:03 3569d3ae-a366-4f33-8667-036e6874b873
brw-rw----. 1 vdsm qemu    253, 24 May 28 14:34 58001deb-908e-4fd2-a1cf-f00b84ac600b
brw-rw----. 1 vdsm qemu    253, 13 May 28 14:34 62e3f367-545b-4608-bc8b-45ac08944acf
brw-rw----. 1 vdsm qemu    253, 23 May 28 14:34 634cbebb-0303-410f-8331-f51274d38e29
brw-rw----. 1 vdsm qemu    253, 12 May 28 14:34 779e2720-cb5b-447a-ac7d-6264a559b26e
brw-rw----. 1 vdsm qemu    253, 20 May 28 14:34 8c44b67c-2408-454b-af43-352cdcc2f6e7
brw-rw----. 1 vdsm qemu    253, 25 May 28 15:19 9af3574d-dc83-485f-b906-0970ad09b660
brw-rw----. 1 vdsm qemu    253, 21 May 28 14:34 9e5a64b4-e3cc-4d4e-aa4f-35a9ae3f2f6f
brw-rw----. 1 vdsm qemu    253, 10 May 24 15:57 a3f315ab-a5c4-4366-bbe7-2fee006fb4ad
brw-rw----. 1 vdsm qemu    253, 27 May 28 15:13 aeb2e4bf-54df-4324-b1ee-283890afe4cf
brw-rw----. 1 vdsm qemu    253, 18 May 28 14:34 b36fbe5b-56e0-4f23-a9b7-cd80f7f5c0ab
brw-rw----. 1 vdsm qemu    253, 14 May 28 14:34 c0414aed-1459-48d7-ae22-5b0215c7e7de
brw-rw----. 1 vdsm qemu    253, 11 May 24 15:57 d83f5aaf-b4ae-4c20-9f05-bf9162c6cbf8
brw-rw----. 1 vdsm qemu    253, 28 May 28 15:13 e6d44b9c-a9ae-426b-8c16-2c0f43e0faed
brw-rw----. 1 vdsm qemu    253, 26 May 28 15:13 ec1f2904-4263-4322-8ae8-35c91759f88d
brw-rw----. 1 vdsm qemu    253, 29 May 28 15:13 f772414d-a1b9-45fc-a8e9-640f62592333
brw-rw----. 1 vdsm sanlock 253,  4 Jun  7 10:47 ids
brw-------. 1 vdsm qemu    253,  8 May 24 17:16 inbox
brw-rw----. 1 vdsm sanlock 253,  5 Jun  7 10:48 leases
brw-rw----. 1 root disk    253,  9 May 30 13:09 master
brw-------. 1 vdsm qemu    253,  3 Jun  5 16:16 metadata
brw-------. 1 vdsm qemu    253,  6 May 30 13:09 outbox
brw-rw----. 1 vdsm sanlock 253,  7 May 24 15:55 xleases
[root@ov200 tmp]#

--- Additional comment from Gianluca Cecchi on 2018-06-07 10:26:47 EDT ---

Hello, 
I have also reproduced the same problem (always on iSCSI based storage domain) on RHV, version 4.2.3.6-0.1.el7.
I have opened the case 02115992 for that and I have uploaded the ovirt-log-collector tar.gz to the case. So you can also crosscheck with that environment data, to get the info and find a solution.
HIH,
Gianluca
Comment 4 Gianluca Cecchi 2018-08-22 13:16:51 UTC 
We are going from 4.2.5, to 4.2.6 to 4.2.7 as target...
Can you detail the difficulties you are having or if it is not considered so much important or if your queue already full or what?
Once a month we go ahead in target... without specifying reason for it.
Not quite nice for the guy that opens a bug because he/she thinks there is a problem with the current implementation.
Or at least in the mean time prevent to make upload of iso to block domain specifying it is yet unsupported, instead of giving the false impression that the iso upload has completed without problems and the user gets problems when trying to consume that iso....

Thanks
Gianluca
Comment 5 Gianluca Cecchi 2018-08-22 13:22:38 UTC 
Also considering that both priority and severity of this bug have been marked as "high"... Lower them if you think they are not that level...
Comment 9 Marina Kalinin 2018-08-29 14:18:47 UTC 
Until the fix -> the recommended workaround is to use ISO domain.
Comment 11 Marina Kalinin 2018-08-29 14:20:27 UTC 
Another workaround is to use NFS Data Domain, that should work as well.
Comment 12 Marina Kalinin 2018-08-29 18:21:48 UTC 
Another option is if a VM is down -> you can change the CD via run once.
Comment 19 Tal Nisan 2018-12-30 08:49:33 UTC 
*** Bug 1588400 has been marked as a duplicate of this bug. ***
Comment 21 Michal Skrivanek 2019-07-02 05:04:23 UTC 
*** Bug 1725910 has been marked as a duplicate of this bug. ***
Comment 22 Nir Soffer 2019-07-04 20:09:36 UTC 
Copying my comments from bug 1588400 - it was closed as duplicate without
looking at my comments.

This looks like engine issue, since vdsm supports preparing cdrom using PDIV
format:

    def prepareVolumePath(self, drive, vmId=None):
        if type(drive) is dict:
            device = drive['device']
            # PDIV drive format
            # Since version 4.2 cdrom may use a PDIV format
            if device in ("cdrom", "disk") and isVdsmImage(drive):
                res = self.irs.prepareImage(
                    drive['domainID'], drive['poolID'],
                    drive['imageID'], drive['volumeID'])

Engine sends a path - this incorrect for two reasons:

1. engine should not assume any path on the host, since the file system layout is
   not part of vdsm API.

2. cdrom on block device does not exist before preparing the image.

2018-05-30 11:57:04,692+02 INFO  [org.ovirt.engine.core.bll.storage.disk.ChangeDiskCommand] (default task-15) [7bb55228-dbf3-49a9-aa03-148433cca656] Running command: ChangeDiskCommand internal: false. Entities affected :  ID: 2e571c77-bae1-4c1c-bf98-effaf9fed741 Type: VMAction group CHANGE_VM_CD with role type USER
2018-05-30 11:57:04,706+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ChangeDiskVDSCommand] (default task-15) [7bb55228-dbf3-49a9-aa03-148433cca656] START, ChangeDiskVDSCommand(HostName = ov200, ChangeDiskVDSCommandParameters:{hostId='d16e723c-b44c-4c1c-be76-c67911e47ccd', vmId='2e571c77-bae1-4c1c-bf98-effaf9fed741', iface='ide', index='2', diskPath='/rhev/data-center/mnt/blockSD/fa33df49-b09d-4f86-9719-ede649542c21/images/321d8f35-854a-4415-a78a-a30b197a29a5/8df78e5b-e79b-4b29-96ae-330fec2cdc9d'}), log id: 62555af

Fix:
- Engine should send PDIV format in the request
- Update vdsm schema to document the option to use PDIV format in VM.changeCD
Comment 23 Fred Rolland 2019-07-16 07:48:41 UTC 
*** Bug 1722474 has been marked as a duplicate of this bug. ***
Comment 24 Tal Nisan 2019-08-19 14:10:04 UTC 
*** Bug 1735778 has been marked as a duplicate of this bug. ***
Comment 32 Robert Scheck 2020-02-17 20:51:51 UTC 
Are there any plans to get this issue finally solved? It is mandatory for Windows installations to be able to change a CD for a running VM in order to select the VirtIO drivers. Cross-filed ticket 02587480 with as escalation request at the Red Hat customer portal (and neither a deprecated NFS ISO domain nor a NFS data domain was usable as workaround).
Comment 37 Dmitry 2020-03-25 10:22:07 UTC 
Actual for version 4.3.8.2-1.el7
Comment 39 Sandro Bonazzola 2020-05-18 14:46:45 UTC 
Moved to 4.4.1 not being marked as blocker for 4.4.0 and we are preparing to GA.
Comment 46 Robert Scheck 2020-06-02 14:49:38 UTC 
Fedor, please explain why you assume that this works (and if so, in which version exactly). According to our knowledge, this issue still exists and has not been fixed in any RHV release - thus we hereby kindly request reopening (or an explanation).
Comment 47 Nir Soffer 2020-06-02 16:49:37 UTC 
Fedor please see comment 22. I don't know about required changes in
vdsm, so this is likely still broken.

If this was fixed please find the patch that fixed it add to the bug.
Comment 54 matthias.leopold 2020-06-22 14:38:33 UTC 
I can confirm this bug still exists in oVirt 4.3.10.
I thought I had to get rid of my dedicated ISO domain as a preparation for an upgrade to oVirt 4.4.x and ran into this bug.
Comment 55 Tal Nisan 2020-07-23 17:31:26 UTC 
(In reply to matthias.leopold from comment #54)
> I can confirm this bug still exists in oVirt 4.3.10.
> I thought I had to get rid of my dedicated ISO domain as a preparation for
> an upgrade to oVirt 4.4.x and ran into this bug.

ISO domains are still supported in 4.4 but are deprecated.
ISO files on data domains are fully supported both on file and block domains with the exception of this bug - changing a CD on a running VM in case the new CD resides on a block domain, will be fixed in 4.4.3
Comment 57 Nir Soffer 2020-08-26 11:10:15 UTC 
Vojta started to work on this.
Comment 58 Marina Kalinin 2020-11-03 21:36:32 UTC 
Vojta, are those 2 bugs related to this?
BZ#1847223 BZ#1835640.
Or this is a totally different issue with the same subject?
Comment 59 Vojtech Juranek 2020-11-04 07:38:08 UTC 
(In reply to Marina Kalinin from comment #58)
> Vojta, are those 2 bugs related to this?
> BZ#1847223 BZ#1835640.
> Or this is a totally different issue with the same subject?

both seems to be unrelated to this issue
Comment 67 Evelina Shames 2021-05-06 08:40:13 UTC 
Created attachment 1780156 [details]
Logs
Comment 76 errata-xmlrpc 2021-06-01 13:22:09 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: RHV Manager security update (ovirt-engine) [ovirt-4.4.6]), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2179
Comment 78 Red Hat Bugzilla 2023-09-18 00:13:48 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days
Note You need to log in before you can comment on or make changes to this bug.