Created attachment 1691587 [details] Added my engine.log and VM logs and VDSM log Description of problem: When importing a VM from a 4.2 cluster with a 4.2 Host attached to it without setting the custom emulated machine as Q35 to a 4.4 cluster with a 4.4 host attached to it cannot run after being imported. * The default emulated machine for 4.2 cluster is set and it is "pc-i440fx-rhel7.5" and the VM after being imported uses the default emulated machine of the 4.4 cluster which is "pc-q35-rhel8.2.0" and uses the default bios type of Q35 with "Q35 chipset with Legacy BIOS". The error in the events is: "VM no_comp_42 is down with error. Exit message: internal error: Bus 0 must be PCI for integrated PIIX3 USB or IDE controllers." Same error in the engine: "2020-05-24 18:13:43,864+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-85) [] EVENT_ID: VM_DOWN_ERROR(119), VM no_changes_4.2 is down with error. Exit message: internal error: Bus 0 must be PCI for integrated PIIX3 USB or IDE controllers." In order to run the VM the user has to change the bios type to "Legacy" and the custom emulated machine to i440fx. I think there should be an automatic change of the bios type and custom emulated machine in those cases or let the user to have clearer message. Version-Release number of selected component (if applicable): RHV 4.4.1-1 with RHEL 8.2 host for the 4.4 cluster and the engine and RHEL 7.6 Host for the 4.2 cluster. How reproducible: 100% Steps to Reproduce: 1. Create 2 data centers in RHV 4.4 one with compatibility level of 4.4 and one with 4.2 and for both data centers create a cluster with the same compatibility level of it's data center. Add hosts to both of those clusters (Which matches their cluster level) and create and attach data domains to both clusters. 2. Create a VM in the 4.2 cluster without choosing custom compatibility level. 3. Detach the data domain from the 4.2 cluster (First put it in maintenance). 4. Attach the data domain to the 4.4 cluster and activate it. 5. Import the VM. 6. Run the VM. Actual results: 1 + 2. Everything was created correctly. 3. The v4 data domain is detached from the 4.2 cluster. 4. The v4 data domain is attached to the 4.4 cluster. 5. The VM was imported. 6. The VM run has failed with the errors mentioned above. Expected results: In step 6, I expect the VM to run the first time without properties changes or a warning to change the custom emulated machine and BIOS type. Additional info:
Reviewed this issue. The failure [1] is due to an invalid Bus ad part of the cdrom device address which is different in compatibility version 4.2 for legacy Bios Types than it is in compatibility version 4.4 whose default should be upgraded to Q35. Reverting the Bios Type and machine type as suggested would not have been correct when their defaults should be Q35. Therefore to address this issue, during importing of the VM the addresses are reset so that the VM can run successfully by receiving updated addresses that are consistent with the version defaults. [1] Bus 0 must be PCI for integrated PIIX3 USB or IDE controllers."
Filed bz 1857116 to address the reported scenario at the documentation level. The fix for this bug addresses the more interesting case of templates/VMs that were exported from a lower cluster version and imported to 4.4 clusters. Tamir, please adjust this bug to address that scenario (export & import) and let's verify that it works.
*** Bug 1856278 has been marked as a duplicate of this bug. ***
I tried to verify the bug on RHV 4.4.1-11 (with engine and host running on RHEL 8 and another host with RHEL 7.6 for the RHV 4.2) In this test I tried Import/Export using OVA. I am still getting the error "Bus 0 must be PCI for integrated PIIX3 USB or IDE controllers" when trying to run an imported VM from 4.2. After getting the error I changed the BIOS type to "Legacy", I could successfully run the VM after that. This is the workaround, so I don't think the bug is a blocker. Steps to Reproduce: 1. Create 2 data centers in RHV 4.4 one with compatibility level of 4.4 and one with 4.2 and for both data centers create a cluster with the same compatibility level of it's data center. Add hosts to both of those clusters (Which matches their cluster level) and create and attach data domains to both clusters. 2. Create a VM in the 4.2 cluster without choosing custom compatibility level, add a disk. 3. Run the VM and wait for it to run. 4. Stop the VM. 5. Export the VM as OVA to a directory on the 4.2 host. 6. Copy the VM OVA file using scp to the 4.4 host. 7. Import the VM using the OVA file. 8. Run the VM. Actual results: 1 + 2. Everything was created correctly. 3. The 4.2 VM run successfully. 4. The 4.2 VM stopped running successfully. 5. The VM was export to an OVA file at the specified location. 6. The OVA file is on the 4.4 Host. 7. The VM was imported. 8. The VM run has failed with the errors mentioned above. I am attaching the logs.
The fix cleared only the address of managed devices but we need to clear the address of unmanaged devices as well
(In reply to Arik from comment #10) > The fix cleared only the address of managed devices but we need to clear the > address of unmanaged devices as well I think the problem is different. I see in the logs IDE controller, but it should be changed to SATA for Q35.
(In reply to Shmuel Melamud from comment #11) > I think the problem is different. I see in the logs IDE controller, but it > should be changed to SATA for Q35. Right, same thing - CompatibilityVersionUpdater iterates the managed devices and those controllers are unmanaged
(In reply to tamir from comment #8) > I tried to verify the bug on RHV 4.4.1-11 is it upgraded or fresh installed?
the RHV 4.4.1-11 was freshly installed before 4 days. I am going to upload or send in any way the OVA file that was export for the same instance of RHV 4.4.1-11, 4.2 cluster, 4.2 host.
After another review, this affects all import flows from <4.4 cluster levels
Verified: ovirt-engine-4.4.1.10-0.1.el8ev vdsm-4.40.22-1.el8ev.x86_64 libvirt-daemon-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64 qemu-kvm-4.2.0-29.module+el8.2.1+7297+a825794d.x86_64 Verification scenario: Steps to Reproduce: 1. Create 2 data centers in RHV 4.4 one with compatibility level of 4.4 and one with 4.2 and for both data centers create a cluster with the same compatibility level of it's data center. Add hosts to both of those clusters (Which matches their cluster level) and create and attach data domains to both clusters. 2. Create a VM in the 4.2 cluster without choosing custom compatibility level, add a disk. 3. Run the VM and wait for it to run. 4. Stop the VM. 5. Export the VM as OVA to a directory on the 4.2 host. 6. Copy the VM OVA file using scp to the 4.4 host. 7. Import the VM using the OVA file. 8. Run the VM. Verify VM is running successfully and VM HW devices changed to Q35 chipset. For example: -------- HW of VM running under cluster with 4.2 compatibility version: [nsimsolo@localhost ~]$ lspci 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01) 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 00:02.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 04) 00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device 00:04.0 Communication controller: Red Hat, Inc. Virtio console 00:05.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon 00:07.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG [nsimsolo@localhost ~]$ -------- HW of VM imported from OVA (created in 4.2 cluster) to cluster with 4.4 compatibility version: [nsimsolo@localhost ~]$ lspci 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller 00:01.0 VGA compatible controller: Red Hat, Inc. QXL paravirtual graphic card (rev 04) 00:02.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:02.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:02.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:02.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:02.4 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:02.5 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:02.6 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:02.7 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:03.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:03.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:03.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:03.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:03.4 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:03.5 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:03.6 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:03.7 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) 01:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01) 02:00.0 USB controller: Red Hat, Inc. QEMU XHCI Host Controller (rev 01) 03:00.0 Communication controller: Red Hat, Inc. Virtio console (rev 01) 04:00.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI (rev 01) 05:00.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon (rev 01) 06:00.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG (rev 01) [nsimsolo@localhost ~]$
I've tested yesterday with: ovirt-engine-4.4.1.10-1.el8.noarch vdsm-4.40.22-1.el8.x86_64 libvirt-daemon-6.0.0-17.el8.x86_64 qemu-kvm-4.2.0-19.el8.x86_64 The problem was still. Imported with OVA VM able to start, but not able to load from boot disk. Tested OVA impor from oVirt 4.3.10 to the oVirt 4.4.1.10.
(In reply to Pavel Zinchuk from comment #17) > I've tested yesterday with: > ovirt-engine-4.4.1.10-1.el8.noarch > vdsm-4.40.22-1.el8.x86_64 > libvirt-daemon-6.0.0-17.el8.x86_64 > qemu-kvm-4.2.0-19.el8.x86_64 > > > The problem was still. Imported with OVA VM able to start, but not able to > load from boot disk. > Tested OVA impor from oVirt 4.3.10 to the oVirt 4.4.1.10. That might be a different problem. Please file a different issue with the relevant (engine, vdsm, import) logs and ideally with the OVA you used.
(In reply to Arik from comment #18) > (In reply to Pavel Zinchuk from comment #17) > > I've tested yesterday with: > > ovirt-engine-4.4.1.10-1.el8.noarch > > vdsm-4.40.22-1.el8.x86_64 > > libvirt-daemon-6.0.0-17.el8.x86_64 > > qemu-kvm-4.2.0-19.el8.x86_64 > > > > > > The problem was still. Imported with OVA VM able to start, but not able to > > load from boot disk. > > Tested OVA impor from oVirt 4.3.10 to the oVirt 4.4.1.10. > > That might be a different problem. > Please file a different issue with the relevant (engine, vdsm, import) logs > and ideally with the OVA you used. Hi Arik I've previously created bug report and provided all required information. Check here https://bugzilla.redhat.com/show_bug.cgi?id=1856278 Bug report https://bugzilla.redhat.com/show_bug.cgi?id=1856278 was closed and marked by you as duplicate to this bug at 2020-07-16 08:19:14 Based on problem descriptions, we have the same problem. Problem, described in bug report https://bugzilla.redhat.com/show_bug.cgi?id=1856278 still not solved. With the latest version of oVirt 4.4.1 we still can't get a working VM that was exported from oVirt 4.3.10 (exported with OVA).
Hi Arik, Please check my previous comment. Need your thought, work on this bug will be continued in this bug report or will be reopened bug report https://bugzilla.redhat.com/show_bug.cgi?id=1856278 ?
So maybe I didn't understand you correctly - I thought you've meant that the VM starts but fails to boot from the disk. That seems like a different issue. bz 1856278 was opened before the fix for this was merged - if it's about the same steps and still reproduced please upload logs and ideally the OVA and we'll look at that.
Arik, I am provided all logs in this bug https://bugzilla.redhat.com/show_bug.cgi?id=1856278 If you think that this is another issue please reopen it and send please to QA. From my side, I have already provided all the information earlier. I don't understand why I still need to perform additional check if QA still has not confirmed the problem described here - https://bugzilla.redhat.com/show_bug.cgi?id=1856278 Issue still persist - VM that imported with OVA from oVirt 4.3 to 4.4 still not operable. In some cases VM don't start at all, sometimes - start but VM don't boot operation system because VM don't see boot disk. I draw attention that VM with Power On status this is not the same that VM with power On status with correctly started guest operation system. Therefore, I think, issue should be additionally investigated.
(In reply to Pavel Zinchuk from comment #22) > I don't understand why I still need to perform additional check if QA still > has not confirmed the problem described here - > https://bugzilla.redhat.com/show_bug.cgi?id=1856278 What's the difference between the reproduction steps in comment 16 to the ones you report in bz 1856278 ?
(In reply to Arik from comment #23) > (In reply to Pavel Zinchuk from comment #22) > > I don't understand why I still need to perform additional check if QA still > > has not confirmed the problem described here - > > https://bugzilla.redhat.com/show_bug.cgi?id=1856278 > > What's the difference between the reproduction steps in comment 16 to the > ones you report in bz 1856278 ? In my case, I have two separate oVirt engine setups: oVirt 4.3.10 and 4.4.1 These details I've provided in the bug report https://bugzilla.redhat.com/show_bug.cgi?id=1856278 After update oVirt 4.4.1.8 to the 4.4.1.10 behaviour has changed insignificantly. VM after import with OVA always start, but operation system not always start correctly. As I mentioned previously: >sometimes - start but VM don't boot operation system because VM don't see boot disk It will be much faster if QA back to my bug report https://bugzilla.redhat.com/show_bug.cgi?id=1856278 and will perform steps to reproduce. Let me remind that in the comment #15 Michal already wrote: >After another review, this affects all import flows from <4.4 cluster level But after this information was confirmed only import from oVirt 4.2 by QA, without oVirt 4.3.10. And test case was not the same, that I've reported. QA tested single oVirt Engine with two clusters (4.2 and 4.4.1). But I have separate oVirt Engine instances with different oVirt versions. Why I wrote about QA?! Because each oVirt minor update cause behaviour change, I can't each time perform full debug in a situation where my bug report was closed and when no one didn't try to recheck import from oVirt 4.3 in this bug report. Arik, please clarify, will be continued work on issue in this bug report or will be continued work under the issue in bug report https://bugzilla.redhat.com/show_bug.cgi?id=1856278 ? One of them should be reopened i guess.
If you can provide the OVA it failed on for you or logs that show it still fails with the fix, we'll be happy to check that. And that scenario in which the VM starts but doesn't boot is also interesting, we'll appreciate getting more information (logs, screenshot) on that as well.
This bugzilla is included in oVirt 4.4.1.1 Async release, published on July 13th 2020. Since the problem described in this bug report should be resolved in oVirt 4.4.1.1 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.