Bug 1846954 - Can't power on guest on rhv4.4 after v2v conversion because of unmatched machine type
Summary: Can't power on guest on rhv4.4 after v2v conversion because of unmatched mach...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Virt
Version: 4.4.0
Hardware: x86_64
OS: Unspecified
high
high
Target Milestone: ovirt-4.4.1
: ---
Assignee: Shmuel Melamud
QA Contact: Beni Pelled
URL:
Whiteboard: V2V_RHV_INT
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-15 09:13 UTC by mxie@redhat.com
Modified: 2020-10-20 05:55 UTC (History)
16 users (show)

Fixed In Version: rhv-4.4.1-10
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-05 06:25:08 UTC
oVirt Team: Virt
Embargoed:
pm-rhel: ovirt-4.4+


Attachments (Terms of Use)
machine-types-rhv4.4.png (1.67 MB, image/png)
2020-06-15 09:13 UTC, mxie@redhat.com
no flags Details
rhv4.4-engine.log (4.45 KB, text/plain)
2020-06-15 09:16 UTC, mxie@redhat.com
no flags Details
rhv4.4-vdsm.log (23.77 KB, text/plain)
2020-06-15 09:24 UTC, mxie@redhat.com
no flags Details
v2v-conversion-to-rhv4.4.log (2.12 MB, text/plain)
2020-06-15 11:59 UTC, mxie@redhat.com
no flags Details
rhv4.4.1.3-engine.log (4.82 KB, text/plain)
2020-06-16 08:14 UTC, mxie@redhat.com
no flags Details
v2v logs - import & engine (116.92 KB, application/x-xz)
2020-07-08 10:11 UTC, Beni Pelled
no flags Details
BIOS-type-new-cluster-by-default.png (100.13 KB, image/png)
2020-07-10 02:22 UTC, mxie@redhat.com
no flags Details
cluster-bios-type-change.png (95.82 KB, image/png)
2020-07-10 02:23 UTC, mxie@redhat.com
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 109937 0 master MERGED core: Chipset update in CompatibilityVersionUpdater 2020-12-20 12:16:09 UTC

Description mxie@redhat.com 2020-06-15 09:13:15 UTC
Created attachment 1697353 [details]
machine-types-rhv4.4.png

Description of problem:
Can't power on guest on rhv4.4 after v2v conversion because of unmatched machine type


Version-Release number of selected component (if applicable):
virt-v2v-1.42.0-4.module+el8.3.0+6798+ad6e66be.x86_64
libguestfs-1.42.0-2.module+el8.3.0+6798+ad6e66be.x86_64
rhv:4.4.1.2-0.10.el8ev


How reproducible:
100%

Steps to Reproduce:
1.Convert a guest from VMware to rhv4.4 by virt-v2v
# virt-v2v -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78   esx7.0-rhel8.2-x86_64 -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt
[   1.2] Opening the source -i libvirt -ic vpx://root.198.169/data/10.73.199.217/?no_verify=1 esx7.0-rhel8.2-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=B5:52:1F:B4:21:09:45:24:51:32:56:F6:63:6A:93:5D:54:08:2D:78
[   3.9] Creating an overlay to protect the source from being modified
[   5.0] Opening the overlay
[  29.1] Inspecting the overlay
[  53.8] Checking for sufficient free disk space in the guest
[  53.8] Estimating space required on target for each disk
[  53.8] Converting Red Hat Enterprise Linux 8.2 (Ootpa) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 208.4] Mapping filesystem data to avoid copying unused and blank areas
[ 209.7] Closing the overlay
[ 210.0] Assigning disks to buses
[ 210.0] Checking if the guest needs BIOS or UEFI to boot
[ 210.0] Initializing the target -o rhv-upload -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -os nfs_data
[ 211.9] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/tmp/v2vnbdkit.GS1pHd/nbdkit4.sock", "file.export": "/" } (raw)
    (100.00/100%)
[ 689.4] Creating output metadata
[ 692.5] Finishing off

2.Power on guest after v2v conversion, but guest can't run due to below error info which is got from engine.log
2020-06-15 14:52:59,681+08 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (EE-ManagedThreadFactory-engine-Thread-955) [9b6fb6fa-8fdf-4bc7-8c88-8002ab9fb71a] Running command: RunVmCommand internal: false. Entities affected :  ID: 801c718d-443b-42d9-a2e2-3ac8fdde0b31 Type: VMAction group RUN_VM with role type USER
2020-06-15 14:52:59,690+08 INFO  [org.ovirt.engine.core.bll.utils.EmulatedMachineUtils] (EE-ManagedThreadFactory-engine-Thread-955) [9b6fb6fa-8fdf-4bc7-8c88-8002ab9fb71a] Emulated machine 'pc-i440fx-rhel8.2.0' which is different than that of the cluster is set for 'esx7.0-rhel8.2-x86_64'(801c718d-443b-42d9-a2e2-3ac8fdde0b31)


Actual results:
As above description

Expected results:
Can power on guest on rhv4.4 after v2v conversion normally

Additional info:
Can't reproduce the problem on rhv4.3

Comment 1 mxie@redhat.com 2020-06-15 09:16:12 UTC
Created attachment 1697366 [details]
rhv4.4-engine.log

Comment 2 mxie@redhat.com 2020-06-15 09:24:39 UTC
Created attachment 1697370 [details]
rhv4.4-vdsm.log

Comment 3 mxie@redhat.com 2020-06-15 11:59:48 UTC
Created attachment 1697412 [details]
v2v-conversion-to-rhv4.4.log

Comment 5 mxie@redhat.com 2020-06-16 08:14:12 UTC
Hi Arik,

   Our rhv env has been updated to latest version 4.4.1.3-0.6.el8ev (repo version is rhv-4.4.1-4), I still can't power on guest on rhv4.4 after v2v conversion, but the error info of engine log is different with rhv-4.4.1.2-0.10.el8ev, details pls check "rhv4.4.1.3-engine.log"

Comment 6 mxie@redhat.com 2020-06-16 08:14:45 UTC
Created attachment 1697570 [details]
rhv4.4.1.3-engine.log

Comment 7 Arik 2020-06-18 09:16:51 UTC
Hi Ming,
Thanks, can you please try on a new cluster?

Comment 8 mxie@redhat.com 2020-06-22 03:46:51 UTC
Hi Arik

  1.Create a new cluster on rhv4.4, set BIOS type as 'cluster default', then convert a guest to rhv by v2v but guest can't power on due to below error.

2020-06-22 11:27:06,504+08 INFO  [org.ovirt.engine.core.bll.RunVmCommand] (EE-ManagedThreadFactory-engine-Thread-66179) [d22fbb3a-757c-49b4-b8ac-2e37a6c3e659] Running command: RunVmCommand internal: false. Entities affected :  ID: c03ee368-fd5a-410c-b15c-8eff17e28cee Type: VMAction group RUN_VM with role type USER
2020-06-22 11:27:06,533+08 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (EE-ManagedThreadFactory-engine-Thread-66179) [d22fbb3a-757c-49b4-b8ac-2e37a6c3e659] Candidate host '10.73.224.195' ('02744911-6da6-4b61-bcac-38ca1e854e1e') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'Emulated-Machine' (correlation id: d22fbb3a-757c-49b4-b8ac-2e37a6c3e659)
2020-06-22 11:27:06,534+08 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (EE-ManagedThreadFactory-engine-Thread-66179) [d22fbb3a-757c-49b4-b8ac-2e37a6c3e659] Can't find VDS to run the VM 'c03ee368-fd5a-410c-b15c-8eff17e28cee' on, so this VM will not be run.


  2. Found guest has BIOS type 'Legacy', then change the BIOS type of cluster to 'legacy', but guest still can't power on due to same error with step1:

2020-06-22 11:31:00,263+08 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (EE-ManagedThreadFactory-engine-Thread-66211) [9a83e8bc-4b82-4b31-8bfd-3d1408995460] Candidate host '10.73.224.195' ('02744911-6da6-4b61-bcac-38ca1e854e1e') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'Emulated-Machine' (correlation id: 9a83e8bc-4b82-4b31-8bfd-3d1408995460)
2020-06-22 11:31:00,263+08 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (EE-ManagedThreadFactory-engine-Thread-66211) [9a83e8bc-4b82-4b31-8bfd-3d1408995460] Can't find VDS to run the VM 'c03ee368-fd5a-410c-b15c-8eff17e28cee' on, so this VM will not be run.

  3.Change the BIOS type of cluster to 'Q35 Chipset with Legacy BIOS', then power on guest again, still can't power on guest due to same error with step1:

2020-06-22 11:37:30,252+08 INFO  [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (EE-ManagedThreadFactory-engine-Thread-66269) [6d0537d8-e99a-4dfd-bad2-de09c3b0f082] Candidate host '10.73.224.195' ('02744911-6da6-4b61-bcac-38ca1e854e1e') was filtered out by 'VAR__FILTERTYPE__INTERNAL' filter 'Emulated-Machine' (correlation id: 6d0537d8-e99a-4dfd-bad2-de09c3b0f082)
2020-06-22 11:37:30,252+08 ERROR [org.ovirt.engine.core.bll.RunVmCommand] (EE-ManagedThreadFactory-engine-Thread-66269) [6d0537d8-e99a-4dfd-bad2-de09c3b0f082] Can't find VDS to run the VM 'c03ee368-fd5a-410c-b15c-8eff17e28cee' on, so this VM will not be run.


  4.After debugging, found the workaround is guest can power on successfully if the BIOS type of guest is changed to 'Q35 Chipset with Legacy BIOS' whatever the BIOS type of cluster

Comment 9 Arik 2020-06-22 10:34:28 UTC
I see, so it seems that it's not only about existing clusters set with invalid i440fx machine types.
Shmuel, any idea?

Comment 10 Shmuel Melamud 2020-06-22 11:14:49 UTC
(In reply to Arik from comment #9)

If VM BIOS type after v2v wasn't 'Cluster default', changes in cluster BIOS type don't affect it. Also I guess that the VM may have customEmulatedMachine set.

Also, please switch on the debug output, so we will be able to see debug messages from EmulatedMachineFilterPolicyUnit containing the exact emulated machine type that is looked for.

Comment 11 mxie@redhat.com 2020-06-22 12:43:37 UTC
(In reply to Shmuel Melamud from comment #10)
> (In reply to Arik from comment #9)
> 
> If VM BIOS type after v2v wasn't 'Cluster default', changes in cluster BIOS
> type don't affect it. Also I guess that the VM may have
> customEmulatedMachine set.
> 
> Also, please switch on the debug output, so we will be able to see debug
> messages from EmulatedMachineFilterPolicyUnit containing the exact emulated
> machine type that is looked for.


Please check attachment file "v2v-conversion-to-rhv4.4.log", v2v will set <BiosType>0</BiosType> for guest during conversion

Comment 12 Arik 2020-06-24 09:55:21 UTC
(In reply to mxie from comment #11)
> Please check attachment file "v2v-conversion-to-rhv4.4.log", v2v will set
> <BiosType>0</BiosType> for guest during conversion

This setting is incorrect - the BiosType should not be set in the generated OVF in order to indicate oVirt that the bios type should be derived from the cluster (CLUSTER_DEFAULT). Currently, it means I440FX_SEA_BIOS would be used.

Tomas, could you please make the change in virt-v2v?

Comment 14 Pino Toscano 2020-06-24 10:15:38 UTC
(In reply to Arik from comment #12)
> (In reply to mxie from comment #11)
> > Please check attachment file "v2v-conversion-to-rhv4.4.log", v2v will set
> > <BiosType>0</BiosType> for guest during conversion
> 
> This setting is incorrect - the BiosType should not be set in the generated
> OVF in order to indicate oVirt that the bios type should be derived from the
> cluster (CLUSTER_DEFAULT). Currently, it means I440FX_SEA_BIOS would be used.

Well no, this is exactly the reason why we set it explicitly: we want to tell oVirt which BIOS version it must use for the guest.
Leaving oVirt use the cluster default means that e.g. a UEFI VM migrated to a BIOS cluster will not boot.

Comment 15 Arik 2020-06-24 10:24:11 UTC
(In reply to Pino Toscano from comment #14)
> Well no, this is exactly the reason why we set it explicitly: we want to
> tell oVirt which BIOS version it must use for the guest.
> Leaving oVirt use the cluster default means that e.g. a UEFI VM migrated to
> a BIOS cluster will not boot.

I see, so maybe bios types that are not interchangeable should be explicitly defined but in this case - is there a reason to limit the bios type to i440fx? we try to get rid of those...

Comment 16 Pino Toscano 2020-06-24 10:45:30 UTC
(In reply to Arik from comment #15)
> (In reply to Pino Toscano from comment #14)
> > Well no, this is exactly the reason why we set it explicitly: we want to
> > tell oVirt which BIOS version it must use for the guest.
> > Leaving oVirt use the cluster default means that e.g. a UEFI VM migrated to
> > a BIOS cluster will not boot.
> 
> I see, so maybe bios types that are not interchangeable should be explicitly
> defined but in this case - is there a reason to limit the bios type to
> i440fx?

Because virt-v2v does not support Q35 yet: bug 1581428.

Comment 17 Arik 2020-06-24 10:53:33 UTC
(In reply to Pino Toscano from comment #16)
> Because virt-v2v does not support Q35 yet: bug 1581428.

Ah I see, but if CD-ROMs are the only exception then it should be fine as we don't set CD-ROMs in RHV according to the output of virt-v2v (we don't have the image that the VM used in the remote environment anyway).
And when Ming just changed the bios type of the VM within RHV, it worked...

Comment 18 Pino Toscano 2020-06-24 12:52:19 UTC
(In reply to Arik from comment #17)
> (In reply to Pino Toscano from comment #16)
> > Because virt-v2v does not support Q35 yet: bug 1581428.
> 
> Ah I see, but if CD-ROMs are the only exception then it should be fine as we
> don't set CD-ROMs in RHV according to the output of virt-v2v (we don't have
> the image that the VM used in the remote environment anyway).
> And when Ming just changed the bios type of the VM within RHV, it worked...

Just because "it worked" it does not mean it will work in all the cases, nor that it is a stable strategy for production.

Also, what virt-v2v does and why is unrelated to the original bug report. virt-v2v has reasons to set the BIOS type, i.e. make sure that the VM will run as intended, no matter what the cluster configuration (which can be changed) is; hence, oVirt must work also in this situation.

Comment 19 Arik 2020-06-24 14:06:27 UTC
(In reply to Pino Toscano from comment #18)
> Just because "it worked" it does not mean it will work in all the cases, nor
> that it is a stable strategy for production.

Sure my point was that it's an evidence that the conversion of non-q35 VMs to q35 within oVirt can work for VMs that were converted using virt-v2v.

> Also, what virt-v2v does and why is unrelated to the original bug report.
> virt-v2v has reasons to set the BIOS type, i.e. make sure that the VM will
> run as intended, no matter what the cluster configuration (which can be
> changed) is; hence, oVirt must work also in this situation.

I see your point but it looks a bit different when we take into account oVirt as the target platform.
If I have a q35 cluster in oVirt, I expect imported VMs to this cluster to be converted to q35 - unless the VMs are explicitly set with a different bios type.
So it would be great if virt-v2v could convert them to q35 but considering that it doesn't, the question is whether those VMs can or should not be converted by oVirt to q35.
When virt-v2v sets the bios type to i440fx it's like saying "this VM is explicitly set to i440fx" although oVirt can do what it does for other non-q35 VMs. oVirt has no way to distinguish that from "this VM is set to i440fx but it can be converted to q35 if the target cluster is set to q35".

So I think the question here is really about whether the VM must be set to i440fx or can be converted by oVirt (considering that the latter works).

Comment 20 Pino Toscano 2020-06-24 14:25:06 UTC
(In reply to Arik from comment #19)
> (In reply to Pino Toscano from comment #18)
> > Just because "it worked" it does not mean it will work in all the cases, nor
> > that it is a stable strategy for production.
> 
> Sure my point was that it's an evidence that the conversion of non-q35 VMs
> to q35 within oVirt can work for VMs that were converted using virt-v2v.

Empirical experience with one VM is not an evidence.

> > Also, what virt-v2v does and why is unrelated to the original bug report.
> > virt-v2v has reasons to set the BIOS type, i.e. make sure that the VM will
> > run as intended, no matter what the cluster configuration (which can be
> > changed) is; hence, oVirt must work also in this situation.
> 
> I see your point but it looks a bit different when we take into account
> oVirt as the target platform.
> If I have a q35 cluster in oVirt, I expect imported VMs to this cluster to
> be converted to q35 - unless the VMs are explicitly set with a different
> bios type.

... which is the point here: for reasons external to oVirt, and even external and _unrelated_ to this bug, a VM is created in oVirt with explicit BiosType=0.

> So it would be great if virt-v2v could convert them to q35 but considering
> that it doesn't, the question is whether those VMs can or should not be
> converted by oVirt to q35.

oVirt ought to not mess up with the explicit configuration made by whoever creates VM, be it automatically (virt-v2v) or manual (by user).

> When virt-v2v sets the bios type to i440fx it's like saying "this VM is
> explicitly set to i440fx" although oVirt can do what it does for other
> non-q35 VMs. oVirt has no way to distinguish that from "this VM is set to
> i440fx but it can be converted to q35 if the target cluster is set to q35".

Exactly: oVirt ought to not mess up with VMs!

> So I think the question here is really about whether the VM must be set to
> i440fx or can be converted by oVirt (considering that the latter works).

So let me reiterate one more time:
- oVirt ought to respect the configuration of the VM, and no mess up with it
- whatever virt-v2v does when converting is _unrelated_ to this bug

Can we please get back to the original point of this bug? Thanks!

Comment 21 Arik 2020-06-24 21:33:03 UTC
Well, it may actually be better to solve this on the oVirt side by introducing an optional bios type property to the OVF.

Example 1 - the common case:
<bios_type>X</bios_type> or <bios_type custom="false">X</bios_type>
This is the case here - virt-v2v configured it for X=i440fx but the user didn't explicitly ask for that.
This may also be the case when the VM is set with CLUSTER_DEFAULT and exported from a cluster that is set to X in oVirt.
It should be read as "The VM is configured for X and it's ok to change it".
If X=i440fx and the target is a q35 cluster, the VM will be converted to q35.

Example 2 - backward compatibility:
No <bios_type> tag.
This may only be the case when the VM is exported from oVirt version prior to the introduction of q35 or generated by an old version of virt-v2v.
It should be handled as: <bios_type>i440fx</bios_type>

Example 3 - customized value:
<bios_type custom="true">X</bios_type>
This may only be the case when the VM is explicitly set to X (regardless of the cluster's settings).
This should be read as "The VM is configured for X and the bios type should not change".

Shmuel, this is similar in a way to your suggestion to also save the cluster's bios type in the OVF -
it enables us to know not only what bios type the VM is configured for but also whether it was set explicitly or not.
What do you think?

Comment 22 Pino Toscano 2020-06-25 06:26:42 UTC
(In reply to Arik from comment #21)
> Well, it may actually be better to solve this on the oVirt side by
> introducing an optional bios type property to the OVF.

Which is what <BiosType> already is?

> Example 1 - the common case:
> <bios_type>X</bios_type> or <bios_type custom="false">X</bios_type>
> This is the case here - virt-v2v configured it for X=i440fx but the user
> didn't explicitly ask for that.

Note that "the user" _is_ virt-v2v in this case.

I honestly do not understand why you keep considering virt-v2v as a different case from the user choice.
Whether virt-v2v creates a VM with a specified bios type, or the user sets one in the GUI, the situation is the same: oVirt must respect that choice, and not change it for any reason. It is exactly as any other property of the VM, be it CPU topology, OS, disks, etc.

> This may also be the case when the VM is set with CLUSTER_DEFAULT and
> exported from a cluster that is set to X in oVirt.
> It should be read as "The VM is configured for X and it's ok to change it".
> If X=i440fx and the target is a q35 cluster, the VM will be converted to q35.

Almost: if a VM has no bios type set, then oVirt ought to not set it when exporting the VM to anywhere; this means that whatever will import that VM again (be it another oVirt, for example) can simply choose the bios type according to their own logic. And this is fine, because whatever created the VM in oVirt in the first place did not set a bios type explicitly.

> Example 2 - backward compatibility:
> No <bios_type> tag.
> This may only be the case when the VM is exported from oVirt version prior
> to the introduction of q35 or generated by an old version of virt-v2v.
> It should be handled as: <bios_type>i440fx</bios_type>

Again, what virt-v2v did is irrelevant.

> Example 3 - customized value:
> <bios_type custom="true">X</bios_type>
> This may only be the case when the VM is explicitly set to X (regardless of
> the cluster's settings).
> This should be read as "The VM is configured for X and the bios type should
> not change".

From what I know, <BiosType> in the OVF already means "custom" in oVirt 4.4, so I'm not sure what adding a new attribute would do (other than forcing users of the REST API to adapt the OVF they create).

Otherwise yes, this is exactly how the situation ought to be.

Comment 23 Arik 2020-06-28 17:54:31 UTC
Thanks Pino, we've heard and appreciate your feedback.
That being said, we believe a better way to move forward with this from the oVirt/RHV point of view would be convert the VM to q35 when the target cluster is a q35 cluster, as we do with other i440fx VMs that are not explicitly set by users (not the tools they use) with a certain bios type.
Adding the 'custom' attribute can be added later on.

Comment 24 Pino Toscano 2020-06-29 06:35:25 UTC
(In reply to Arik from comment #23)
> That being said, we believe a better way to move forward with this from the
> oVirt/RHV point of view would be convert the VM to q35 when the target
> cluster is a q35 cluster, as we do with other i440fx VMs that are not
> explicitly set by users (not the tools they use) with a certain bios type.

Again, this is a dangerous idea, especially when the VM is created with a bios type different than q35.
Please do _not_ second-guess what was the user intention, and respect the settings explicitly specified.
Changing devices for the VM may require changes in the guest, which is something virt-v2v (or whoever prepares & uploads the VM to oVirt) must do.

Comment 25 Richard W.M. Jones 2020-06-29 09:32:26 UTC
I think I'm more confused by this bug than I started.  What exactly does
<BiosType> control?  Is it misnamed and really controlling the machine type
(i440fx vs q35)?  Or the firmware (BIOS vs UEFI)?  Or a bit of both?

Comment 26 Arik 2020-06-29 10:38:42 UTC
Hi Richard,
A combination of both.
The values that the BiosType within the OVF can be mapped to are:

I440FX_SEA_BIOS(1, ChipsetType.I440FX, false),
Q35_SEA_BIOS(2, ChipsetType.Q35, false),
Q35_OVMF(3, ChipsetType.Q35, true),
Q35_SECURE_BOOT(4, ChipsetType.Q35, true);

(note the index within the OVF is mapped to the first parameter minus 1)

So '0' within the OVF is mapped to I440FX_SEA_BIOS that declares the VM is set with BIOS and i440fx machine type.
Similarly, '1' within the OVF is mapped to Q35_SEA_BIOS that declares the VM is set with BIOS and q35 machine type.

But in oVirt/RHV, not only VMs but also clusters are set with the aforementioned value.
Clusters are set with one of the above (when there's an active host in the cluster) while VMs default to CLUSTER_DEFAULT.

Clusters from previous versions of oVirt/RHV are mapped to I440FX_SEA_BIOS.
However as we want to push users to q35, as from 4.4 q35 is the new default and we introduced code that converts VMs from previous versions (that are assumed to use BIOS and i440fx machine type) to BIOS and q35 machine type (Q35_SEA_BIOS). This logic is applied when upgrading existing clusters and when importing VMs that were exported from older versions.

So when importing VMs from VMware to a q35 cluster in oVirt 4.4, users would expect the VMs to be converted to q35.
It would be great if virt-v2v would do that for us, but in the meantime, as long as virt-v2v produces an ovirt-compatible i440fx VM, we can "compensate" that by applying the above mentioned logic to convert it from I440FX_SEA_BIOS to Q35_SEA_BIOS when the target cluster is q35.

Comment 27 Michal Skrivanek 2020-06-29 13:07:01 UTC
regardless if virt-v2v provides q35 output we have already supported "update" of i440fx VMs to Q35 within oVirt. The user expectation is that if they bring a VM into a new cluster today it's going to be a q35 VM.

virt-v2v produces currently OVF with <BiosType>0</BiosType> which in 4.4 would mean q35 by default. That's the mismatch here.

Pino, Rich - for the use case of command line virt-v2v if you really want not to support q35 I think you should inspect the target cluster and if it is not I440FX_SEA_BIOS refuse to convert, or explicitly generate BiosType 1.

For integrated v2v I wonder what do we do today? Tomas? The expectation here would be that the imported VM is a q35 VM (or rather that it matches the cluster setting). If virt-v2v doesn't support that it could be done as "import as i440fx and then switch to q35" internally like we do for existing VMs.
Same for IMS.

Comment 28 Arik 2020-06-29 13:15:09 UTC
(In reply to Michal Skrivanek from comment #27)
> virt-v2v produces currently OVF with <BiosType>0</BiosType> which in 4.4
> would mean q35 by default. That's the mismatch here.

Michal, note that '0' in the OVF doesn't mean CLUSTER_DEFAULT but I440FX_SEA_BIOS -
which is what we expect it to be, the configuration of the VM.
We can add a 'custom' (or similar) field to the OVF to know if it is derived from the cluster or a custom settings,
but that can be done separately.

Comment 29 Tomáš Golembiovský 2020-06-29 14:32:23 UTC
(In reply to Michal Skrivanek from comment #27)
> regardless if virt-v2v provides q35 output we have already supported
> "update" of i440fx VMs to Q35 within oVirt. The user expectation is that if
> they bring a VM into a new cluster today it's going to be a q35 VM.
> 
> virt-v2v produces currently OVF with <BiosType>0</BiosType> which in 4.4
> would mean q35 by default. That's the mismatch here.

For OVF the values are one less, so 0 is not cluster default but i440fx. When the <BiosType> is missing then it is treated as cluster default.


> 
> Pino, Rich - for the use case of command line virt-v2v if you really want
> not to support q35 I think you should inspect the target cluster and if it
> is not I440FX_SEA_BIOS refuse to convert, or explicitly generate BiosType 1.
> 
> For integrated v2v I wonder what do we do today? Tomas?

It should match whatever virt-v2v stores in <BiosType> too. I don't think we are using the internal conversion mechanism.

> The expectation here
> would be that the imported VM is a q35 VM (or rather that it matches the
> cluster setting). If virt-v2v doesn't support that it could be done as
> "import as i440fx and then switch to q35" internally like we do for existing
> VMs.
> Same for IMS.

Comment 30 Michal Skrivanek 2020-06-30 07:56:53 UTC
yes, sorry for the mix up.

Still, until virt-v2v support q35 we have to have some compromise. oVirt moved to q35 by default and so it's IMO bettter to "convert" the output of virt-v2v to a q35 VM if the cluster is configured that way, same as we would do for 4.3->4.4 upgrade of existing VMs.
There's no need to modify anything if the target cluster matches virt-v2v, i.e. when it's i440fx+seabios

Comment 31 Pino Toscano 2020-06-30 08:17:28 UTC
(In reply to Michal Skrivanek from comment #30)
> Still, until virt-v2v support q35 we have to have some compromise.

"compromised" is different than "breaking"

> oVirt
> moved to q35 by default and so it's IMO bettter to "convert" the output of
> virt-v2v to a q35 VM if the cluster is configured that way, same as we would
> do for 4.3->4.4 upgrade of existing VMs.

Look Michal and Arik: I understand the desire to move away from i440fx to q35 as much as possible, and I agree with the idea. However, it must be done only when it makes sense and it is doable.
oVirt does nothing about the guest other than what it is configured (it does not do any kind of inspection), so arbitrary hardware changes may break the guest.
Linux guests may have less problems, OTOH Windows is way more picky and fragile. virt-v2v does not do it yet because... it is not easy, otherwise we'd had done it 2 years ago.

Also, the oVirt REST API (i.e. what virt-v2v uses) is a generic way to do non-interactive changes in oVirt, including creating new guests.
Let's assume that somebody prepares a VM outside oVirt, and then they want to import it using the REST API:
- they upload the disk(s)
- they prepare a OVF describing the VM
- they create the VM
At this point, the user expects that the VM is in oVirt _exactly_ as it was configured, not with some parts of the hardware changed by oVirt using some internal logic it is not possible to override. This is a very bad (and dangerous) user experience.

(In reply to Michal Skrivanek from comment #27)
> Pino, Rich - for the use case of command line virt-v2v if you really want
> not to support q35 I think you should inspect the target cluster and if it
> is not I440FX_SEA_BIOS refuse to convert, or explicitly generate BiosType 1.

Sigh, this has nothing to do with "not wanting to support q35", really. This is exactly what bug 1581428 is about.

Comment 33 Michal Skrivanek 2020-06-30 08:41:51 UTC
(In reply to Pino Toscano from comment #31)
> Look Michal and Arik: I understand the desire to move away from i440fx to
> q35 as much as possible, and I agree with the idea. However, it must be done
> only when it makes sense and it is doable.
> oVirt does nothing about the guest other than what it is configured (it does
> not do any kind of inspection), so arbitrary hardware changes may break the
> guest.

that's correct. that's the reason for compatiblity levels where we guarantee no changes, and "update" process of changing the cluster level which may(and often does) make arbitrary hardware changes

> Linux guests may have less problems, OTOH Windows is way more picky and
> fragile. virt-v2v does not do it yet because... it is not easy, otherwise
> we'd had done it 2 years ago.

that's fair, and not really in conflict with what i'm saying. I only say that oVirt itself supports both, and it supports the transition from i440fx to q35 and vice versa with a very simplistic best effort process. This has nothing to do with virt-v2v and exists independently.
 
> Also, the oVirt REST API (i.e. what virt-v2v uses) is a generic way to do
> non-interactive changes in oVirt, including creating new guests.
> Let's assume that somebody prepares a VM outside oVirt, and then they want
> to import it using the REST API:
> - they upload the disk(s)
> - they prepare a OVF describing the VM
> - they create the VM
> At this point, the user expects that the VM is in oVirt _exactly_ as it was
> configured, not with some parts of the hardware changed by oVirt using some
> internal logic it is not possible to override. This is a very bad (and
> dangerous) user experience.

right. We do respect that decision and in this case we would be violating that. I'm still not sure about this particular case though, it still may make sense to do that. I'm not fond of virt-v2v creating tons of VMs using a per-VM override which is intended for exceptional use.

> (In reply to Michal Skrivanek from comment #27)
> > Pino, Rich - for the use case of command line virt-v2v if you really want
> > not to support q35 I think you should inspect the target cluster and if it
> > is not I440FX_SEA_BIOS refuse to convert, or explicitly generate BiosType 1.
> 
> Sigh, this has nothing to do with "not wanting to support q35", really. This
> is exactly what bug 1581428 is about.

sorry, that was before i got corrected that the enum is (sadly) different from internal use. You are using "0" which does mean i440fx+seabios so from this perspective this is perfectly correct.

Comment 34 Pino Toscano 2020-06-30 08:53:23 UTC
(In reply to Michal Skrivanek from comment #33)
> > Linux guests may have less problems, OTOH Windows is way more picky and
> > fragile. virt-v2v does not do it yet because... it is not easy, otherwise
> > we'd had done it 2 years ago.
> 
> that's fair, and not really in conflict with what i'm saying. I only say
> that oVirt itself supports both, and it supports the transition from i440fx
> to q35 and vice versa with a very simplistic best effort process.

From my POV, a "best effort" should always bring some kind of improvement, even minimal, but never regress and break things.
If oVirt does not have the way to know that the logic used for the "best effort" is safe, then it ought to rather not apply it.

> > Also, the oVirt REST API (i.e. what virt-v2v uses) is a generic way to do
> > non-interactive changes in oVirt, including creating new guests.
> > Let's assume that somebody prepares a VM outside oVirt, and then they want
> > to import it using the REST API:
> > - they upload the disk(s)
> > - they prepare a OVF describing the VM
> > - they create the VM
> > At this point, the user expects that the VM is in oVirt _exactly_ as it was
> > configured, not with some parts of the hardware changed by oVirt using some
> > internal logic it is not possible to override. This is a very bad (and
> > dangerous) user experience.
> 
> right. We do respect that decision and in this case we would be violating
> that.

"we respect the user decision yet we arbitratly break it"
this reeks incoherency from miles...

> I'm still not sure about this particular case though, it still may
> make sense to do that. I'm not fond of virt-v2v creating tons of VMs using a
> per-VM override which is intended for exceptional use.

Anything/anyone writing a VM definition as OVF, or even changing the settings via the UI, is explicitly signalling: "I don't want to use the default because it may change, and I rather need this". The reason for the explicit choice are outside oVirt's realm, and this is why oVirt ought to not mess up with them.
virt-v2v already sets lots of properties of the VM (topology, RAM, disks, network interfaces), or even create wanted devices (memory balooon, RNG device), to make sure they are used as requested, and not depend on the oVirt configuration. The choice of the bios type follows the same logic: we know we cannot convert the VM so it will run on a q35 machine type, so we explicitly request i440fx.

Comment 38 Sandro Bonazzola 2020-07-02 13:23:50 UTC
Missed final 4.4.1 build, can you re-target to 4.4.2?

Comment 40 Beni Pelled 2020-07-08 10:11:05 UTC
Verified with:
- ovirt-engine-4.4.1.7-0.3.el8ev.noarch
- vdsm-4.40.22-1.el8ev.x86_64
- libvirt-6.0.0-25.module+el8.2.1+7154+47ffd890.x86_64

Verification steps:
1. Import a VM (RHEL 8.2) from VMware (6.5)

Result:
- VM imported successfully and works as expected (logs attached)

Comment 41 Beni Pelled 2020-07-08 10:11:33 UTC
Created attachment 1700268 [details]
v2v logs - import & engine

Comment 42 mxie@redhat.com 2020-07-08 15:38:00 UTC
Test the bug with below builds:
rhv:4.4.1.7-0.3.el8ev
vdsm-4.40.22-1.el8ev.x86_64

Steps:
1.Update rhv and rhv-node to 4.4.1.7-0.3 and create a new cluster which has BIOS Type 'cluster default' on rhv env

2.Convert a non-uefi guest from VMware to rhv4.4 by virt-v2v
#virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt esx6.7-win2019-x86_64

3.Check the non-efi guest on rhv, can power on guest successfully and BIOS Type of non-uefi guest is 'Cluster default'


4.Convert a uefi guest from VMware to rhv4.4 by virt-v2v
#virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -ip /home/passwd  -o rhv-upload -oo rhv-direct -oc https://dell-per740-22.lab.eng.pek2.redhat.com/ovirt-engine/api -op /home/rhvpasswd -of raw -os nfs_data -b ovirtmgmt esx6.7-win10-x86_64-efi

5.Check the efi guest after v2v conversion, can power on guest successfully but BIOS Type of uefi guest is still 'Cluster default', already filed bug1855001 to track this problem

Comment 43 Michal Skrivanek 2020-07-09 17:58:25 UTC
The cluster shouldn't have "Cluster default", that sounds like old bug from earlier version(we didn't fix upgrade cleanly because those versions were not released). If you create a new cluster now and once you add a host tto it there should no longer be Cluster Default BIOS type in Cluster's settings. Can you please try that? If it still shows Cluster Defaul, can you please explicitly select Q35 Legacy Bios? And try convert then.
This shouldn't happen for clean installs anymore, the cluster should always be Q35 Legacy Bios once there's a host added to it.

Comment 44 mxie@redhat.com 2020-07-10 02:22:06 UTC
(In reply to Michal Skrivanek from comment #43)
> The cluster shouldn't have "Cluster default", that sounds like old bug from
> earlier version(we didn't fix upgrade cleanly because those versions were
> not released). If you create a new cluster now and once you add a host tto
> it there should no longer be Cluster Default BIOS type in Cluster's
> settings. Can you please try that? If it still shows Cluster Defaul, can you
> please explicitly select Q35 Legacy Bios? And try convert then.
> This shouldn't happen for clean installs anymore, the cluster should always
> be Q35 Legacy Bios once there's a host added to it.

Hi Michal, pls check screenshot "BIOS-type-in-new-cluster.png", a new cluster will has BIOS type "Cluster Default" by default. and you're right, BIOS type of cluster will be changed from 'Cluster Default' to 'Q35 Chipset with Legacy BIOS' if move the host to the new cluster, pls check 'cluster-bios-type-change.png'. I'm sure the result of comment42 is correct (I just didn't notice BIOS type of the cluster will change, BIOS type of cluster is already 'Q35 Chipset with Legacy BIOS' when I used v2v to convert guests), all guests will have BIOS type 'Cluster default' after v2v conversion no matter the guest is UEFI guest or not

Comment 45 mxie@redhat.com 2020-07-10 02:22:48 UTC
Created attachment 1700508 [details]
BIOS-type-new-cluster-by-default.png

Comment 46 mxie@redhat.com 2020-07-10 02:23:17 UTC
Created attachment 1700509 [details]
cluster-bios-type-change.png

Comment 47 Arik 2020-07-15 16:05:18 UTC
(In reply to mxie from comment #44)
> all guests will have BIOS type 'Cluster default' after v2v
> conversion no matter the guest is UEFI guest or not

This part should be handled by bz 1726558

Comment 48 Sandro Bonazzola 2020-08-05 06:25:08 UTC
This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.