Bug 2141534

Summary: Can't launch uefi volume and image as instance on OSP17
Product: Red Hat OpenStack Reporter: mxie <mxie>
Component: openstack-novaAssignee: OSP DFG:Compute <osp-dfg-compute>
Status: CLOSED DUPLICATE QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 17.0 (Wallaby)CC: alifshit, chhu, dasmith, eglynn, hongzliu, jhakimra, jparker, juzhou, kchamart, lersek, mzhan, rjones, sbauza, sgordon, tyan, tzheng, vromanso, vwu, xiaodwan
Target Milestone: ---Keywords: Regression, TestBlocker
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: V2V_OSP_INT
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-02-07 00:40:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description mxie@redhat.com 2022-11-10 06:42:37 UTC
Description of problem:
Can't launch uefi volume and image as instance on OSP17

Version-Release number of selected component (if applicable):
OSP: 17.0_20220908.1

How reproducible:
100%

Steps to Reproduce:
Scenario1:
1.1 Prepare a uefi image on OSP17, the image has hw_firmware_type='uefi' and hw_machine_type='q35'
$ openstack image show RHEL-9.2.0-20221108.4-x86_64-ovmf.qcow2
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| checksum         | d3a448e0f7a1edb85723e2e728a7e9c6                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| container_format | bare                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| created_at       | 2022-11-10T04:55:48Z                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| disk_format      | qcow2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| file             | /v2/images/b262d82f-2085-4e03-961c-c126898fb9fe/file                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| id               | b262d82f-2085-4e03-961c-c126898fb9fe                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| min_disk         | 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| min_ram          | 0                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| name             | RHEL-9.2.0-20221108.4-x86_64-ovmf.qcow2                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| owner            | 456e4d27fb4948dcbdc4f904c486751c                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| properties       | direct_url='swift+config://ref1/glance/b262d82f-2085-4e03-961c-c126898fb9fe', hw_firmware_type='uefi', hw_machine_type='q35', os_hash_algo='sha512', os_hash_value='83d8ed57cfe1a01d265ce09a1fb955817a7f54588278b15aa2f10939c597a0dba648b322d81d8af73416545341c4021e5a568970f447308168c44286aa6eee14', os_hidden='False', owner_specified.openstack.md5='', owner_specified.openstack.object='images/RHEL-9.2.0-20221108.4-x86_64-ovmf.qcow2', owner_specified.openstack.sha256='', stores='default_backend' |
| protected        | False                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| schema           | /v2/schemas/image                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| size             | 1013186560                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| status           | active                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| tags             |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| updated_at       | 2022-11-10T06:21:54Z                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| virtual_size     | 10737418240                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| visibility       | shared                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

1.2 Launch the image as instance with openstack command
$   openstack server create --flavor small --image RHEL-9.2.0-20221108.4-x86_64-ovmf.qcow2 --nic net-id=e083ff7a-f61b-4ddd-b169-abf153fa51f1 launch-uefi-image
+-------------------------------------+------------------------------------------------------------------------------------+
| Field                               | Value                                                                              |
+-------------------------------------+------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                                             |
| OS-EXT-AZ:availability_zone         |                                                                                    |
| OS-EXT-SRV-ATTR:host                | None                                                                               |
| OS-EXT-SRV-ATTR:hostname            | launch-uefi-image                                                                  |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                                                               |
| OS-EXT-SRV-ATTR:instance_name       |                                                                                    |
| OS-EXT-SRV-ATTR:kernel_id           |                                                                                    |
| OS-EXT-SRV-ATTR:launch_index        | 0                                                                                  |
| OS-EXT-SRV-ATTR:ramdisk_id          |                                                                                    |
| OS-EXT-SRV-ATTR:reservation_id      | r-jtwo8xrk                                                                         |
| OS-EXT-SRV-ATTR:root_device_name    | None                                                                               |
| OS-EXT-SRV-ATTR:user_data           | None                                                                               |
| OS-EXT-STS:power_state              | NOSTATE                                                                            |
| OS-EXT-STS:task_state               | scheduling                                                                         |
| OS-EXT-STS:vm_state                 | building                                                                           |
| OS-SRV-USG:launched_at              | None                                                                               |
| OS-SRV-USG:terminated_at            | None                                                                               |
| accessIPv4                          |                                                                                    |
| accessIPv6                          |                                                                                    |
| addresses                           |                                                                                    |
| adminPass                           | p9cYcDgAUPzD                                                                       |
| config_drive                        |                                                                                    |
| created                             | 2022-11-10T06:36:39Z                                                               |
| description                         | None                                                                               |
| flavor                              | disk='20', ephemeral='0', , original_name='small', ram='2048', swap='0', vcpus='2' |
| hostId                              |                                                                                    |
| host_status                         |                                                                                    |
| id                                  | 09178b9f-3ae2-4c13-953d-926404853ae7                                               |
| image                               | RHEL-9.2.0-20221108.4-x86_64-ovmf.qcow2 (b262d82f-2085-4e03-961c-c126898fb9fe)     |
| key_name                            | None                                                                               |
| locked                              | False                                                                              |
| locked_reason                       | None                                                                               |
| name                                | launch-uefi-image                                                                  |
| progress                            | 0                                                                                  |
| project_id                          | 456e4d27fb4948dcbdc4f904c486751c                                                   |
| properties                          |                                                                                    |
| security_groups                     | name='default'                                                                     |
| server_groups                       | []                                                                                 |
| status                              | BUILD                                                                              |
| tags                                |                                                                                    |
| trusted_image_certificates          | None                                                                               |
| updated                             | 2022-11-10T06:36:39Z                                                               |
| user_id                             | b1ae29d10dcc4546a341dba556aafff4                                                   |
| volumes_attached                    |                                                                                    |
+-------------------------------------+------------------------------------------------------------------------------------+

1.3  Found status of instance shows error after launching
$ openstack server list
+--------------------------------------+---------------------+--------+---------------------------------------------+-----------------------------------------+--------+
| ID                                   | Name                | Status | Networks                                    | Image                                   | Flavor |
+--------------------------------------+---------------------+--------+---------------------------------------------+-----------------------------------------+--------+
| 09178b9f-3ae2-4c13-953d-926404853ae7 | launch-uefi-image   | ERROR  |                                             | RHEL-9.2.0-20221108.4-x86_64-ovmf.qcow2 | small  |
....


Scenario2:
2.1 Convert a uefi guest from ova file to OSP17 by v2v
# virt-v2v -i ova esx7.0-rhel9.1-x86_64-uefi -o openstack -oo server-id=v2v-appliance
[   0.0] Setting up the source: -i ova esx7.0-rhel9.1-x86_64-uefi
virt-v2v: warning: making OVA directory public readable to work around 
libvirt bug https://bugzilla.redhat.com/1045069
[  10.0] Opening the source
[  17.3] Inspecting the source
[  23.2] Checking for sufficient free disk space in the guest
[  23.2] Converting Red Hat Enterprise Linux 9.1 Beta (Plow) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 143.3] Mapping filesystem data to avoid copying unused and blank areas
[ 144.5] Closing the overlay
[ 144.8] Assigning disks to buses
[ 144.8] Checking if the guest needs BIOS or UEFI to boot
virt-v2v: This guest requires UEFI on the target to boot.
[ 144.8] Setting up the destination: -o openstack -oo server-id=v2v-appliance
[ 166.5] Copying disk 1/1
█ 100% [****************************************]
[ 428.4] Creating output metadata
[ 432.3] Finishing off

2.2 Lanuch the volume as instance after v2v conversion on OSP17 undercloud server or OSP17 GUI dashboard
(overcloud) [stack@dell-per740-06 ~]$ openstack server create --flavor small --volume esx7.0-rhel9.1-x86_64-uefi-sda --availability-zone nova  esx7-0-rhel9-1-x86_64-uefi --nic net-id=e083ff7a-f61b-4ddd-b169-abf153fa51f1
+-------------------------------------+------------------------------------------------------------------------------------+
| Field                               | Value                                                                              |
+-------------------------------------+------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                                             |
| OS-EXT-AZ:availability_zone         | nova                                                                               |
| OS-EXT-SRV-ATTR:host                | None                                                                               |
| OS-EXT-SRV-ATTR:hostname            | esx7-0-rhel9-1-x86-64-uefi                                                         |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                                                               |
| OS-EXT-SRV-ATTR:instance_name       |                                                                                    |
| OS-EXT-SRV-ATTR:kernel_id           |                                                                                    |
| OS-EXT-SRV-ATTR:launch_index        | 0                                                                                  |
| OS-EXT-SRV-ATTR:ramdisk_id          |                                                                                    |
| OS-EXT-SRV-ATTR:reservation_id      | r-2q5ozrqv                                                                         |
| OS-EXT-SRV-ATTR:root_device_name    | None                                                                               |
| OS-EXT-SRV-ATTR:user_data           | None                                                                               |
| OS-EXT-STS:power_state              | NOSTATE                                                                            |
| OS-EXT-STS:task_state               | scheduling                                                                         |
| OS-EXT-STS:vm_state                 | building                                                                           |
| OS-SRV-USG:launched_at              | None                                                                               |
| OS-SRV-USG:terminated_at            | None                                                                               |
| accessIPv4                          |                                                                                    |
| accessIPv6                          |                                                                                    |
| addresses                           |                                                                                    |
| adminPass                           | jH6bmLZvyM46                                                                       |
| config_drive                        |                                                                                    |
| created                             | 2022-11-10T03:37:52Z                                                               |
| description                         | None                                                                               |
| flavor                              | disk='20', ephemeral='0', , original_name='small', ram='2048', swap='0', vcpus='2' |
| hostId                              |                                                                                    |
| host_status                         |                                                                                    |
| id                                  | a82a7ec5-4871-4f94-95c0-18fec86f7820                                               |
| image                               | N/A (booted from volume)                                                           |
| key_name                            | None                                                                               |
| locked                              | False                                                                              |
| locked_reason                       | None                                                                               |
| name                                | esx7-0-rhel9-1-x86_64-uefi                                                         |
| progress                            | 0                                                                                  |
| project_id                          | 456e4d27fb4948dcbdc4f904c486751c                                                   |
| properties                          |                                                                                    |
| security_groups                     | name='default'                                                                     |
| server_groups                       | []                                                                                 |
| status                              | BUILD                                                                              |
| tags                                |                                                                                    |
| trusted_image_certificates          | None                                                                               |
| updated                             | 2022-11-10T03:37:52Z                                                               |
| user_id                             | b1ae29d10dcc4546a341dba556aafff4                                                   |
| volumes_attached                    |                                                                                    |
+-------------------------------------+------------------------------------------------------------------------------------+

2.3 Found status of instance shows error after launching

(overcloud) [stack@dell-per740-06 ~]$ openstack server list
+--------------------------------------+----------------------------+--------+---------------------------------------------+------------------------------+--------+
| ID                                   | Name                       | Status | Networks                                    | Image                        | Flavor |
+--------------------------------------+----------------------------+--------+---------------------------------------------+------------------------------+--------+
| a82a7ec5-4871-4f94-95c0-18fec86f7820 | esx7-0-rhel9-1-x86_64-uefi | ERROR  |                                             | N/A (booted from volume)     | small  |
.....



Actual results:
As above description

Expected results:
Can launch uefi volume as instance on OSP17

Additional info:
1.Can launch uefi volumes and images as instances on OSP16.2 environment successfully
2.Can launch non-uefi volume and images as instances on OSP17 environment successfully

Comment 2 Richard W.M. Jones 2022-11-15 09:59:37 UTC
Do we believe this is a regression in nova itself which has
nothing to do with virt-v2v, but affects virt-v2v?

Comment 3 Kashyap Chamarthy 2022-11-16 15:43:04 UTC
Hi, Rich.  We needed two fixes here; one in OSP, one in RHEL.  Both are available.

In short, these are the versions to be used:

  - openstack-tripleo-common-15.4.1-0.20221114190240.51f6577.el9osttrunk
  - edk2-20220126gitbb1bba3d77-3.el9_0.1

Context
-------

The firmware descriptor files, "50-edk2-ovmf-amdsev.json" and "50-edk2-ovmf-cc.json"  shipped as part of "edk2-ovmf" RHEL package hard-codes "pc-q35-rhel8.5.0"; and didn't have 9.0 machine type.  That is now fixed as part this[1] and this 9.0.0.z clone[2] in RHEL.  And this patch[3] one in the installer tool, TripleO.  I'm not 100% sure if [3] is required.  I'll ask OSP QE, James.  He verified a similar bug[4] recently.



[1] https://bugzilla.redhat.com/show_bug.cgi?id=2090752 -- Add RHEL 8.5, 8,6 and 9.x machine types to firmware descriptor files 50-edk2-ovmf-{amdsev,cc}.json
[2] [rhel-9.0.0.z] clone pf [1]: https://bugzilla.redhat.com/show_bug.cgi?id=2109988 
[3] https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-common/+/426556/ -- Update ovmf-amdsev from ovmf-cc to boot UEFI VMs
[4] https://bugzilla.redhat.com/show_bug.cgi?id=2109644#c7 -- [RHOS 17] UEFI guest deployment fails due to UEFINotSupported exception.

Comment 4 Kashyap Chamarthy 2022-11-16 16:14:36 UTC
(In reply to Kashyap Chamarthy from comment #3)
> Hi, Rich.  We needed two fixes here; one in OSP, one in RHEL.  Both are
> available.
> 
> In short, these are the versions to be used:
> 
>   - openstack-tripleo-common-15.4.1-0.20221114190240.51f6577.el9osttrunk
>   - edk2-20220126gitbb1bba3d77-3.el9_0.1
> 
> Context
> -------
> 
> The firmware descriptor files, "50-edk2-ovmf-amdsev.json" and
> "50-edk2-ovmf-cc.json"  shipped as part of "edk2-ovmf" RHEL package
> hard-codes "pc-q35-rhel8.5.0"; and didn't have 9.0 machine type.  That is
> now fixed as part this[1] and this 9.0.0.z clone[2] in RHEL.  And this
> patch[3] one in the installer tool, TripleO.  I'm not 100% sure if [3] is
> required.  I'll ask OSP QE, James.  He verified a similar bug[4] recently.

James: if you wanted add anything extra here, please go ahead.

> 
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=2090752 -- Add RHEL 8.5, 8,6
> and 9.x machine types to firmware descriptor files
> 50-edk2-ovmf-{amdsev,cc}.json
> [2] [rhel-9.0.0.z] clone pf [1]:
> https://bugzilla.redhat.com/show_bug.cgi?id=2109988 
> [3]
> https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-common/+/
> 426556/ -- Update ovmf-amdsev from ovmf-cc to boot UEFI VMs
> [4] https://bugzilla.redhat.com/show_bug.cgi?id=2109644#c7 -- [RHOS 17] UEFI
> guest deployment fails due to UEFINotSupported exception.

Comment 5 James Parker 2022-11-16 16:16:47 UTC
I believe that covers it Kashyap, looking at the failure logs for this BZ compared to [1], this looks to be a duplicate of [2].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2109644#c3
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2109644

Comment 6 mxie@redhat.com 2022-11-17 06:20:45 UTC
(In reply to Kashyap Chamarthy from comment #3)
> Hi, Rich.  We needed two fixes here; one in OSP, one in RHEL.  Both are
> available.
> 
> In short, these are the versions to be used:
> 
>   - openstack-tripleo-common-15.4.1-0.20221114190240.51f6577.el9osttrunk
>   - edk2-20220126gitbb1bba3d77-3.el9_0.1
> 

Sorry, I don't understand what I need to do, by the way, the versions of the above packages in my OSP17.0 are:

openstack-tripleo-common-15.4.1-0.20220705010409.51f6577.el9ost.noarch
edk2-ovmf-20220126gitbb1bba3d77-3.el9_0.1.noarch

Comment 7 Kashyap Chamarthy 2022-11-30 14:34:32 UTC
(In reply to mxie from comment #6)
> (In reply to Kashyap Chamarthy from comment #3)
> > Hi, Rich.  We needed two fixes here; one in OSP, one in RHEL.  Both are
> > available.
> > 
> > In short, these are the versions to be used:
> > 
> >   - openstack-tripleo-common-15.4.1-0.20221114190240.51f6577.el9osttrunk
> >   - edk2-20220126gitbb1bba3d77-3.el9_0.1
> > 
> 
> Sorry, I don't understand what I need to do, by the way, the versions of the
> above packages in my OSP17.0 are:
> 
> openstack-tripleo-common-15.4.1-0.20220705010409.51f6577.el9ost.noarch
> edk2-ovmf-20220126gitbb1bba3d77-3.el9_0.1.noarch

@jparker Can you please provide testing guidance to mxie, here?

Comment 8 James Parker 2022-11-30 14:51:09 UTC
The BZ tracking is tracking two fixes landing as Kashyap mentioned earlier. The fix for edk2-ovmf should already be present in current puddles but the other change [1] for tripleo-common has yet to land.  This BZ [2] is currently tracking the overall status of both and is in MODIFIED state, so both fixes should land in the next phase3 puddle.  If testing is urgent you can unofficially verify this by applying [1] to the Undercloud and redeploying or you can wait until [2] is ON_QA and redeploy/retest with the associated puddle.


[1] https://review.opendev.org/c/openstack/tripleo-common/+/855142/
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2109644

Comment 9 Kashyap Chamarthy 2022-12-07 15:59:24 UTC
Hi mxie, does James' answer in comment#8 provide enough info for you to test?

Comment 10 mxie@redhat.com 2022-12-08 04:53:57 UTC
(In reply to Kashyap Chamarthy from comment #9)
> Hi mxie, does James' answer in comment#8 provide enough info for you to test?

Yes, I will continue to do v2v testing on OSP17.1 recently, I think fixed packages will be included in OSP17.1 env, thanks!

Comment 11 Artom Lifshitz 2023-02-07 00:40:21 UTC

*** This bug has been marked as a duplicate of bug 2109644 ***