Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1749310

Summary: UEFI whole disk image will not boot - "Failed to switch root: Specified switch root path /sysroot does not seem to be an OS tree"
Product: Red Hat OpenStack Reporter: Bob Fournier <bfournie>
Component: openstack-tripleo-commonAssignee: Adriano Petrich <apetrich>
Status: CLOSED CURRENTRELEASE QA Contact: Alexander Chuzhoy <sasha>
Severity: medium Docs Contact:
Priority: medium    
Version: 15.0 (Stein)CC: dwane.pottratz, mburns, slinaber
Target Milestone: z2Keywords: Triaged, ZStream
Target Release: 15.0 (Stein)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-12-08 15:02:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1680572    
Bug Blocks:    

Description Bob Fournier 2019-09-05 10:43:20 UTC
Description of problem:

Problem when booting the UEFI hardened whole disk image which was created with the command:
$ openstack overcloud image build --image-name overcloud-hardened-uefi-full --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-uefi-python3.yaml --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-uefi-rhel8.yaml

The deployed image isn't booting correctly, it boots into emergency mode with the error:
systemd[1]: Starting Switch Root...
systemctl[421]: Failed to switch root: Specified switch root path /sysroot does not seem to be an OS tree. os-release file is missing.
initrd-switch-root.service: Failed with result exit-code

Version-Release number of selected component (if applicable):

openstack-tripleo-common-10.8.1-0.20190831030439.300785c.el8ost.noarch


How reproducible:

Every time.


Steps to Reproduce:
1. Create whole disk image as above and using instructions at https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/director_installation_and_usage/appe-whole_disk_images#appe-Uploading-Security-Hardened-Whole-Disk-Image
2. Replace current overcloud-full image
3. Deploy with new image

Actual results:

Deployed node boots into emergency mode with error as above.


Expected results:

Node boots correctly.

Comment 1 Bob Fournier 2019-09-05 10:47:23 UTC
Problem looks like it is due to RHEL issue https://bugzilla.redhat.com/show_bug.cgi?id=1680572, fix is available in RHEL 8.1 - grub2-2.02-74, RHEL-8.1.0-20190701.0.

Comment 4 Bob Fournier 2019-11-11 13:18:03 UTC
Should be fixed when using grub2-2.02-74, RHEL-8.1.0-20190701.0 for the image.

Comment 5 Bob Fournier 2019-12-02 21:00:53 UTC
*** Bug 1778859 has been marked as a duplicate of this bug. ***

Comment 6 dwane.pottratz 2019-12-03 19:58:02 UTC
Hi Bob,

I am not able to create an image that will boot UEFI on bare metal using openstack ironic to deploy the image.

I tried the following using diskimage-builder. (version 2.30.0)
export DIB_LOCAL_IMAGE=/<path>/rhel-guest-image-8.2-87.x86_64.qcow2
export DIB_RELEASE=8
export DIB_YUM_REPO_CONF="
/<path-to-partner-repo-definitions>/RHEL-8-appstream.repo \
/<path-to-partner-repo-definitions>/RHEL-8-baseos.repo \
/<path-to-partner-repo-definitions>/RHEL-8-crb.repo \
/<path-to-partner-repo-definitions>/RHEL-8-highavailability.repo \
/<path-to-partner-repo-definitions>/RHEL-8-resilientstorage.repo \
/<path-to-partner-repo-definitions>/RHEL-8-rt.repo \
/<path-to-partner-repo-definitions>/RHEL-8-saphana.repo \
/<path-to-partner-repo-definitions>/RHEL-8-sap.repo
"
disk-image-create -o spe-rhel-8 rhel vm block-device-efi

I think this is similar to what TripleO does in the undercloud, correct?

What are some log that I might be able to collect to help with this?  The image is booting.  I don't get a logon prompt on the machine.  I am not able to ping the network.

Thanks,
Dwane

Comment 7 Bob Fournier 2019-12-05 19:11:40 UTC
Dwayne - you should be using the openstack overcloud image build command to create the image (which calls disk-image-create).

Are you able to view logs here? https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/director_installation_and_usage/appe-whole_disk_images#appe-Uploading-Security-Hardened-Whole-Disk-Image

If not I can pull out the relevant sections.

Comment 8 dwane.pottratz 2019-12-05 20:05:02 UTC
Hi Bob,

I am really trying to build an image that ironic can deploy to nodes for instances for end users.  I am using a packstack install and have images that work for centos and others.

I created bug 1778859 for it and it was marked as a duplicate of this bug.

Dwane

Comment 9 Bob Fournier 2019-12-05 20:16:24 UTC
So you can't use "openstack overcloud image" commands?  

There was an issue with booting RHEL 8.0 only, that's only what this bug addresses.  The images were created using the commands in comment 1 and the output on the node console with 8.0 was specifically for the title of this bug.  It sounds like you may not be hitting that as the image boots.  Are you able to capture the output in a console?

The best place to get help for this Ironic issue may be on #openstack-ironic on freenode.

Comment 10 Bob Fournier 2019-12-08 15:02:40 UTC
I've verified that this issue is no longer occurring when using a RHEL guest image for 8.1.

This is the procedure I used to boot a UEFI whole disk image:

$ curl http://rhos-qe-mirror-tlv.usersys.redhat.com/brewroot/packages/rhel-guest-image/8.1/310/images/rhel-guest-image-8.1-310.x86_64.qcow2 -o rhel-guest-image-8.1-310.x86_64.qcow2
$ export DIB_LOCAL_IMAGE=/home/stack/rhel-guest-image-8.1-310.x86_64.qcow2 
$ export DIB_YUM_REPO_CONF="/etc/yum.repos.d/rhos-release-16.repo /etc/yum.repos.d/rhos-release-rhel-8.1.repo"

$ openstack overcloud image build --image-name overcloud-hardened-uefi-full --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-uefi-python3.yaml --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-uefi-rhel8.yaml
cp -r images/ whole-disk-images

$ mv overcloud-hardened-uefi-full.qcow2 whole-disk-images/overcloud-full.qcow2 
$ openstack image delete overcloud-full
$ openstack image delete overcloud-full-initrd
$ openstack image delete overcloud-full-vmlinuz
$ openstack overcloud image upload --image-path /home/stack/whole-disk-images/ --whole-disk
$ openstack server create --flavor control --image overcloud-full --key default dell

$ openstack server list
+--------------------------------------+------+--------+-----------------------------+----------------+---------+
| ID                                   | Name | Status | Networks                    | Image          | Flavor  |
+--------------------------------------+------+--------+-----------------------------+----------------+---------+
| d3e75a0e-c3c4-4584-a2f4-5eb01cb6a959 | dell | ACTIVE | ctlplane=fe32:dead:beef::1f | overcloud-full | control |
+--------------------------------------+------+--------+-----------------------------+----------------+---------+

$ ping6 fe32:dead:beef::1f
PING fe32:dead:beef::1f(fe32:dead:beef::1f) 56 data bytes
64 bytes from fe32:dead:beef::1f: icmp_seq=1 ttl=64 time=0.514 ms
64 bytes from fe32:dead:beef::1f: icmp_seq=2 ttl=64 time=0.614 ms

Comment 11 Bob Fournier 2019-12-08 15:04:40 UTC
Dwayne - let us know if this procedure works for you.

Comment 12 dwane.pottratz 2019-12-09 17:39:24 UTC
Hi Bob,

I don't have a 8.1 '310' image available to me. I do have rhel-guest-image-8.1-263.x86_64.qcow2

I am able to create the image using disk-image-builder, but it will not boot.  It looks like the kernel get stuck.  Here are the last few lines from the boot.

[   30.180212] ata1.00: Enabling discard_zeroes_data
[   30.243332] sd 0:0:0:0: [sda] 937703088 512-byte logical blocks: (480 GB/447 GiB)
[   30.339826] sd 0:0:0:0: [sda] 4096-byte physical blocks
[   30.409229] sd 0:0:0:0: [sda] Write Protect is off
[   30.473241] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doest support DPO or FUA
[   30.588500] ata1.00: Enabling discard_zeroes_data
[   30.653899]  sda: sda1 sda2 sda3 sda4
[   30.705099] ata1.00: Enabling discard_zeroes_data
[   30.768296] sd 0:0:0:0: [sda] Attached SCSI disk
[   30.837125] sr 7:0:0:0: [sr0] scsi3-mmc drive: 48x/48x writer dvd-ram cd-rw xa/form2 cdda tray
[   30.947187] cdrom: Uniform CD-ROM driver Revision: 3.20

Here is what I used to create the image:
export DIB_LOCAL_IMAGE=/tmp/rhel-guest-image-8.1-263.x86_64.qcow2
export DIB_RELEASE=8
export DIB_YUM_REPO_CONF="
/tmp/RHEL-8-appstream.repo \
/tmp/RHEL-8-baseos.repo \
/tmp/RHEL-8-crb.repo \
/tmp/RHEL-8-highavailability.repo \
/tmp/RHEL-8-resilientstorage.repo \
/tmp/RHEL-8-rt.repo \
/tmp/RHEL-8-saphana.repo \
/tmp/RHEL-8-sap.repo
"
disk-image-create -o spe-rhel-8 rhel vm block-device-efi

Dwane

Comment 13 Bob Fournier 2019-12-09 18:53:11 UTC
It should not matter which version of 8.1 you are using, rhel-guest-image-8.1-263.x86_64.qcow2 should be fine.  

I recommend using the openstack commands to generate the image per Comment 10 which will use disk-image-create under the covers.

Comment 14 dwane.pottratz 2019-12-09 19:43:47 UTC
Hi Bob,

(In reply to Bob Fournier from comment #13)

> I recommend using the openstack commands to generate the image per Comment
> 10 which will use disk-image-create under the covers.

I did my install of openstack with packstack, not triple0.  I don't have the openstack overcloud command to create images.  I also don't have the yaml files, config files, etc associated with tripleO install.

Is there something else I can try?

Thanks,
Dwane

Comment 15 Bob Fournier 2019-12-09 20:16:03 UTC
So bug 1778859 was marked a duplicate of this because it was hitting the same UEFI issue with RHEL 8.0 that this bug addresses, namely the "Failed to switch root" problem which was resolved in 8.1.

It looks like with 8.1 that you are having an additional issue now when using disk-image-create directly.  I haven't built images directly using the DIB commands,  perhaps the best way is to open a new bug for RHEL 8.1 for diskimage-builder and we can track it that way. Ian Weinand specifically would probably be the person who can help with that.  Or we can continue the thread in the other RHEL bug that you opened (I can't seem to find the bug number right now).

It may also be useful to follow what tripleo uses for parameters to the "openstack overcloud image build" when it calls disk-image-create here:
https://github.com/openstack/tripleo-common/blob/master/tripleo_common/image/image_builder.py#L89

Comment 16 dwane.pottratz 2020-01-06 16:00:20 UTC
Is there anything else I need to do to this bug?  I keep getting waiting for response emails.

Comment 17 Bob Fournier 2020-01-06 16:09:45 UTC
Can you add a link to the bug that you created for disk-image-builder?  This bug is specifically for tripleo-common and since you are not using tripleo it isn't relevant.

Comment 18 dwane.pottratz 2020-01-06 16:13:52 UTC
(In reply to Bob Fournier from comment #17)
> Can you add a link to the bug that you created for disk-image-builder?  This
> bug is specifically for tripleo-common and since you are not using tripleo
> it isn't relevant.

https://bugzilla.redhat.com/show_bug.cgi?id=1781885

Comment 20 Red Hat Bugzilla 2023-09-14 05:42:52 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days