Bug 2059429

Summary: Slow booting when using the fdo-linux-client
Product: Red Hat Enterprise Linux 9 Reporter: Ben Breard <bbreard>
Component: osbuild-composerAssignee: Image Builder team <osbuilders>
Status: CLOSED ERRATA QA Contact: Xiaofeng Wang <xiaofwan>
Severity: high Docs Contact:
Priority: unspecified    
Version: 9.0CC: amurdaca, jrusz, lmiksik, obudai, perobins, pvlasin, xiaofwan, yih
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: osbuild-composer-46.2-1.el9_0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2062132 2062820 (view as bug list) Environment:
Last Closed: 2022-05-17 13:30:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2062132, 2062820    

Description Ben Breard 2022-03-01 02:30:30 UTC
Description of problem:

When installing a RHEL for Edge system using the simplified installer, the boot time is very long. A system that should take ~10-15 seconds to boot, will take about 2 minutes. What ever is going wrong happens incredibly early in the boot process. When this is happening, the UEFI splash screen will stay for a *long* time. I've tried power cycling, disabling the fdo unit - nothing seems to help. 

This issue may belong to the coreos-installer. 

Version-Release number of selected component (if applicable):
RHEL 9 nightlies 
fdo-client-0.3.0-1.el9

How reproducible:
100%

Steps to Reproduce:
1. create an edge image using the simplified-installer type
2. install to disk
3. watch the boot time

Actual results:
The system will take a relatively long time to boot. Also, systemd-analyze blame will report jobs are still running.

Expected results:
Boot speeds should not be negatively impacted with using the simplified installer and FDO

Additional info:

Comment 1 Antonio Murdaca 2022-03-01 08:28:39 UTC
(In reply to Ben Breard from comment #0)
> Description of problem:
> 
> When installing a RHEL for Edge system using the simplified installer, the
> boot time is very long. A system that should take ~10-15 seconds to boot,
> will take about 2 minutes. What ever is going wrong happens incredibly early
> in the boot process. When this is happening, the UEFI splash screen will
> stay for a *long* time. I've tried power cycling, disabling the fdo unit -
> nothing seems to help. 

the fact that disabling the fdo-client unit does nothing makes me think this isn't the issue
maybe we should take a look at the package/service set we ship but I'm positive they are the same as we ship with the Anaconda installer
have you tested both anaconda and simpl installer?

> 
> This issue may belong to the coreos-installer. 
> 
> Version-Release number of selected component (if applicable):
> RHEL 9 nightlies 
> fdo-client-0.3.0-1.el9
> 
> How reproducible:
> 100%
> 
> Steps to Reproduce:
> 1. create an edge image using the simplified-installer type
> 2. install to disk
> 3. watch the boot time
> 
> Actual results:
> The system will take a relatively long time to boot. Also, systemd-analyze
> blame will report jobs are still running.
> 
> Expected results:
> Boot speeds should not be negatively impacted with using the simplified
> installer and FDO
> 
> Additional info:

Comment 2 Antonio Murdaca 2022-03-01 08:31:45 UTC
also, afaict, the slow down is on "real" devices (like my fitlet2) - on virtual machine this doesn't happen (they boot just as fast as you'd expect)

Comment 3 Antonio Murdaca 2022-03-01 08:52:48 UTC
ok, I think I've found the difference between simplified installer and anaconda and it boils down to https://bugzilla.redhat.com/show_bug.cgi?id=1839923 - so w/o reading up that bz, we're adding console= kargs in the system image with simplified installer where anaconda doesn't and leave console to the kernel - I'm creating a patch for that after testing it out

Comment 4 Antonio Murdaca 2022-03-01 08:56:18 UTC
Ben, the workaround is:

sudo rpm-ostree kargs --delete "console=tty0"
sudo rpm-ostree kargs --delete "console=ttyS0"

The system will boot just as normal speed afterwards

Comment 5 Antonio Murdaca 2022-03-01 08:59:31 UTC
Fixed in https://github.com/osbuild/osbuild-composer/pull/2377

Comment 16 Xiaofeng Wang 2022-04-05 13:32:33 UTC
Verified on:
osbuild-composer-dnf-json-46.2-1.el9_0.x86_64
osbuild-composer-core-46.2-1.el9_0.x86_64
python3-osbuild-53-1.el9_0.noarch
osbuild-selinux-53-1.el9_0.noarch
osbuild-53-1.el9_0.noarch
osbuild-luks2-53-1.el9_0.noarch
osbuild-lvm2-53-1.el9_0.noarch
osbuild-ostree-53-1.el9_0.noarch
osbuild-composer-worker-46.2-1.el9_0.x86_64
osbuild-composer-46.2-1.el9_0.x86_64

Comment 18 errata-xmlrpc 2022-05-17 13:30:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: osbuild-composer), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2522