Bug 1712511

related: https://github.com/coreos/coreos-installer/issues/13

moving to next release.

So... I'm going to treat this as an RFE for the bare metal installer.  It happens to reference packet.net, but not as a true supported platform.  Rather, it is for the case where packet is used to simulate bare metal installs for UPI testing.

The RFE in this case is to:

A) Support passing an arbitrary console device for the _install_ boot (that is, for booting into coreos-installer)

B) Persisting this option into the ignition boot and future boots.

I believe A) is already supported, by virtue of being able to set the PXE, libvirt or CDROM boot command line for whatever invocation of coreos-install you select.

Let me suggest re-purposing this bug as a request to implement B) - "Persist boot time console options into installed system" - It will nearly always be the case that a modification to the console for install is also going to be useful, or even required, for future boots.

Thoughts?

I'm gonna ask Dusty to give a first impression on this via the magic of NEEDINFO.

The discussion on this topic has continued upstream in https://github.com/coreos/coreos-installer/issues/13

There are three ways to do this:

- Support persisting specific kernel arguments to the next boot
    - We do this for network based args today
    - Subsequent boots require networking to be set up by ignition, which means if we choose this option you'd have to set up the console config via ignition as well
- Support persisting specific kernel arguments permanently
     - We are currently doing this for ignition.platform.id, but could make it more generic
        - See https://github.com/coreos/coreos-installer/pull/60
- Support persisting arbitrary kernel arguments by adding arguments into the BLS config
    - What we do for ignition.platform.id, but make it more generic such that any kernel arg could be specified

It's unlikely that we will be able to implement changes related to this BZ in the 4.3 timeframe, given other priorities.  Additionally, the `coreos-installer` has undergone a major re-write, so it should be easier to add this kind of ability in the future.

Resetting assignee for now

Having been bitten by this, any method that allows me to persist kernel parameters without having to do so via MCO would be great because I'm finding them necessary to debug issues that occur prior to MCO configuration application. Catching grub before it boots and editing kernel parameters via a serial console is not a pleasurable experience :-)

Whether that's implicitly persisting all parameters minus a blacklist like 'coreos.inst*', an explicit prefix like 'coreos.inst.karg.*' doesn't matter to me.

We won't be able to change anything here for 4.4, however the new `coreos-installer` has better options for changing the kernel args of disk images before they are deployed.  That will be delivered as part of 4.5.

(In reply to Scott Dodson from comment #9)
> Having been bitten by this, any method that allows me to persist kernel
> parameters without having to do so via MCO would be great because I'm
> finding them necessary to debug issues that occur prior to MCO configuration
> application. Catching grub before it boots and editing kernel parameters via
> a serial console is not a pleasurable experience :-)

We're still discussing in https://github.com/coreos/coreos-installer/issues/13 on the best way forward. Meanwhile though, with the new coreos-installer, one can run any code one wants after the installation is complete. As mentioned in https://github.com/coreos/coreos-installer/issues/13#issuecomment-617998519, it's possible to mount the /boot partition and add any kargs one wants to the kernel cmdline in the BLS config at `$bootmnt/loader/entries/*.conf`.

Let's repurpose this ticket for the more general RFE of supporting persistent kargs. For now though, going to retarget this for 4.6.0.

*** Bug 1832255 has been marked as a duplicate of this bug. ***

Had a chat with David about this specifically as it relates to Packet and `console=ttyS1`. For the immediate short-term, we can use the MCO day-1 kargs support to add it: https://github.com/openshift/installer/blob/master/docs/user/customization.md#nodes-with-custom-kernel-arguments. This will help debug issues that happen post bootstrap reboot.

We are also almost at the point where we can use the new coreos-installer. Once that's in, as mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1712511#c11, the workflow would look something like:

1. Change the PXE config to contain `console=ttyS1` and `ignition.config.url=$url`.
2. Write an Ignition config hosted at `$url` which runs coreos-installer, and then mounts `/dev/disk/by-label/boot`, and appends `console=ttyS1` to the `options` line of the BLS entries at `/boot/loader/entries/*.conf`.

This RHBZ (and the upstream ticket https://github.com/coreos/coreos-installer/issues/13) is about making this workflow nicer.

Fixing this will require the new `coreos-installer`, which will be delivered as part of RHCOS/OCP 4.6

Upstream discussions now moved to https://github.com/coreos/fedora-coreos-tracker/issues/533.

I'm still working on this, but it'll spill over into the next sprint.

This is fixed in https://github.com/coreos/coreos-installer/pull/268. Now needs a new coreos-installer release and respin.

We should keep this in POST until it is delivered as part of an RHCOS build.

coreos-installer v0.3.0 or later should hit RHCOS 4.6 in the next sprint.

coreos-installer 0.4.0 is already in current builds.

Verified on 4.6.0-0.nightly-2020-08-04-074017 which contains RHEL CoreOS 46.82.202008032211-0.

== Live PXE boot RHCOS == 
PXE Config

```
default menu.c32
 prompt 1
 timeout 9
 ONTIMEOUT 1
 menu title ######## PXE Boot Menu ########
 label 1
 menu label ^1) Install Worker Node
 menu default
 kernel rhcos/kernel
 append initrd=rhcos/initramfs.img,rhcos/rootfs nomodeset ignition.firstboot ignition.platform.id=metal ignition.config.url=http://192.168.7.77:8080/ignition/worker.ign
```

== Run coreos-installer with --append-karg ==

[core@master2 ~]$ sudo coreos-installer install /dev/vda --append-karg foo=bar --ignition-url http://192.168.7.77:8080/ignition/master.ign --insecure-ignition
Installing Red Hat Enterprise Linux CoreOS 46.82.202008032211-0 (Ootpa) x86_64 (512-byte sectors)
> Read disk 3.5 GiB/3.5 GiB (100%)    
Writing Ignition config
Modifying kernel arguments
Install complete.
[core@master2 ~]$ sudo systemctl reboot
[core@master2 ~]$ Connection to master2 closed by remote host.
Connection to master2 closed.
[root@helper pxelinux.cfg]# rm -rf ~/.ssh/known_hosts 

== Verify karg is added ==
[root@helper pxelinux.cfg]# ssh core@master2
The authenticity of host 'master2 (192.168.7.23)' can't be established.
ECDSA key fingerprint is SHA256:0csSiBWN9fOuukctzFDA1222lRhfTSww7ajMqLyTrcI.
ECDSA key fingerprint is MD5:5b:31:86:7b:1c:83:3e:5f:59:84:5a:9f:eb:1f:27:02.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master2,192.168.7.23' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 46.82.202008032211-0
  Part of OpenShift 4.6, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.6/architecture/architecture-rhcos.html

---
[core@master2 ~]$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-5c3797397db1bafbc8c328137628f4008d4157b555477436f6ad87a10a00a388/vmlinuz-4.18.0-211.el8.x86_64 rhcos.root=crypt_rootfs random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ignition.platform.id=metal rd.luks.options=discard ignition.firstboot rd.neednet=1 ostree=/ostree/boot.1/rhcos/5c3797397db1bafbc8c328137628f4008d4157b555477436f6ad87a10a00a388/0 foo=bar
[core@master2 ~]$ Connection to master2 closed by remote host.
Connection to master2 closed.
[root@helper pxelinux.cfg]# ssh core@master2
Red Hat Enterprise Linux CoreOS 46.82.202008032211-0
  Part of OpenShift 4.6, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.6/architecture/architecture-rhcos.html

---
Last login: Wed Aug  5 14:13:08 2020 from 192.168.7.77
[core@master2 ~]$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-5c3797397db1bafbc8c328137628f4008d4157b555477436f6ad87a10a00a388/vmlinuz-4.18.0-211.el8.x86_64 rhcos.root=crypt_rootfs random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ignition.platform.id=metal rd.luks.options=discard ostree=/ostree/boot.0/rhcos/5c3797397db1bafbc8c328137628f4008d4157b555477436f6ad87a10a00a388/0 foo=bar






== Same setup, but test appending and deleting karg ==

[core@worker0 ~]$ sudo coreos-installer install /dev/vda --ignition-url http://192.168.7.77:8080/ignition/worker.ign --insecure-ignition --append-karg foo=bar --append-karg goo=car --delete-karg console=tty0
Installing Red Hat Enterprise Linux CoreOS 46.82.202008032211-0 (Ootpa) x86_64 (512-byte sectors)
> Read disk 3.5 GiB/3.5 GiB (100%)    
Writing Ignition config
Modifying kernel arguments
Install complete.
[core@worker0 ~]$ sudo systemctl reboot
[core@worker0 ~]$ Connection to worker0 closed by remote host.
Connection to worker0 closed.

[root@helper openshift]# rm -rf ~/.ssh/known_hosts 
[root@helper openshift]# ssh core@worker0
The authenticity of host 'worker0 (192.168.7.11)' can't be established.
ECDSA key fingerprint is SHA256:YeC6MsQVFOGzt/BJrdjaVmxTRv0C2FotHJgaTTdGqSA.
ECDSA key fingerprint is MD5:7c:02:27:74:e0:d4:98:ac:7d:8b:7d:13:7c:93:62:66.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'worker0,192.168.7.11' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 46.82.202008032211-0
  Part of OpenShift 4.6, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.6/architecture/architecture-rhcos.html

---
[core@worker0 ~]$ 
[core@worker0 ~]$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-5c3797397db1bafbc8c328137628f4008d4157b555477436f6ad87a10a00a388/vmlinuz-4.18.0-211.el8.x86_64 rhcos.root=crypt_rootfs random.trust_cpu=on console=ttyS0,115200n8 ignition.platform.id=metal rd.luks.options=discard ostree=/ostree/boot.0/rhcos/5c3797397db1bafbc8c328137628f4008d4157b555477436f6ad87a10a00a388/0 foo=bar goo=car

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196