Bug 1804772

Summary: rpm-ostree should support --append and --delete of kernel args simultaneously
Product: OpenShift Container Platform Reporter: Marc Sluiter <msluiter>
Component: RHCOSAssignee: Colin Walters <walters>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: high Docs Contact:
Priority: high    
Version: 4.4CC: bbreard, dshchedr, dustymabe, fsimonce, imcleod, jligon, markmc, miabbott, nstielau, perobins, smilner, walters
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rpm-ostree-2019.6-8.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-04 11:37:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1807487    
Bug Blocks: 1771572    

Description Marc Sluiter 2020-02-19 15:30:13 UTC
Description of problem:

Currently it is not possible to add and delete kernel args simultaneously with MachineConfigs [0]. This is already fixed in rpm-ostree [1], but the fix is not available in RHCOS for OCP 4.4 yet.

[0] https://github.com/openshift/machine-config-operator/issues/1265
[1] https://github.com/coreos/rpm-ostree/pull/1940

Version-Release number of selected component (if applicable):
OCP 4.4 nightly

How reproducible:

always

Steps to Reproduce:
1. Create a MachineConfig with kernel args, wait for MachineConfigPool getting updated.
2. Update the MachineConfig with added and deleted kernel args

Actual results:
MachineConfigPool is stuck in degraded state

Expected results:
MachineConfigPool updated again, nodes rebooted with updated kernel args.

Additional info:

Comment 2 Micah Abbott 2020-02-20 16:28:52 UTC
Related - https://bugzilla.redhat.com/show_bug.cgi?id=1776823

Comment 4 Micah Abbott 2020-02-25 14:07:28 UTC
The KNI folks would like this fix as part of 4.4, ahead of the RHEL 8.2 release.

Colin, can we get a fixed version of `rpm-ostree` built and included in the RHAOS 4.4 puddle?

Comment 5 Colin Walters 2020-02-25 23:26:12 UTC
So that's only in 2019.7, but the latest errata https://errata.devel.redhat.com/advisory/47351 is for 2019.6.  

The tracker bug for that is: https://bugzilla.redhat.com/show_bug.cgi?id=1756192

I think at this point we need an exception or something to do another build?

Comment 6 Colin Walters 2020-02-26 13:43:52 UTC
OK I filed a new bug https://bugzilla.redhat.com/show_bug.cgi?id=1807487

Comment 12 Micah Abbott 2020-03-12 12:58:42 UTC
The fixed package landed in RHCOS 44.81.202003062006-0; all subsequent builds should have the fix.

Comment 13 Micah Abbott 2020-03-12 14:23:36 UTC
This can be moved to VERIFIED once it hits ON_QA


```
Red Hat Enterprise Linux CoreOS 44.81.202003110830-0
  Part of OpenShift 4.4, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.4/architecture/architecture-rhcos.html

---
Last login: Thu Mar 12 14:15:36 2020 from 192.168.124.1
[core@localhost ~]$ rpm-ostree kargs
rhcos.root=crypt_rootfs console=tty0 console=ttyS0,115200n8 ignition.platform.id=qemu rd.luks.options=discard $ignition_firstboot ostree=/ostree/boot.0/rhcos/8e33da004cf9c41250a7fb2f8f125e9767bfe7343252ae5d2488b08a858f8d8a/0 foo=bar bar=foo pizza=life
[core@localhost ~]$ sudo rpm-ostree kargs --delete pizza=life --append beer=life
Staging deployment... done
Kernel arguments updated.
Run "systemctl reboot" to start a reboot
[core@localhost ~]$ rpm-ostree kargs
rhcos.root=crypt_rootfs console=tty0 console=ttyS0,115200n8 ignition.platform.id=qemu rd.luks.options=discard $ignition_firstboot ostree=/ostree/boot.0/rhcos/8e33da004cf9c41250a7fb2f8f125e9767bfe7343252ae5d2488b08a858f8d8a/0 foo=bar bar=foo beer=life
[core@localhost ~]$ sudo systemctl reboot
...
Warning: Permanently added '192.168.124.105' (ECDSA) to the list of known hosts.
Red Hat Enterprise Linux CoreOS 44.81.202003110830-0
  Part of OpenShift 4.4, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.4/architecture/architecture-rhcos.html

---
Last login: Thu Mar 12 14:21:22 2020 from 192.168.124.1
[core@localhost ~]$ rpm-ostree kargs
rhcos.root=crypt_rootfs console=tty0 console=ttyS0,115200n8 ignition.platform.id=qemu rd.luks.options=discard $ignition_firstboot ostree=/ostree/boot.1/rhcos/8e33da004cf9c41250a7fb2f8f125e9767bfe7343252ae5d2488b08a858f8d8a/0 foo=bar bar=foo beer=life
[core@localhost ~]$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-8e33da004cf9c41250a7fb2f8f125e9767bfe7343252ae5d2488b08a858f8d8a/vmlinuz-4.18.0-147.5.1.el8_1.x86_64 rhcos.root=crypt_rootfs console=tty0 console=ttyS0,115200n8 ignition.platform.id=qemu rd.luks.options=discard ostree=/ostree/boot.1/rhcos/8e33da004cf9c41250a7fb2f8f125e9767bfe7343252ae5d2488b08a858f8d8a/0 foo=bar bar=foo beer=life
```

Comment 17 Micah Abbott 2020-03-12 15:02:51 UTC
Marking VERIFIED per comment #13

Comment 19 errata-xmlrpc 2020-05-04 11:37:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581