Bug 1776823

Summary: rpm-ostree should preserve order of kernel boot parameters
Product: OpenShift Container Platform Reporter: Artyom <alukiano>
Component: RHCOSAssignee: Colin Walters <walters>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.3.0CC: augol, bbreard, dustymabe, fsimonce, imcleod, jligon, miabbott, nstielau, perobins, sgordon, walters
Target Milestone: ---Keywords: Reopened
Target Release: 4.4.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-13 21:53:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1771572    

Description Artyom 2019-11-26 13:06:58 UTC
Description of problem:
When I append hugepages boot parameters via
rpm-ostree kargs --append=hugepagesz=1G --append=hugepages=4 --append=hugepagesz=2M --append=hugepages=1024

After the reboot I have:
$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-73879866f874786fe64bcd407218b9ba9045c8b5a19078e0e467e0e5481ce5ed/vmlinuz-4.18.0-147.rt24.93.el8.x86_64 console=tty0 console=ttyS0,115200n8 rd.luks.options=discard ostree=/ostree/boot.1/rhcos/73879866f874786fe64bcd407218b9ba9045c8b5a19078e0e467e0e5481ce5ed/0 ignition.platform.id=openstack hugepagesz=1G hugepagesz=2M hugepages=4 hugepages=1024

rpm-ostree ordered parameters lexicographically, but the order is important in case of hugepages.

Version-Release number of selected component (if applicable):
rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● pivot://registry.svc.ci.openshift.org/ocp/4.3-2019-11-19-095016@sha256:e3154258679cab548b1c82d159f98e9863b76ce60932a4f1b25433924cdc55be
              CustomOrigin: Managed by machine-config-operator
                   Version: 43.81.201911190320.0 (2019-11-19T03:25:34Z)
       RemovedBasePackages: kernel-core kernel-modules kernel kernel-modules-extra 4.18.0-147.0.3.el8_1
      ReplacedBasePackages: microcode_ctl 4:20190618-1.20191112.1.el8_1 -> 4:20190918-3.rhcos.1.el8
             LocalPackages: kernel-rt-modules-4.18.0-147.rt24.93.el8.x86_64 kernel-rt-core-4.18.0-147.rt24.93.el8.x86_64
                            kernel-rt-modules-extra-4.18.0-147.rt24.93.el8.x86_64

  pivot://registry.svc.ci.openshift.org/ocp/4.3-2019-11-19-095016@sha256:e3154258679cab548b1c82d159f98e9863b76ce60932a4f1b25433924cdc55be
              CustomOrigin: Managed by machine-config-operator
                   Version: 43.81.201911190320.0 (2019-11-19T03:25:34Z)
       RemovedBasePackages: kernel-core kernel-modules kernel kernel-modules-extra 4.18.0-147.0.3.el8_1
      ReplacedBasePackages: microcode_ctl 4:20190618-1.20191112.1.el8_1 -> 4:20190918-3.rhcos.1.el8
             LocalPackages: kernel-rt-modules-4.18.0-147.rt24.93.el8.x86_64 kernel-rt-core-4.18.0-147.rt24.93.el8.x86_64
                            kernel-rt-modules-extra-4.18.0-147.rt24.93.el8.x86_64

How reproducible:
Always

Steps to Reproduce:
1. On the existing system run rpm-ostree kargs --append=hugepagesz=1G --append=hugepages=4 --append=hugepagesz=2M --append=hugepages=1024
2. systemctl reboot
3. cat /proc/cmdline

Actual results:
$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-73879866f874786fe64bcd407218b9ba9045c8b5a19078e0e467e0e5481ce5ed/vmlinuz-4.18.0-147.rt24.93.el8.x86_64 console=tty0 console=ttyS0,115200n8 rd.luks.options=discard ostree=/ostree/boot.1/rhcos/73879866f874786fe64bcd407218b9ba9045c8b5a19078e0e467e0e5481ce5ed/0 ignition.platform.id=openstack hugepagesz=1G hugepagesz=2M hugepages=4 hugepages=1024

Expected results:
$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-73879866f874786fe64bcd407218b9ba9045c8b5a19078e0e467e0e5481ce5ed/vmlinuz-4.18.0-147.rt24.93.el8.x86_64 console=tty0 console=ttyS0,115200n8 rd.luks.options=discard ostree=/ostree/boot.1/rhcos/73879866f874786fe64bcd407218b9ba9045c8b5a19078e0e467e0e5481ce5ed/0 ignition.platform.id=openstack hugepagesz=1G hugepages=4 hugepagesz=2M hugepages=1024

Additional info:
The issue already fix on U/S https://github.com/ostreedev/ostree/issues/1859

Comment 2 Colin Walters 2020-01-30 21:45:53 UTC
This will be fixed when we rebase to RHEL 8.2.

Comment 3 Micah Abbott 2020-01-30 21:51:35 UTC
The rebase to 8.2 won't happen for 4.4; pushing to 4.5.

Comment 4 Micah Abbott 2020-02-24 18:26:52 UTC
(In reply to Micah Abbott from comment #3)
> The rebase to 8.2 won't happen for 4.4; pushing to 4.5.

I guess I am a liar about pushing this out.

The KNI folks need this support as part of 4.4, ahead of the RHEL 8.2 release.

Colin, can we get a new build of `rpm-ostree` made with the fix for this and tagged into the 4.4 RHAOS puddle?

Comment 6 Micah Abbott 2020-02-25 14:06:34 UTC
(In reply to Micah Abbott from comment #4)

> Colin, can we get a new build of `rpm-ostree` made with the fix for this and
> tagged into the 4.4 RHAOS puddle?

Err...I meant `ostree`, but `rpm-ostree` would be good as well (see BZ#1804772)

Comment 7 Colin Walters 2020-02-26 21:24:37 UTC
ART please ask RCM to:

koji -p brew tag-build rhaos-4.4-rhel-8 ostree-2019.6-2.el8
koji -p brew tag-build rhaos-4.5-rhel-8 ostree-2019.6-2.el8

Comment 11 Micah Abbott 2020-03-12 20:17:07 UTC
Moving to MODIFIED so it can be attached to the errata.

Comment 15 Michael Nguyen 2020-03-13 00:47:19 UTC
$ oc image info -a ../all-the-pull-secrets.json $(oc adm release -a ../all-the-pull-secrets.json info --image-for=machine-os-content registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-03-12-152413)
Name:       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:470051ce2a761323af4e972d93d547551c3670f5f41993b3bfcbd1006da6afce
Media Type: application/vnd.docker.distribution.manifest.v2+json
Created:    1d ago
Image Size: 801.1MB
OS:         linux
Arch:       amd64
Entrypoint: /noentry
Labels:     com.coreos.ostree-commit=e6285cbf6737188a7d3ee3efa580d0af5b3694f6b164f9dd8fe81c061206e9f2
            io.buildah.version=1.14.0
            io.openshift.build.version-display-names=machine-os=Red Hat Enterprise Linux CoreOS
            io.openshift.build.versions=machine-os=44.81.202003110830-0
            version=44.81.202003110830-0


---
[core@localhost ~]$ sudo rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
* ostree://e6285cbf6737188a7d3ee3efa580d0af5b3694f6b164f9dd8fe81c061206e9f2
                   Version: 44.81.202003110830-0 (2020-03-11T08:36:13Z)
[core@localhost ~]$ sudo rpm-ostree kargs --append=pagesz=1G --append=pages=4 -append=pagesz=2M --append=pages=1024
Staging deployment... done
Kernel arguments updated.
Run "systemctl reboot" to start a reboot
[core@localhost ~]$ 

[core@localhost ~]$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-8e33da004cf9c41250a7fb2f8f125e9767bfe7343252ae5d2488b08a858f8d8a/vmlinuz-4.18.0-147.5.1.el8_1.x86_64 rhcos.root=crypt_rootfs console=tty0 console=ttyS0,115200n8 ignition.platform.id=qemu rd.luks.options=discard ostree=/ostree/boot.0/rhcos/8e33da004cf9c41250a7fb2f8f125e9767bfe7343252ae5d2488b08a858f8d8a/0 pagesz=1G pages=4 pagesz=2M pages=1024

[core@localhost ~]$ rpm -q ostree
ostree-2019.6-2.el8.x86_64

Comment 17 errata-xmlrpc 2020-05-13 21:53:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581

Comment 18 Red Hat Bugzilla 2023-09-15 00:19:54 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days