Description of problem: build RHEL8 image using diskimage-builder creating an image without initramfs or vmlinuz files if kernel was updated during the process Version-Release number of selected component (if applicable): How reproducible: everytime Steps to Reproduce: 1. DIB_LOCAL_IMAGE="rhel-8.0-update-1-x86_64-kvm.qcow2" 2. REG_REPOS="rhel-8-for-x86_64-baseos-rpms,rhel-8-for-x86_64-appstream-rpms,rhel-8-for-x86_64-highavailability-rpms,ansible-2.8-for-rhel-8-x86_64-rpms,openstack-beta-for-rhel-8-x86_64-rpms,fast-datapath-for-rhel-8-x86_64-rpms" 2. disk-image-create rhel 3. Actual results: qcow2 image without any vmlinuz or initramfs Expected results: qcow2 image with any vmlinuz or initramfs Additional info:
Created attachment 1600605 [details] build log diskimage-builder log file inserted "ls -l /boot" so it can be looked in the log
From the logs here, it appears that "clean-old-kernels" actually cleared the current kernel, not old kernels --- 2019-08-05 06:29:26.049 | dib-run-parts Running /tmp/in_target.d/finalise.d/01-clean-old-kernels 2019-08-05 06:29:26.052 | + set -eu 2019-08-05 06:29:26.052 | + set -o pipefail 2019-08-05 06:29:26.052 | + '[' 0 -ne 0 ']' 2019-08-05 06:29:26.052 | + YUM=dnf 2019-08-05 06:29:26.052 | + ls -l /boot/ 2019-08-05 06:29:26.054 | total 89120 2019-08-05 06:29:26.054 | -rw------- 1 root root 3752924 Jun 14 09:29 System.map-4.18.0-80.4.2.el8_0.x86_64 2019-08-05 06:29:26.054 | -rw-r--r-- 1 root root 180948 Jun 14 09:29 config-4.18.0-80.4.2.el8_0.x86_64 2019-08-05 06:29:26.054 | drwxr-xr-x 3 root root 4096 Jun 18 13:04 efi 2019-08-05 06:29:26.054 | drwxr-xr-x 2 root root 4096 Jun 18 13:11 grub 2019-08-05 06:29:26.054 | drwx------ 4 root root 4096 Jun 18 13:10 grub2 2019-08-05 06:29:26.054 | -rw------- 1 root root 48186115 Jun 18 13:08 initramfs-0-rescue-e1b088c6ab1348649040e16d35eebe6b.img 2019-08-05 06:29:26.054 | -rw------- 1 root root 23366266 Jun 18 13:11 initramfs-4.18.0-80.4.2.el8_0.x86_64.img 2019-08-05 06:29:26.054 | drwxr-xr-x 3 root root 4096 Jun 18 13:06 loader 2019-08-05 06:29:26.054 | -rwxr-xr-x 1 root root 7868768 Jun 18 13:07 vmlinuz-0-rescue-e1b088c6ab1348649040e16d35eebe6b 2019-08-05 06:29:26.054 | -rwxr-xr-x 1 root root 7868768 Jun 14 09:29 vmlinuz-4.18.0-80.4.2.el8_0.x86_64 2019-08-05 06:29:26.054 | + [[ dnf == \d\n\f ]] 2019-08-05 06:29:26.055 | ++ dnf repoquery --installonly --latest-limit=-1 -q 2019-08-05 06:29:28.924 | + _old_kernels='kernel-0:4.18.0-80.4.2.el8_0.x86_64 2019-08-05 06:29:28.924 | kernel-core-0:4.18.0-80.4.2.el8_0.x86_64 2019-08-05 06:29:28.924 | kernel-modules-0:4.18.0-80.4.2.el8_0.x86_64' 2019-08-05 06:29:28.924 | + [[ -n kernel-0:4.18.0-80.4.2.el8_0.x86_64 2019-08-05 06:29:28.924 | kernel-core-0:4.18.0-80.4.2.el8_0.x86_64 2019-08-05 06:29:28.924 | kernel-modules-0:4.18.0-80.4.2.el8_0.x86_64 ]] 2019-08-05 06:29:28.924 | + dnf remove -y kernel-0:4.18.0-80.4.2.el8_0.x86_64 kernel-core-0:4.18.0-80.4.2.el8_0.x86_64 kernel-modules-0:4.18.0-80.4.2.el8_0.x86_64 2019-08-05 06:29:29.437 | Updating Subscription Management repositories. 2019-08-05 06:29:31.749 | Dependencies resolved. 2019-08-05 06:29:31.753 | ================================================================================ 2019-08-05 06:29:31.753 | Package Arch Version Repository Size 2019-08-05 06:29:31.753 | ================================================================================ 2019-08-05 06:29:31.753 | Removing: 2019-08-05 06:29:31.753 | kernel x86_64 4.18.0-80.4.2.el8_0 @anaconda 0 2019-08-05 06:29:31.753 | kernel-core x86_64 4.18.0-80.4.2.el8_0 @anaconda 57 M 2019-08-05 06:29:31.753 | kernel-modules x86_64 4.18.0-80.4.2.el8_0 @anaconda 19 M 2019-08-05 06:29:31.753 | 2019-08-05 06:29:31.753 | Transaction Summary 2019-08-05 06:29:31.753 | ================================================================================ 2019-08-05 06:29:31.753 | Remove 3 Packages 2019-08-05 06:29:31.753 | 2019-08-05 06:29:31.753 | Freed space: 76 M 2019-08-05 06:29:31.753 | Running transaction check 2019-08-05 06:29:31.978 | Transaction check succeeded. 2019-08-05 06:29:31.978 | Running transaction test 2019-08-05 06:29:32.024 | Transaction test succeeded. 2019-08-05 06:29:32.024 | Running transaction 2019-08-05 06:29:32.120 | Preparing : 1/1 2019-08-05 06:29:32.121 | Erasing : kernel-4.18.0-80.4.2.el8_0.x86_64 1/3 2019-08-05 06:29:32.256 | Running scriptlet: kernel-4.18.0-80.4.2.el8_0.x86_64 1/3 2019-08-05 06:29:32.256 | Erasing : kernel-modules-4.18.0-80.4.2.el8_0.x86_64 2/3 2019-08-05 06:29:33.997 | Running scriptlet: kernel-modules-4.18.0-80.4.2.el8_0.x86_64 2/3 2019-08-05 06:29:34.181 | Running scriptlet: kernel-core-4.18.0-80.4.2.el8_0.x86_64 3/3 2019-08-05 06:29:34.198 | Erasing : kernel-core-4.18.0-80.4.2.el8_0.x86_64 3/3 2019-08-05 06:29:35.658 | Running scriptlet: kernel-core-4.18.0-80.4.2.el8_0.x86_64 3/3 2019-08-05 06:29:35.659 | Verifying : kernel-4.18.0-80.4.2.el8_0.x86_64 1/3 2019-08-05 06:29:35.659 | Verifying : kernel-core-4.18.0-80.4.2.el8_0.x86_64 2/3 2019-08-05 06:29:35.852 | Verifying : kernel-modules-4.18.0-80.4.2.el8_0.x86_64 3/3 2019-08-05 06:29:35.852 | Installed products updated. 2019-08-05 06:29:35.991 | 2019-08-05 06:29:35.992 | Removed: 2019-08-05 06:29:35.992 | kernel-4.18.0-80.4.2.el8_0.x86_64 2019-08-05 06:29:35.992 | kernel-core-4.18.0-80.4.2.el8_0.x86_64 2019-08-05 06:29:35.992 | kernel-modules-4.18.0-80.4.2.el8_0.x86_64 2019-08-05 06:29:35.992 | --- That really comes down to just --- _old_kernels="$(dnf repoquery --installonly --latest-limit=-1 -q)" --- AFAIK this dib code hasn't changed in a long time, so I don't think it's new behaviour as such. But perhaps there's a missing assumption in the code for this case ... [1] https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/redhat-common/finalise.d/01-clean-old-kernels#L19
Checking a recent f29 build [1], which uses the same dnf path in this element; the repoquery does not return anything. note the centos7 path ues the old yum path so isn't equivalent --- 2019-07-01 09:13:02.398 | dib-run-parts Running /tmp/in_target.d/finalise.d/01-clean-old-kernels 2019-07-01 09:13:02.406 | + set -eu 2019-07-01 09:13:02.406 | + set -o pipefail 2019-07-01 09:13:02.406 | + '[' 0 -ne 0 ']' 2019-07-01 09:13:02.406 | + YUM=dnf 2019-07-01 09:13:02.406 | + [[ dnf == \d\n\f ]] 2019-07-01 09:13:02.407 | ++ dnf repoquery --installonly --latest-limit=-1 -q 2019-07-01 09:13:02.994 | + _old_kernels= 2019-07-01 09:13:02.994 | + [[ -n '' ]] 2019-07-01 09:13:02.996 | dib-run-parts 01-clean-old-kernels completed --- [1] http://logs.openstack.org/72/668372/1/check/dib-nodepool-functional-openstack-fedora-29-src/bbe68f4/nodepool/builds/test-image-0000000001.log
(In reply to Ian Wienand from comment #3) > From the logs here, it appears that "clean-old-kernels" actually cleared the > current kernel, not old kernels > > --- > 2019-08-05 06:29:26.049 | dib-run-parts Running > /tmp/in_target.d/finalise.d/01-clean-old-kernels > 2019-08-05 06:29:26.052 | + set -eu > 2019-08-05 06:29:26.052 | + set -o pipefail > 2019-08-05 06:29:26.052 | + '[' 0 -ne 0 ']' > 2019-08-05 06:29:26.052 | + YUM=dnf > 2019-08-05 06:29:26.052 | + ls -l /boot/ > 2019-08-05 06:29:26.054 | total 89120 > 2019-08-05 06:29:26.054 | -rw------- 1 root root 3752924 Jun 14 09:29 > System.map-4.18.0-80.4.2.el8_0.x86_64 > 2019-08-05 06:29:26.054 | -rw-r--r-- 1 root root 180948 Jun 14 09:29 > config-4.18.0-80.4.2.el8_0.x86_64 > 2019-08-05 06:29:26.054 | drwxr-xr-x 3 root root 4096 Jun 18 13:04 efi > 2019-08-05 06:29:26.054 | drwxr-xr-x 2 root root 4096 Jun 18 13:11 grub > 2019-08-05 06:29:26.054 | drwx------ 4 root root 4096 Jun 18 13:10 grub2 > 2019-08-05 06:29:26.054 | -rw------- 1 root root 48186115 Jun 18 13:08 > initramfs-0-rescue-e1b088c6ab1348649040e16d35eebe6b.img > 2019-08-05 06:29:26.054 | -rw------- 1 root root 23366266 Jun 18 13:11 > initramfs-4.18.0-80.4.2.el8_0.x86_64.img > 2019-08-05 06:29:26.054 | drwxr-xr-x 3 root root 4096 Jun 18 13:06 loader > 2019-08-05 06:29:26.054 | -rwxr-xr-x 1 root root 7868768 Jun 18 13:07 > vmlinuz-0-rescue-e1b088c6ab1348649040e16d35eebe6b > 2019-08-05 06:29:26.054 | -rwxr-xr-x 1 root root 7868768 Jun 14 09:29 > vmlinuz-4.18.0-80.4.2.el8_0.x86_64 > 2019-08-05 06:29:26.054 | + [[ dnf == \d\n\f ]] > 2019-08-05 06:29:26.055 | ++ dnf repoquery --installonly --latest-limit=-1 -q > 2019-08-05 06:29:28.924 | + _old_kernels='kernel-0:4.18.0-80.4.2.el8_0.x86_64 > 2019-08-05 06:29:28.924 | kernel-core-0:4.18.0-80.4.2.el8_0.x86_64 > 2019-08-05 06:29:28.924 | kernel-modules-0:4.18.0-80.4.2.el8_0.x86_64' > 2019-08-05 06:29:28.924 | + [[ -n kernel-0:4.18.0-80.4.2.el8_0.x86_64 > 2019-08-05 06:29:28.924 | kernel-core-0:4.18.0-80.4.2.el8_0.x86_64 > 2019-08-05 06:29:28.924 | kernel-modules-0:4.18.0-80.4.2.el8_0.x86_64 ]] > 2019-08-05 06:29:28.924 | + dnf remove -y > kernel-0:4.18.0-80.4.2.el8_0.x86_64 kernel-core-0:4.18.0-80.4.2.el8_0.x86_64 > kernel-modules-0:4.18.0-80.4.2.el8_0.x86_64 > 2019-08-05 06:29:29.437 | Updating Subscription Management repositories. > 2019-08-05 06:29:31.749 | Dependencies resolved. > 2019-08-05 06:29:31.753 | > ============================================================================= > === > 2019-08-05 06:29:31.753 | Package Arch Version > Repository Size > 2019-08-05 06:29:31.753 | > ============================================================================= > === > 2019-08-05 06:29:31.753 | Removing: > 2019-08-05 06:29:31.753 | kernel x86_64 > 4.18.0-80.4.2.el8_0 @anaconda 0 > 2019-08-05 06:29:31.753 | kernel-core x86_64 > 4.18.0-80.4.2.el8_0 @anaconda 57 M > 2019-08-05 06:29:31.753 | kernel-modules x86_64 > 4.18.0-80.4.2.el8_0 @anaconda 19 M > 2019-08-05 06:29:31.753 | > 2019-08-05 06:29:31.753 | Transaction Summary > 2019-08-05 06:29:31.753 | > ============================================================================= > === > 2019-08-05 06:29:31.753 | Remove 3 Packages > 2019-08-05 06:29:31.753 | > 2019-08-05 06:29:31.753 | Freed space: 76 M > 2019-08-05 06:29:31.753 | Running transaction check > 2019-08-05 06:29:31.978 | Transaction check succeeded. > 2019-08-05 06:29:31.978 | Running transaction test > 2019-08-05 06:29:32.024 | Transaction test succeeded. > 2019-08-05 06:29:32.024 | Running transaction > 2019-08-05 06:29:32.120 | Preparing : > 1/1 > 2019-08-05 06:29:32.121 | Erasing : > kernel-4.18.0-80.4.2.el8_0.x86_64 1/3 > 2019-08-05 06:29:32.256 | Running scriptlet: > kernel-4.18.0-80.4.2.el8_0.x86_64 1/3 > 2019-08-05 06:29:32.256 | Erasing : > kernel-modules-4.18.0-80.4.2.el8_0.x86_64 2/3 > 2019-08-05 06:29:33.997 | Running scriptlet: > kernel-modules-4.18.0-80.4.2.el8_0.x86_64 2/3 > 2019-08-05 06:29:34.181 | Running scriptlet: > kernel-core-4.18.0-80.4.2.el8_0.x86_64 3/3 > 2019-08-05 06:29:34.198 | Erasing : > kernel-core-4.18.0-80.4.2.el8_0.x86_64 3/3 > 2019-08-05 06:29:35.658 | Running scriptlet: > kernel-core-4.18.0-80.4.2.el8_0.x86_64 3/3 > 2019-08-05 06:29:35.659 | Verifying : > kernel-4.18.0-80.4.2.el8_0.x86_64 1/3 > 2019-08-05 06:29:35.659 | Verifying : > kernel-core-4.18.0-80.4.2.el8_0.x86_64 2/3 > 2019-08-05 06:29:35.852 | Verifying : > kernel-modules-4.18.0-80.4.2.el8_0.x86_64 3/3 > 2019-08-05 06:29:35.852 | Installed products updated. > 2019-08-05 06:29:35.991 | > 2019-08-05 06:29:35.992 | Removed: > 2019-08-05 06:29:35.992 | kernel-4.18.0-80.4.2.el8_0.x86_64 > > 2019-08-05 06:29:35.992 | kernel-core-4.18.0-80.4.2.el8_0.x86_64 > > 2019-08-05 06:29:35.992 | kernel-modules-4.18.0-80.4.2.el8_0.x86_64 > > 2019-08-05 06:29:35.992 | > --- > > That really comes down to just > > --- > _old_kernels="$(dnf repoquery --installonly --latest-limit=-1 -q)" > --- > > AFAIK this dib code hasn't changed in a long time, so I don't think it's new > behaviour as such. But perhaps there's a missing assumption in the code for > this case ... > > [1] > https://opendev.org/openstack/diskimage-builder/src/branch/master/ > diskimage_builder/elements/redhat-common/finalise.d/01-clean-old-kernels#L19 when installing kernel update in chroot env it does not create the files initramfs and vmlinuz see kernel-x86_644.18.0-80.7.1.el8_0 is the update but the files never created although it was said to be install successfully by dnf
arrgghh, so I went down the rabbit hole on this... To make a long story short; kernel-core-4.18.0-80.7.1.el8_0.x86_64.rpm does not actually ship anything in /boot The files in /boot actually get installed via a long path starting at kernel-install that ends up at /usr/lib/kernel/install.d/20-grub.install That has an interesting line at the top [1] --- if ! [[ $KERNEL_INSTALL_MACHINE_ID ]]; then exit 0 fi --- ... guess what; we have no /etc/machine-id, this variable is not set, this script exits silently, no kernel is installed but the package is, the old kernel is removed, you end up with no kernel in the final image! [1] https://src.fedoraproject.org/rpms/grub2/blob/f30/f/20-grub.install#_3
Oh, and I actually debugged this once before 2 years ago and forgot about it! :) https://review.opendev.org/#/c/504300/
Could you try https://review.opendev.org/#/c/675056/1 and confirm that works? See my notes in a comment there, but this seems to get the right kernels installed for me
(In reply to Ian Wienand from comment #8) > Could you try > > https://review.opendev.org/#/c/675056/1 > > and confirm that works? See my notes in a comment there, but this seems to > get the right kernels installed for me its did the trick. able to boot into image but boot entry are not correct /boot/grub/grub.conf: default=0 timeout=0 title Red Hat Enterprise Linux 8 (4.18.0-80.4.2.el8_0.x86_64) root (hd0) kernel /boot/vmlinuz-4.18.0-80.4.2.el8_0.x86_64 ro root=UUID=9c3d4ce1-341c-401d-8f8f-8e9c15c00f40 console=hvc0 LANG=en_US.UTF-8 initrd /boot/initramfs-4.18.0-80.4.2.el8_0.x86_64.img
Thanks for trying it! > /boot/grub/grub.conf: I do not believe this is used in RHEL8, right (i'm not sure why it's there...)? I think it follows the "boot loader specification" where bits of /boot/loader/entries/ contains the kernels that are booted? on my build, I see the config in there (for both kernels at first, then only the final one) So I think that if it's booting, it's getting the right info. As an additional check, if you like, could build with DIB_DISABLE_KERNEL_CLEANUP=1 which will leave both kernels, and hopepfully you can switch between them. [1] https://systemd.io/BOOT_LOADER_SPECIFICATION
i meant that upon server boot loading grub present also the old kernel that has been removed in the image build as an viable option. here is the cmd output sudo ls -l /boot/loader/entries/ total 16 -rw-r--r--. 1 root root 408 Aug 7 11:47 5034848f983f4a9eba05ba212e9eb3d9-0-rescue.conf -rw-r--r--. 1 root root 371 Aug 7 11:47 5034848f983f4a9eba05ba212e9eb3d9-4.18.0-80.7.1.el8_0.x86_64.conf -rw-r--r--. 1 root root 408 Jun 18 20:08 e1b088c6ab1348649040e16d35eebe6b-0-rescue.conf -rw-r--r--. 1 root root 371 Jun 18 20:08 e1b088c6ab1348649040e16d35eebe6b-4.18.0-80.4.2.el8_0.x86_64.conf
> i meant that upon server boot loading grub present also the old kernel that has been removed in the image build as an viable option. Interesting; yes I agree that's not quite right. However, I think it's something a bit out of our control, i think it has to do with the kernel/grub/systemd scripts not wanting to touch the entries for a different machine-id. it replicates on the "factory" image too ... I've filed bug#1739286 I imagine though, if it is by default booting in to the right kernel, this doesn't really cause an operational issue?
yes no operational issue, unless someone will manually choose this dead entry. can we merge the code?
OSP15 Trunk is building diskimage-builder from the upstream master branch since upstream is "branchless". Also latest upstream tag 2.26.1 includes the fix https://review.opendev.org/675056 "Create /etc/machine-id for RHEL images", so I'm adding latest OSP15 Trunk candidate build in fixed-in-verson: NB this will not make OSP15 GA since it's not marked as blocker!
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0643