Bug 1783571
Summary: | Rebase to BOOM 1.1 with kernel caching support | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 8 | Reporter: | Jonathan Earl Brassow <jbrassow> |
Component: | boom-boot | Assignee: | Bryn M. Reeves <bmr> |
Status: | CLOSED ERRATA | QA Contact: | cluster-qe <cluster-qe> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 8.1 | CC: | agk, bmr, cmarthal, jbrassow, jmagrini, mcsontos, rbednar, rhandlin |
Target Milestone: | rc | Keywords: | Rebase |
Target Release: | 8.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | boom-boot-1.1-2.el8 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2020-11-04 01:57:39 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1825061 |
Description
Jonathan Earl Brassow
2019-12-13 22:14:59 UTC
The caching feature adds a new python module, boom.cache to the Boom package. This implements an optional caching facility for boot images (kernels, initramfs, etc.). The cache preserves the content of files such as /boot/vmlinux-$(VERSION) and /boot/initramfs-$(VERSION).img so that boom boot entries that reference these files can still be used following major system upgrades that may purge content from /boot. The cache is physically located (in the default configuration) at /boot/boom/cache, and consists of image files (named with an SHA1 hash and a .img extension), and a single JSON metadata file, "cacheindex.json": $ tree boom/cache boom/cache/ ├── 07828df1013c4a5024ecfacae217a74ab5345ea6.img ├── 087beab60378488439a972d1144fef22f2afb1b0.img ├── 1562375e4d022e814ba39521d0852e490b7c07f8.img ├── 3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3.img ├── 3cb9fccc94bb6e40ba08d6be5a471b217fe0d5c7.img ├── 537f67b4c6efe26a873cafc78bea72a6ec32a874.img ├── 537f67b4c6efe26a873cafc78bea72a6ec32a875.img ├── 6c9313159c9d13dbbba3204e39bec870d0ab168d.img ├── a0ce183662af53087fd786e95b43e670c42fe9a2.img └── cacheindex.json 0 directories, 10 files The test suite will include validation of the cache internal state and API calls: for QE purposes we should be looking at the operations that users will carry out: 1. Confirm that a cache entry is created automatically when a new boot entry is created -> "boom create ..." sets up the entry when automatic caching is enabled (this will be the default) -> confirm via tool output ("boom cache list") output should contain matching /vmlinuz-$(VERSION) and /initramfs-$(VERSION).img 2. Confirm that the proper /boot/vmlinuz-$(VERSION) and /boot/initramfs-$(VERSION).img contents are created on a restore operation -> "boom cache restore" command, exact details tbd 3. Confirming that the files are restored should be sufficient but if feasible it may also be useful to attempt a boot following an upgrade/restore (I can help with this) I can provide details of the JSON format if it's useful but it's probably a bit low-level: the main thing to note is that there are three "path" values associated with the entries in the cache. They are referred to in the code (and occasionally tools/docs) as "img_path", "boot_path", and "cache_path": # img_path - The path to an image as specified in a BLS snippet, or an # argument to cache_path(), relative to the root of the # /boot file system. E.g. "/vmlinuz-5.0.0". # # boot_path - The absolute path to an image relative to the host root # file system. E.g. "/boot/vmlinuz-5.0.0". # # cache_path - The absolute path to a cached boot image relative to the # host root file system. E.g. # "/boot/boom/cache/1562375e4d022e814ba39521d0852e490b7c07f8.img" It shouldn't be necessary to poke into this stuff but some details might help to simplify test cases? Additional stuff that may be helpful to test (this will be covered by the test suite as well) would be to confirm that images (in both /boot and /boot/boom/cache) are cleaned up when the corresponding boom boot entries are removed. I can add extra details on this if it seems useful. The Coverity warning is now fixed upstream: commit d788f82746826d091abcdf76b5f4884c22e3b7de (HEAD -> master, origin/master) Author: Bryn M. Reeves <bmr> Date: Wed May 20 15:14:27 2020 +0100 boom.cache: use correct variable in load_cache() warning branch Signed-off-by: Bryn M. Reeves <bmr> diff --git a/boom/cache.py b/boom/cache.py index 1dc5e07..cf7de47 100644 --- a/boom/cache.py +++ b/boom/cache.py @@ -238,7 +238,7 @@ def load_cache(verify=True, digests=False): images = cachedata["images"] for image_id in images.keys(): if image_id not in ids: - _log_warn("Found orphan image_id '%s'" % image) + _log_warn("Found orphan image_id '%s'" % image_id) # clean up? _log_debug("Loaded %d cache paths and %d images" % There are four patches now in master that would be good to include in the RHEL8 build: commit 2b32c40344ff82ba5d7bb959af9b9a97a1a4c142 (HEAD -> master, origin/master) Author: Bryn M. Reeves <bmr> Date: Fri May 22 17:43:49 2020 +0100 boom.bootloader: decode bytes output of grub2-editenv Signed-off-by: Bryn M. Reeves <bmr> commit 50559a4991068a0189e8c5d97c9ed395b3d02bad Author: Bryn M. Reeves <bmr> Date: Fri May 22 17:17:15 2020 +0100 boom: raise exceptions in load_profiles_for_class() if --debug Raise exceptions so that the command handler prints the full backtrace when --debug is enabled. Signed-off-by: Bryn M. Reeves <bmr> commit 235e8e64b7d0c6ba7dcabd899bb7181596e9a137 Author: Bryn M. Reeves <bmr> Date: Fri May 22 17:15:15 2020 +0100 boom.bootloader: raise exceptions in load_entries() if --debug Raise exceptions so that the command handler prints the full backtrace when --debug is enabled. Signed-off-by: Bryn M. Reeves <bmr> commit d788f82746826d091abcdf76b5f4884c22e3b7de Author: Bryn M. Reeves <bmr> Date: Wed May 20 15:14:27 2020 +0100 boom.cache: use correct variable in load_cache() warning branch Signed-off-by: Bryn M. Reeves <bmr> Two of these are for supportability: they enable more detailed debugging output in the event that a BootEntry, OsProfile or HostProfile fails to load: boom.bootloader: raise exceptions in load_entries() if --debug boom: raise exceptions in load_profiles_for_class() if --debug This means that we can get first-fault debug data from customers directly without needing to provide a debug build or complex instructions. The other two fix two minor bugs: boom.cache: use correct variable in load_cache() warning branch Fixes an exception printing a warning message when a specific type of inconsistency is detected in the cache metadata. This is not expected to occur other than as a result of file system corruption or external modifications of the cacheindex.json file but but the patch improves error reporting & avoids the need for any manual cleanup in the event that it did occur. boom.bootloader: decode bytes output of grub2-editenv Fixes a Unicode decode error affecting Python3 (so only affects RHEL8) that causes an error loading "foreign" (i.e. non-boom managed) BLS boot entries. Normally these are loaded as read-only (so they can be cloned but not edited): without this patch these entries are not loaded since an exception is raised obtaining the value of Grub2 environment variables used in the entry definition. BEFORE: # boom list -V INFO - Could not load BootEntry '/boot/loader/entries/b49e623f993c4fd7b200c5e350f9cacc-4.18.0-107.el8.x86_64.conf': a bytes-like object is required, not 'str' INFO - Could not load BootEntry '/boot/loader/entries/b49e623f993c4fd7b200c5e350f9cacc-0-rescue.conf': a bytes-like object is required, not 'str' INFO - Could not load BootEntry '/boot/loader/entries/b49e623f993c4fd7b200c5e350f9cacc-4.18.0-167.el8.x86_64.conf': a bytes-like object is required, not 'str' BootID Version Name RootDevice Options MachineID b7e8a97 4.18.0-167.el8.x86_64 Red Hat Enterprise Linux /dev/rhel/root root=/dev/rhel/root ro rd.lvm.lv=rhel/root b49e623f9966 AFTER: # boom list -V INFO - Marking unknown boot entry as read-only: b49e623f993c4fd7b200c5e350f9cacc-4.18.0-107.el8.x86_64.conf INFO - Entry file name does not match boot_id: b49e623f993c4fd7b200c5e350f9cacc-0-rescue.conf INFO - Marking unknown boot entry as read-only: b49e623f993c4fd7b200c5e350f9cacc-4.18.0-167.el8.x86_64.conf BootID Version Name RootDevice Options MachineID 30c3610 4.18.0-107.el8.x86_64 Red Hat Enterprise Linux /dev/mapper/rhel-root $kernelopts $tuned_params d5479b2 4.18.0-167.el8.x86_64 Red Hat Enterprise Linux /dev/mapper/rhel-root $kernelopts $tuned_params b7e8a97 4.18.0-167.el8.x86_64 Red Hat Enterprise Linux /dev/rhel/root root=/dev/rhel/root BootEntry 30c3610 and d5479b2 are system-defined BLS entries that are affected by this bug. Adding the Rebase keyword. This was pm-acked by rhandlin so I guess it is the right thing to do. The test suite includes unit tests for the newly added functionality - there should be 100% pass with the included suite (~416 tests although the exact number may vary because some tests are skipped if the environment is unsuitable). On my system with current master I see: Ran 416 tests in 27.067s OK (skipped=4) And coverage looks like: $ coverage report Name Stmts Miss Cover ----------------------------------------- boom/__init__.py 5 0 100% boom/_boom.py 388 9 98% boom/bootloader.py 942 87 91% boom/cache.py 358 26 93% boom/command.py 1348 216 84% boom/config.py 118 26 78% boom/hostprofile.py 465 50 89% boom/legacy.py 167 48 71% boom/osprofile.py 606 75 88% boom/report.py 492 56 89% ----------------------------------------- TOTAL 4889 593 88% The cache-specific tests are in tests/cache_tests.py: there are currently 39 unit cases in that module. Thanks Bryn for the unit test results. Our sanity checks look good as well. Marking this rebase bug verified. boom-boot-1.2-2.el8 BUILT: Mon Jun 29 08:27:02 CDT 2020 kernel-4.18.0-234.el8 BUILT: Thu Aug 20 12:01:26 CDT 2020 lvm2-2.03.09-5.el8 BUILT: Wed Aug 12 15:51:50 CDT 2020 lvm2-libs-2.03.09-5.el8 BUILT: Wed Aug 12 15:51:50 CDT 2020 SCENARIO - [boom_sanity] Create a snapshot and perform some boom sanity only checks as this wont be an actual bootable snapshot Making origin volume lvcreate --yes -L 300M snapper -n origin Making snapshot of origin volume lvcreate --yes -s /dev/snapper/origin -c 128 -n boom_snap -L 100M Check 0 a. Verify a new profile can be created (this is basically a dup of the above cmd) boom profile create --from-host --uname-pattern Linux Check 0 b. Verify a new profile can be created automatically from just current host heuristic boom profile create --from-host Check 0 c. Verify a new profile can be created boom profile create --from-host --uname-pattern el8 Check 1. Verify boom not configured for grub yet warning (Bug 1542952) Boom already configured for grub. Skipping check. Generating grub configuration file ... done Check 2 a. Verify version numbers not matching an OS profile provide a helpful warning (Bug 1771584#c5) boom create --title nomatchversion --rootlv snapper/boom_snap --version 2.5.0 Check 2 b. Verify fake OS profiles are properly identified (Bug 1540266) boom create --profile fake_profile --title title --version 3.10-272.el7 --rootlv snapper/boom_snap WARNING - Boom grub2 integration is disabled in '/boot/../etc/default/boom' Check 3. Verify --rootlv and --root-device variations are both accepted (Bug 1543188) --rootlv: boom create --title 1 --rootlv /dev/snapper/boom_snap boom list b4d0685 BootID Version Name RootDevice b4d0685 4.18.0-234.el8.x86_64 Red Hat Enterprise Linux /dev/snapper/boom_snap boom entry show b4d0685 Boot Entry (boot_id=b4d0685) title 1 machine-id 4329f0e5c0004df9b09a707f827dc31c version 4.18.0-234.el8.x86_64 linux /vmlinuz-4.18.0-234.el8.x86_64 initrd /initramfs-4.18.0-234.el8.x86_64.img options root=/dev/snapper/boom_snap ro rd.lvm.lv=snapper/boom_snap rhgb quiet grub_users $grub_users grub_arg --unrestricted grub_class kernel boom delete b4d0685 --root-lv: boom create --title 2 --root-lv /dev/snapper/boom_snap boom list 91a537d BootID Version Name RootDevice 91a537d 4.18.0-234.el8.x86_64 Red Hat Enterprise Linux /dev/snapper/boom_snap boom entry show 91a537d Boot Entry (boot_id=91a537d) title 2 machine-id 4329f0e5c0004df9b09a707f827dc31c version 4.18.0-234.el8.x86_64 linux /vmlinuz-4.18.0-234.el8.x86_64 initrd /initramfs-4.18.0-234.el8.x86_64.img options root=/dev/snapper/boom_snap ro rd.lvm.lv=snapper/boom_snap rhgb quiet grub_users $grub_users grub_arg --unrestricted grub_class kernel boom delete 91a537d --rootdevice: boom create --title 3 --rootdevice /dev/snapper/boom_snap boom list 1c61767 BootID Version Name RootDevice 1c61767 4.18.0-234.el8.x86_64 Red Hat Enterprise Linux /dev/snapper/boom_snap boom entry show 1c61767 Boot Entry (boot_id=1c61767) title 3 machine-id 4329f0e5c0004df9b09a707f827dc31c version 4.18.0-234.el8.x86_64 linux /vmlinuz-4.18.0-234.el8.x86_64 initrd /initramfs-4.18.0-234.el8.x86_64.img options root=/dev/snapper/boom_snap ro rd.lvm.lv=snapper/boom_snap rhgb quiet grub_users $grub_users grub_arg --unrestricted grub_class kernel boom delete 1c61767 --root-device: boom create --title 4 --root-device /dev/snapper/boom_snap boom list 594f8c4 BootID Version Name RootDevice 594f8c4 4.18.0-234.el8.x86_64 Red Hat Enterprise Linux /dev/snapper/boom_snap boom entry show 594f8c4 Boot Entry (boot_id=594f8c4) title 4 machine-id 4329f0e5c0004df9b09a707f827dc31c version 4.18.0-234.el8.x86_64 linux /vmlinuz-4.18.0-234.el8.x86_64 initrd /initramfs-4.18.0-234.el8.x86_64.img options root=/dev/snapper/boom_snap ro rd.lvm.lv=snapper/boom_snap rhgb quiet grub_users $grub_users grub_arg --unrestricted grub_class kernel boom delete 594f8c4 Check 4. Verify --root-lv and --root-device verify the lv/device exists (Bug 1543186) boom create --title no-lv --root-lv snapper/nonlv WARNING - Boom grub2 integration is disabled in '/boot/../etc/default/boom' boom create --title no-dev --root-device /dev/sdxyz WARNING - Boom grub2 integration is disabled in '/boot/../etc/default/boom' boom create --title 5 --no-dev --root-lv snapper/nonlv WARNING - Boom grub2 integration is disabled in '/boot/../etc/default/boom' boom delete 623d07e Check 5. Verify proper use of user providing --rootlv /dev/* format (Bug 1544914) boom create --title 6 --rootlv /dev/snapper/boom_snap WARNING - Boom grub2 integration is disabled in '/boot/../etc/default/boom' Created entry with boot_id 4c059a8: title 6 machine-id 4329f0e5c0004df9b09a707f827dc31c version 4.18.0-234.el8.x86_64 linux /vmlinuz-4.18.0-234.el8.x86_64 initrd /initramfs-4.18.0-234.el8.x86_64.img options root=/dev/snapper/boom_snap ro rd.lvm.lv=snapper/boom_snap rhgb quiet grub_users $grub_users grub_arg --unrestricted grub_class kernel boom delete 4c059a8 Check 6. Verify proper rd.lvm.lv entry w/ --root-device option (Also listed in Bug 1544914) boom create --title 7 --root-device /dev/snapper/boom_snap WARNING - Boom grub2 integration is disabled in '/boot/../etc/default/boom' Created entry with boot_id 372913e: title 7 machine-id 4329f0e5c0004df9b09a707f827dc31c version 4.18.0-234.el8.x86_64 linux /vmlinuz-4.18.0-234.el8.x86_64 initrd /initramfs-4.18.0-234.el8.x86_64.img options root=/dev/snapper/boom_snap ro rd.lvm.lv=snapper/boom_snap rhgb quiet grub_users $grub_users grub_arg --unrestricted grub_class kernel regression check for bug 1840488 boom remove 372913e boom delete 372913e Check 7. Verify --grub-arg, grub-class, grub-users, and grub-id specified arguments (r7 rfe 1777364 / r8 rfe 1848123) Removing volume snapper/boom_snap Removing origin snapper/origin Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (boom-boot bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4532 |