RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1302220 - package-cleanup --oldkernels does not clean up rescue images
Summary: package-cleanup --oldkernels does not clean up rescue images
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: dracut
Version: 7.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Lukáš Nykrýn
QA Contact: qe-baseos-daemons
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-01-27 08:56 UTC by Robert Buchholz
Modified: 2020-12-15 07:39 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-12-15 07:39:51 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Robert Buchholz 2016-01-27 08:56:07 UTC 
Description of problem:
The boot partition as created by default on most of our infrastructure is less than 500 MB large. As the system is being maintained, rescue images keep piling up the boot partition overflows. They are not being cleaned even by "package-cleanup" which supposedly keeps only a certain number of kernels around.

Version-Release number of selected component (if applicable):
yum-utils-1.1.31-34.el7

How reproducible:
Install CentOS 7 a year ago

Steps to Reproduce:
1. Update every day
2. Run "package-cleanup --oldkernels --assumeyes" every day

Actual results:
/boot is filling up:
-rw-r--r--  1 root root   126430  5. Jan 17:17 config-3.10.0-327.4.4.el7.x86_64
-rw-r--r--  1 root root   126430 25. Jan 23:16 config-3.10.0-327.4.5.el7.x86_64
drwxr-xr-x. 2 root root     1024  2. Okt 10:30 grub
drwx------. 6 root root     1024 26. Jan 10:44 grub2
-rw-------  1 root root 43549536  7. Jan 10:31 initramfs-0-rescue-6ccdcb4d68f3455eb20f6fba65bacd9b.img
-rw-r--r--. 1 root root 25192418 10. Jul 2014  initramfs-0-rescue-74f877a555c841639922c44507d67b89.img
-rw-------  1 root root 37138965 10. Feb 2015  initramfs-0-rescue-7ca9daf9b6cb43bbaba0a50ac27211d5.img
-rw-------  1 root root 39797158  1. Apr 2015  initramfs-0-rescue-de83ff89d95b416f8b8c69758d0eb3cd.img
-rw-------  1 root root 39809543  7. Aug 10:31 initramfs-0-rescue-f038bc83c6ff40069c43de3e7b93a902.img
-rw-------  1 root root 42932055  7. Jan 10:30 initramfs-3.10.0-327.4.4.el7.x86_64.img
-rw-------  1 root root 42930922 26. Jan 10:30 initramfs-3.10.0-327.4.5.el7.x86_64.img
-rw-------  1 root root 18451242 26. Jan 10:36 initramfs-3.10.0-327.4.5.el7.x86_64kdump.img
-rw-r--r--. 1 root root   602522 15. Dez 10:35 initrd-plymouth.img
drwx------  2 root root    12288  5. Feb 2015  lost+found
-rw-r--r--  1 root root   252630  5. Jan 17:20 symvers-3.10.0-327.4.4.el7.x86_64.gz
-rw-r--r--  1 root root   252630 25. Jan 23:19 symvers-3.10.0-327.4.5.el7.x86_64.gz
-rw-------  1 root root  2963007  5. Jan 17:17 System.map-3.10.0-327.4.4.el7.x86_64
-rw-------  1 root root  2963007 25. Jan 23:16 System.map-3.10.0-327.4.5.el7.x86_64
-rwxr-xr-x  1 root root  5154640  7. Jan 10:31 vmlinuz-0-rescue-6ccdcb4d68f3455eb20f6fba65bacd9b
-rwxr-xr-x. 1 root root  4902656 10. Jul 2014  vmlinuz-0-rescue-74f877a555c841639922c44507d67b89
-rwxr-xr-x  1 root root  4906464 10. Feb 2015  vmlinuz-0-rescue-7ca9daf9b6cb43bbaba0a50ac27211d5
-rwxr-xr-x  1 root root  5029008  1. Apr 2015  vmlinuz-0-rescue-de83ff89d95b416f8b8c69758d0eb3cd
-rwxr-xr-x  1 root root  5029744  7. Aug 10:32 vmlinuz-0-rescue-f038bc83c6ff40069c43de3e7b93a902
-rwxr-xr-x  1 root root  5154640  5. Jan 17:17 vmlinuz-3.10.0-327.4.4.el7.x86_64
-rw-r--r--  1 root root      170  5. Jan 17:17 .vmlinuz-3.10.0-327.4.4.el7.x86_64.hmac
-rwxr-xr-x  1 root root  5154640 25. Jan 23:16 vmlinuz-3.10.0-327.4.5.el7.x86_64
-rw-r--r--  1 root root      170 25. Jan 23:16 .vmlinuz-3.10.0-327.4.5.el7.x86_64.hmac


Expected results:
/boot is not filling up

Additional info:
I am not exactly sure this is a bug in package-cleanup or in the kernel/dracut to create these rescue images in the first place.
Comment 2 Valentina Mukhamedzhanova 2016-02-12 14:39:25 UTC 
This is not a bug in package-cleanup, the old kernel packages get erased successfully. Rescue images don't belong to any package.

AFAIK there's supposed to be only one rescue image, I'm not sure how you ended up with several. Switching to dracut.
Comment 3 Robert Buchholz 2016-04-26 09:57:02 UTC 
I've narrowed down this problem, but have not gotten to the root cause. It seems occasionally, some of our servers regenerate their machine-id. I cannot pinpoint exactly when and why this is happening. The rescue images are just a symptom.

The /etc/kernel/postinst.d/51-dracut-rescue-postinst.sh script makes sure that a rescue image exists for the given machine-id. The timing of the rescue images corresponds exactly to the kernel updates.
What baffles me is how+why the machine-id keeps changing. Here's some more information.

# ls -lt /boot/*rescue*
-rwxr-xr-x  1 root root  5154640  7. Jan 10:31 /boot/vmlinuz-0-rescue-83a7188675ae463388fb3cc58f4aa5ce
-rw-------  1 root root 43825394  7. Jan 10:31 /boot/initramfs-0-rescue-83a7188675ae463388fb3cc58f4aa5ce.img
-rwxr-xr-x  1 root root  5154832 15. Dez 10:36 /boot/vmlinuz-0-rescue-01fe33c960a14b1fbc3edf9edad103ec
-rw-------  1 root root 43824001 15. Dez 10:36 /boot/initramfs-0-rescue-01fe33c960a14b1fbc3edf9edad103ec.img
-rwxr-xr-x  1 root root  5029328  4. Nov 10:33 /boot/vmlinuz-0-rescue-45de284498324e81b2e6ab94921b6f04
-rw-------  1 root root 40060449  4. Nov 10:33 /boot/initramfs-0-rescue-45de284498324e81b2e6ab94921b6f04.img
-rwxr-xr-x  1 root root  5029232 16. Sep 2015  /boot/vmlinuz-0-rescue-9496a40d458a48b9b6beec488e78f83c
-rw-------  1 root root 40055878 16. Sep 2015  /boot/initramfs-0-rescue-9496a40d458a48b9b6beec488e78f83c.img
-rwxr-xr-x  1 root root  5029200 25. Jun 2015  /boot/vmlinuz-0-rescue-15dfc153fa2e45ee9f6d5522483fc1ae
-rw-------  1 root root 40051089 25. Jun 2015  /boot/initramfs-0-rescue-15dfc153fa2e45ee9f6d5522483fc1ae.img

# stat /etc/machine-id 
  File: ‘/etc/machine-id’
  Size: 33        	Blocks: 8          IO Block: 4096   regular file
Device: 902h/2306d	Inode: 49021059    Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2016-04-25 12:02:56.999783597 +0200
Modify: 2015-12-15 11:45:29.065688379 +0100
Change: 2015-12-15 11:45:29.065688379 +0100
 Birth: -

# grep kernel /var/log/yum.log*
Apr 30 15:09:22 Installed: kernel-headers-3.10.0-229.1.2.el7.x86_64
May 14 10:30:26 Updated: kernel-tools-libs-3.10.0-229.4.2.el7.x86_64
May 14 10:30:27 Updated: kernel-tools-3.10.0-229.4.2.el7.x86_64
May 14 10:30:27 Updated: kernel-headers-3.10.0-229.4.2.el7.x86_64
May 14 10:30:35 Installed: kernel-3.10.0-229.4.2.el7.x86_64
Jun 25 10:30:51 Updated: kernel-tools-libs-3.10.0-229.7.2.el7.x86_64
Jun 25 10:31:07 Updated: kernel-tools-3.10.0-229.7.2.el7.x86_64
Jun 25 10:31:20 Installed: kernel-3.10.0-229.7.2.el7.x86_64
Jun 25 10:31:22 Updated: kernel-headers-3.10.0-229.7.2.el7.x86_64
Aug 07 10:30:22 Updated: kernel-tools-libs-3.10.0-229.11.1.el7.x86_64
Aug 07 10:30:23 Updated: kernel-tools-3.10.0-229.11.1.el7.x86_64
Aug 07 10:30:40 Installed: kernel-3.10.0-229.11.1.el7.x86_64
Aug 07 10:30:43 Updated: kernel-headers-3.10.0-229.11.1.el7.x86_64
Sep 16 10:30:21 Updated: kernel-tools-libs-3.10.0-229.14.1.el7.x86_64
Sep 16 10:30:24 Updated: kernel-tools-3.10.0-229.14.1.el7.x86_64
Sep 16 10:30:46 Installed: kernel-3.10.0-229.14.1.el7.x86_64
Sep 16 10:30:47 Updated: kernel-headers-3.10.0-229.14.1.el7.x86_64
Sep 22 10:30:15 Erased: kernel
Sep 22 10:30:18 Erased: kernel
Sep 22 10:30:22 Erased: kernel
Nov 04 10:30:23 Updated: kernel-tools-libs-3.10.0-229.20.1.el7.x86_64
Nov 04 10:30:40 Updated: kernel-tools-3.10.0-229.20.1.el7.x86_64
Nov 04 10:30:41 Updated: kernel-headers-3.10.0-229.20.1.el7.x86_64
Nov 04 10:30:56 Installed: kernel-3.10.0-229.20.1.el7.x86_64
Nov 04 10:33:47 Erased: kernel-3.10.0-229.11.1.el7.x86_64
Dec 15 10:31:47 Updated: kernel-tools-libs-3.10.0-327.3.1.el7.x86_64
Dec 15 10:31:57 Updated: kernel-headers-3.10.0-327.3.1.el7.x86_64
Dec 15 10:32:10 Installed: kernel-3.10.0-327.3.1.el7.x86_64
Dec 15 10:32:13 Updated: kernel-tools-3.10.0-327.3.1.el7.x86_64
Dec 15 10:37:05 Erased: kernel-3.10.0-229.14.1.el7.x86_64
Jan 07 10:30:19 Updated: kernel-tools-libs-3.10.0-327.4.4.el7.x86_64
Jan 07 10:30:20 Updated: kernel-tools-3.10.0-327.4.4.el7.x86_64
Jan 07 10:30:29 Installed: kernel-3.10.0-327.4.4.el7.x86_64
Jan 07 10:30:31 Updated: kernel-headers-3.10.0-327.4.4.el7.x86_64
Jan 07 10:31:51 Erased: kernel-3.10.0-229.20.1.el7.x86_64
Jan 26 10:30:07 Updated: kernel-tools-libs-3.10.0-327.4.5.el7.x86_64
Jan 26 10:30:08 Updated: kernel-tools-3.10.0-327.4.5.el7.x86_64
Jan 26 10:30:17 Installed: kernel-3.10.0-327.4.5.el7.x86_64
Jan 26 10:30:17 Updated: kernel-headers-3.10.0-327.4.5.el7.x86_64
Jan 27 10:30:19 Erased: kernel-3.10.0-327.3.1.el7.x86_64
Feb 17 10:30:30 Updated: kernel-tools-libs-3.10.0-327.10.1.el7.x86_64
Feb 17 10:30:31 Updated: kernel-headers-3.10.0-327.10.1.el7.x86_64
Feb 17 10:30:34 Updated: kernel-tools-3.10.0-327.10.1.el7.x86_64
Feb 17 10:30:54 Installed: kernel-3.10.0-327.10.1.el7.x86_64
Feb 17 10:31:56 Erased: kernel-3.10.0-327.4.4.el7.x86_64
Apr 01 10:38:35 Updated: kernel-tools-libs-3.10.0-327.13.1.el7.x86_64
Apr 01 10:38:39 Updated: kernel-tools-3.10.0-327.13.1.el7.x86_64
Apr 01 10:38:49 Installed: kernel-3.10.0-327.13.1.el7.x86_64
Apr 01 10:38:52 Updated: kernel-headers-3.10.0-327.13.1.el7.x86_64
Apr 02 10:30:11 Erased: kernel-3.10.0-327.4.5.el7.x86_64

# grep systemd /var/log/yum.log*
May 13 10:30:11 Updated: systemd-libs-208-20.el7_1.3.x86_64
May 13 10:30:14 Updated: systemd-208-20.el7_1.3.x86_64
May 13 10:30:27 Updated: systemd-sysv-208-20.el7_1.3.x86_64
Jun 25 10:30:31 Updated: systemd-libs-208-20.el7_1.5.x86_64
Jun 25 10:30:34 Updated: systemd-208-20.el7_1.5.x86_64
Jun 25 10:31:10 Updated: systemd-sysv-208-20.el7_1.5.x86_64
Sep 16 10:30:13 Updated: systemd-libs-208-20.el7_1.6.x86_64
Sep 16 10:30:16 Updated: systemd-208-20.el7_1.6.x86_64
Sep 16 10:30:36 Updated: systemd-sysv-208-20.el7_1.6.x86_64
Dec 15 10:30:42 Updated: systemd-libs-219-19.el7.x86_64
Dec 15 10:31:35 Updated: systemd-219-19.el7.x86_64
Dec 15 10:31:37 Updated: systemd-sysv-219-19.el7.x86_64
Feb 17 10:30:21 Updated: systemd-libs-219-19.el7_2.4.x86_64
Feb 17 10:30:24 Updated: systemd-219-19.el7_2.4.x86_64
Feb 17 10:30:27 Updated: systemd-sysv-219-19.el7_2.4.x86_64
Apr 01 10:38:24 Updated: systemd-libs-219-19.el7_2.7.x86_64
Apr 01 10:38:28 Updated: systemd-219-19.el7_2.7.x86_64
Apr 01 10:38:49 Updated: systemd-sysv-219-19.el7_2.7.x86_64
Comment 7 RHEL Program Management 2020-12-15 07:39:51 UTC 
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
Note You need to log in before you can comment on or make changes to this bug.