Bug 1302220

Summary:	package-cleanup --oldkernels does not clean up rescue images
Product:	Red Hat Enterprise Linux 7	Reporter:	Robert Buchholz <rbu>
Component:	dracut	Assignee:	Lukáš Nykrýn <lnykryn>
Status:	CLOSED WONTFIX	QA Contact:	qe-baseos-daemons
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.2	CC:	dracut-maint-list, jan.public, spamfaenger
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-12-15 07:39:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Robert Buchholz 2016-01-27 08:56:07 UTC

Description of problem:
The boot partition as created by default on most of our infrastructure is less than 500 MB large. As the system is being maintained, rescue images keep piling up the boot partition overflows. They are not being cleaned even by "package-cleanup" which supposedly keeps only a certain number of kernels around.

Version-Release number of selected component (if applicable):
yum-utils-1.1.31-34.el7

How reproducible:
Install CentOS 7 a year ago

Steps to Reproduce:
1. Update every day
2. Run "package-cleanup --oldkernels --assumeyes" every day

Actual results:
/boot is filling up:
-rw-r--r--  1 root root   126430  5. Jan 17:17 config-3.10.0-327.4.4.el7.x86_64
-rw-r--r--  1 root root   126430 25. Jan 23:16 config-3.10.0-327.4.5.el7.x86_64
drwxr-xr-x. 2 root root     1024  2. Okt 10:30 grub
drwx------. 6 root root     1024 26. Jan 10:44 grub2
-rw-------  1 root root 43549536  7. Jan 10:31 initramfs-0-rescue-6ccdcb4d68f3455eb20f6fba65bacd9b.img
-rw-r--r--. 1 root root 25192418 10. Jul 2014  initramfs-0-rescue-74f877a555c841639922c44507d67b89.img
-rw-------  1 root root 37138965 10. Feb 2015  initramfs-0-rescue-7ca9daf9b6cb43bbaba0a50ac27211d5.img
-rw-------  1 root root 39797158  1. Apr 2015  initramfs-0-rescue-de83ff89d95b416f8b8c69758d0eb3cd.img
-rw-------  1 root root 39809543  7. Aug 10:31 initramfs-0-rescue-f038bc83c6ff40069c43de3e7b93a902.img
-rw-------  1 root root 42932055  7. Jan 10:30 initramfs-3.10.0-327.4.4.el7.x86_64.img
-rw-------  1 root root 42930922 26. Jan 10:30 initramfs-3.10.0-327.4.5.el7.x86_64.img
-rw-------  1 root root 18451242 26. Jan 10:36 initramfs-3.10.0-327.4.5.el7.x86_64kdump.img
-rw-r--r--. 1 root root   602522 15. Dez 10:35 initrd-plymouth.img
drwx------  2 root root    12288  5. Feb 2015  lost+found
-rw-r--r--  1 root root   252630  5. Jan 17:20 symvers-3.10.0-327.4.4.el7.x86_64.gz
-rw-r--r--  1 root root   252630 25. Jan 23:19 symvers-3.10.0-327.4.5.el7.x86_64.gz
-rw-------  1 root root  2963007  5. Jan 17:17 System.map-3.10.0-327.4.4.el7.x86_64
-rw-------  1 root root  2963007 25. Jan 23:16 System.map-3.10.0-327.4.5.el7.x86_64
-rwxr-xr-x  1 root root  5154640  7. Jan 10:31 vmlinuz-0-rescue-6ccdcb4d68f3455eb20f6fba65bacd9b
-rwxr-xr-x. 1 root root  4902656 10. Jul 2014  vmlinuz-0-rescue-74f877a555c841639922c44507d67b89
-rwxr-xr-x  1 root root  4906464 10. Feb 2015  vmlinuz-0-rescue-7ca9daf9b6cb43bbaba0a50ac27211d5
-rwxr-xr-x  1 root root  5029008  1. Apr 2015  vmlinuz-0-rescue-de83ff89d95b416f8b8c69758d0eb3cd
-rwxr-xr-x  1 root root  5029744  7. Aug 10:32 vmlinuz-0-rescue-f038bc83c6ff40069c43de3e7b93a902
-rwxr-xr-x  1 root root  5154640  5. Jan 17:17 vmlinuz-3.10.0-327.4.4.el7.x86_64
-rw-r--r--  1 root root      170  5. Jan 17:17 .vmlinuz-3.10.0-327.4.4.el7.x86_64.hmac
-rwxr-xr-x  1 root root  5154640 25. Jan 23:16 vmlinuz-3.10.0-327.4.5.el7.x86_64
-rw-r--r--  1 root root      170 25. Jan 23:16 .vmlinuz-3.10.0-327.4.5.el7.x86_64.hmac


Expected results:
/boot is not filling up

Additional info:
I am not exactly sure this is a bug in package-cleanup or in the kernel/dracut to create these rescue images in the first place.

Comment 2 Valentina Mukhamedzhanova 2016-02-12 14:39:25 UTC

This is not a bug in package-cleanup, the old kernel packages get erased successfully. Rescue images don't belong to any package.

AFAIK there's supposed to be only one rescue image, I'm not sure how you ended up with several. Switching to dracut.

Comment 3 Robert Buchholz 2016-04-26 09:57:02 UTC

I've narrowed down this problem, but have not gotten to the root cause. It seems occasionally, some of our servers regenerate their machine-id. I cannot pinpoint exactly when and why this is happening. The rescue images are just a symptom.

The /etc/kernel/postinst.d/51-dracut-rescue-postinst.sh script makes sure that a rescue image exists for the given machine-id. The timing of the rescue images corresponds exactly to the kernel updates.
What baffles me is how+why the machine-id keeps changing. Here's some more information.

# ls -lt /boot/*rescue*
-rwxr-xr-x  1 root root  5154640  7. Jan 10:31 /boot/vmlinuz-0-rescue-83a7188675ae463388fb3cc58f4aa5ce
-rw-------  1 root root 43825394  7. Jan 10:31 /boot/initramfs-0-rescue-83a7188675ae463388fb3cc58f4aa5ce.img
-rwxr-xr-x  1 root root  5154832 15. Dez 10:36 /boot/vmlinuz-0-rescue-01fe33c960a14b1fbc3edf9edad103ec
-rw-------  1 root root 43824001 15. Dez 10:36 /boot/initramfs-0-rescue-01fe33c960a14b1fbc3edf9edad103ec.img
-rwxr-xr-x  1 root root  5029328  4. Nov 10:33 /boot/vmlinuz-0-rescue-45de284498324e81b2e6ab94921b6f04
-rw-------  1 root root 40060449  4. Nov 10:33 /boot/initramfs-0-rescue-45de284498324e81b2e6ab94921b6f04.img
-rwxr-xr-x  1 root root  5029232 16. Sep 2015  /boot/vmlinuz-0-rescue-9496a40d458a48b9b6beec488e78f83c
-rw-------  1 root root 40055878 16. Sep 2015  /boot/initramfs-0-rescue-9496a40d458a48b9b6beec488e78f83c.img
-rwxr-xr-x  1 root root  5029200 25. Jun 2015  /boot/vmlinuz-0-rescue-15dfc153fa2e45ee9f6d5522483fc1ae
-rw-------  1 root root 40051089 25. Jun 2015  /boot/initramfs-0-rescue-15dfc153fa2e45ee9f6d5522483fc1ae.img

# stat /etc/machine-id 
  File: ‘/etc/machine-id’
  Size: 33        	Blocks: 8          IO Block: 4096   regular file
Device: 902h/2306d	Inode: 49021059    Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2016-04-25 12:02:56.999783597 +0200
Modify: 2015-12-15 11:45:29.065688379 +0100
Change: 2015-12-15 11:45:29.065688379 +0100
 Birth: -

# grep kernel /var/log/yum.log*
Apr 30 15:09:22 Installed: kernel-headers-3.10.0-229.1.2.el7.x86_64
May 14 10:30:26 Updated: kernel-tools-libs-3.10.0-229.4.2.el7.x86_64
May 14 10:30:27 Updated: kernel-tools-3.10.0-229.4.2.el7.x86_64
May 14 10:30:27 Updated: kernel-headers-3.10.0-229.4.2.el7.x86_64
May 14 10:30:35 Installed: kernel-3.10.0-229.4.2.el7.x86_64
Jun 25 10:30:51 Updated: kernel-tools-libs-3.10.0-229.7.2.el7.x86_64
Jun 25 10:31:07 Updated: kernel-tools-3.10.0-229.7.2.el7.x86_64
Jun 25 10:31:20 Installed: kernel-3.10.0-229.7.2.el7.x86_64
Jun 25 10:31:22 Updated: kernel-headers-3.10.0-229.7.2.el7.x86_64
Aug 07 10:30:22 Updated: kernel-tools-libs-3.10.0-229.11.1.el7.x86_64
Aug 07 10:30:23 Updated: kernel-tools-3.10.0-229.11.1.el7.x86_64
Aug 07 10:30:40 Installed: kernel-3.10.0-229.11.1.el7.x86_64
Aug 07 10:30:43 Updated: kernel-headers-3.10.0-229.11.1.el7.x86_64
Sep 16 10:30:21 Updated: kernel-tools-libs-3.10.0-229.14.1.el7.x86_64
Sep 16 10:30:24 Updated: kernel-tools-3.10.0-229.14.1.el7.x86_64
Sep 16 10:30:46 Installed: kernel-3.10.0-229.14.1.el7.x86_64
Sep 16 10:30:47 Updated: kernel-headers-3.10.0-229.14.1.el7.x86_64
Sep 22 10:30:15 Erased: kernel
Sep 22 10:30:18 Erased: kernel
Sep 22 10:30:22 Erased: kernel
Nov 04 10:30:23 Updated: kernel-tools-libs-3.10.0-229.20.1.el7.x86_64
Nov 04 10:30:40 Updated: kernel-tools-3.10.0-229.20.1.el7.x86_64
Nov 04 10:30:41 Updated: kernel-headers-3.10.0-229.20.1.el7.x86_64
Nov 04 10:30:56 Installed: kernel-3.10.0-229.20.1.el7.x86_64
Nov 04 10:33:47 Erased: kernel-3.10.0-229.11.1.el7.x86_64
Dec 15 10:31:47 Updated: kernel-tools-libs-3.10.0-327.3.1.el7.x86_64
Dec 15 10:31:57 Updated: kernel-headers-3.10.0-327.3.1.el7.x86_64
Dec 15 10:32:10 Installed: kernel-3.10.0-327.3.1.el7.x86_64
Dec 15 10:32:13 Updated: kernel-tools-3.10.0-327.3.1.el7.x86_64
Dec 15 10:37:05 Erased: kernel-3.10.0-229.14.1.el7.x86_64
Jan 07 10:30:19 Updated: kernel-tools-libs-3.10.0-327.4.4.el7.x86_64
Jan 07 10:30:20 Updated: kernel-tools-3.10.0-327.4.4.el7.x86_64
Jan 07 10:30:29 Installed: kernel-3.10.0-327.4.4.el7.x86_64
Jan 07 10:30:31 Updated: kernel-headers-3.10.0-327.4.4.el7.x86_64
Jan 07 10:31:51 Erased: kernel-3.10.0-229.20.1.el7.x86_64
Jan 26 10:30:07 Updated: kernel-tools-libs-3.10.0-327.4.5.el7.x86_64
Jan 26 10:30:08 Updated: kernel-tools-3.10.0-327.4.5.el7.x86_64
Jan 26 10:30:17 Installed: kernel-3.10.0-327.4.5.el7.x86_64
Jan 26 10:30:17 Updated: kernel-headers-3.10.0-327.4.5.el7.x86_64
Jan 27 10:30:19 Erased: kernel-3.10.0-327.3.1.el7.x86_64
Feb 17 10:30:30 Updated: kernel-tools-libs-3.10.0-327.10.1.el7.x86_64
Feb 17 10:30:31 Updated: kernel-headers-3.10.0-327.10.1.el7.x86_64
Feb 17 10:30:34 Updated: kernel-tools-3.10.0-327.10.1.el7.x86_64
Feb 17 10:30:54 Installed: kernel-3.10.0-327.10.1.el7.x86_64
Feb 17 10:31:56 Erased: kernel-3.10.0-327.4.4.el7.x86_64
Apr 01 10:38:35 Updated: kernel-tools-libs-3.10.0-327.13.1.el7.x86_64
Apr 01 10:38:39 Updated: kernel-tools-3.10.0-327.13.1.el7.x86_64
Apr 01 10:38:49 Installed: kernel-3.10.0-327.13.1.el7.x86_64
Apr 01 10:38:52 Updated: kernel-headers-3.10.0-327.13.1.el7.x86_64
Apr 02 10:30:11 Erased: kernel-3.10.0-327.4.5.el7.x86_64

# grep systemd /var/log/yum.log*
May 13 10:30:11 Updated: systemd-libs-208-20.el7_1.3.x86_64
May 13 10:30:14 Updated: systemd-208-20.el7_1.3.x86_64
May 13 10:30:27 Updated: systemd-sysv-208-20.el7_1.3.x86_64
Jun 25 10:30:31 Updated: systemd-libs-208-20.el7_1.5.x86_64
Jun 25 10:30:34 Updated: systemd-208-20.el7_1.5.x86_64
Jun 25 10:31:10 Updated: systemd-sysv-208-20.el7_1.5.x86_64
Sep 16 10:30:13 Updated: systemd-libs-208-20.el7_1.6.x86_64
Sep 16 10:30:16 Updated: systemd-208-20.el7_1.6.x86_64
Sep 16 10:30:36 Updated: systemd-sysv-208-20.el7_1.6.x86_64
Dec 15 10:30:42 Updated: systemd-libs-219-19.el7.x86_64
Dec 15 10:31:35 Updated: systemd-219-19.el7.x86_64
Dec 15 10:31:37 Updated: systemd-sysv-219-19.el7.x86_64
Feb 17 10:30:21 Updated: systemd-libs-219-19.el7_2.4.x86_64
Feb 17 10:30:24 Updated: systemd-219-19.el7_2.4.x86_64
Feb 17 10:30:27 Updated: systemd-sysv-219-19.el7_2.4.x86_64
Apr 01 10:38:24 Updated: systemd-libs-219-19.el7_2.7.x86_64
Apr 01 10:38:28 Updated: systemd-219-19.el7_2.7.x86_64
Apr 01 10:38:49 Updated: systemd-sysv-219-19.el7_2.7.x86_64

Comment 7 RHEL Program Management 2020-12-15 07:39:51 UTC

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.