Bug 1302220 - package-cleanup --oldkernels does not clean up rescue images
package-cleanup --oldkernels does not clean up rescue images
Status: NEW
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: dracut (Show other bugs)
7.2
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Lukáš Nykrýn
Release Test Team
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-27 03:56 EST by Robert Buchholz
Modified: 2017-08-21 04:36 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Robert Buchholz 2016-01-27 03:56:07 EST
Description of problem:
The boot partition as created by default on most of our infrastructure is less than 500 MB large. As the system is being maintained, rescue images keep piling up the boot partition overflows. They are not being cleaned even by "package-cleanup" which supposedly keeps only a certain number of kernels around.

Version-Release number of selected component (if applicable):
yum-utils-1.1.31-34.el7

How reproducible:
Install CentOS 7 a year ago

Steps to Reproduce:
1. Update every day
2. Run "package-cleanup --oldkernels --assumeyes" every day

Actual results:
/boot is filling up:
-rw-r--r--  1 root root   126430  5. Jan 17:17 config-3.10.0-327.4.4.el7.x86_64
-rw-r--r--  1 root root   126430 25. Jan 23:16 config-3.10.0-327.4.5.el7.x86_64
drwxr-xr-x. 2 root root     1024  2. Okt 10:30 grub
drwx------. 6 root root     1024 26. Jan 10:44 grub2
-rw-------  1 root root 43549536  7. Jan 10:31 initramfs-0-rescue-6ccdcb4d68f3455eb20f6fba65bacd9b.img
-rw-r--r--. 1 root root 25192418 10. Jul 2014  initramfs-0-rescue-74f877a555c841639922c44507d67b89.img
-rw-------  1 root root 37138965 10. Feb 2015  initramfs-0-rescue-7ca9daf9b6cb43bbaba0a50ac27211d5.img
-rw-------  1 root root 39797158  1. Apr 2015  initramfs-0-rescue-de83ff89d95b416f8b8c69758d0eb3cd.img
-rw-------  1 root root 39809543  7. Aug 10:31 initramfs-0-rescue-f038bc83c6ff40069c43de3e7b93a902.img
-rw-------  1 root root 42932055  7. Jan 10:30 initramfs-3.10.0-327.4.4.el7.x86_64.img
-rw-------  1 root root 42930922 26. Jan 10:30 initramfs-3.10.0-327.4.5.el7.x86_64.img
-rw-------  1 root root 18451242 26. Jan 10:36 initramfs-3.10.0-327.4.5.el7.x86_64kdump.img
-rw-r--r--. 1 root root   602522 15. Dez 10:35 initrd-plymouth.img
drwx------  2 root root    12288  5. Feb 2015  lost+found
-rw-r--r--  1 root root   252630  5. Jan 17:20 symvers-3.10.0-327.4.4.el7.x86_64.gz
-rw-r--r--  1 root root   252630 25. Jan 23:19 symvers-3.10.0-327.4.5.el7.x86_64.gz
-rw-------  1 root root  2963007  5. Jan 17:17 System.map-3.10.0-327.4.4.el7.x86_64
-rw-------  1 root root  2963007 25. Jan 23:16 System.map-3.10.0-327.4.5.el7.x86_64
-rwxr-xr-x  1 root root  5154640  7. Jan 10:31 vmlinuz-0-rescue-6ccdcb4d68f3455eb20f6fba65bacd9b
-rwxr-xr-x. 1 root root  4902656 10. Jul 2014  vmlinuz-0-rescue-74f877a555c841639922c44507d67b89
-rwxr-xr-x  1 root root  4906464 10. Feb 2015  vmlinuz-0-rescue-7ca9daf9b6cb43bbaba0a50ac27211d5
-rwxr-xr-x  1 root root  5029008  1. Apr 2015  vmlinuz-0-rescue-de83ff89d95b416f8b8c69758d0eb3cd
-rwxr-xr-x  1 root root  5029744  7. Aug 10:32 vmlinuz-0-rescue-f038bc83c6ff40069c43de3e7b93a902
-rwxr-xr-x  1 root root  5154640  5. Jan 17:17 vmlinuz-3.10.0-327.4.4.el7.x86_64
-rw-r--r--  1 root root      170  5. Jan 17:17 .vmlinuz-3.10.0-327.4.4.el7.x86_64.hmac
-rwxr-xr-x  1 root root  5154640 25. Jan 23:16 vmlinuz-3.10.0-327.4.5.el7.x86_64
-rw-r--r--  1 root root      170 25. Jan 23:16 .vmlinuz-3.10.0-327.4.5.el7.x86_64.hmac


Expected results:
/boot is not filling up

Additional info:
I am not exactly sure this is a bug in package-cleanup or in the kernel/dracut to create these rescue images in the first place.
Comment 2 Valentina Mukhamedzhanova 2016-02-12 09:39:25 EST
This is not a bug in package-cleanup, the old kernel packages get erased successfully. Rescue images don't belong to any package.

AFAIK there's supposed to be only one rescue image, I'm not sure how you ended up with several. Switching to dracut.
Comment 3 Robert Buchholz 2016-04-26 05:57:02 EDT
I've narrowed down this problem, but have not gotten to the root cause. It seems occasionally, some of our servers regenerate their machine-id. I cannot pinpoint exactly when and why this is happening. The rescue images are just a symptom.

The /etc/kernel/postinst.d/51-dracut-rescue-postinst.sh script makes sure that a rescue image exists for the given machine-id. The timing of the rescue images corresponds exactly to the kernel updates.
What baffles me is how+why the machine-id keeps changing. Here's some more information.

# ls -lt /boot/*rescue*
-rwxr-xr-x  1 root root  5154640  7. Jan 10:31 /boot/vmlinuz-0-rescue-83a7188675ae463388fb3cc58f4aa5ce
-rw-------  1 root root 43825394  7. Jan 10:31 /boot/initramfs-0-rescue-83a7188675ae463388fb3cc58f4aa5ce.img
-rwxr-xr-x  1 root root  5154832 15. Dez 10:36 /boot/vmlinuz-0-rescue-01fe33c960a14b1fbc3edf9edad103ec
-rw-------  1 root root 43824001 15. Dez 10:36 /boot/initramfs-0-rescue-01fe33c960a14b1fbc3edf9edad103ec.img
-rwxr-xr-x  1 root root  5029328  4. Nov 10:33 /boot/vmlinuz-0-rescue-45de284498324e81b2e6ab94921b6f04
-rw-------  1 root root 40060449  4. Nov 10:33 /boot/initramfs-0-rescue-45de284498324e81b2e6ab94921b6f04.img
-rwxr-xr-x  1 root root  5029232 16. Sep 2015  /boot/vmlinuz-0-rescue-9496a40d458a48b9b6beec488e78f83c
-rw-------  1 root root 40055878 16. Sep 2015  /boot/initramfs-0-rescue-9496a40d458a48b9b6beec488e78f83c.img
-rwxr-xr-x  1 root root  5029200 25. Jun 2015  /boot/vmlinuz-0-rescue-15dfc153fa2e45ee9f6d5522483fc1ae
-rw-------  1 root root 40051089 25. Jun 2015  /boot/initramfs-0-rescue-15dfc153fa2e45ee9f6d5522483fc1ae.img

# stat /etc/machine-id 
  File: ‘/etc/machine-id’
  Size: 33        	Blocks: 8          IO Block: 4096   regular file
Device: 902h/2306d	Inode: 49021059    Links: 1
Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2016-04-25 12:02:56.999783597 +0200
Modify: 2015-12-15 11:45:29.065688379 +0100
Change: 2015-12-15 11:45:29.065688379 +0100
 Birth: -

# grep kernel /var/log/yum.log*
Apr 30 15:09:22 Installed: kernel-headers-3.10.0-229.1.2.el7.x86_64
May 14 10:30:26 Updated: kernel-tools-libs-3.10.0-229.4.2.el7.x86_64
May 14 10:30:27 Updated: kernel-tools-3.10.0-229.4.2.el7.x86_64
May 14 10:30:27 Updated: kernel-headers-3.10.0-229.4.2.el7.x86_64
May 14 10:30:35 Installed: kernel-3.10.0-229.4.2.el7.x86_64
Jun 25 10:30:51 Updated: kernel-tools-libs-3.10.0-229.7.2.el7.x86_64
Jun 25 10:31:07 Updated: kernel-tools-3.10.0-229.7.2.el7.x86_64
Jun 25 10:31:20 Installed: kernel-3.10.0-229.7.2.el7.x86_64
Jun 25 10:31:22 Updated: kernel-headers-3.10.0-229.7.2.el7.x86_64
Aug 07 10:30:22 Updated: kernel-tools-libs-3.10.0-229.11.1.el7.x86_64
Aug 07 10:30:23 Updated: kernel-tools-3.10.0-229.11.1.el7.x86_64
Aug 07 10:30:40 Installed: kernel-3.10.0-229.11.1.el7.x86_64
Aug 07 10:30:43 Updated: kernel-headers-3.10.0-229.11.1.el7.x86_64
Sep 16 10:30:21 Updated: kernel-tools-libs-3.10.0-229.14.1.el7.x86_64
Sep 16 10:30:24 Updated: kernel-tools-3.10.0-229.14.1.el7.x86_64
Sep 16 10:30:46 Installed: kernel-3.10.0-229.14.1.el7.x86_64
Sep 16 10:30:47 Updated: kernel-headers-3.10.0-229.14.1.el7.x86_64
Sep 22 10:30:15 Erased: kernel
Sep 22 10:30:18 Erased: kernel
Sep 22 10:30:22 Erased: kernel
Nov 04 10:30:23 Updated: kernel-tools-libs-3.10.0-229.20.1.el7.x86_64
Nov 04 10:30:40 Updated: kernel-tools-3.10.0-229.20.1.el7.x86_64
Nov 04 10:30:41 Updated: kernel-headers-3.10.0-229.20.1.el7.x86_64
Nov 04 10:30:56 Installed: kernel-3.10.0-229.20.1.el7.x86_64
Nov 04 10:33:47 Erased: kernel-3.10.0-229.11.1.el7.x86_64
Dec 15 10:31:47 Updated: kernel-tools-libs-3.10.0-327.3.1.el7.x86_64
Dec 15 10:31:57 Updated: kernel-headers-3.10.0-327.3.1.el7.x86_64
Dec 15 10:32:10 Installed: kernel-3.10.0-327.3.1.el7.x86_64
Dec 15 10:32:13 Updated: kernel-tools-3.10.0-327.3.1.el7.x86_64
Dec 15 10:37:05 Erased: kernel-3.10.0-229.14.1.el7.x86_64
Jan 07 10:30:19 Updated: kernel-tools-libs-3.10.0-327.4.4.el7.x86_64
Jan 07 10:30:20 Updated: kernel-tools-3.10.0-327.4.4.el7.x86_64
Jan 07 10:30:29 Installed: kernel-3.10.0-327.4.4.el7.x86_64
Jan 07 10:30:31 Updated: kernel-headers-3.10.0-327.4.4.el7.x86_64
Jan 07 10:31:51 Erased: kernel-3.10.0-229.20.1.el7.x86_64
Jan 26 10:30:07 Updated: kernel-tools-libs-3.10.0-327.4.5.el7.x86_64
Jan 26 10:30:08 Updated: kernel-tools-3.10.0-327.4.5.el7.x86_64
Jan 26 10:30:17 Installed: kernel-3.10.0-327.4.5.el7.x86_64
Jan 26 10:30:17 Updated: kernel-headers-3.10.0-327.4.5.el7.x86_64
Jan 27 10:30:19 Erased: kernel-3.10.0-327.3.1.el7.x86_64
Feb 17 10:30:30 Updated: kernel-tools-libs-3.10.0-327.10.1.el7.x86_64
Feb 17 10:30:31 Updated: kernel-headers-3.10.0-327.10.1.el7.x86_64
Feb 17 10:30:34 Updated: kernel-tools-3.10.0-327.10.1.el7.x86_64
Feb 17 10:30:54 Installed: kernel-3.10.0-327.10.1.el7.x86_64
Feb 17 10:31:56 Erased: kernel-3.10.0-327.4.4.el7.x86_64
Apr 01 10:38:35 Updated: kernel-tools-libs-3.10.0-327.13.1.el7.x86_64
Apr 01 10:38:39 Updated: kernel-tools-3.10.0-327.13.1.el7.x86_64
Apr 01 10:38:49 Installed: kernel-3.10.0-327.13.1.el7.x86_64
Apr 01 10:38:52 Updated: kernel-headers-3.10.0-327.13.1.el7.x86_64
Apr 02 10:30:11 Erased: kernel-3.10.0-327.4.5.el7.x86_64

# grep systemd /var/log/yum.log*
May 13 10:30:11 Updated: systemd-libs-208-20.el7_1.3.x86_64
May 13 10:30:14 Updated: systemd-208-20.el7_1.3.x86_64
May 13 10:30:27 Updated: systemd-sysv-208-20.el7_1.3.x86_64
Jun 25 10:30:31 Updated: systemd-libs-208-20.el7_1.5.x86_64
Jun 25 10:30:34 Updated: systemd-208-20.el7_1.5.x86_64
Jun 25 10:31:10 Updated: systemd-sysv-208-20.el7_1.5.x86_64
Sep 16 10:30:13 Updated: systemd-libs-208-20.el7_1.6.x86_64
Sep 16 10:30:16 Updated: systemd-208-20.el7_1.6.x86_64
Sep 16 10:30:36 Updated: systemd-sysv-208-20.el7_1.6.x86_64
Dec 15 10:30:42 Updated: systemd-libs-219-19.el7.x86_64
Dec 15 10:31:35 Updated: systemd-219-19.el7.x86_64
Dec 15 10:31:37 Updated: systemd-sysv-219-19.el7.x86_64
Feb 17 10:30:21 Updated: systemd-libs-219-19.el7_2.4.x86_64
Feb 17 10:30:24 Updated: systemd-219-19.el7_2.4.x86_64
Feb 17 10:30:27 Updated: systemd-sysv-219-19.el7_2.4.x86_64
Apr 01 10:38:24 Updated: systemd-libs-219-19.el7_2.7.x86_64
Apr 01 10:38:28 Updated: systemd-219-19.el7_2.7.x86_64
Apr 01 10:38:49 Updated: systemd-sysv-219-19.el7_2.7.x86_64

Note You need to log in before you can comment on or make changes to this bug.