1533931 – the /boot partition grows after each update until it's at 100% causing boot loop.

Bug 1533931 - the /boot partition grows after each update until it's at 100% causing boot loop.

Summary: the /boot partition grows after each update until it's at 100% causing boot l...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Virtualization Manager
Classification:	Red Hat
Component:	imgbased
Sub Component:
Version:	4.1.9
Hardware:	All
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	ovirt-4.1.9
Target Release:	---
Assignee:	Ryan Barry
QA Contact:	Huijuan Zhao
Docs Contact:
URL:
Whiteboard:
Depends On:	1533871
Blocks:
TreeView+	depends on / blocked

Reported:	2018-01-12 15:01 UTC by Federico Sun
Modified:	2021-03-11 19:39 UTC (History)
CC List:	20 users (show)
Fixed In Version:	imgbased-0.9.52-0.1.el7ev
Doc Type:	Bug Fix
Doc Text:	Previously, Red Hat Virtualization Host (RHVH) copied the kernel and initrd from a subdirectory into /boot to mitigate limitations in some platform utilities, however, these files were not cleaned up after RHVH layers were removed. As a result, after a large number of updates, /boot could fill up, leaving the system in an unbootable state. In this release, RHVH cleans extraneous boot files after layer are removed, which prevents /boot from reaching full capacity.
Clone Of:	1533871
Environment:
Last Closed:	2018-02-01 15:06:17 UTC
oVirt Team:	Node
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Knowledge Base (Solution)	3319511	None	None	None	2018-01-12 20:57:31 UTC
Red Hat Product Errata	RHBA-2018:0263	normal	SHIPPED_LIVE	redhat-virtualization-host bug fix, and enhancement update for RHV 4.1.9	2018-02-01 20:06:45 UTC
oVirt gerrit	86283	master	MERGED	osupdater: clean up the files from removed boot dirs	2018-01-15 12:35:10 UTC
oVirt gerrit	86350	ovirt-4.2	MERGED	osupdater: clean up the files from removed boot dirs	2018-01-15 12:35:40 UTC
oVirt gerrit	86351	ovirt-4.1	MERGED	osupdater: clean up the files from removed boot dirs	2018-01-15 12:36:57 UTC

Description Federico Sun 2018-01-12 15:01:01 UTC

+++ This bug was initially created as a clone of Bug #1533871 +++

Description of problem:

Starting with RHVH 4.1 when updating to a new version, it will remove the oldest /boot/rhvh-4.1-xxxx but failed to clean up the kernel/systemmap/initramfs that are in /boot.

This will eventually leads to /boot using 100% and fail to apply any new upgrades. 


Version-Release number of selected component (if applicable):

all RHVH 4.1 images

How reproducible:

100%

Steps to Reproduce:
1. Install from scratch with RHVH-4.1.0-20170417.0.iso. Take note of the content of /boot and its size.

2. install update from 4.1-20170616.0.el7_3. Newer version vmlinuz/system.map/initramfs is placed under /boot. The size grows.


3. install next update 4.1-20170706.0.el7_3. Same behavior. 


Actual results:

 /boot partition usage is at 100%. Updates don't fail to apply. Upon reboot, it enters a boot loop.


Expected results:

With each update it would clean up the older kernels under /boot or not putting them there at all. 


Additional info:

Not affecting 4.0. Because it does not place vmlinuz/system.map/initramfs under /boot

See attached rhvh41_boot_partition_leaking.txt for full output.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2018-01-12 09:55:53 EST ---

Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

Comment 2 Huijuan Zhao 2018-01-15 08:11:48 UTC

QE can reproduce this issue.

Comment 4 Huijuan Zhao 2018-01-16 07:07:06 UTC

This issue is fixed in imgbased-0.9.52-0.1.el7ev.noarch. 

Test version:
Version 1: rhvh-4.1-0.20170417.0
Version 2: rhvh-4.1-0.20170616.0
Version 3: rhvh-4.1-0.20170706.0
Version 4: rhvh-4.1-0.20180115.0

Test steps:
1. First install rhvh-4.1-0.20170417.0
[root@test41 ~]# imgbase layout
rhvh-4.1-0.20170417.0
 +- rhvh-4.1-0.20170417.0+1

[root@dhcp-8-155 ~]# df -h /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       976M  165M  745M  19% /boot

[root@dhcp-8-155 ~]# ls -l /boot
total 84212
-rw-r--r--. 1 root root   137701 Jan 15 22:28 config-3.10.0-514.16.1.el7.x86_64
drwxr-xr-x. 3 root root     4096 Apr 17  2017 efi
-rw-r--r--. 1 root root   192572 Apr  5  2016 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Apr 17  2017 extlinux
drwx------. 6 root root     4096 Jan 15 22:32 grub2
-rw-------. 1 root root 54997252 Jan 15 22:32 initramfs-3.10.0-514.16.1.el7.x86_64.img
-rw-------. 1 root root 20904865 Jan 15 22:33 initramfs-3.10.0-514.16.1.el7.x86_64kdump.img
-rw-r--r--. 1 root root   613623 Apr 17  2017 initrd-plymouth.img
drwx------. 2 root root    16384 Jan 15 22:24 lost+found
-rw-r--r--. 1 root root   190896 Apr  5  2016 memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Jan 15 22:29 rhvh-4.1-0.20170417.0+1
-rw-r--r--. 1 root root   277943 Jan 15 22:28 symvers-3.10.0-514.16.1.el7.x86_64.gz
-rw-------. 1 root root  3113648 Jan 15 22:28 System.map-3.10.0-514.16.1.el7.x86_64
-rw-r--r--. 1 root root   342147 Aug  9  2016 tboot.gz
-rw-r--r--. 1 root root    13062 Aug  9  2016 tboot-syms
-rwxr-xr-x. 1 root root  5394272 Jan 15 22:32 vmlinuz-3.10.0-514.16.1.el7.x86_64

2. Then upgrade host to rhvh-4.1-0.20170616.0
[root@dhcp-8-155 ~]# imgbase w
[INFO] You are on rhvh-4.1-0.20170616.0+1
[root@dhcp-8-155 ~]# imgbase layout
rhvh-4.1-0.20170417.0
 +- rhvh-4.1-0.20170417.0+1
rhvh-4.1-0.20170616.0
 +- rhvh-4.1-0.20170616.0+1

[root@dhcp-8-155 ~]# df -h /boot
Filesystem                                 Size  Used Avail Use% Mounted on
/dev/mapper/TOSHIBA_DT01ACA100_33A6GMAMS1  976M  310M  600M  35% /boot

[root@dhcp-8-155 ~]# ls -l /boot/
total 169760
-rw-r--r--. 1 root root   137701 Jan 15 22:28 config-3.10.0-514.16.1.el7.x86_64
-rw-r--r--. 1 root root   137701 Jan 16 00:57 config-3.10.0-514.21.2.el7.x86_64
drwxr-xr-x. 3 root root     4096 Apr 17  2017 efi
-rw-r--r--. 1 root root   192572 Apr  5  2016 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Apr 17  2017 extlinux
drwx------. 6 root root     4096 Jan 16 00:58 grub2
-rw-------. 1 root root 54997252 Jan 15 22:32 initramfs-3.10.0-514.16.1.el7.x86_64.img
-rw-------. 1 root root 20904865 Jan 15 22:33 initramfs-3.10.0-514.16.1.el7.x86_64kdump.img
-rw-------. 1 root root 55009837 Jan 16 01:03 initramfs-3.10.0-514.21.2.el7.x86_64.img
-rw-------. 1 root root 23648229 Jan 16 01:04 initramfs-3.10.0-514.21.2.el7.x86_64kdump.img
-rw-r--r--. 1 root root   613623 Apr 17  2017 initrd-plymouth.img
drwx------. 2 root root    16384 Jan 15 22:24 lost+found
-rw-r--r--. 1 root root   190896 Apr  5  2016 memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Jan 15 22:29 rhvh-4.1-0.20170417.0+1
drwxr-xr-x. 2 root root     4096 Jan 16 00:58 rhvh-4.1-0.20170616.0+1
-rw-r--r--. 1 root root   277943 Jan 15 22:28 symvers-3.10.0-514.16.1.el7.x86_64.gz
-rw-r--r--. 1 root root   277955 Jan 16 00:57 symvers-3.10.0-514.21.2.el7.x86_64.gz
-rw-------. 1 root root  3113648 Jan 15 22:28 System.map-3.10.0-514.16.1.el7.x86_64
-rw-------. 1 root root  3114214 Jan 16 00:57 System.map-3.10.0-514.21.2.el7.x86_64
-rw-r--r--. 1 root root   342147 Aug  9  2016 tboot.gz
-rw-r--r--. 1 root root    13062 Aug  9  2016 tboot-syms
-rwxr-xr-x. 1 root root  5394272 Jan 15 22:32 vmlinuz-3.10.0-514.16.1.el7.x86_64
-rwxr-xr-x. 1 root root  5396288 Jan 16 01:03 vmlinuz-3.10.0-514.21.2.el7.x86_64

3. Then upgrade to rhvh-4.1-0.20170706.0
[root@dhcp-8-155 ~]# imgbase w
2018-01-16 01:30:39,424 [INFO] You are on rhvh-4.1-0.20170706.0+1
[root@dhcp-8-155 ~]# imgbase layout
rhvh-4.1-0.20170616.0
 +- rhvh-4.1-0.20170616.0+1
rhvh-4.1-0.20170706.0
 +- rhvh-4.1-0.20170706.0+1

[root@dhcp-8-155 ~]# df -h /boot
Filesystem                                 Size  Used Avail Use% Mounted on
/dev/mapper/TOSHIBA_DT01ACA100_33A6GMAMS1  976M  371M  539M  41% /boot

[root@dhcp-8-155 ~]# ls -l /boot
total 232200
-rw-r--r--. 1 root root   137701 Jan 15 22:28 config-3.10.0-514.16.1.el7.x86_64
-rw-r--r--. 1 root root   137701 Jan 16 00:57 config-3.10.0-514.21.2.el7.x86_64
-rw-r--r--. 1 root root   137701 Jan 16 01:17 config-3.10.0-514.26.1.el7.x86_64
drwxr-xr-x. 3 root root     4096 Apr 17  2017 efi
-rw-r--r--. 1 root root   192572 Apr  5  2016 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Apr 17  2017 extlinux
drwx------. 6 root root     4096 Jan 16 01:20 grub2
-rw-------. 1 root root 54997252 Jan 15 22:32 initramfs-3.10.0-514.16.1.el7.x86_64.img
-rw-------. 1 root root 20904865 Jan 15 22:33 initramfs-3.10.0-514.16.1.el7.x86_64kdump.img
-rw-------. 1 root root 55009837 Jan 16 01:03 initramfs-3.10.0-514.21.2.el7.x86_64.img
-rw-------. 1 root root 23648229 Jan 16 01:04 initramfs-3.10.0-514.21.2.el7.x86_64kdump.img
-rw-------. 1 root root 55004925 Jan 16 01:29 initramfs-3.10.0-514.26.1.el7.x86_64.img
-rw-r--r--. 1 root root   613623 Apr 17  2017 initrd-plymouth.img
drwx------. 2 root root    16384 Jan 15 22:24 lost+found
-rw-r--r--. 1 root root   190896 Apr  5  2016 memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Jan 16 00:58 rhvh-4.1-0.20170616.0+1
drwxr-xr-x. 2 root root     4096 Jan 16 01:18 rhvh-4.1-0.20170706.0+1
-rw-r--r--. 1 root root   277943 Jan 15 22:28 symvers-3.10.0-514.16.1.el7.x86_64.gz
-rw-r--r--. 1 root root   277955 Jan 16 00:57 symvers-3.10.0-514.21.2.el7.x86_64.gz
-rw-r--r--. 1 root root   277955 Jan 16 01:17 symvers-3.10.0-514.26.1.el7.x86_64.gz
-rw-------. 1 root root  3113648 Jan 15 22:28 System.map-3.10.0-514.16.1.el7.x86_64
-rw-------. 1 root root  3114214 Jan 16 00:57 System.map-3.10.0-514.21.2.el7.x86_64
-rw-------. 1 root root  3114429 Jan 16 01:17 System.map-3.10.0-514.26.1.el7.x86_64
-rw-r--r--. 1 root root   342147 Aug  9  2016 tboot.gz
-rw-r--r--. 1 root root    13062 Aug  9  2016 tboot-syms
-rwxr-xr-x. 1 root root  5394272 Jan 15 22:32 vmlinuz-3.10.0-514.16.1.el7.x86_64
-rwxr-xr-x. 1 root root  5396288 Jan 16 01:03 vmlinuz-3.10.0-514.21.2.el7.x86_64
-rwxr-xr-x. 1 root root  5395424 Jan 16 01:29 vmlinuz-3.10.0-514.26.1.el7.x86_64

4. Then upgrade to rhvh-4.1-0.20180115.0
[root@dhcp-8-155 ~]# imgbase w
You are on rhvh-4.1-0.20180115.0+1
[root@dhcp-8-155 ~]# imgbase layout
rhvh-4.1-0.20170706.0
 +- rhvh-4.1-0.20170706.0+1
rhvh-4.1-0.20180115.0
 +- rhvh-4.1-0.20180115.0+1

[root@dhcp-8-155 ~]# df -h /boot
Filesystem                                 Size  Used Avail Use% Mounted on
/dev/mapper/TOSHIBA_DT01ACA100_33A6GMAMS1  976M  403M  506M  45% /boot

[root@dhcp-8-155 ~]# ls -l /boot/
total 260120
-rw-r--r--. 1 root root   137701 Jan 15 22:28 config-3.10.0-514.16.1.el7.x86_64
-rw-r--r--. 1 root root   137701 Jan 16 01:17 config-3.10.0-514.26.1.el7.x86_64
-rw-r--r--. 1 root root   140915 Jan 16 01:40 config-3.10.0-693.11.6.el7.x86_64
drwxr-xr-x. 3 root root     4096 Apr 17  2017 efi
-rw-r--r--. 1 root root   192572 Apr  5  2016 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Apr 17  2017 extlinux
drwx------. 6 root root     4096 Jan 16 01:41 grub2
-rw-------. 1 root root 54997252 Jan 15 22:32 initramfs-3.10.0-514.16.1.el7.x86_64.img
-rw-------. 1 root root 20904865 Jan 15 22:33 initramfs-3.10.0-514.16.1.el7.x86_64kdump.img
-rw-------. 1 root root 55004925 Jan 16 01:29 initramfs-3.10.0-514.26.1.el7.x86_64.img
-rw-------. 1 root root 23641824 Jan 16 01:31 initramfs-3.10.0-514.26.1.el7.x86_64kdump.img
-rw-------. 1 root root 60098747 Jan 16 01:44 initramfs-3.10.0-693.11.6.el7.x86_64.img
-rw-------. 1 root root 22877401 Jan 16 01:45 initramfs-3.10.0-693.11.6.el7.x86_64kdump.img
-rw-r--r--. 1 root root   613623 Apr 17  2017 initrd-plymouth.img
drwx------. 2 root root    16384 Jan 15 22:24 lost+found
-rw-r--r--. 1 root root   190896 Apr  5  2016 memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Jan 16 01:18 rhvh-4.1-0.20170706.0+1
drwxr-xr-x. 2 root root     4096 Jan 16 01:40 rhvh-4.1-0.20180115.0+1
-rw-r--r--. 1 root root   277943 Jan 15 22:28 symvers-3.10.0-514.16.1.el7.x86_64.gz
-rw-r--r--. 1 root root   277955 Jan 16 01:17 symvers-3.10.0-514.26.1.el7.x86_64.gz
-rw-r--r--. 1 root root   293110 Jan 16 01:40 symvers-3.10.0-693.11.6.el7.x86_64.gz
-rw-------. 1 root root  3113648 Jan 15 22:28 System.map-3.10.0-514.16.1.el7.x86_64
-rw-------. 1 root root  3114429 Jan 16 01:17 System.map-3.10.0-514.26.1.el7.x86_64
-rw-------. 1 root root  3232490 Jan 16 01:40 System.map-3.10.0-693.11.6.el7.x86_64
-rw-r--r--. 1 root root   342147 Aug  9  2016 tboot.gz
-rw-r--r--. 1 root root    13062 Aug  9  2016 tboot-syms
-rwxr-xr-x. 1 root root  5394272 Jan 15 22:32 vmlinuz-3.10.0-514.16.1.el7.x86_64
-rwxr-xr-x. 1 root root  5395424 Jan 16 01:29 vmlinuz-3.10.0-514.26.1.el7.x86_64
-rwxr-xr-x. 1 root root  5889712 Jan 16 01:44 vmlinuz-3.10.0-693.11.6.el7.x86_64


Test results:
The files related with rhvh-4.1-0.20170616.0 were deleted in /boot.
But files related with rhvh-4.1-0.20170417.0 were still there.
It means only can delete the current older build(rhvh-4.1-0.20170616.0) related files after third upgrades, do not care the oldest builds(rhvh-4.1-0.20170417.0) files.

From QE side, I think it is reasonable.

Ryan, is this the expected result?

Comment 6 Huijuan Zhao 2018-01-17 10:32:56 UTC

According to comment 4, change the status to VERIFIED.

Comment 9 errata-xmlrpc 2018-02-01 15:06:17 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0263

Comment 10 Franta Kust 2019-05-16 12:54:56 UTC

BZ<2>Jira re-sync

Note You need to log in before you can comment on or make changes to this bug.

adevolder
bugs
cshao
dfediuck
dguo
dossow
eheftman
huzhao
mgoldboi
mkalinin
qiyuan
rbarry
sbonazzo
virt-bugs
weiwang
yaniwang
ycui
yisong
yturgema
yzhao