Bug 1705979

Summary: Kernel and initramdisk files are not deleted on TFTP
Product: Red Hat Satellite Reporter: Oliver Falk <ofalk>
Component: ProvisioningAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED WONTFIX QA Contact: Ondrej Gajdusek <ogajduse>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.5.0CC: inecas, lzap, peter.vreman, sshtein
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1827176 (view as bug list) Environment:
Last Closed: 2021-07-09 17:02:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1122832, 1827176    

Description Oliver Falk 2019-05-03 10:42:10 UTC
Description of problem:
The  /var/lib/tftpboot/boot is growing a lot, it adds a lot of new files:
- The old files with the old naming convention are still kept, should those not be deleted if not used anymore?
- The files with the same content are not hardlinked taking up a lot of space.

Example:
    # du -h /var/lib/tftpboot/boot
    1.4G    /var/lib/tftpboot/boot

    # ls -l /var/lib/tftpboot/boot
    total 1365152
    -rw-r--r--. 1 foreman-proxy foreman-proxy 41039717 Nov  9 16:20 RedHat-6.10-x86_64-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  4316752 Nov  9 16:20 RedHat-6.10-x86_64-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 41411274 Nov 10 03:02 RedHat-6.9-x86_64-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  4276496 Nov 10 03:02 RedHat-6.9-x86_64-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 44701632 Nov 10 03:12 RedHat-7.3-x86_64-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  5391264 Nov 10 03:12 RedHat-7.3-x86_64-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 49763300 Nov  9 17:11 RedHat-7.4-x86_64-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  5875184 Nov  9 17:11 RedHat-7.4-x86_64-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54525200 Nov  9 17:24 RedHat-7.5-x86_64-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6381872 Nov  9 17:24 RedHat-7.5-x86_64-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 RedHat-7.6-x86_64-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 RedHat-7.6-x86_64-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 41039717 Nov  9 16:20 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-534-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  4316752 Nov  9 16:20 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-534-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 41039717 Nov  9 16:20 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-556-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  4316752 Nov  9 16:20 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-556-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 41039717 Nov  9 16:20 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-583-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  4316752 Nov  9 16:20 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-583-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 44701632 Nov 10 03:12 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-616-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  5391264 Nov 10 03:12 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-616-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 44701632 Nov 10 03:12 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-652-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  5391264 Nov 10 03:12 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-652-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 44701632 Nov 10 03:12 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-688-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  5391264 Nov 10 03:12 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-688-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 49763300 Nov  9 17:11 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-4-795-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  5875184 Nov  9 17:11 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-4-795-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54525200 Nov  9 17:24 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-5-891-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6381872 Nov  9 17:24 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-5-891-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1027-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1027-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1060-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1060-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1096-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1096-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1148-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1148-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-13132-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-13132-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1318-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1318-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-4300-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-4300-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-5674-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-5674-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-923-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-923-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-963-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-963-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-989-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-989-vmlinuz

    # md5sum /var/lib/tftpboot/boot/* | sort
    1ff03802bb4b852a95f191a5c0247df7  /var/lib/tftpboot/boot/RedHat-7.5-x86_64-initrd.img
    1ff03802bb4b852a95f191a5c0247df7  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-5-891-initrd.img
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/RedHat-7.6-x86_64-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1027-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1060-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1096-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1148-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-13132-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1318-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-4300-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-5674-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-923-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-963-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-989-vmlinuz
    431c9c6e39fce34ffc3eb93df21e865a  /var/lib/tftpboot/boot/RedHat-7.5-x86_64-vmlinuz
    431c9c6e39fce34ffc3eb93df21e865a  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-5-891-vmlinuz
    ad945bbd0365fcab439a8ab65a133881  /var/lib/tftpboot/boot/RedHat-6.10-x86_64-vmlinuz
    ad945bbd0365fcab439a8ab65a133881  /var/lib/tftpboot/boot/red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-534-vmlinuz
    ad945bbd0365fcab439a8ab65a133881  /var/lib/tftpboot/boot/red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-556-vmlinuz
    ad945bbd0365fcab439a8ab65a133881  /var/lib/tftpboot/boot/red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-583-vmlinuz
    b113220e7b9f392f79a11ade59c3ba52  /var/lib/tftpboot/boot/RedHat-6.10-x86_64-initrd.img
    b113220e7b9f392f79a11ade59c3ba52  /var/lib/tftpboot/boot/red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-534-initrd.img
    b113220e7b9f392f79a11ade59c3ba52  /var/lib/tftpboot/boot/red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-556-initrd.img
    b113220e7b9f392f79a11ade59c3ba52  /var/lib/tftpboot/boot/red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-583-initrd.img
    c48b15aa91c08801a72063a54788686f  /var/lib/tftpboot/boot/RedHat-6.9-x86_64-vmlinuz
    cd156a477a7584d108569673df46de64  /var/lib/tftpboot/boot/RedHat-6.9-x86_64-initrd.img
    e0342f5226d7f35b2b12fb5f5506c13f  /var/lib/tftpboot/boot/RedHat-7.3-x86_64-initrd.img
    e0342f5226d7f35b2b12fb5f5506c13f  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-616-initrd.img
    e0342f5226d7f35b2b12fb5f5506c13f  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-652-initrd.img
    e0342f5226d7f35b2b12fb5f5506c13f  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-688-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/RedHat-7.6-x86_64-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1027-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1060-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1096-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1148-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-13132-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1318-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-4300-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-5674-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-923-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-963-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-989-initrd.img
    f0bc285eb44f9e363d1327f99d28d976  /var/lib/tftpboot/boot/RedHat-7.4-x86_64-initrd.img
    f0bc285eb44f9e363d1327f99d28d976  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-4-795-initrd.img
    f15a0fb05249b3d1daa46ec179e9928e  /var/lib/tftpboot/boot/RedHat-7.4-x86_64-vmlinuz
    f15a0fb05249b3d1daa46ec179e9928e  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-4-795-vmlinuz
    f3558ce1e32775a86d9b367a11c58ccc  /var/lib/tftpboot/boot/RedHat-7.3-x86_64-vmlinuz
    f3558ce1e32775a86d9b367a11c58ccc  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-616-vmlinuz
    f3558ce1e32775a86d9b367a11c58ccc  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-652-vmlinuz
    f3558ce1e32775a86d9b367a11c58ccc  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-688-vmlinuz


Version-Release number of selected component (if applicable): 6.5


How reproducible: Always


Steps to Reproduce:
1. Use deployment via TFTP
2. Check tftp/boot

Actual results:
Space wasted.


Expected results:
If there is no process to track if these files are still needed or not, there should be at least a process that does a hardlink run from time to time.
Is there a manual process that one can trigger to get rid of old files there?


Additional info:
Customer case will be linked.

Comment 4 Lukas Zapletal 2019-07-04 10:34:57 UTC
Hello, this was implemented to avoid corrupt files when OS variant is changed (Server/Workstation) or when content update is promoted.

It is safe to delete all red-hat-enterprise-linux-* files, new hosts which enter build mode will cause redownload of the boot files from Pulp so this is safe to do. Possibly deletion might break hosts which are already in the build mode or downloading the files at the very same moment, again exiting and entering build mode and rebooting will redownload both files.

We want to improve this and only create new file when contents changed, but this will not be delivered even in 6.7 yet. Please create a KBASE for other customers who can eventually run into this. Using deduplication FS or utility is also an option of there are space concerns.

Comment 5 Peter Vreman 2019-07-16 11:32:28 UTC
It in-use files is a risk then the cleaning can be done during an Upgrade.

In the above posted listings there are also old files like 'RedHat-6.10-x86_64-initrd.img' that do not match the new naming convention. With a fresh instalaltion i do not see the 'RedHat-*' anymore. Why are there still there an not remvoed during upgrade if obsolete?

Comment 6 Lukas Zapletal 2019-07-17 09:51:00 UTC
Peter,

there are actually two different naming conventions. RedHat-* are for operating systems which are associated to normal installation media (URL), in this case naming convention is simple OperatingSystem-Version-architecture-XXXXXXX where X are hash of URL basically.

The second convention (long) is for hosts with Content Source (Katello-based) content. In that case, product name is used together with Content View database ID. Honestly, we did not think this through correctly and haven't realized that this will create many files as CVs are being promoted/published. So the solution is only half-baked.

The final solution we want to provide is to simply drop the XXXXXX and database ID and replace it with a simple hash from Last-Modified and ETAG HTTP headers. So only when content really changes there will be new (unique) name. We are not there yet.

SOLUTION: List all hosts which are in build mode in Satellite UI/CLI. If there are none, you can delete ALL those TFTP files. If there are few, you can delete them and then exit and enter build mode back again to redownload files.

Comment 12 Mike McCune 2021-06-08 21:14:22 UTC
Upon review of our valid but aging backlog the Satellite Team has concluded that this Bugzilla does not meet the criteria for a resolution in the near term, and are planning to close in a month. This message may be a repeat of a previous update and the bug is again being considered to be closed. If you have any concerns about this, please contact your Red Hat Account team.  Thank you.

Comment 13 Mike McCune 2021-07-09 17:02:24 UTC
Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this feel free to contact your Red Hat Account Team. Thank you.