Bug 1827176

Summary: [RFE] Clean up old Kernel and initramfs files from tftp/boot via satellite-maintain
Product: Red Hat Satellite Reporter: Oliver Falk <ofalk>
Component: Satellite MaintainAssignee: Suraj Patil <supatil>
Status: CLOSED ERRATA QA Contact: Jameer Pathan <jpathan>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.7.0CC: apatel, aupadhye, gtalreja, inecas, kgaikwad, lzap, pcreech, peter.vreman, rplevka, satellite6-bugs, sshtein, supatil
Target Milestone: 6.8.0Keywords: FutureFeature, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-foreman_maintain-0.6.10 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1705979 Environment:
Last Closed: 2020-10-27 12:38:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1705979    
Bug Blocks: 1122832    

Description Oliver Falk 2020-04-23 12:05:53 UTC 
+++ This bug was initially created as a clone of Bug #1705979 +++

Description of problem:
The  /var/lib/tftpboot/boot is growing a lot, it adds a lot of new files:
- The old files with the old naming convention are still kept, should those not be deleted if not used anymore?
- The files with the same content are not hardlinked taking up a lot of space.

Example:
    # du -h /var/lib/tftpboot/boot
    1.4G    /var/lib/tftpboot/boot

    # ls -l /var/lib/tftpboot/boot
    total 1365152
    -rw-r--r--. 1 foreman-proxy foreman-proxy 41039717 Nov  9 16:20 RedHat-6.10-x86_64-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  4316752 Nov  9 16:20 RedHat-6.10-x86_64-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 41411274 Nov 10 03:02 RedHat-6.9-x86_64-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  4276496 Nov 10 03:02 RedHat-6.9-x86_64-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 44701632 Nov 10 03:12 RedHat-7.3-x86_64-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  5391264 Nov 10 03:12 RedHat-7.3-x86_64-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 49763300 Nov  9 17:11 RedHat-7.4-x86_64-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  5875184 Nov  9 17:11 RedHat-7.4-x86_64-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54525200 Nov  9 17:24 RedHat-7.5-x86_64-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6381872 Nov  9 17:24 RedHat-7.5-x86_64-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 RedHat-7.6-x86_64-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 RedHat-7.6-x86_64-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 41039717 Nov  9 16:20 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-534-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  4316752 Nov  9 16:20 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-534-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 41039717 Nov  9 16:20 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-556-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  4316752 Nov  9 16:20 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-556-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 41039717 Nov  9 16:20 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-583-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  4316752 Nov  9 16:20 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-583-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 44701632 Nov 10 03:12 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-616-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  5391264 Nov 10 03:12 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-616-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 44701632 Nov 10 03:12 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-652-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  5391264 Nov 10 03:12 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-652-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 44701632 Nov 10 03:12 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-688-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  5391264 Nov 10 03:12 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-688-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 49763300 Nov  9 17:11 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-4-795-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  5875184 Nov  9 17:11 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-4-795-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54525200 Nov  9 17:24 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-5-891-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6381872 Nov  9 17:24 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-5-891-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1027-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1027-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1060-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1060-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1096-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1096-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1148-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1148-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-13132-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-13132-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1318-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1318-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-4300-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-4300-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-5674-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-5674-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-923-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-923-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-963-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-963-vmlinuz
    -rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-989-initrd.img
    -rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Nov 10 03:14 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-989-vmlinuz

    # md5sum /var/lib/tftpboot/boot/* | sort
    1ff03802bb4b852a95f191a5c0247df7  /var/lib/tftpboot/boot/RedHat-7.5-x86_64-initrd.img
    1ff03802bb4b852a95f191a5c0247df7  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-5-891-initrd.img
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/RedHat-7.6-x86_64-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1027-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1060-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1096-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1148-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-13132-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1318-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-4300-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-5674-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-923-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-963-vmlinuz
    36eff2be463ab8e70518c802647a96c5  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-989-vmlinuz
    431c9c6e39fce34ffc3eb93df21e865a  /var/lib/tftpboot/boot/RedHat-7.5-x86_64-vmlinuz
    431c9c6e39fce34ffc3eb93df21e865a  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-5-891-vmlinuz
    ad945bbd0365fcab439a8ab65a133881  /var/lib/tftpboot/boot/RedHat-6.10-x86_64-vmlinuz
    ad945bbd0365fcab439a8ab65a133881  /var/lib/tftpboot/boot/red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-534-vmlinuz
    ad945bbd0365fcab439a8ab65a133881  /var/lib/tftpboot/boot/red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-556-vmlinuz
    ad945bbd0365fcab439a8ab65a133881  /var/lib/tftpboot/boot/red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-583-vmlinuz
    b113220e7b9f392f79a11ade59c3ba52  /var/lib/tftpboot/boot/RedHat-6.10-x86_64-initrd.img
    b113220e7b9f392f79a11ade59c3ba52  /var/lib/tftpboot/boot/red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-534-initrd.img
    b113220e7b9f392f79a11ade59c3ba52  /var/lib/tftpboot/boot/red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-556-initrd.img
    b113220e7b9f392f79a11ade59c3ba52  /var/lib/tftpboot/boot/red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-583-initrd.img
    c48b15aa91c08801a72063a54788686f  /var/lib/tftpboot/boot/RedHat-6.9-x86_64-vmlinuz
    cd156a477a7584d108569673df46de64  /var/lib/tftpboot/boot/RedHat-6.9-x86_64-initrd.img
    e0342f5226d7f35b2b12fb5f5506c13f  /var/lib/tftpboot/boot/RedHat-7.3-x86_64-initrd.img
    e0342f5226d7f35b2b12fb5f5506c13f  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-616-initrd.img
    e0342f5226d7f35b2b12fb5f5506c13f  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-652-initrd.img
    e0342f5226d7f35b2b12fb5f5506c13f  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-688-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/RedHat-7.6-x86_64-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1027-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1060-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1096-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1148-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-13132-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1318-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-4300-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-5674-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-923-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-963-initrd.img
    e15958ee629978a716218bc3595bcfef  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-989-initrd.img
    f0bc285eb44f9e363d1327f99d28d976  /var/lib/tftpboot/boot/RedHat-7.4-x86_64-initrd.img
    f0bc285eb44f9e363d1327f99d28d976  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-4-795-initrd.img
    f15a0fb05249b3d1daa46ec179e9928e  /var/lib/tftpboot/boot/RedHat-7.4-x86_64-vmlinuz
    f15a0fb05249b3d1daa46ec179e9928e  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-4-795-vmlinuz
    f3558ce1e32775a86d9b367a11c58ccc  /var/lib/tftpboot/boot/RedHat-7.3-x86_64-vmlinuz
    f3558ce1e32775a86d9b367a11c58ccc  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-616-vmlinuz
    f3558ce1e32775a86d9b367a11c58ccc  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-652-vmlinuz
    f3558ce1e32775a86d9b367a11c58ccc  /var/lib/tftpboot/boot/red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-688-vmlinuz


Version-Release number of selected component (if applicable): 6.5


How reproducible: Always


Steps to Reproduce:
1. Use deployment via TFTP
2. Check tftp/boot

Actual results:
Space wasted.


Expected results:
If there is no process to track if these files are still needed or not, there should be at least a process that does a hardlink run from time to time.
Is there a manual process that one can trigger to get rid of old files there?
Comment 3 Oliver Falk 2020-04-23 12:10:29 UTC 
As suggested over in the other BZ, I'm opening this one in order to address/add the functionality to clean out old files in tftp/boot with satellite-maintain.
Comment 5 Lukas Zapletal 2020-04-27 09:33:11 UTC 
To give you little bit more context:

Implement a command that checks if there are any /var/lib/tftpboot/boot/*-{vmlinuz,initrd.img} files older than two weeks (configurable via option). Provide a resolution (fix) - delete those files. Make this check to be part of the standard suite of checks. Note this directory can be missing if TFTP module is not set up - skip the check in that case.
Comment 6 Peter Vreman 2020-04-27 09:41:21 UTC 
Lukas,

Deleting everything older than 2 weeks cannot be done, at least not what i quickly see as an user:

--------
[crash/LI] root@li-lc-2222:/var/lib/tftpboot/boot# ls -lrt
total 2411476
-rw-r--r--. 1 foreman-proxy foreman-proxy  4316752 Jun 18  2019 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-415-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 41039717 Jun 18  2019 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-415-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  4316752 Jun 18  2019 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-389-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 41039717 Jun 18  2019 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-389-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  4316752 Jun 18  2019 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-361-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 41039717 Jun 18  2019 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-361-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  5391264 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-520-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 44701632 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-520-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  5391264 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-484-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 44701632 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-484-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  5391264 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-448-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 44701632 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-3-448-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-990-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-990-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-9713-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-9713-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-960-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-960-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-900-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-900-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-840-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-840-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-750-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-750-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-718-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-718-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-682-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-682-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-646-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-646-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-612-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-612-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-582-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-582-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-552-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-552-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6635920 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1073-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 54799220 Jun 18  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-6-1073-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6730032 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9641-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57746484 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9641-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6730032 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9595-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57746484 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9595-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6730032 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9206-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57746484 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9206-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6730032 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9167-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57746484 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9167-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6730032 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9106-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57746484 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9106-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6730032 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9067-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57746484 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9067-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6730032 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9037-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57746484 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-9037-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6730032 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-32557-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57746484 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-32557-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6730032 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-32461-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57746484 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-32461-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6730032 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-24252-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57746484 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-24252-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6730032 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-19531-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57746484 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-19531-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6730032 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-18625-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57746484 Aug 14  2019 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-18625-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  8106848 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-27021-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 62248424 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-27021-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  8106848 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26974-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 62248424 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26974-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  8106848 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26941-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 62248424 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26941-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  8106848 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26250-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 62248424 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26250-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  8106848 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26216-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 62248424 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26216-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6762800 Apr  2 13:44 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-8-45614-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57894232 Apr  2 13:44 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-8-45614-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6762800 Apr  2 13:44 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-8-44873-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57894232 Apr  2 13:44 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-8-44873-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6762800 Apr  2 13:44 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-8-44829-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57894232 Apr  2 13:44 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-8-44829-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6762800 Apr  2 13:44 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-8-43766-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57894232 Apr  2 13:44 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-8-43766-initrd.img
[crash/LI] root@li-lc-2222:
-------------

If you implement what you mentioned above to delete files older than 2 weeks then All files will be deleted and then any TFTP is broken until a build-pxe-default is executed and the boot images are recreated.

Peter
Comment 7 Peter Vreman 2020-04-27 09:52:17 UTC 
And secondly the original issue is not only partly solved by cleanup of obsolete images.
Because after cleanup of all files (2.4GB) and then call to build-pxe-default to rebuild all TFTP images i still have 1.4GB of images:

----
[crash/LI] root@li-lc-2222:/var/lib/tftpboot# sudo du -ms /var/lib/tftpboot/boot
1383    /var/lib/tftpboot/boot
----

Why do i have 5x the same image for red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-* without using e.g. hardlinking (the ls -l below shows '1' for files that have the same hash in column 2)?

-----
[crash/LI] root@li-lc-2222:/var/lib/tftpboot/boot# ls -l red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-*-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy 62248424 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26216-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy 62248424 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26250-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy 62248424 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26941-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy 62248424 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26974-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy 62248424 Nov  8 13:38 red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-27021-initrd.img

[crash/LI] root@li-lc-2222:/var/lib/tftpboot/boot# sha1sum red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-*-initrd.img
24506ad7a7de868e5e991491d5cad102a5e1d1ec  red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26216-initrd.img
24506ad7a7de868e5e991491d5cad102a5e1d1ec  red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26250-initrd.img
24506ad7a7de868e5e991491d5cad102a5e1d1ec  red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26941-initrd.img
24506ad7a7de868e5e991491d5cad102a5e1d1ec  red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-26974-initrd.img
24506ad7a7de868e5e991491d5cad102a5e1d1ec  red-hat-enterprise-linux-8-for-x86_64-baseos-kickstart-8-1-27021-initrd.img
[crash/LI] root@li-lc-2222:/var/lib/tftpboot/boot#
-----
Comment 8 Lukas Zapletal 2020-04-27 11:14:12 UTC 
The reason why this is so clunky is because PXE handling is done by Foreman core while content handling (Content Views) is done by Katello plugin and this is not integrated deeply. Ideally, we would calculate SHA1 sum of the content and name files accordingly, problem is filenames are created by Foreman not Smart Proxy and without access to the content. Foreman would need to download them, calculate SHA1 sum thus filename and pass this to smart proxy. This seems to be too complicated. Thus the proper solution is to completely redesign this.

One partial solution could be that we keep everything (the API) as is but proxy would download and store files under their content hash name and create only symlinks pointing to these files. But then it's just a different problem - you would end up with many symlinks. Granted it wouldn't consume too much space then but with high rate of promotions you would still end up with many symlinks. A cleanup process would still be needed. Is your bigger concern the size or amount of files?

Now, you are right that I haven't considered hosts which are in build mode. But hosts are only valid for given amount of time (token duration) and they expire and the only way to get out of that is to exit and enter build mode which is a process which re-downloads TFTP files. Therefore I am going to update instructions for the foreman maintain team:

- Connect to Satellite API or DB and find token_duration setting (amount of minutes host provisioning tokens are valid).
- Implement a maintain command that checks if there are any /var/lib/tftpboot/boot/*-{vmlinuz,initrd.img} files older than N minutes (token duration setting found above).
- Provide a resolution (fix) - delete those files.
- Make this check to be part of the standard suite of checks.
- Skip this check if:
 - the TFTP directory is missing
 - token_duration is set to 0 (tokens are disabled)

I hope this creates better experience until we find time to redesign TFTP.
Comment 9 Suraj Patil 2020-05-11 04:30:28 UTC 
Created redmine issue https://projects.theforeman.org/issues/29727 from this bug
Comment 10 Bryan Kearney 2020-06-25 08:05:00 UTC 
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/29727 has been resolved.
Comment 13 Jameer Pathan 2020-08-12 08:33:00 UTC 
FailedQA

Tested with:
- Satellite 6.8.0 snap 11
- rubygem-foreman_maintain-0.6.8-1.el7sat.noarch

Test steps:
- Run "foreman-maintain health check --label check-tftp-storage"

Observations/Issues found:

- Check deleted all files older than token_duration setting from /var/lib/tftpboot/boot/ instead 
of only deleting those matching /var/lib/tftpboot/boot/*-{vmlinuz,initrd.img}(Ref: comment#8)
- It also deleted fdi-image subdirectory.

Additional info:
# ll /var/lib/tftpboot/boot/
total 4.0K
drwxr-xr-x. 2 foreman-proxy root 4.0K Aug  5 06:37 fdi-image
# ll /var/lib/tftpboot/boot/fdi-image/
total 227M
-rw-r--r--. 1 foreman-proxy root 220M Aug  5 06:37 foreman-discovery-image-3.6.1-2.iso-img
-rw-r--r--. 1 foreman-proxy root 6.4M Aug  5 06:37 foreman-discovery-image-3.6.1-2.iso-vmlinuz
lrwxrwxrwx. 1 foreman-proxy root   39 Aug  5 06:37 initrd0.img -> foreman-discovery-image-3.6.1-2.iso-img
lrwxrwxrwx. 1 foreman-proxy root   43 Aug  5 06:37 vmlinuz0 -> foreman-discovery-image-3.6.1-2.iso-vmlinuz
# foreman-maintain health check --label check-tftp-storage
Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Clean old Kernel and initramfs files from tftp-boot:                  [FAIL]
There are old initrd and vmlinuz files present in tftp
--------------------------------------------------------------------------------
Continue with step [Remove the files]?, [y(yes), n(no), q(quit)] y
Remove the files:                                                     [OK]      
--------------------------------------------------------------------------------
Rerunning the check after fix procedure
Clean old Kernel and initramfs files from tftp-boot:                  [OK]
--------------------------------------------------------------------------------

# ll /var/lib/tftpboot/boot/
total 0
# ll /var/lib/tftpboot/boot/fdi-image/
ls: cannot access /var/lib/tftpboot/boot/fdi-image/: No such file or directory
Comment 16 Jameer Pathan 2020-09-11 09:17:11 UTC 
Verified:

Verified with:
- Satellite 6.8.0 snap 15
- rubygem-foreman_maintain-0.6.10-1.el7sat.noarch

Test steps:
- Have some *-{vmlinuz,initrd.img} files in /var/lib/tftpboot/boot/ directory 
which are older than token_duration settings.
- Run "foreman-maintain health check --label check-tftp-storage"

Observation:
- check-tftp-storage check deletes files older than token_duration time, 
matching *-{vmlinuz,initrd.img} pattern from /var/lib/tftpboot/boot/ directory.

[root@qe-sat6-feature-rhel7 ~]# ll /var/lib/tftpboot/boot/
total 107272
drwxr-xr-x. 2 foreman-proxy root              4096 Sep 10 17:24 fdi-image
-rw-r--r--. 1 root          root                 0 Sep 11 04:56 red-hat-enterprise-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy 41039717 Sep 10 17:56 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-28-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  4316752 Sep 10 17:56 red-hat-enterprise-linux-6-server-kickstart-x86_64-6-10-28-vmlinuz
-rw-r--r--. 1 foreman-proxy foreman-proxy 57746484 Sep 10 17:58 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-19-initrd.img
-rw-r--r--. 1 foreman-proxy foreman-proxy  6730032 Sep 10 17:57 red-hat-enterprise-linux-7-server-kickstart-x86_64-7-7-19-vmlinuz
-rw-r--r--. 1 root          root                 0 Sep 11 04:56 red-hat-enterprise-vmlinuz
-rw-r--r--. 1 root          root                 0 Sep 11 04:55 test.yml


[root@qe-sat6-feature-rhel7 ~]# hammer settings list --search token_duration
---------------|----------------|-------|---------------------------------------------------------------------------------
NAME           | FULL NAME      | VALUE | DESCRIPTION                                                                     
---------------|----------------|-------|---------------------------------------------------------------------------------
token_duration | Token duration | 360   | Time in minutes installation tokens should be valid for, 0 to disable token g...
---------------|----------------|-------|---------------------------------------------------------------------------------


[root@qe-sat6-feature-rhel7 ~]# date
Fri Sep 11 04:58:47 EDT 2020

[root@qe-sat6-feature-rhel7 ~]# foreman-maintain health check --label check-tftp-storage
Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Clean old Kernel and initramfs files from tftp-boot:                  [FAIL]
There are old initrd and vmlinuz files present in tftp
--------------------------------------------------------------------------------
Continue with step [Remove the files]?, [y(yes), n(no), q(quit)] y
Remove the files:                                                     [OK]      
--------------------------------------------------------------------------------
Rerunning the check after fix procedure
Clean old Kernel and initramfs files from tftp-boot:                  [OK]
--------------------------------------------------------------------------------

[root@qe-sat6-feature-rhel7 ~]# ll /var/lib/tftpboot/boot/
total 4
drwxr-xr-x. 2 foreman-proxy root 4096 Sep 10 17:24 fdi-image
-rw-r--r--. 1 root          root    0 Sep 11 04:56 red-hat-enterprise-initrd.img
-rw-r--r--. 1 root          root    0 Sep 11 04:56 red-hat-enterprise-vmlinuz
-rw-r--r--. 1 root          root    0 Sep 11 04:55 test.yml

[root@qe-sat6-feature-rhel7 ~]# foreman-maintain health check --label check-tftp-storage
Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Clean old Kernel and initramfs files from tftp-boot:                  [OK]
--------------------------------------------------------------------------------
Comment 19 errata-xmlrpc 2020-10-27 12:38:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.8 Satellite Maintenance Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4365