Bug 1571290

Hey, known bug, WIP, workaround possible - one liner.

*** This bug has been marked as a duplicate of bug 1447963 ***

Lukas, this issue is different than the BZ1447963 describes.
This is that it created a tftp boot files with 7.5 in the name but the content from 6.4.

In the Sat6.3 log it is also visible that foreman passes really the wrong source and destination with differnt rhel versions to the smart proxy. This is still going wrong with the oneliner 'wget remove -c' fix.

----------------------
2018-04-17 11:41:30 2f02471c [app] [D] RestClient.post "https://li-lc-1578.hag.hilti.com:9090/tftp/fetch_boot_file", "prefix=boot%2F RedHat-7.5-x86_64 &path=http%3A%2F%2Fli-lc-1578.hag.hilti.com%2Fpulp%2Frepos%2FHilti%2FHostgroup%2Fhgp-crash_sat-6_2-ci%2Fcontent%2Fdist%2Frhel%2Fserver%2F7%2F 7.4 %2Fx86_64%2Fkickstart%2F%2Fimages%2Fpxeboot%2Finitrd.img", "Accept"=>"application/json", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"231", "Content-Type"=>"application/x-www-form-urlencoded" ------ 
----------------------

To fix this issue you need to full change of http://projects.theforeman.org/issues/19389 to create the mediums with a unique id once and then create symlinks to if for each hostgroup and host.

Peter,

thanks for detailed analysis. The planned patch will help to download the correct content, but the symlink will still be named incorrectly, which is RedHat-7.5-x86_64-xxx at the moment.

The filename is determined from Operating System family name, OS version and Architecture, after successful sync they are both concatenated together and this forms the prefix. If you have Operating System with family set to "Red Hat" and version set to 7.5, then it will create this file.

Can you tell me how did you get to this state where you have this inconsistency? What was the upstream URL during the initial sync?

I am flipping to the Katello team for further investigation, the TFTP problem is consequence of the wrong version.

Peter, some updates.

I re-reviewed the PR you refer to and I can confirm that at this stage, it will help to workaround the TFTP corruption. But we would like to solve this incorrect versioning in Katello.

We found out that 7.4 and 7.5 RHEL Workstation kickstarts have incorrect kernel/initramdisk (they are from 7.3), we have an internal ticket and release engineering is investigating this issue already.

Can you tell me if you synced Workstation kickstart in this case to rule out this issue?

Also can you confirm me that the reproduce steps from the description are correct? We will try here.

Hoi Lukas,

The root cause is simple there is no validation that the KickstartRepo inside the ContentView matches the Foreman OS:
- Foreman uses the OS version from the Hostgroup
- Katello provides the ContentView


Reproduction:
- Remove the /var/lib/tftpboot/boot/RedHat-7.5*
- Create a Hostgroup with OS version set to RHEL7.5
- Assign a ContentView with only RHEL6.9 Kickstart repo included
- Use Hammer or API to set the Kickstart repo to the Kickstart repo in the Contentview
- Create default PXE template

------------
2018-04-26 16:09:35 cd90838b [app] [D] RestClient.post "https://li-lc-1578.hag.hilti.com:9090/tftp/fetch_boot_file", "prefix=boot%2FRedHat-7.5-x86_64&path=http%3A%2F%2Fli-lc-1578.hag.hilti.com%2Fpulp%2Frepos%2FHilti%2FHostgroup%2Fhgp-crash_rhel-6_9-ci%2Fcontent%2Fdist%2Frhel%2Fserver%2F6%2F6.9%2Fx86_64%2Fkickstart%2F%2Fimages%2Fpxeboot%2Finitrd.img", "Accept"=>"application/json", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"232", "Content-Type"=>"application/x-www-form-urlencoded"
------------------

Maybe a separate BZ: The kernel and initrd files are being ask to download for every Hostgroup. E.g. in my case with almost 200 Hostgroups the files there are 380 Rest calls to (re)download the 'same' rhel6.9,7.3,7.4,7.5 boot images

[crash/LI] root@li-lc-1578:~# grep '2018-04-26 16:09.*:9090/tftp/fetch_boot_file' /var/log/foreman/production.log -c
380

Additional info on the API calls i use:
- Set medium_id=null
- Set kickstart_repo_id to repo from haivng Kickstart in the name CV+LC

Real-World Scenario where this hits me with every RHEL minor release since the Sat6.2.x that has 'Synced Content' as medium:

- I have already a RHEL7Server ContentView with kcikstart repo RHEL7.y
- When the next RHEL7.x minor comes out i sync the new repos
- When then i have an the last OS 7.x there which i then use when i have to attach to '7Server' (RFE for better support in BZ1267885)
- Then i update my Hostgroups and Hosts with the latest info of 7Server that is then RHEL7.x, but the Contentview stays with RHEL7.y
- Then the PXE menu is build
- Now you have a RHEL7.x-initrd with the content of RHEL7.y-initd.
   - The first time you are lucky that it is a 100% dupe

Now step two:
- I update my RHEL7Server ContentView with kcikstart repo RHEL7.x
- Update the Hostgroups for the new RHEL7.x kickstart inside the CV
- Build PXE menu
- Now you are in bad luck, because it is going to 'dowmload continue' because of the 'wget -c' the previous RHEL7.y-initrd to the size of RHEL7.x-initrd. And there you have the corrupted boot file (matching the BZ1447963)

Thanks Peter. This is indeed a weak part of the implementation we need to work on. A reasonable goal would be:

1) Get the referenced https://github.com/theforeman/foreman/pull/5244 PR merged and ship as part of Satellite 6.4 to minimize corruptions

2) Work on refactoring of Operating System vs ContentView Kickstart vs download handling.

For the record, I started discussion on this topic at:

https://community.theforeman.org/t/rfc-change-tftp-file-naming-pattern/8955

Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this, please do not reopen. Instead, feel free to contact Red Hat Technical Support. Thank you.