Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2120684 - Initial if clause in dhcpd.conf causing PXE-based discovery errors
Summary: Initial if clause in dhcpd.conf causing PXE-based discovery errors
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Documentation
Version: 6.12.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: 6.12.0
Assignee: Malhar Jivrajani
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-08-23 14:17 UTC by Griffin Sullivan
Modified: 2022-11-16 14:57 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-11-16 14:57:18 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 35520 0 Normal New Invalid remote_ip on production setup 2022-09-13 11:36:50 UTC
Red Hat Issue Tracker SAT-12482 0 None None None 2022-08-25 14:22:19 UTC
Red Hat Issue Tracker SATDOC-1159 0 None None None 2022-09-15 06:16:55 UTC

Description Griffin Sullivan 2022-08-23 14:17:55 UTC 
Description of problem:
On 6.12 there is a change to how dhcpd.conf is configured which is causing discovery to fail on providers using iPXE (RHV and libvirt). 

dhcpd.conf in 6.12 adds the first if statement here (6.11 has everything but the first if):

if exists user-class and option user-class = "iPXE" {
  filename "http://sat-url:8000/unattended/iPXE";
} elsif option architecture = 00:06 {
  filename "grub2/shim.efi";
} elsif option architecture = 00:07 {
  filename "grub2/shim.efi";
} elsif option architecture = 00:09 {
  filename "grub2/shim.efi";
} else {
  filename "pxelinux.0";
}

Version-Release number of selected component (if applicable):
6.12

How reproducible:
100%

Steps to Reproduce:
1. Setup discovery on Satellite
2. Create a blank vm in RHV that is ready for discovery
3. Boot the vm

Notes:
Curl-ing the filename for the first if clause works on any other machine in our network, but the error is only seen when run on the satellite.

Here was the change in dhcp: https://github.com/theforeman/puppet-dhcp/pull/150
Comment 2 Lukas Zapletal 2022-08-25 06:47:41 UTC 
I can confirm the bug, the filename "http://sat-url:8000/unattended/iPXE"; statement added in that patch needs to be removed. Since this was community-contributed, the best approach would be to make this configurable option in puppet and default to not including it. I do not see any other option than this, iPXE is built in libvirt/RHV and thus will always fall in the if statement.

I am not sure why X-Forwarded-For header in unattended controller does not work for you, the smart proxy should be in that list. Can you check that the host that is physically forwarding the request (Capsule) is registered in Capsules and the URL that is there resolves back to the IP address of the Capsule? Because then the controller would return 404, I am not sure if that would change a thing in the workflow tho.

Ewoud? Ideas?
Comment 3 Lukas Zapletal 2022-08-25 06:49:06 UTC 
What I mean is that for your https://capsule.xyz registered in the Satellite - Capsules page, when you do DNS PTR request, you get the IP address that matches the REMOTE_IP (10.1.2.48). Because this is how Satellite knows that it can actually use X-Forwarded-For header safely.
Comment 4 Brad Buckingham 2022-08-25 14:14:48 UTC 
Is this a regression from 6.11?  I am assuming it may be based upon comment 2.
Comment 5 Griffin Sullivan 2022-08-25 14:30:21 UTC 
Yes this is a regression in the behavior from 6.11. Our discovery process has not changed at all between versions.
Comment 6 Leos Stejskal 2022-08-30 13:43:10 UTC 
Since the issue is regression I propose to remove it from the 6.12 scope and move it to the backlog for now
Comment 7 Leos Stejskal 2022-08-30 13:48:54 UTC 
I was too hasty with my previous comment, please ignore it
Comment 8 Lukas Zapletal 2022-09-01 09:42:58 UTC 
I wonder if changing the URL to the boostrap entry would help to fix the issue.

if exists user-class and option user-class = "iPXE" {
  filename "http://sat-url:8000/unattended/iPXE?bootstrap=1";
} elsif ...

The bootstrap endpoint should ignore REMOTE_IP and serve a template that explicitly sends out MAC address of the interface, if that's not matched it should give 404. But I believe unattended will still try REMOTE_IP and cause the issue when proxy is registered in the inventory.
Comment 9 Leos Stejskal 2022-09-13 11:36:49 UTC 
Created redmine issue https://projects.theforeman.org/issues/35520 from this bug
Comment 10 Ewoud Kohl van Wijngaarden 2022-09-14 15:47:01 UTC 
I did some more thorough investigation.

The root cause is that the templates feature is enabled on the Satellite, but the Foreman Proxy isn't in trusted_proxies. This means the remote IP is being set to the Foreman Proxy's IP instead of the actual machine.

The templates feature is not enabled by default on Satellite, it is on Capsules.

The correct solution is to use the installer:

    satellite-installer --foreman-trusted-proxies 127.0.0.0/8  --foreman-trusted-proxies ::1  --foreman-trusted-proxies FOREMAN_PROXY_IP

The value for FOREMAN_PROXY_IP should be the source IP the Foreman Proxy uses to connect to Foreman. It should also be repeated for every Foreman Proxy with the templates and/or registration feature enabled. Alternatively, an IP range is also accepted.

This should be properly documented. The installer should also make it easier. For example, it can always include the localhost IPs so users don't need to specify those.
Comment 11 Leos Stejskal 2022-09-15 06:13:07 UTC 
Moving to documentation component,
problem is not in code but in the misconfiguration.

See:
https://github.com/theforeman/foreman/pull/9422#issuecomment-1246947431
https://github.com/theforeman/foreman-documentation/pull/1626
Comment 13 Malhar Jivrajani 2022-10-03 14:56:00 UTC 
Link to the Preview Doc: https://dxp-docp-prod.apps.ext-waf.spoke.prod.us-west-2.aws.paas.redhat.com/documentation/en-us/red_hat_satellite/6.12/html-single/provisioning_hosts/index?lb_target=preview&check_logged_in=1
Comment 14 Malhar Jivrajani 2022-11-16 14:57:18 UTC 
Link to the published document: https://access.redhat.com/documentation/en-us/red_hat_satellite/6.12/html/provisioning_hosts/provisioning_virtual_machines_in_vmware_provisioning#Using_VMware_Cloud_Init_and_Userdata_Templates_for_Provisioning_provisioning
Note You need to log in before you can comment on or make changes to this bug.