RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1858344 - Hostname not set from DHCP with ipv6
Summary: Hostname not set from DHCP with ipv6
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: NetworkManager
Version: 8.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: 8.3
Assignee: Thomas Haller
QA Contact: Desktop QE
URL:
Whiteboard:
Depends On:
Blocks: 1883571
TreeView+ depends on / blocked
 
Reported: 2020-07-17 15:45 UTC by Ben Nemec
Modified: 2020-11-04 01:48 UTC (History)
9 users (show)

Fixed In Version: NetworkManager-1.26.0-4.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1883571 (view as bug list)
Environment:
Last Closed: 2020-11-04 01:47:57 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Trace logs from failing deployment (547.65 KB, text/plain)
2020-07-20 19:37 UTC, Ben Nemec 		no flags Details
Trace logs from dhclient deployment (3.88 MB, text/plain)
2020-07-24 19:02 UTC, Ben Nemec 		no flags Details
Reproducer script (1.25 KB, application/x-shellscript)
2020-07-26 12:30 UTC, Beniamino Galvani 		no flags Details
View All

Description Ben Nemec 2020-07-17 15:45:15 UTC 
Description of problem: We want node hostnames to be set via DHCP when possible so external DNS records are not required for NetworkManager to go look up the hostname for the node. However, it was recently reported that when there is no hostname available from the external DNS server, nodes are coming up with a hostname of localhost.


How reproducible: Always


Steps to Reproduce:
1. In https://github.com/openshift/machine-config-operator/blob/master/templates/common/baremetal/files/baremetal-NetworkManager-kni-conf.yaml add "hostname-mode=dhcp" under the [main] section. This forces NetworkManager to only consider DHCP-provided hostnames.
2. Deploy in dev-scripts with the above patch.


Actual results: Nodes come up with a hostname of localhost, which prevents the deployment from proceeding.


Expected results: Nodes get their correct hostnames from DHCP.


Additional info: This problem does not occur with ipv4. My suspicion is that it is related to the change from dhclient to the NetworkManager internal DHCP client, but I don't have a good way to verify that since dhclient is no longer present on the images.
Comment 1 Ben Nemec 2020-07-17 19:13:58 UTC 
I confirmed that this is related to the switch to the internal DHCP client. On 4.5, which still uses dhclient, this works as expected. It's only ipv6 on 4.6 that is a problem.
Comment 2 Ben Nemec 2020-07-20 19:37:50 UTC 
Created attachment 1701806 [details]
Trace logs from failing deployment
Comment 3 Ben Nemec 2020-07-20 19:40:42 UTC 
Sending to NetworkManager team so they can take a look.
Comment 4 Beniamino Galvani 2020-07-21 09:33:25 UTC 
The internal DHCPv6 client currently doesn't parse the FQDN option returned by server; this must be implemented.
Comment 5 Ben Nemec 2020-07-21 13:57:55 UTC 
Would this only be for future releases then? If it's never going to be supported in 8.2 we'll need to have some discussions within our team about how to deal with that. Thanks.
Comment 6 Beniamino Galvani 2020-07-21 14:46:39 UTC 
(In reply to Ben Nemec from comment #1)
> I confirmed that this is related to the switch to the internal DHCP client.
> On 4.5, which still uses dhclient, this works as expected. It's only ipv6 on
> 4.6 that is a problem.

When you say that it works on 4.5 with dhclient, do you mean using DHCPv4 or
DHCPv6? And did you use dhclient alone or NM with the dhclient backend?

From a look at the NM code, it seems to me that getting an hostname through
DHCPv6 was never really functional, even when using dhcp=dhclient.

(In reply to Ben Nemec from comment #5)
> Would this only be for future releases then? If it's never going to be
> supported in 8.2 we'll need to have some discussions within our team about
> how to deal with that. Thanks.

Yes, it would be for future releases. If the issue is critical to openshift, we can
discuss about backporting it to 8.2 as a z-stream.
Comment 7 Ben Nemec 2020-07-21 20:28:22 UTC 
(In reply to Beniamino Galvani from comment #6)
> (In reply to Ben Nemec from comment #1)
> > I confirmed that this is related to the switch to the internal DHCP client.
> > On 4.5, which still uses dhclient, this works as expected. It's only ipv6 on
> > 4.6 that is a problem.
> 
> When you say that it works on 4.5 with dhclient, do you mean using DHCPv4 or
> DHCPv6? And did you use dhclient alone or NM with the dhclient backend?
> 
> From a look at the NM code, it seems to me that getting an hostname through
> DHCPv6 was never really functional, even when using dhcp=dhclient.

Both were DHCPv6. I set hostname-mode=dhcp to force them to use DHCP for hostnames, and with dhclient it worked and with internal it did not. I suppose it's also possible that the dhclient path still pulled it from DNS for some reason.

> 
> (In reply to Ben Nemec from comment #5)
> > Would this only be for future releases then? If it's never going to be
> > supported in 8.2 we'll need to have some discussions within our team about
> > how to deal with that. Thanks.
> 
> Yes, it would be for future releases. If the issue is critical to openshift,
> we can
> discuss about backporting it to 8.2 as a z-stream.

Okay, thanks. We'll have to discuss our options then.
Comment 8 Beniamino Galvani 2020-07-24 13:08:41 UTC 
(In reply to Ben Nemec from comment #7)
> (In reply to Beniamino Galvani from comment #6)
> > (In reply to Ben Nemec from comment #1)
> > > I confirmed that this is related to the switch to the internal DHCP client.
> > > On 4.5, which still uses dhclient, this works as expected. It's only ipv6 on
> > > 4.6 that is a problem.
> > 
> > When you say that it works on 4.5 with dhclient, do you mean using DHCPv4 or
> > DHCPv6? And did you use dhclient alone or NM with the dhclient backend?
> > 
> > From a look at the NM code, it seems to me that getting an hostname through
> > DHCPv6 was never really functional, even when using dhcp=dhclient.
> 
> Both were DHCPv6. I set hostname-mode=dhcp to force them to use DHCP for
> hostnames, and with dhclient it worked and with internal it did not. I
> suppose it's also possible that the dhclient path still pulled it from DNS
> for some reason.

That shouldn't happen, and from what I see NM doesn't support getting the hostname from DHCPv6 when using dhclient.

Do you happen to have NM logs, preferably at trace level?
Comment 9 Ben Nemec 2020-07-24 19:02:09 UTC 
Created attachment 1702379 [details]
Trace logs from dhclient deployment

I've attached trace logs from a dhclient-based run. From what I can see, the difference may be that dhclient provides the necessary information for our dispatcher script to set the hostname, where the internal client does not. That would explain why it worked even though NM doesn't support setting the hostname itself.

I think the relevant part of the dispatcher script is probably this: https://github.com/openshift/machine-config-operator/blob/master/templates/common/baremetal/files/NetworkManager-resolv-prepender.yaml#L22
Comment 10 Beniamino Galvani 2020-07-26 12:28:45 UTC 
Right, it works with dhclient because NM exports the FQDN option, while it doesn't when using the internal client.

This merge request adds support to the internal client to parse the FQDN option and set it as transient hostname. This means that you won't need a dispatcher script to do it. To prevent this, set hostname-mode=none in the configuration.

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/589
Comment 11 Beniamino Galvani 2020-07-26 12:30:29 UTC 
Created attachment 1702439 [details]
Reproducer script
Comment 14 Vladimir Benes 2020-08-20 13:52:17 UTC 
CI test added:
https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/merge_requests/623
Comment 15 Antonio Cardace 2020-09-24 14:51:02 UTC 
Hi Ben,

I'm afraid it's too late to include this in 8.2.1, if this is important we're still in time to include it in 8.2.0.4 EUS but you can you please provide an explanation of why this is needed in RHEL 8.2?

Thanks,
Antonio
Comment 16 Ben Nemec 2020-09-24 17:13:54 UTC 
OpenShift 4.6 is based on RHEL 8.2, and in that release we had to move to the internal DHCP client. As a result, it has broken ipv6 deployments in some environments that were relying on the behavior of dhclient in regards to the hostname. If we can get this backported then we don't have to do any hacky workarounds to fix the problem.
Comment 17 Antonio Cardace 2020-09-25 10:20:01 UTC 
(In reply to Ben Nemec from comment #16)
> OpenShift 4.6 is based on RHEL 8.2, and in that release we had to move to
> the internal DHCP client. As a result, it has broken ipv6 deployments in
> some environments that were relying on the behavior of dhclient in regards
> to the hostname. If we can get this backported then we don't have to do any
> hacky workarounds to fix the problem.

The next batch update for 8.2.0 is 8.2.0.4 (EUS), you can request the zstream for that ITR if you think this is important.
Comment 18 Thomas Haller 2020-09-25 11:46:30 UTC 
(In reply to Ben Nemec from comment #16)
> OpenShift 4.6 is based on RHEL 8.2, and in that release we had to move to
> the internal DHCP client. As a result, it has broken ipv6 deployments in
> some environments that were relying on the behavior of dhclient in regards
> to the hostname. If we can get this backported then we don't have to do any
> hacky workarounds to fix the problem.

Note that it's fine to continue using dhcp=dhclient. What changed is the default, that doesn't mean for rhel-8.2 that the other option is less maintained (or worse than it was before). In fact, dhclient DHCP plugin is infinitely configurable (because it can run any script), and as such there will be always use-cases that the internal DHCP plugin cannot satisfy.

Have a file:

  [main]
  dhcp=dhclient

in /{etc,usr/lib}/NeworkManager/conf.d/60-openshift-use-dhclient.conf.
Comment 19 Ben Nemec 2020-09-25 14:47:52 UTC 
(In reply to Antonio Cardace from comment #17)
> (In reply to Ben Nemec from comment #16)
> > OpenShift 4.6 is based on RHEL 8.2, and in that release we had to move to
> > the internal DHCP client. As a result, it has broken ipv6 deployments in
> > some environments that were relying on the behavior of dhclient in regards
> > to the hostname. If we can get this backported then we don't have to do any
> > hacky workarounds to fix the problem.
> 
> The next batch update for 8.2.0 is 8.2.0.4 (EUS), you can request the
> zstream for that ITR if you think this is important.

Okay, I moved it back to 8.2.0. ipv6 is an important feature for us.
Comment 20 Ben Nemec 2020-09-25 14:49:52 UTC 
(In reply to Thomas Haller from comment #18)
> Note that it's fine to continue using dhcp=dhclient. What changed is the
> default, that doesn't mean for rhel-8.2 that the other option is less
> maintained (or worse than it was before). In fact, dhclient DHCP plugin is
> infinitely configurable (because it can run any script), and as such there
> will be always use-cases that the internal DHCP plugin cannot satisfy.

Unfortunately dhclient was completely removed from our base image for this release. That's why we had to switch to the internal client.
Comment 25 errata-xmlrpc 2020-11-04 01:47:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4499
Note You need to log in before you can comment on or make changes to this bug.