In many instances the internal dhcp option for NetworkManager fails to correctly bootstrap or acquire a lease. For many customers, dhclient was used to correctly respond to their infrastructure in order to acquire a new or existing lease. RHCOS 4.6 seems to have removed dhclient, or there seems to be an inconsistency in how dhcp-client is installed. The removal of this has resulted in numerous customers who depend on this unable to upgrade or build: [root@master-0 ~]# cat /etc/os-release NAME="Red Hat Enterprise Linux CoreOS" VERSION="46.82.202011210620-0" VERSION_ID="4.6" OPENSHIFT_VERSION="4.6" RHEL_VERSION="8.2" PRETTY_NAME="Red Hat Enterprise Linux CoreOS 46.82.202011210620-0 (Ootpa)" ID="rhcos" ID_LIKE="rhel fedora" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform" REDHAT_BUGZILLA_PRODUCT_VERSION="4.6" REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform" REDHAT_SUPPORT_PRODUCT_VERSION="4.6" OSTREE_VERSION='46.82.202011210620-0' [root@master-0 ~]# rpm -qil dhcp-client Name : dhcp-client Epoch : 12 Version : 4.3.6 Release : 40.el8 Architecture: x86_64 Install Date: Sat 21 Nov 2020 06:22:23 AM UTC Group : Unspecified Size : 530668 License : ISC Signature : RSA/SHA256, Wed 27 Nov 2019 10:25:25 AM UTC, Key ID 199e2f91fd431d51 Source RPM : dhcp-4.3.6-40.el8.src.rpm Build Date : Tue 26 Nov 2019 10:54:50 AM UTC Build Host : x86-vm-08.build.eng.bos.redhat.com Relocations : (not relocatable) Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla> Vendor : Red Hat, Inc. URL : http://isc.org/products/DHCP/ Summary : Provides the ISC DHCP client daemon and dhclient-script Description : DHCP (Dynamic Host Configuration Protocol) is a protocol which allows individual devices on an IP network to get their own network configuration information (IP address, subnetmask, broadcast address, etc.) from a DHCP server. The overall purpose of DHCP is to make it easier to administer a large network. This package provides the ISC DHCP client. /etc/NetworkManager /etc/NetworkManager/dispatcher.d /etc/NetworkManager/dispatcher.d/11-dhclient /etc/dhcp /etc/dhcp/dhclient.conf /etc/dhcp/dhclient.d /usr/lib/.build-id /usr/lib/.build-id/a6 /usr/lib/.build-id/a6/da683182abfc3bf1ba3c74217b8dce70ceb83e /usr/lib64/pm-utils/sleep.d/56dhclient /usr/sbin/dhclient /usr/sbin/dhclient-script /usr/share/doc/dhcp-client /usr/share/doc/dhcp-client/README.dhclient.d /usr/share/doc/dhcp-client/dhclient-enter-hooks /usr/share/doc/dhcp-client/dhclient.conf.example /usr/share/doc/dhcp-client/dhclient6.conf.example /usr/share/man/man5/dhclient.conf.5.gz /usr/share/man/man5/dhclient.leases.5.gz /usr/share/man/man8/dhclient-script.8.gz /usr/share/man/man8/dhclient.8.gz /var/lib/dhclient [root@master-0 ~]# ls -l /usr/sbin/dhclient ls: cannot access '/usr/sbin/dhclient': No such file or directory [root@master-0 ~]# ls -l /usr/lib/.build-id/a6/da683182abfc3bf1ba3c74217b8dce70ceb83e lrwxrwxrwx. 3 root root 29 Nov 9 23:08 /usr/lib/.build-id/a6/da683182abfc3bf1ba3c74217b8dce70ceb83e -> ../../../../usr/sbin/dhclient This breaks for customers who many have setup a machine config to force dhcp to dhclient: [root@master-0 ~]# cat /etc/NetworkManager/conf.d/10-dhcp-config.conf [main] no-auto-default=* dhcp=dhclient
Hey Jonathan, Any chance we could gain some insight into what use cases are missing that make NetworkManager's internal DHCP agent insufficient? We'd like to follow up on these with the NetworkManager team.
There's a few corner cases where we're still finding dhclient succeeds where the internal NetworkManager dhcp client fails. Given the pervasiveness of dhclient over the years, it's desirable to have it as a fallback in case anything goes awry in a customer environment where dhclient works and the internal dhcp fails.
> What do you suggest as the next steps to investigate the issue? It seems to me this is the same issue as: https://bugzilla.redhat.com/show_bug.cgi?id=1883521#c52 https://github.com/openshift/openshift-docs/pull/26455 What is the content of /etc/dhcp/dhclient.conf on 4.5?
This issue has been discussed with PM, CEE, and Engineering and we've concluded that the best path forward is to provide the `dhclient` binary in RHCOS for 4.6 and 4.7. We will include a deprecation notice as part of the OCP 4.7 release notes, as we would like to remove the binary from RHCOS as part of OCP 4.8. Note: this plan does not cover including the `dhclient` binary in the RHCOS initrd for 4.6 or 4.7. However, we will continue to default to using the internal NetworkManager DHCP client in 4.6, 4.7 and future releases. This means that customers that wish to continue to use `dhclient` for their DHCP needs in 4.6 and 4.7, will need to provide a config file in `/etc/NetworkManager/conf.d` that instructs NetworkManager to use `dhclient`, *before* upgrading to 4.6. For example: ``` $ cat /etc/NetworkManager/conf.d/dhcp-client.conf [main] dhcp=dhclient ``` This can be done with a properly formatted MachineConfig. We've speculated that an incorrectly configured `dhclient.conf` file could explain why DHCP requests in RHCOS 4.5 were using something other than the MAC address as the DHCP client ID. When the upgrade to 4.6 happened, the default behavior of the internal NetworkManager DHCP client uses the MAC address and therefore a different IP was assigned. However, without the entirety of any `dhclient.conf` on the nodes and/or config files under `/etc/NetworkManager` it is purely speculation. As part of fixing and verifying this BZ, we would need a proper reproducer that we can execute to prove that the addition of the `dhclient` binary will solve the issue during the upgrade from RHCOS 4.5 to RHCOS 4.6 (with `dhclient` binary). @Jonathan is this something that you could help us with? Lastly, this BZ should be used to track the inclusion of the `dhclient` binary in RHCOS 4.7; I will clone this BZ to track the inclusion of the `dhclient` binary in RHCOS 4.6.z
Higher priority work has prevented from this issue being worked on; adding UpcomingSprint keyword
Checked 47.83.202101301239-0 which is the latest referenced ART build. [core@cosa-devsh ~]$ ls /usr/sbin/dhclient /usr/sbin/dhclient-script ls: cannot access '/usr/sbin/dhclient': No such file or directory ls: cannot access '/usr/sbin/dhclient-script': No such file or directory [core@cosa-devsh ~]$ However, checking the latest prow passing image in CI (47.83.202102031754-0) I do see the change: [core@cosa-devsh ~]$ ls /usr/sbin/dhclient /usr/sbin/dhclient-script /usr/sbin/dhclient /usr/sbin/dhclient-script [core@cosa-devsh ~]$ Marking as verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633