Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2216530

Summary: Dhcp-client(isc-dhclient-4.4.2b1) packaged with RHEL 9.2 is not compatible with windows server 2019 acting as a dhcpv6 server
Product: Red Hat Enterprise Linux 9 Reporter: rnegi.nb
Component: dhcpAssignee: Martin Osvald 🛹 <mosvald>
Status: CLOSED MIGRATED QA Contact: rhel-cs-infra-services-qe <rhel-cs-infra-services-qe>
Severity: high Docs Contact:
Priority: unspecified    
Version: 9.2CC: mosvald
Target Milestone: rcKeywords: MigratedToJIRA
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-09-21 18:41:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
strace logs from rhel 9.2.
none
strace logs from ubuntu 20.04 none

Description rnegi.nb 2023-06-21 17:11:10 UTC 
Description of problem:
Dhcp-client(isc-dhclient-4.4.2b1) packaged with RHEL 9.2 is not compatible with windows server 2019 acting as a dhcpv6 server.dhcp-client is unable to process DHCPv6 advertise packet coming from Dhcpv6 server. dhcp-client works fine with the dhcpv6 server running in any linux VM(VYOS,ubuntu).

Version-Release number of selected component (if applicable):

dhcp-client : isc-dhclient-4.4.2b1


How reproducible: Always reproducible


Steps to Reproduce:
1.Install dhcp-client package in RHEL 9.2 
2.run dhclient6 with command "dhclient -6 -v <intf_name>".interface from client server are back to back connected.start the dhcpv6 server on windows server VM (2019).
3.Capture ipv6 packets either in client or in server .

Actual results: from the packet captures and verbose logs from client, we are observing that server is sending dhcpv6 advertise packet but somehow client is not able to process it and send dhcpv6 request packet .


Expected results:

Verbose logs from dhclient should show the processing of dhcpv6 advertise packet from dhcpv6 server and  client should send dhcpv6 request packet to server.

Additional info:Issue is there in centos8/rockylinux 9/RHEL 8/RHEL 9.2 linux versions. dhclient packaged in ubuntu linux distribution is working as expected.There might be a good chance that ubuntu/debian community has applied patches on top of isc-dhcp-client package.
Comment 1 Martin Osvald 🛹 2023-06-22 09:50:22 UTC 
Thank you for reporting this issue.

Please, due to the fact, I don't have access to windows server machine right now, could you provide the following details to help me debug this issue further?

Q1. What is the version of Ubuntu and dhcp package you did the test with?
Q2. Would you be able to share dhclient verbose output from both RHEL and Ubuntu?
Q3. Could you also share from both RHEL and Ubuntu dhclient-*.log output of the below commands?

RHEL# strace -qfTttvys 4096 -o /tmp/dhclient-rhel.log dhclient -d -6 -v <intf_name>
Ubuntu# strace -qfTttvys 4096 -o /tmp/dhclient-ubuntu.log dhclient -d -6 -v <intf_name>

Thank you!
Comment 2 rnegi.nb 2023-06-23 08:26:42 UTC 
Hi Martin,
Thanks for looking into this.My answers are below:

A1: ubuntu version (20.4.6 LTS) and dhclient package is 4.4.1.

ubuntu@ubuntu17-ise-Server:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.6 LTS
Release:        20.04
Codename:       focal



ubuntu@ubuntu17-ise-Server:~$ dhclient --version
isc-dhclient-4.4.1


A2: Yes i have collected the verbose output of both RHEL and ubuntu machines.

//////////////////////////////////////////////////////////verbose output from ubuntu VM (20.4)//////////////////////////////////////////////////////


ubuntu@ubuntu17-ise-Server:~$ sudo dhclient -6 -v ens224
Internet Systems Consortium DHCP Client 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on Socket/ens224
Sending on   Socket/ens224
PRC: Soliciting for leases (INIT).
XMT: Forming Solicit, 0 ms elapsed.
XMT:  X-- IA_NA 29:c9:18:88
XMT:  | X-- Request renew in  +3600
XMT:  | X-- Request rebind in +5400
XMT:  | X-- Request address 2019:2018::a74c:d480:e68a:c78b.
XMT:  | | X-- Request preferred in +7200
XMT:  | | X-- Request valid in     +10800
XMT: Solicit on ens224, interval 1050ms.
RCV: Advertise message on ens224 from 2019:2018::1.
RCV:  X-- Preference 1.
RCV:  X-- IA_NA 29:c9:18:88
RCV:  | X-- starts 1687503592
RCV:  | X-- t1 - renew  +900
RCV:  | X-- t2 - rebind +1440
RCV:  | X-- [Options]
RCV:  | | X-- IAADDR 2019:2018::a74c:d480:e68a:c78b
RCV:  | | | X-- Preferred lifetime 1800.
RCV:  | | | X-- Max lifetime 1800.
RCV:  X-- Server ID: 00:01:00:00:63:17:2d:a3:00:0c:29:e0:86:97
RCV:  Advertisement recorded.
PRC: Selecting best advertised lease.
PRC: Considering best lease.
PRC:  X-- Initial candidate 00:01:00:00:63:17:2d:a3:00:0c:29:e0:86:97 (s: 10104, p: 1).
XMT: Forming Request, 0 ms elapsed.
XMT:  X-- IA_NA 29:c9:18:88
XMT:  | X-- Requested renew  +3600
XMT:  | X-- Requested rebind +5400
XMT:  | | X-- IAADDR 2019:2018::a74c:d480:e68a:c78b
XMT:  | | | X-- Preferred lifetime +7200
XMT:  | | | X-- Max lifetime +7500
XMT:  V IA_NA appended.
XMT: Request on ens224, interval 1000ms.
RCV: Reply message on ens224 from 2019:2018::1.
RCV:  X-- IA_NA 29:c9:18:88
RCV:  | X-- starts 1687503593
RCV:  | X-- t1 - renew  +900
RCV:  | X-- t2 - rebind +1440
RCV:  | X-- [Options]
RCV:  | | X-- IAADDR 2019:2018::a74c:d480:e68a:c78b
RCV:  | | | X-- Preferred lifetime 1800.
RCV:  | | | X-- Max lifetime 1800.
RCV:  X-- Server ID: 00:01:00:00:63:17:2d:a3:00:0c:29:e0:86:97
PRC: Bound to lease 00:01:00:00:63:17:2d:a3:00:0c:29:e0:86:97.


//////////////////////////////////////////////////////verbose message of dhclient package from RHEL 9.2////////////////////////////////////////////////

[root@localhost ~]# dhclient -6 -v ens224
Internet Systems Consortium DHCP Client 4.4.2b1
Copyright 2004-2019 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

grep: /etc/sysconfig/network-scripts/ifcfg-*: No such file or directory
grep: /etc/sysconfig/network-scripts/ifcfg-*: No such file or directory
grep: /etc/sysconfig/network-scripts/ifcfg-*: No such file or directory
Listening on Socket/ens224
Sending on   Socket/ens224
PRC: Soliciting for leases (INIT).
XMT: Forming Solicit, 0 ms elapsed.
XMT:  X-- IA_NA 56:83:ca:d3
XMT:  | X-- Request renew in  +3600
XMT:  | X-- Request rebind in +5400
XMT: Solicit on ens224, interval 1090ms.
XMT: Forming Solicit, 1090 ms elapsed.
XMT:  X-- IA_NA 56:83:ca:d3
XMT:  | X-- Request renew in  +3600
XMT:  | X-- Request rebind in +5400
XMT: Solicit on ens224, interval 2170ms.
XMT: Forming Solicit, 3260 ms elapsed.
XMT:  X-- IA_NA 56:83:ca:d3
XMT:  | X-- Request renew in  +3600
XMT:  | X-- Request rebind in +5400
XMT: Solicit on ens224, interval 4190ms.
XMT: Forming Solicit, 7450 ms elapsed.
XMT:  X-- IA_NA 56:83:ca:d3
XMT:  | X-- Request renew in  +3600
XMT:  | X-- Request rebind in +5400
XMT: Solicit on ens224, interval 8060ms.



A3. strace logs from both the machines RHEL and ubuntu are attached as attachments.

RHEL 9.2 : dhclient-rhel.log
Ubuntu 20.04: dhclient-ubuntu.log

NOTE: based on my debugging so far what is observed is that dhclient is somehow silently dropping the dhcpv6 advertise packet coming from windows dhcpv6 server.

@martin we can use free version of windows 2019 server iso and spawn dhcpv6 server in it.Please refer this link https://archive.org/details/en_windows_server_2019_x64_dvd_4cb967d8_202209
Comment 3 rnegi.nb 2023-06-23 08:28:04 UTC 
Created attachment 1972200 [details]
strace logs from rhel 9.2.
Comment 4 rnegi.nb 2023-06-23 08:28:52 UTC 
Created attachment 1972201 [details]
strace logs from ubuntu 20.04
Comment 5 rnegi.nb 2023-06-28 17:25:01 UTC 
Hi Martin,
Do you need any other information?
Regards,
Rahul
Comment 6 rnegi.nb 2023-07-10 07:07:24 UTC 
Hi Martin,
Could you please update the latest on this issue?

Regards,
rahul
Comment 7 rnegi.nb 2023-07-20 08:57:52 UTC 
Hi @mosvald,
Could you please help?

Regards,
rahul
Comment 8 Martin Osvald 🛹 2023-09-05 08:15:17 UTC 
I am sorry for the long delay in my reply and thank you for providing the requested files.


According to the strace output RHEL9 machine never received a reply from Windows dhcp server and it timed out. The below shows dhclient on RHEL9 trying to wait for a reply, but none comes:

~~~
106619 13:39:57.957414 write(2</dev/pts/1>, "XMT: Solicit on ens224, interval 1010ms.", 40) = 40 <0.000045>
...
106619 13:39:57.957983 sendmsg(5<socket:[702235]>, {msg_name={sa_family=AF_INET6, sin6_port=htons(547), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "ff02::1:2", &sin6_addr), sin6_scope_id=if_nametoindex("ens224")}, msg_namelen=28, msg_iov=[{iov_base="\0017\3446\0\1\0\22\0\4\4\313\7\272F\312@w\252\275\351\0^sy\253\0\6\0\4\0\27\0\30\0\10\0\2\0\0\0\3\0\fV\203\312\323\0\0\16\20\0\0\25\30", iov_len=56}], msg_iovlen=1, msg_control=[{cmsg_len=36, cmsg_level=SOL_IPV6, cmsg_type=0x32}], msg_controllen=40, msg_flags=0}, 0) = 56 <0.000103>
106619 13:39:57.958252 pselect6(6, [5<socket:[702235]>], [], NULL, {tv_sec=0, tv_nsec=0}, NULL) = 0 (Timeout) <0.000053>
106619 13:39:57.958412 pselect6(6, [5<socket:[702235]>], [], NULL, {tv_sec=1, tv_nsec=9839000}, NULL) = 0 (Timeout) <1.011040>

...

106619 13:39:58.971225 write(2</dev/pts/1>, "XMT: Solicit on ens224, interval 2100ms.", 40) = 40 <0.000063>
...
106619 13:39:58.972100 sendmsg(5<socket:[702235]>, {msg_name={sa_family=AF_INET6, sin6_port=htons(547), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "ff02::1:2", &sin6_addr), sin6_scope_id=if_nametoindex("ens224")}, msg_namelen=28, msg_iov=[{iov_base="\0017\3446\0\1\0\22\0\4\4\313\7\272F\312@w\252\275\351\0^sy\253\0\6\0\4\0\27\0\30\0\10\0\2\0e\0\3\0\fV\203\312\323\0\0\16\20\0\0\25\30", iov_len=56}], msg_iovlen=1, msg_control=[{cmsg_len=36, cmsg_level=SOL_IPV6, cmsg_type=0x32}], msg_controllen=40, msg_flags=0}, 0) = 56 <0.000088>
106619 13:39:58.972310 pselect6(6, [5<socket:[702235]>], [], NULL, {tv_sec=0, tv_nsec=0}, NULL) = 0 (Timeout) <0.000040>
106619 13:39:58.972416 pselect6(6, [5<socket:[702235]>], [], NULL, {tv_sec=2, tv_nsec=99859000}, NULL) = 0 (Timeout) <2.102163>

... The above repeats repeatedly
~~~


I have compared the RHEL9 patched 4.4.2b1 and Ubuntu patched 4.4.1-2.1ubuntu5.20.04.5 dhcp6 related code executed prior to sending these solicit messages and found nothing different except the generation of client DUID where on Ubuntu it is by default LLT (Link Local Address type of DUID) and on RHEL9 is UUID (RFC6355, this feature introduced with bug 560361, https://bugzilla.redhat.com/show_bug.cgi?id=560361#c40).

Because this client DUID is sent to the dhcp server we should test whether this change in behavior doesn't influence anything on the Windows DHCP server side.

For RHEL9 to match the Ubuntu behavior in this regard, please, could you run the dhclient on RHEL9 with the extra '-D LLT' parameter and see if it changes anything?

~~~
# dhclient -d -6 -D LLT -v ens224
~~~

If this doesn't change anything, we will have to obtain network capture from the Windows DHCP server to see if it really sends anything back to the dhclient.
Comment 9 RHEL Program Management 2023-09-21 18:20:27 UTC 
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.
Comment 10 RHEL Program Management 2023-09-21 18:41:02 UTC 
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.