Bug 1959461

Summary: NetworkManager ignoring Classless routes
Product: Red Hat Enterprise Linux 8 Reporter: mcolombo
Component: NetworkManagerAssignee: Beniamino Galvani <bgalvani>
Status: CLOSED ERRATA QA Contact: Vladimir Benes <vbenes>
Severity: medium Docs Contact:
Priority: medium    
Version: 8.3CC: acardace, atragler, bgalvani, cutaylor, fpokryvk, lrintel, rkhan, sukulkar, thaller, till, vbenes
Target Milestone: betaKeywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: NetworkManager-1.32.0-0.4.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-09 19:30:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
NM journal logs with debug enabled
none
pcap while NM debug was running and also subsequent dhclient was run. none

Description mcolombo 2021-05-11 14:41:18 UTC 
Created attachment 1782054 [details]
NM journal logs with debug enabled

Created attachment 1782054 [details]
NM journal logs with debug enabled

Description of problem:
NetworkMAnager does not populate classless routes provided by dhcp server.

Version-Release number of selected component (if applicable):
NetworkManager-1.22.8-4.el8 and later

How reproducible:
Every time

Steps to Reproduce:
1.Configure NetworkManager profile containing requested_ms_classless_static_routes = 1
requested_rfc3442_classless_static_routes = 1
2.reboot
3.confirm classless routes are not present in routing table

Actual results:
Classless routes not present in routing table post dhcp request

Expected results:
classless routes be present after dhcp request. 

Additional info:
Gathered NetworkManager debug logs and pcap while making dhcp request. We can see the dhcp server provides the classless routes, but NetworkManager ignores these. 

This also worked as expected in RHEL 8.0 and 8.1. Customer simply notes runing dhclient the classless routes will be populated.
Comment 1 mcolombo 2021-05-11 14:43:27 UTC 
Created attachment 1782055 [details]
pcap while NM debug was running and also subsequent dhclient was run.
Comment 2 mcolombo 2021-05-11 14:46:49 UTC 
Initial Analysis:
=================
# grep dhcp NM_logs.txt 
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4110] dhcp-init: Using DHCP client 'internal'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4723] dhcp4 (ens3): activation: beginning transaction (timeout in 45 seconds)
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4864] dhcp4 (ens3): option dhcp_lease_time      => '604800'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4864] dhcp4 (ens3): option domain_name          => 'us01-odc.synopsys.com'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4864] dhcp4 (ens3): option domain_name_servers  => '10.228.0.1 10.228.0.2'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4864] dhcp4 (ens3): option expiry               => '1620874288'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4864] dhcp4 (ens3): option host_name            => 'temp-qscr-rh82--yesu-0505-1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4865] dhcp4 (ens3): option interface_mtu        => '1500'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4865] dhcp4 (ens3): option ip_address           => '10.197.200.104'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4865] dhcp4 (ens3): option next_server          => '10.197.200.13'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4865] dhcp4 (ens3): option requested_broadcast_address => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4865] dhcp4 (ens3): option requested_domain_name => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4865] dhcp4 (ens3): option requested_domain_name_servers => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4865] dhcp4 (ens3): option requested_domain_search => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4865] dhcp4 (ens3): option requested_host_name  => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4865] dhcp4 (ens3): option requested_interface_mtu => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4865] dhcp4 (ens3): option requested_ms_classless_static_routes => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4865] dhcp4 (ens3): option requested_nis_domain => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4865] dhcp4 (ens3): option requested_nis_servers => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4865] dhcp4 (ens3): option requested_ntp_servers => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4866] dhcp4 (ens3): option requested_rfc3442_classless_static_routes => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4866] dhcp4 (ens3): option requested_root_path  => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4866] dhcp4 (ens3): option requested_routers    => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4866] dhcp4 (ens3): option requested_static_routes => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4866] dhcp4 (ens3): option requested_subnet_mask => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4866] dhcp4 (ens3): option requested_time_offset => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4866] dhcp4 (ens3): option requested_wpad       => '1'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4866] dhcp4 (ens3): option routers              => '10.197.203.254'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4866] dhcp4 (ens3): option subnet_mask          => '255.255.252.0'
May 05 19:51:28 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620269488.4866] dhcp4 (ens3): state changed unknown -> bound

-------- the request in the capture starts here ------

May 05 12:56:34 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620244594.9458] dhcp4 (ens3): canceled DHCP transaction
May 05 12:56:34 temp-qscr-rh82--yesu-0505-1 NetworkManager[1112]: <info>  [1620244594.9458] dhcp4 (ens3): state changed bound -> done
May 05 12:56:34 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <debug> [1620244594.9982] platform: (ens3) signal: route   4   added: 0.0.0.0/0 via 10.197.203.254 dev 2 metric 100 mss 0 rt-src rt-dhcp scope global
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <debug> [1620244595.2752] dhcp-init: enabled DHCP client 'dhclient'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <debug> [1620244595.2752] dhcp-init: enabled DHCP client 'internal'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <debug> [1620244595.2752] dhcp-init: enabled DHCP client 'systemd' (undocumented internal plugin)
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <debug> [1620244595.2752] dhcp-init: enabled DHCP client 'nettools' (undocumented internal plugin)
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2752] dhcp-init: Using DHCP client 'internal'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <debug> [1620244595.2883] device[d18462b73bff3c25] (ens3): ipv4.dhcp-client-id: no explicit client-id configured
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2884] dhcp4 (ens3): activation: beginning transaction (timeout in 45 seconds)
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <debug> [1620244595.2897] dhcp4 (ens3): send REQUEST to 255.255.255.255
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <debug> [1620244595.2923] dhcp4 (ens3): received ACK of 10.197.200.104 from 10.197.200.11  <---- here's the ack
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2938] dhcp4 (ens3): option dhcp_lease_time      => '604800'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2938] dhcp4 (ens3): option domain_name          => 'us01-odc.synopsys.com'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2938] dhcp4 (ens3): option domain_name_servers  => '10.228.0.1 10.228.0.2'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2938] dhcp4 (ens3): option expiry               => '1620849395'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2938] dhcp4 (ens3): option host_name            => 'temp-qscr-rh82--yesu-0505-1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2938] dhcp4 (ens3): option interface_mtu        => '1500'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2938] dhcp4 (ens3): option ip_address           => '10.197.200.104'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option next_server          => '10.197.200.11'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option requested_broadcast_address => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option requested_domain_name => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option requested_domain_name_servers => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option requested_domain_search => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option requested_host_name  => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option requested_interface_mtu => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option requested_ms_classless_static_routes => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option requested_nis_domain => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option requested_nis_servers => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option requested_ntp_servers => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option requested_rfc3442_classless_static_routes => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option requested_root_path  => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2939] dhcp4 (ens3): option requested_routers    => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2940] dhcp4 (ens3): option requested_static_routes => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2940] dhcp4 (ens3): option requested_subnet_mask => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2940] dhcp4 (ens3): option requested_time_offset => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2940] dhcp4 (ens3): option requested_wpad       => '1'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2940] dhcp4 (ens3): option routers              => '10.197.203.254'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2940] dhcp4 (ens3): option subnet_mask          => '255.255.252.0'
May 05 12:56:35 temp-qscr-rh82--yesu-0505-1 NetworkManager[8744]: <info>  [1620244595.2940] dhcp4 (ens3): state changed unknown -> bound


# tshark -r temp-qscr-rh82--yesu-0505-1.pcap udp.port==67
  219  32.401402      0.0.0.0 → 255.255.255.255 DHCP 353 DHCP Request  - Transaction ID 0x9c82808c
  221  32.404001 10.197.200.11 → 10.197.200.104 DHCP 525 DHCP ACK      - Transaction ID 0x9c82808c
  222  32.404047 10.197.200.104 → 10.197.200.11 ICMP 553 Destination unreachable (Port unreachable)  <---- why are we returning this?
  223  32.404504 10.197.200.13 → 10.197.200.104 DHCP 525 DHCP ACK      - Transaction ID 0x9c82808c
  224  32.404522 10.197.200.104 → 10.197.200.13 ICMP 553 Destination unreachable (Port unreachable)  <---- why are we returning this?

------ this is where dhclient was run and classroutes are populated----

  384  50.863207      0.0.0.0 → 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0x9ac9cd60
  385  50.865118 10.197.200.13 → 10.197.200.104 DHCP 522 DHCP Offer    - Transaction ID 0x9ac9cd60
  386  50.865298      0.0.0.0 → 255.255.255.255 DHCP 342 DHCP Request  - Transaction ID 0x9ac9cd60
  387  50.865402 10.197.200.12 → 10.197.200.104 DHCP 522 DHCP Offer    - Transaction ID 0x9ac9cd60
  388  50.865410 10.197.200.11 → 10.197.200.104 DHCP 522 DHCP Offer    - Transaction ID 0x9ac9cd60
  389  50.867638 10.197.200.13 → 10.197.200.104 DHCP 522 DHCP ACK      - Transaction ID 0x9ac9cd60



Also why does it look like multiple dhcp servers are responding?


Further investigation:
======================


      Lets take a look at frames 219,221 and 222


# tshark -V -r temp-qscr-rh82--yesu-0505-1.pcap frame.number==219
Frame 219: 353 bytes on wire (2824 bits), 353 bytes captured (2824 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: May  5, 2021 15:56:35.289590000 EDT
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1620244595.289590000 seconds
    [Time delta from previous captured frame: 0.228047000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 32.401402000 seconds]
    Frame Number: 219
    Frame Length: 353 bytes (2824 bits)
    Capture Length: 353 bytes (2824 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:udp:bootp]
Ethernet II, Src: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
        Address: Broadcast (ff:ff:ff:ff:ff:ff)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a)
        Address: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 0.0.0.0, Dst: 255.255.255.255
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0xc0 (DSCP: CS6, ECN: Not-ECT)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (48)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 339
    Identification: 0x0000 (0)
    Flags: 0x02 (Don't Fragment)
        0... .... = Reserved bit: Not set
        .1.. .... = Don't fragment: Set
        ..0. .... = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: UDP (17)
    Header checksum: 0x38db [correct]
    [Header checksum status: Good]
    [Calculated Checksum: 0x38db]
    Source: 0.0.0.0
    Destination: 255.255.255.255
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
User Datagram Protocol, Src Port: 68, Dst Port: 67
    Source Port: 68
    Destination Port: 67
    Length: 319
    Checksum: 0xfd90 [unverified]
    [Checksum Status: Unverified]
    [Stream index: 4]
Bootstrap Protocol (Request)
    Message type: Boot Request (1)
    Hardware type: Ethernet (0x01)
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x9c82808c
    Seconds elapsed: 1
    Bootp flags: 0x0000 (Unicast)
        0... .... .... .... = Broadcast flag: Unicast
        .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0
    Your (client) IP address: 0.0.0.0
    Next server IP address: 0.0.0.0
    Relay agent IP address: 0.0.0.0
    Client MAC address: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: DHCP
    Option: (53) DHCP Message Type (Request)
        Length: 1
        DHCP: Request (3)
    Option: (61) Client identifier
        Length: 7
        Hardware type: Ethernet (0x01)
        Client MAC address: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a)
    Option: (55) Parameter Request List
        Length: 17
        Parameter Request List Item: (1) Subnet Mask
        Parameter Request List Item: (2) Time Offset
        Parameter Request List Item: (6) Domain Name Server
        Parameter Request List Item: (12) Host Name
        Parameter Request List Item: (15) Domain Name
        Parameter Request List Item: (26) Interface MTU
        Parameter Request List Item: (28) Broadcast Address
        Parameter Request List Item: (121) Classless Static Route
        Parameter Request List Item: (3) Router
        Parameter Request List Item: (33) Static Route
        Parameter Request List Item: (40) Network Information Service Domain
        Parameter Request List Item: (41) Network Information Service Servers
        Parameter Request List Item: (42) Network Time Protocol Servers
        Parameter Request List Item: (119) Domain Search
        Parameter Request List Item: (249) Private/Classless Static Route (Microsoft)
        Parameter Request List Item: (252) Private/Proxy autodiscovery
        Parameter Request List Item: (17) Root Path
    Option: (57) Maximum DHCP Message Size
        Length: 2
        Maximum DHCP Message Size: 576
    Option: (50) Requested IP Address
        Length: 4
        Requested IP Address: 10.197.200.104
    Option: (12) Host Name
        Length: 27
        Host Name: temp-qscr-rh82--yesu-0505-1
    Option: (255) End
        Option End: 255



# tshark -V -r temp-qscr-rh82--yesu-0505-1.pcap frame.number==221
Frame 221: 525 bytes on wire (4200 bits), 525 bytes captured (4200 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: May  5, 2021 15:56:35.292189000 EDT
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1620244595.292189000 seconds
    [Time delta from previous captured frame: 0.000442000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 32.404001000 seconds]
    Frame Number: 221
    Frame Length: 525 bytes (4200 bits)
    Capture Length: 525 bytes (4200 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:udp:bootp]
Ethernet II, Src: fa:16:3e:09:31:7c (fa:16:3e:09:31:7c), Dst: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a)
    Destination: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a)
        Address: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: fa:16:3e:09:31:7c (fa:16:3e:09:31:7c)
        Address: fa:16:3e:09:31:7c (fa:16:3e:09:31:7c)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.197.200.11, Dst: 10.197.200.104
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0xc0 (DSCP: CS6, ECN: Not-ECT)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (48)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 511
    Identification: 0xb8f8 (47352)
    Flags: 0x00
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: UDP (17)
    Header checksum: 0x1938 [correct]
    [Header checksum status: Good]
    [Calculated Checksum: 0x1938]
    Source: 10.197.200.11
    Destination: 10.197.200.104
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
User Datagram Protocol, Src Port: 67, Dst Port: 68
    Source Port: 67
    Destination Port: 68
    Length: 491
    Checksum: 0xe0aa [unverified]
    [Checksum Status: Unverified]
    [Stream index: 5]
Bootstrap Protocol (ACK)
    Message type: Boot Reply (2)
    Hardware type: Ethernet (0x01)
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x9c82808c
    Seconds elapsed: 1
    Bootp flags: 0x0000 (Unicast)
        0... .... .... .... = Broadcast flag: Unicast
        .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0
    Your (client) IP address: 10.197.200.104
    Next server IP address: 10.197.200.11
    Relay agent IP address: 0.0.0.0
    Client MAC address: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a)
    Client hardware address padding: 00000000000000000000
    Server name option overloaded by DHCP
        [Expert Info (Note/Protocol): Server name option overloaded by DHCP]
            [Server name option overloaded by DHCP]
            [Severity level: Note]
            [Group: Protocol]
    Boot file name option overloaded by DHCP
        [Expert Info (Note/Protocol): Boot file name option overloaded by DHCP]
            [Boot file name option overloaded by DHCP]
            [Severity level: Note]
            [Group: Protocol]
    Magic cookie: DHCP
    Option: (53) DHCP Message Type (ACK)
        Length: 1
        DHCP: ACK (5)
    Option: (54) DHCP Server Identifier
        Length: 4
        DHCP Server Identifier: 10.197.200.11
    Option: (51) IP Address Lease Time
        Length: 4
        IP Address Lease Time: (604800s) 7 days
    Option: (58) Renewal Time Value
        Length: 4
        Renewal Time Value: (302400s) 3 days, 12 hours
    Option: (59) Rebinding Time Value
        Length: 4
        Rebinding Time Value: (529200s) 6 days, 3 hours
    Option: (1) Subnet Mask
        Length: 4
        Subnet Mask: 255.255.252.0
    Option: (28) Broadcast Address
        Length: 4
        Broadcast Address: 10.197.203.255
    Option: (15) Domain Name
        Length: 21
        Domain Name: us01-odc.synopsys.com
    Option: (12) Host Name
        Length: 27
        Host Name: temp-qscr-rh82--yesu-0505-1
    Option: (3) Router
        Length: 4
        Router: 10.197.203.254
    Option: (249) Private/Classless Static Route (Microsoft)   <------ The classless routes are in the ack from the dhcp server. 
        Length: 126
         10.185.64.0/22-10.197.203.254
         10.197.192.0/22-10.197.203.254
         10.197.196.0/22-10.197.203.254
         10.197.204.0/22-10.197.203.254
         10.197.208.0/22-10.197.203.254
         10.197.212.0/22-10.197.203.254
         10.197.216.0/22-10.197.203.254
         10.197.224.0/22-10.197.203.254
         10.197.228.0/22-10.197.203.254
         10.197.232.0/22-10.197.203.254
         10.197.236.0/22-10.197.203.254
         10.197.240.0/22-10.197.203.254
         10.197.244.0/22-10.197.203.254
         10.197.252.0/22-10.197.203.254
         169.254.169.254/32-10.197.200.11
         default-10.197.203.254
    Option: (52) Option Overload
        Length: 1
        Option Overload: Boot file and server host names hold options (3)
    Boot file name option overload
    Option: (255) End
        Option End (Overload): 255
    Server host name option overload
    Option: (255) End
        Option End (Overload): 255
    Option: (6) Domain Name Server
        Length: 8
        Domain Name Server: 10.228.0.1
        Domain Name Server: 10.228.0.2
    Option: (26) Interface MTU
        Length: 2
        Interface MTU: 1500
    Option: (255) End
        Option End: 255


# tshark -V -r temp-qscr-rh82--yesu-0505-1.pcap frame.number==222
Frame 222: 553 bytes on wire (4424 bits), 553 bytes captured (4424 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: May  5, 2021 15:56:35.292235000 EDT
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1620244595.292235000 seconds
    [Time delta from previous captured frame: 0.000046000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 32.404047000 seconds]
    Frame Number: 222
    Frame Length: 553 bytes (4424 bits)
    Capture Length: 553 bytes (4424 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:icmp:ip:udp:bootp]
Ethernet II, Src: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a), Dst: fa:16:3e:09:31:7c (fa:16:3e:09:31:7c)
    Destination: fa:16:3e:09:31:7c (fa:16:3e:09:31:7c)
        Address: fa:16:3e:09:31:7c (fa:16:3e:09:31:7c)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a)
        Address: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.197.200.104, Dst: 10.197.200.11
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0xc0 (DSCP: CS6, ECN: Not-ECT)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (48)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 539
    Identification: 0xc73b (51003)
    Flags: 0x00
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: ICMP (1)
    Header checksum: 0x0ae9 [correct]
    [Header checksum status: Good]
    [Calculated Checksum: 0x0ae9]
    Source: 10.197.200.104
    Destination: 10.197.200.11
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
Internet Control Message Protocol
    Type: 3 (Destination unreachable)
    Code: 3 (Port unreachable)
    Checksum: 0xa4f7 [correct]
    [Checksum Status: Good]
    Unused: 00000000
    Internet Protocol Version 4, Src: 10.197.200.11, Dst: 10.197.200.104
        0100 .... = Version: 4
        .... 0101 = Header Length: 20 bytes (5)
        Differentiated Services Field: 0xc0 (DSCP: CS6, ECN: Not-ECT)
            1100 00.. = Differentiated Services Codepoint: Class Selector 6 (48)
            .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
        Total Length: 511
        Identification: 0xb8f8 (47352)
        Flags: 0x00
            0... .... = Reserved bit: Not set
            .0.. .... = Don't fragment: Not set
            ..0. .... = More fragments: Not set
        Fragment offset: 0
        Time to live: 64
        Protocol: UDP (17)
        Header checksum: 0x1938 [in ICMP error packet]
        [Header checksum status: Unverified]
        Source: 10.197.200.11
        Destination: 10.197.200.104
        [Source GeoIP: Unknown]
        [Destination GeoIP: Unknown]
    User Datagram Protocol, Src Port: 67, Dst Port: 68
        Source Port: 67
        Destination Port: 68
        Length: 491
        Checksum: 0xe0aa [unverified]
        [Checksum Status: Unverified]
        [Stream index: 5]
    Bootstrap Protocol (ACK)
        Message type: Boot Reply (2)
        Hardware type: Ethernet (0x01)
        Hardware address length: 6
        Hops: 0
        Transaction ID: 0x9c82808c
        Seconds elapsed: 1
        Bootp flags: 0x0000 (Unicast)
            0... .... .... .... = Broadcast flag: Unicast
            .000 0000 0000 0000 = Reserved flags: 0x0000
        Client IP address: 0.0.0.0
        Your (client) IP address: 10.197.200.104
        Next server IP address: 10.197.200.11
        Relay agent IP address: 0.0.0.0
        Client MAC address: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a)
        Client hardware address padding: 00000000000000000000
        Server name option overloaded by DHCP
            [Expert Info (Note/Protocol): Server name option overloaded by DHCP]
                [Server name option overloaded by DHCP]
                [Severity level: Note]
                [Group: Protocol]
        Boot file name option overloaded by DHCP
            [Expert Info (Note/Protocol): Boot file name option overloaded by DHCP]
                [Boot file name option overloaded by DHCP]
                [Severity level: Note]
                [Group: Protocol]
        Magic cookie: DHCP
        Option: (53) DHCP Message Type (ACK)
            Length: 1
            DHCP: ACK (5)
        Option: (54) DHCP Server Identifier
            Length: 4
            DHCP Server Identifier: 10.197.200.11
        Option: (51) IP Address Lease Time
            Length: 4
            IP Address Lease Time: (604800s) 7 days
        Option: (58) Renewal Time Value
            Length: 4
            Renewal Time Value: (302400s) 3 days, 12 hours
        Option: (59) Rebinding Time Value
            Length: 4
            Rebinding Time Value: (529200s) 6 days, 3 hours
        Option: (1) Subnet Mask
            Length: 4
            Subnet Mask: 255.255.252.0
        Option: (28) Broadcast Address
            Length: 4
            Broadcast Address: 10.197.203.255
        Option: (15) Domain Name
            Length: 21
            Domain Name: us01-odc.synopsys.com
        Option: (12) Host Name
            Length: 27
            Host Name: temp-qscr-rh82--yesu-0505-1
        Option: (3) Router
            Length: 4
            Router: 10.197.203.254
        Option: (249) Private/Classless Static Route (Microsoft)
            Length: 126
             10.185.64.0/22-10.197.203.254
             10.197.192.0/22-10.197.203.254
             10.197.196.0/22-10.197.203.254
             10.197.204.0/22-10.197.203.254
             10.197.208.0/22-10.197.203.254
             10.197.212.0/22-10.197.203.254
             10.197.216.0/22-10.197.203.254
             10.197.224.0/22-10.197.203.254
             10.197.228.0/22-10.197.203.254
             10.197.232.0/22-10.197.203.254
             10.197.236.0/22-10.197.203.254
             10.197.240.0/22-10.197.203.254
             10.197.244.0/22-10.197.203.254
             10.197.252.0/22-10.197.203.254
             169.254.169.254/32-10.197.200.11
             default-10.197.203.254
        Option: (52) Option Overload
            Length: 1
            Option Overload: Boot file and server host names hold options (3)
        Boot file name option overload
        Option: (255) End
            Option End (Overload): 255
        Server host name option overload
        Option: (255) End
            Option End (Overload): 255
        Option: (6) Domain Name Server
            Length: 8
            Domain Name Server: 10.228.0.1
            Domain Name Server: 10.228.0.2
        Option: (26) Interface MTU
            Length: 2
            Interface MTU: 1500
        Option: (255) End
            Option End: 255



- Colombo
Comment 3 Beniamino Galvani 2021-05-11 15:31:00 UTC 
> # tshark -r temp-qscr-rh82--yesu-0505-1.pcap udp.port==67
>  219  32.401402      0.0.0.0 → 255.255.255.255 DHCP 353 DHCP Request  - Transaction ID 0x9c82808c
>  221  32.404001 10.197.200.11 → 10.197.200.104 DHCP 525 DHCP ACK      - Transaction ID 0x9c82808c
>  222  32.404047 10.197.200.104 → 10.197.200.11 ICMP 553 Destination unreachable (Port unreachable)  <---- why are we returning this?

I don't know, I will investigate.

> Also why does it look like multiple dhcp servers are responding?

It is ok to have multiple DHCP servers for redundancy; the client
chooses an offer among all the replies.

>    Option: (249) Private/Classless Static Route (Microsoft)   <------ The classless routes are in the ack from the dhcp server.

Routes are usually pushed through option 33 (Static Route [1]) or
option 121 (Classless Static Route [2]). Option 249 is a Microsoft
extension [3] currently not supported by the internal client of
NM. Since the format of the Microsoft option is the same as the
standard one, it should be easy to add support for the option to
the internal client.

[1] https://datatracker.ietf.org/doc/html/rfc2132#section-5.8
[2] https://datatracker.ietf.org/doc/html/rfc3442
[3] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dhcpe/f9c19c79-1c7f-4746-b555-0c0fc523f3f9
Comment 4 mcolombo 2021-05-11 16:00:30 UTC 
(In reply to Beniamino Galvani from comment #3)
> > # tshark -r temp-qscr-rh82--yesu-0505-1.pcap udp.port==67
> >  219  32.401402      0.0.0.0 → 255.255.255.255 DHCP 353 DHCP Request  - Transaction ID 0x9c82808c
> >  221  32.404001 10.197.200.11 → 10.197.200.104 DHCP 525 DHCP ACK      - Transaction ID 0x9c82808c
> >  222  32.404047 10.197.200.104 → 10.197.200.11 ICMP 553 Destination unreachable (Port unreachable)  <---- why are we returning this?
> 
> I don't know, I will investigate.
> 
> > Also why does it look like multiple dhcp servers are responding?
> 
> It is ok to have multiple DHCP servers for redundancy; the client
> chooses an offer among all the replies.
> 
> >    Option: (249) Private/Classless Static Route (Microsoft)   <------ The classless routes are in the ack from the dhcp server.
> 
> Routes are usually pushed through option 33 (Static Route [1]) or
> option 121 (Classless Static Route [2]). Option 249 is a Microsoft
> extension [3] currently not supported by the internal client of
> NM. Since the format of the Microsoft option is the same as the
> standard one, it should be easy to add support for the option to
> the internal client.
> 
> [1] https://datatracker.ietf.org/doc/html/rfc2132#section-5.8
> [2] https://datatracker.ietf.org/doc/html/rfc3442
> [3]
> https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dhcpe/
> f9c19c79-1c7f-4746-b555-0c0fc523f3f9


Thank you for this. I believe it was part of my missing breadcrumb. NetworkManager actually requests the option 249 classless routes. This can be seen in frame 219. If this is not supported by internal why are we requesting it. 

# tshark -V -r temp-qscr-rh82--yesu-0505-1.pcap frame.number==219
Frame 219: 353 bytes on wire (2824 bits), 353 bytes captured (2824 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: May  5, 2021 15:56:35.289590000 EDT
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1620244595.289590000 seconds
    [Time delta from previous captured frame: 0.228047000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 32.401402000 seconds]
    Frame Number: 219
    Frame Length: 353 bytes (2824 bits)
    Capture Length: 353 bytes (2824 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:udp:bootp]
Ethernet II, Src: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
        Address: Broadcast (ff:ff:ff:ff:ff:ff)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a)
        Address: fa:16:3e:a0:09:3a (fa:16:3e:a0:09:3a)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 0.0.0.0, Dst: 255.255.255.255

----- snip -----

    Option: (55) Parameter Request List
        Length: 17
        Parameter Request List Item: (1) Subnet Mask
        Parameter Request List Item: (2) Time Offset
        Parameter Request List Item: (6) Domain Name Server
        Parameter Request List Item: (12) Host Name
        Parameter Request List Item: (15) Domain Name
        Parameter Request List Item: (26) Interface MTU
        Parameter Request List Item: (28) Broadcast Address
        Parameter Request List Item: (121) Classless Static Route
        Parameter Request List Item: (3) Router
        Parameter Request List Item: (33) Static Route
        Parameter Request List Item: (40) Network Information Service Domain
        Parameter Request List Item: (41) Network Information Service Servers
        Parameter Request List Item: (42) Network Time Protocol Servers
        Parameter Request List Item: (119) Domain Search
        Parameter Request List Item: (249) Private/Classless Static Route (Microsoft)   <-------
        Parameter Request List Item: (252) Private/Proxy autodiscovery
        Parameter Request List Item: (17) Root Path
    Option: (57) Maximum DHCP Message Size
        Length: 2
        Maximum DHCP Message Size: 576
    Option: (50) Requested IP Address
        Length: 4
        Requested IP Address: 10.197.200.104
    Option: (12) Host Name
        Length: 27
        Host Name: temp-qscr-rh82--yesu-0505-1
    Option: (255) End
        Option End: 255



Additionally looking at the profile in NetworkManager I can see that this is also set to 1. 


# grep classless sos_commands/networkmanager/nmcli_con_show_id_System_ens3 
DHCP4.OPTION[15]:                       requested_ms_classless_static_routes = 1    <------ I assume that is what this is.
DHCP4.OPTION[19]:                       requested_rfc3442_classless_static_routes = 1


- Colombo
Comment 5 Beniamino Galvani 2021-05-11 17:27:08 UTC 
> Thank you for this. I believe it was part of my missing breadcrumb. NetworkManager actually requests the option 249 classless routes. This can be seen in frame 219. If this is not supported by internal why are we requesting it.

Good point, we request the option but don't parse it. This upstream merge request adds the missing bits.

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/851

As a workaround, you can switch to the 'dhclient' DHCP backend by setting dhcp=dhclient in the [main] section of NetworkManager.conf.
Comment 12 errata-xmlrpc 2021-11-09 19:30:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: NetworkManager security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4361