Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1656520 - NetworkManager keeps the activated status on DHCP renew failure
Summary: NetworkManager keeps the activated status on DHCP renew failure
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: NetworkManager
Version: 7.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Beniamino Galvani
QA Contact: Desktop QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-05 16:58 UTC by Matteo Croce
Modified: 2020-04-23 07:28 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-23 07:28:25 UTC
Target Upstream Version:


Attachments (Terms of Use)
journalctl -b -u NetworkManager (422.68 KB, text/plain)
2018-12-05 17:53 UTC, Matteo Croce
no flags Details

Description Matteo Croce 2018-12-05 16:58:57 UTC
When a DHCP lease expires and the DHCP server doesn't respond, NetworkManager keep the connection in the activated status, despite that the interface doesn't have any IPv4 or IPv6 address:

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: em3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 24:6e:96:5e:e5:9c brd ff:ff:ff:ff:ff:ff
3: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 24:6e:96:5e:e5:98 brd ff:ff:ff:ff:ff:ff
4: em4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 24:6e:96:5e:e5:9d brd ff:ff:ff:ff:ff:ff
5: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 24:6e:96:5e:e5:9a brd ff:ff:ff:ff:ff:ff

# nmcli
em3: connected to System em3
        "Intel I350"
        ethernet (igb), 24:6E:96:5E:E5:9C, hw, mtu 1500
        ip4 default, ip6 default
        route6 2620:52:0:13bc::/64
...

# nmcli connection show System\ em3
connection.id:                          System em3
connection.uuid:                        a66791f8-3ca1-46d0-862a-6bd8ef55b680
connection.stable-id:                   --
connection.type:                        802-3-ethernet
connection.interface-name:              em3
connection.autoconnect:                 yes
connection.autoconnect-priority:        0
connection.autoconnect-retries:         -1 (default)
connection.auth-retries:                -1
connection.timestamp:                   1544028942
connection.read-only:                   no
connection.permissions:                 --
connection.zone:                        --
connection.master:                      --
connection.slave-type:                  --
connection.autoconnect-slaves:          -1 (default)
connection.secondaries:                 --
connection.gateway-ping-timeout:        0
connection.metered:                     unknown
connection.lldp:                        default
connection.mdns:                        -1 (default)
802-3-ethernet.port:                    --
802-3-ethernet.speed:                   0
802-3-ethernet.duplex:                  --
802-3-ethernet.auto-negotiate:          no
802-3-ethernet.mac-address:             --
802-3-ethernet.cloned-mac-address:      --
802-3-ethernet.generate-mac-address-mask:--
802-3-ethernet.mac-address-blacklist:   --
802-3-ethernet.mtu:                     auto
802-3-ethernet.s390-subchannels:        --
802-3-ethernet.s390-nettype:            --
802-3-ethernet.s390-options:            --
802-3-ethernet.wake-on-lan:             default
802-3-ethernet.wake-on-lan-password:    --
ipv4.method:                            auto
ipv4.dns:                               --
ipv4.dns-search:                        --
ipv4.dns-options:                       ""
ipv4.dns-priority:                      0
ipv4.addresses:                         --
ipv4.gateway:                           --
ipv4.routes:                            --
ipv4.route-metric:                      -1
ipv4.route-table:                       0 (unspec)
ipv4.ignore-auto-routes:                no
ipv4.ignore-auto-dns:                   no
ipv4.dhcp-client-id:                    --
ipv4.dhcp-timeout:                      0 (default)
ipv4.dhcp-send-hostname:                yes
ipv4.dhcp-hostname:                     wsfd-netdev76.ntdv.lab.eng.bos.redhat.com
ipv4.dhcp-fqdn:                         --
ipv4.never-default:                     no
ipv4.may-fail:                          yes
ipv4.dad-timeout:                       -1 (default)
ipv6.method:                            auto
ipv6.dns:                               --
ipv6.dns-search:                        --
ipv6.dns-options:                       ""
ipv6.dns-priority:                      0
ipv6.addresses:                         --
ipv6.gateway:                           --
ipv6.routes:                            --
ipv6.route-metric:                      -1
ipv6.route-table:                       0 (unspec)
ipv6.ignore-auto-routes:                no
ipv6.ignore-auto-dns:                   no
ipv6.never-default:                     no
ipv6.may-fail:                          yes
ipv6.ip6-privacy:                       -1 (unknown)
ipv6.addr-gen-mode:                     eui64
ipv6.dhcp-duid:                         --
ipv6.dhcp-send-hostname:                yes
ipv6.dhcp-hostname:                     wsfd-netdev76.ntdv.lab.eng.bos.redhat.com
ipv6.token:                             --
proxy.method:                           none
proxy.browser-only:                     no
proxy.pac-url:                          --
proxy.pac-script:                       --
GENERAL.NAME:                           System em3
GENERAL.UUID:                           a66791f8-3ca1-46d0-862a-6bd8ef55b680
GENERAL.DEVICES:                        em3
GENERAL.STATE:                          activated
GENERAL.DEFAULT:                        yes
GENERAL.DEFAULT6:                       yes
GENERAL.SPEC-OBJECT:                    --
GENERAL.VPN:                            no
GENERAL.DBUS-PATH:                      /org/freedesktop/NetworkManager/ActiveConnection/1
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/Settings/3
GENERAL.ZONE:                           --
GENERAL.MASTER-PATH:                    --
IP4.GATEWAY:                            --
IP4.DNS[1]:                             10.16.36.29
IP4.DNS[2]:                             10.11.5.19
IP4.DNS[3]:                             10.5.30.160
IP4.DOMAIN[1]:                          ntdv.lab.eng.bos.redhat.com
DHCP4.OPTION[1]:                        broadcast_address = 10.19.189.255
DHCP4.OPTION[2]:                        dhcp_lease_time = 86400
DHCP4.OPTION[3]:                        dhcp_message_type = 5
DHCP4.OPTION[4]:                        dhcp_server_identifier = 10.16.36.29
DHCP4.OPTION[5]:                        domain_name = ntdv.lab.eng.bos.redhat.com
DHCP4.OPTION[6]:                        domain_name_servers = 10.16.36.29 10.11.5.19 10.5.30.160
DHCP4.OPTION[7]:                        expiry = 1544040347
DHCP4.OPTION[8]:                        filename = pxelinux.0
DHCP4.OPTION[9]:                        ip_address = 10.19.188.157
DHCP4.OPTION[10]:                       network_number = 10.19.188.0
DHCP4.OPTION[11]:                       next_server = 10.19.42.13
DHCP4.OPTION[12]:                       ntp_servers = 10.19.189.254 10.5.26.10 10.5.27.10
DHCP4.OPTION[13]:                       requested_broadcast_address = 1
DHCP4.OPTION[14]:                       requested_classless_static_routes = 1
DHCP4.OPTION[15]:                       requested_domain_name = 1
DHCP4.OPTION[16]:                       requested_domain_name_servers = 1
DHCP4.OPTION[17]:                       requested_domain_search = 1
DHCP4.OPTION[18]:                       requested_host_name = 1
DHCP4.OPTION[19]:                       requested_interface_mtu = 1
DHCP4.OPTION[20]:                       requested_ms_classless_static_routes = 1
DHCP4.OPTION[21]:                       requested_nis_domain = 1
DHCP4.OPTION[22]:                       requested_nis_servers = 1
DHCP4.OPTION[23]:                       requested_ntp_servers = 1
DHCP4.OPTION[24]:                       requested_rfc3442_classless_static_routes = 1
DHCP4.OPTION[25]:                       requested_routers = 1
DHCP4.OPTION[26]:                       requested_static_routes = 1
DHCP4.OPTION[27]:                       requested_subnet_mask = 1
DHCP4.OPTION[28]:                       requested_time_offset = 1
DHCP4.OPTION[29]:                       requested_wpad = 1
DHCP4.OPTION[30]:                       routers = 10.19.189.254
DHCP4.OPTION[31]:                       subnet_mask = 255.255.254.0
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = 2620:52:0:13bc::/64, nh = ::, mt = 100

Comment 2 Matteo Croce 2018-12-05 17:35:17 UTC
I have discovered what triggered the bug.
The DHCP server was not responding because the IP and subnet of the DHCP server changed between the offer and the renew.
So NetworkManager was still sending the renew to the old IP address. Bringing the connection down and up manually restored the connectivity.

Comment 3 Matteo Croce 2018-12-05 17:53:49 UTC
Created attachment 1511834 [details]
journalctl -b -u NetworkManager

Comment 4 Beniamino Galvani 2018-12-06 07:55:10 UTC
Hi,

at 'Dec 04 21:05:47' interface em3 gets a lease with 1 day lifetime:

  dhcp4 (em3):   address 10.19.188.157
  dhcp4 (em3):   plen 23 (255.255.254.0)
  dhcp4 (em3):   gateway 10.19.189.254
  dhcp4 (em3):   lease time 86400

and dhclient schedules the renewal in ~9 hours:

  dhclient[11256]: bound to 10.19.188.157 -- renewal in 33801 seconds.

At 'Dec 05 06:29:08' dhclient starts trying to renew the lease without success:

  dhclient[11256]: DHCPREQUEST on em3 to 10.16.36.29 port 67 (xid=0x147aa2dc)
  dhclient[11256]: DHCPREQUEST on em3 to 10.16.36.29 port 67 (xid=0x147aa2dc)
  ...

and keeps doing so until it is able to get a new lease at 'Dec 05 18:05:53':

  dhclient[11256]: DHCPREQUEST on em3 to 255.255.255.255 port 67 (xid=0x147aa2dc)
  dhclient[11256]: DHCPACK from 10.19.189.254 (xid=0x147aa2dc)

During this interval the old lease is still valid and so, even this
can't be seen from the log, em3 should still have an address.

After one minute, at 'Dec 05 18:06:58', the connection is manually
restarted. Do you remember at which time you ran 'ip a' and saw there
was no address?

Comment 5 Matteo Croce 2018-12-06 10:18:52 UTC
Hi,

I checked the connection via IPMI console at 17:15, and the lease obtained was because I manually ran 'dhclient em3'.

Before running dhclient, with tcpdump I saw DHCP requests going out, so I tought that the DHCP server was down and run dhclient manually.

Comment 7 Beniamino Galvani 2020-04-23 07:28:25 UTC
Hi,

from the logs attached it is not possible to understand what the
problem is.  According to logs, the lease was renewed before the
expiry time and so the IPv4 address should have always been there.
I don't know why it was removed.

I'm closing this bz for now. Please reopen if you are able to
reproduce the problem; in such case, please also increase NM logging
level to TRACE and attach the log. Thanks!


Note You need to log in before you can comment on or make changes to this bug.