Bug 1227635 - DHCPNAK after neutron-dhcp-agent restart
Summary: DHCPNAK after neutron-dhcp-agent restart
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 6.0 (Juno)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z4
: 6.0 (Juno)
Assignee: Nir Magnezi
QA Contact: Toni Freger
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-03 08:16 UTC by Nir Magnezi
Modified: 2022-07-09 07:41 UTC (History)
11 users (show)

Fixed In Version: openstack-neutron-2014.2.3-6.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, dnsmasq did not save lease information in persistent storage, and when it was restarted, the lease information was lost. This behavior was a result of the removal of the dnsmasq '--dhcp-script' option under BZ#1202392. As a result, instances were stuck in the network boot process for a long period of time. In addition, NACK messages were noted in the dnsmasq log. This update addresses this issue by removing the authoritative option, so that NAKs are not sent in response to DHCPREQUESTs to other servers. This change is expected tp prevent dnsmasq from NAKing clients renewing leases issued before it was restarted/rescheduled, with the result that no DHCPNAK messages can be found in the log files.
Clone Of: 1227633
Environment:
Last Closed: 2015-08-24 20:15:40 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1345947 0 None None None Never
Launchpad 1457900 0 None None None Never
Red Hat Issue Tracker OSP-16634 0 None None None 2022-07-09 07:41:38 UTC
Red Hat Knowledge Base (Solution) 1584893 0 None None None Never
Red Hat Product Errata RHSA-2015:1680 0 normal SHIPPED_LIVE Moderate: openstack-neutron security and bug fix update 2015-08-25 00:12:31 UTC

Description Nir Magnezi 2015-06-03 08:16:38 UTC
+++ This bug was initially created as a clone of Bug #1227633 +++

Description of problem:
=======================

After rolling out a configuration change, we restarted neutron-dhcp-agent service, and then dnsmasq logs start flooding: DHCPNAK ... lease not found.
DHCPNAK is replied by dnsmasq for all DHCPREQUEST renews from all VMs. However the MAC and IP pairs exist in host files.
The log flooding increases when more and more VMs start renewing and they keep retrying until IP expire and send DHCPDISCOVER and reinit the IP.
The log flooding gradually disappears when the VMs IP expire and send DHCPDISCOVER, to which dnsmasq respond DHCPOFFER properly.

Analysis:
=========
I noticed that option --leasefile-ro is used in dnsmasq command when started by neutron dhcp-agent. According to dnsmasq manual, this option should be used together with --dhcp-script to customize the lease database. However, the option --dhcp-script was removed when fixing bug 1202392.
Because of this, dnsmasq will not save lease information in persistent storage, and when it is restarted, lease information is lost.

Solution:
=========
Simply replace --leasefile-ro by --dhcp-leasefile=<path to dhcp runtime files>/lease would solve the problem. (patch attached)

Comment 8 Toni Freger 2015-08-17 10:36:03 UTC
Tested on Rhel7.1 puddle from 2015-08-13.1


dhcp-common-4.2.5-36.el7.x86_64
dhcp-libs-4.2.5-36.el7.x86_64

openstack-neutron-2014.2.3-9.el7ost.noarch
openstack-neutron-common-2014.2.3-9.el7ost.noarch
python-neutronclient-2.3.9-1.el7ost.noarch
python-neutron-2014.2.3-9.el7ost.noarch
openstack-neutron-openvswitch-2014.2.3-9.el7ost.noarch

The DHCP messages from /var/log/messages file of the agents.

Agent#1:

Aug 17 06:23:48 networker2 dnsmasq-dhcp[21035]: DHCPDISCOVER(tap29cff8bc-cc) fa:16:3e:a8:b9:1a
Aug 17 06:23:48 networker2 dnsmasq-dhcp[21035]: DHCPOFFER(tap29cff8bc-cc) 80.80.80.5 fa:16:3e:a8:b9:1a
Aug 17 06:23:48 networker2 dnsmasq-dhcp[21035]: DHCPREQUEST(tap29cff8bc-cc) 80.80.80.5 fa:16:3e:a8:b9:1a
Aug 17 06:23:48 networker2 dnsmasq-dhcp[21035]: DHCPACK(tap29cff8bc-cc) 80.80.80.5 fa:16:3e:a8:b9:1a host-80-80-80-5

Agent#2

Aug 17 06:23:49 networker1 dnsmasq-dhcp[28502]: DHCPDISCOVER(tap6b8cb71c-b4) fa:16:3e:a8:b9:1a
Aug 17 06:23:49 networker1 dnsmasq-dhcp[28502]: DHCPOFFER(tap6b8cb71c-b4) 80.80.80.5 fa:16:3e:a8:b9:1a

Comment 10 errata-xmlrpc 2015-08-24 20:15:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1680.html


Note You need to log in before you can comment on or make changes to this bug.