Bug 1227633

Summary: DHCPNAK after neutron-dhcp-agent restart
Product: Red Hat OpenStack Reporter: Nir Magnezi <nmagnezi>
Component: openstack-neutronAssignee: Nir Magnezi <nmagnezi>
Status: CLOSED ERRATA QA Contact: Toni Freger <tfreger>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: chrisw, ihrachys, lpeer, mlopes, nyechiel, sclewis, tfreger, yeylon
Target Milestone: ga   
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-2015.1.0-2.el7ost Doc Type: Bug Fix
Doc Text:
Previously, dnsmasq did not save lease information in persistent storage, and when it was restarted, the lease information was lost. This behavior was a result of the removal of the dnsmasq '--dhcp-script' option under BZ#1202392. As a result, instances were stuck in the network boot process for a long period of time. In addition, NACK messages were noted in the dnsmasq log. This update addresses this issue by removing the authoritative option, so that NAKs are not sent in response to DHCPREQUESTs to other servers. This change is expected to prevent dnsmasq from NAKing clients renewing leases issued before it was restarted/rescheduled, with the result that no DHCPNAK messages can be found in the log files.
 Story Points: ---
Clone Of:
: 1227635 1227638 (view as bug list) Environment:
Last Closed: 2015-08-05 13:24:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nir Magnezi 2015-06-03 08:10:50 UTC 
Description of problem:
=======================

After rolling out a configuration change, we restarted neutron-dhcp-agent service, and then dnsmasq logs start flooding: DHCPNAK ... lease not found.
DHCPNAK is replied by dnsmasq for all DHCPREQUEST renews from all VMs. However the MAC and IP pairs exist in host files.
The log flooding increases when more and more VMs start renewing and they keep retrying until IP expire and send DHCPDISCOVER and reinit the IP.
The log flooding gradually disappears when the VMs IP expire and send DHCPDISCOVER, to which dnsmasq respond DHCPOFFER properly.

Analysis:
=========
I noticed that option --leasefile-ro is used in dnsmasq command when started by neutron dhcp-agent. According to dnsmasq manual, this option should be used together with --dhcp-script to customize the lease database. However, the option --dhcp-script was removed when fixing bug 1202392.
Because of this, dnsmasq will not save lease information in persistent storage, and when it is restarted, lease information is lost.

Solution:
=========
Simply replace --leasefile-ro by --dhcp-leasefile=<path to dhcp runtime files>/lease would solve the problem. (patch attached)
Comment 7 Toni Freger 2015-06-17 06:55:45 UTC 
Verified on RHEL7.1
puddle from rel-eng/OpenStack/7.0-RHEL-7-Beta/2015-06-12.1/

After the lease was renewed no DHCPNAK messages were found.

The DHCP messages from /var/log/messages file of the agents.

Agent1 :

Jun 17 09:46:24 rose11 dnsmasq-dhcp[20559]: DHCPDISCOVER(tapc705ecac-68) fa:16:3e:4a:27:39
Jun 17 09:46:24 rose11 dnsmasq-dhcp[20559]: DHCPOFFER(tapc705ecac-68) 20.20.20.4 fa:16:3e:4a:27:39
Jun 17 09:46:24 rose11 dnsmasq-dhcp[20559]: DHCPREQUEST(tapc705ecac-68) 20.20.20.4 fa:16:3e:4a:27:39
Jun 17 09:46:24 rose11 dnsmasq-dhcp[20559]: DHCPACK(tapc705ecac-68) 20.20.20.4 fa:16:3e:4a:27:39 host-20-20-20-4


Agent2: 

Jun 17 09:46:24 rose12 dnsmasq-dhcp[20043]: DHCPDISCOVER(tapd266cd9a-8b) fa:16:3e:4a:27:39
Jun 17 09:46:24 rose12 dnsmasq-dhcp[20043]: DHCPOFFER(tapd266cd9a-8b) 20.20.20.4 fa:16:3e:4a:27:39
Comment 9 errata-xmlrpc 2015-08-05 13:24:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1548