Description of problem:

VM's are unable to obtain DHCP addresses


Version-Release number of selected component (if applicable):

Red Hat Openstack 7.0

How reproducible:

Reproducible everytime in the attached case's customer environment

Steps to Reproduce:
1. Run "dhclient enp10s0" in the overcloud host
2. The DHCP service is able to assign IP addresses.
3. But the VM's are uanble to get it

Actual results:
The VM's should get an IP address


Expected results:


Additional info:

Here is my analysis of the SOS reports.


Going through the flow for the DHCP allocation, the follwing components and their logs are attached

1) Nova compute- looks to be fine

2) Neutron Server

There are two noteworthy things in the sos reports for neutron_server regarding this
a) 14403 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Could not find token
b) [req-56fca01f-9be7-48c1-b3b6-362ec91b9215 ] show failed (client error): Port 0b07657f-a23c-4508-b53a-335a75337fe2 could not be found

This is where we can see the first trace of the problem we are having. So the problem could either be port not being created or not being found due to failed authentication. We would have to check the logs futher to see where the problem actually lies.

3) DHCP Agent

A log here states that unable to sync network state with trace saying further
task <bound method DhcpAgentWithStateReport._report_state of <neutron.agent.dhcp.agent.DhcpAgentWithStateReport object at 0x34fcb50>> run outlasted interval by 30.06 sec
Timed out waiting for a reply to message ID : ...

4) Dnsmasq


The neutron server not finding the port looks to be the error but then if the neutron server has created the port itself, it should fail in creation itself.