Bug 1267702 - Undercloud Neutron DHCP Agent Fails After Changing the IP Subnet of Provisioning Network
Undercloud Neutron DHCP Agent Fails After Changing the IP Subnet of Provision...
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
medium Severity unspecified
: ---
: 10.0 (Newton)
Assigned To: James Slagle
Shai Revivo
: Triaged
Depends On:
  Show dependency treegraph
Reported: 2015-09-30 13:39 EDT by Dan Sneddon
Modified: 2016-10-09 18:25 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-08-19 15:40:02 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Dan Sneddon 2015-09-30 13:39:53 EDT
Description of problem:
If an Undercloud has already been built and installed, then reinstalling with a new provisioning subnet creates a broken Undercloud which can't issue DHCP addresses during deployment.

Version-Release number of selected component (if applicable):
OSP 7 GA, OSP-D 7.0

How reproducible:

Steps to Reproduce:
1. Install undercloud with provisioning on subnet X
2. Change provisioning network to subnet Y
3. Delete all OpenStack-related RPMs and reinstall undercloud

Actual results:
The rebuilt undercloud is unable to issue DHCP addresses for the overcloud hosts on deployment. Discovery works fine.

Expected results:
The rebuilt undercloud should deploy OpenStack with the new provisioning net.

Additional info:
There was still a file in /etc/ironic-discoverd/dnsmasq.conf that had one line, but since dicsovery is working this is probably not related:


Not sure what else could be left over after uninstalling all the OpenStack RPMs, but we are going to rebuild the undercloud OS and start over.
Comment 2 Dan Sneddon 2015-09-30 13:55:38 EDT
There was a more complete cleanup done than just deleting the RPMs. Here is the exact procedure that was followed before reinstalling the undercloud:


systemctl kill `systemctl | grep openstack | awk '{print $1}'`

systemctl kill `systemctl | grep neutron | awk '{print $1}'`

systemctl kill `systemctl | grep nova | awk '{print $1}'`

systemctl kill `systemctl | grep keystone | awk '{print $1}'`

systemctl kill `systemctl | grep glance | awk '{print $1}'`

systemctl kill `systemctl | grep swift | awk '{print $1}'`


yum remove -y rabbitmq-server mariadb haproxy openvswitch memcached python-memcached python-pymemcache keepalived python-tooz puppet ahc-tools $(rpm -qa | egrep "openstack|nova|glance|swift|tuskar|keystone|openvswitch|ceilometer")


rm -rf /var/lib/mysql /var/lib/rabbitmq /var/ilb/ceilometer /var/lib/puppet /var/lib/neutron /var/lib/nova /var/lib/glance /var/lib/ironic-discoverd /var/lib/openvswitch /var/lib/heat-cfntools /var/lib/haproxy /var/lib/keystone


rm -rf /var/log/heat /var/log/nova /var/log/neutron /var/log/glance /var/log/keystone /var/log/mariadb /var/log/openvswitch /var/log/ironic-discoverd /var/log/ceilometer /var/log/horizon


rm -rf /opt/stack/* /root/stackrc ~stack/stackrc ~stack/.emacs ~stack/.cache ~stack/.config/ ~stack/.pki ~stack/.novaclient/ ~stack/.instack/ /etc/my.cnf.d


rm -rf /etc/haproxy /etc/swift /etc/neutron /etc/nova /etc/openstack-dashboard /etc/ironic-discoverd /etc/ironic /etc/heat /etc/ceilometer /etc/glance /etc/tuskar /etc/puppet /etc/sysconfig/memcached.rpmsave /etc/rabbitmq /etc/tripleo /etc/haproxy  /etc/keepalived /etc/openstack-dashboard /etc/keystone /etc/os-net-config


\rm /etc/sysconfig/network-scripts/ifcfg-br-ctlplane

\rm /etc/sysconfig/network-scripts/route-br-ctlplane

\rm /etc/sysconfig/network-scripts/route-eth1

\rm /etc/sysconfig/network-scripts/ifcfg-eth1


Comment 3 James Slagle 2015-09-30 17:43:35 EDT
I'm not sure if this is actually your issue or not, but when you reproduce the issue can you check the if the vlan tag of 1 is in the openflow output of br-ctlplane by running the following on the undercloud:

sudo ovs-ofctl dump-flows br-ctlplane

You should see something like:

[stack@instack-deployed ~]$ sudo ovs-ofctl dump-flows br-ctlplane                                               
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=371167.585s, table=0, n_packets=66855637, n_bytes=34600578799, idle_age=0, hard_age=65534, priority=0 actions=NORMAL
 cookie=0x0, duration=371166.894s, table=0, n_packets=52, n_bytes=13851, idle_age=22780, hard_age=65534, priority=4,in_port=2,dl_vlan=1 actions=strip_vlan,NORMAL
 cookie=0x0, duration=371167.532s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,in_port=2 actions=drop

I've seen this issue once before, and the line with dl_vlan=1 was missing. I think the code in neutron that creates the flows, only runs on initial subnet creation, and not always on subnet update, or maybe it's at the neutron network layer. This might be a valid neutron bug I'm not sure.

Further, the whole issue should be fixed in y1 where the new behavior is to delete and recreate the neutron network and subnet whenever openstack undercloud install is run, but only if no ports are in use on the subnet (other than the one for neutron-dhcp-agent of course). I think this would also fix the above issue.
Comment 7 Mike Burns 2016-04-07 16:50:54 EDT
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.
Comment 9 Mike Burns 2016-08-19 15:40:02 EDT
based on comment 3, this appears to be fixed in current versions.  Please re-open if that is not the case.

Note You need to log in before you can comment on or make changes to this bug.