Description of problem:

Neutron seems to have decided it doesn't want to manage iptables on a node, even after restarting the service, it won't rebuild the iptables rules

[root@wcmsc2-l-rh-cmp-14 ~]# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 240K   68M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 000 accept related established rules */ state RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            /* 001 accept all icmp */ state NEW
  308 22612 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0            /* 002 accept all to lo interface */ state NEW
    4   240 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22 /* 003 accept ssh */ state NEW
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 123 /* 105 ntp */ state NEW
 1805  163K ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 4789 /* 118 neutron vxlan networks */ state NEW
  140 12740 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 161 /* 127 snmp */ state NEW
    0     0 ACCEPT     47   --  *      *       0.0.0.0/0            0.0.0.0/0            /* 136 neutron gre networks */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 16509,16514,49152:49215,5900:5999 /* 200 nova_libvirt */ state NEW
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
  388  215K REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 998 log all */ LOG flags 0 level 4
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 999 drop all */ state NEW

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 235K packets, 40M bytes)
 pkts bytes target     prot opt in     out     source               destination

This seems related to openflow / opendaylight

relevant log details are these:

2017-06-19 16:11:09.096 584077 WARNING oslo_rootwrap.client [req-431da185-1901-4939-9ba9-8f4b744b4ebb - - - - -] Leaving behind already spawned process with pid 584103, root should kill it if it's still there (I can't)
2017-06-19 16:11:09.198 584077 INFO oslo_rootwrap.client [req-431da185-1901-4939-9ba9-8f4b744b4ebb - - - - -] Spawned new rootwrap daemon process with pid=584402
2017-06-19 16:11:09.214 584077 ERROR neutron.agent.linux.async_process [-] Error received from [ovsdb-client monitor Interface name,ofport,external_ids --format=json]: None
2017-06-19 16:11:09.214 584077 ERROR neutron.agent.linux.async_process [-] Process [ovsdb-client monitor Interface name,ofport,external_ids --format=json] dies due to the error: None
2017-06-19 16:11:09.216 584077 ERROR ryu.lib.hub [-] hub: uncaught exception: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ryu/base/app_manager.py", line 545, in close
    self.uninstantiate(app_name)
  File "/usr/lib/python2.7/site-packages/ryu/base/app_manager.py", line 528, in uninstantiate
    app = self.applications.pop(name)
KeyError: 'ofctl_service'

2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl [-] Post-commit checks failed
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl Traceback (most recent call last):
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 149, in post_commit
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl     self.do_post_commit(txn)
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 171, in do_post_commit
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl     'timeout': self.timeout})
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl TimeoutException: Commands [SetControllerCommand(bridge=br-int, targets=['tcp:127.0.0.1:6633'])] exceeded timeout 10 seconds post-commit
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl [-] Post-commit checks failed
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl Traceback (most recent call last):
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 149, in post_commit
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl     self.do_post_commit(txn)
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 171, in do_post_commit
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl     'timeout': self.timeout})
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl TimeoutException: Commands [DbSetCommand(table=Controller, col_values=(('connection_mode', 'out-of-band'),), record=c559694e-2d47-4a33-bfbb-1e2c3690878b)] exceeded timeout 10 seconds post-commit

2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl
2017-06-19 16:11:37.398 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int has datapath-ID 0000e2f6186fb445
2017-06-19 16:12:07.411 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:12:07.412 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:12:37.425 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:12:37.426 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:13:07.441 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:13:07.443 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:13:37.457 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:13:37.458 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:14:07.471 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:14:07.473 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:14:37.488 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:14:37.489 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:15:07.505 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:15:07.506 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:15:37.520 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:15:37.521 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:16:07.536 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:16:07.537 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:16:37.552 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:16:37.553 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445

Version-Release number of selected component (if applicable):
puppet-opendaylight-3.7.0-1.b2d8d9dgit.el7ost.noarch
And neutron packages
openstack-neutron-9.1.0-8.el7ost.noarch
openstack-neutron-bigswitch-agent-9.40.0-1.1.el7ost.noarch
openstack-neutron-bigswitch-lldp-9.40.0-1.1.el7ost.noarch
openstack-neutron-common-9.1.0-8.el7ost.noarch
openstack-neutron-lbaas-9.1.0-1.el7ost.noarch
openstack-neutron-metering-agent-9.1.0-8.el7ost.noarch
openstack-neutron-ml2-9.1.0-8.el7ost.noarch
openstack-neutron-openvswitch-9.1.0-8.el7ost.noarch
openstack-neutron-sriov-nic-agent-9.1.0-8.el7ost.noarch
puppet-neutron-9.4.2-1.el7ost.noarch
python-neutron-9.1.0-8.el7ost.noarch
python-neutron-lbaas-9.1.0-1.el7ost.noarch
python-neutron-lib-0.4.0-1.el7ost.noarch
python-neutron-tests-9.1.0-8.el7ost.noarch
python-neutronclient-6.0.0-2.el7ost.noarch

How reproducible:
Customer says he has seen this a couple of times on lab enviroment but is not sure how to reproduce the issue.