This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1467420 - Neutron not managing iptables
Neutron not managing iptables
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-opendaylight (Show other bugs)
10.0 (Newton)
x86_64 Linux
high Severity medium
: ---
: ---
Assigned To: RHOS Maint
nlevinki
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-03 14:58 EDT by Siggy Sigwald
Modified: 2017-07-20 04:35 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-20 04:35:35 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Siggy Sigwald 2017-07-03 14:58:34 EDT
Description of problem:

Neutron seems to have decided it doesn't want to manage iptables on a node, even after restarting the service, it won't rebuild the iptables rules

[root@wcmsc2-l-rh-cmp-14 ~]# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 240K   68M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 000 accept related established rules */ state RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            /* 001 accept all icmp */ state NEW
  308 22612 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0            /* 002 accept all to lo interface */ state NEW
    4   240 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22 /* 003 accept ssh */ state NEW
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 123 /* 105 ntp */ state NEW
 1805  163K ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 4789 /* 118 neutron vxlan networks */ state NEW
  140 12740 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 161 /* 127 snmp */ state NEW
    0     0 ACCEPT     47   --  *      *       0.0.0.0/0            0.0.0.0/0            /* 136 neutron gre networks */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 16509,16514,49152:49215,5900:5999 /* 200 nova_libvirt */ state NEW
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW tcp dpt:22
  388  215K REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 998 log all */ LOG flags 0 level 4
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* 999 drop all */ state NEW

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 235K packets, 40M bytes)
 pkts bytes target     prot opt in     out     source               destination

This seems related to openflow / opendaylight

relevant log details are these:

2017-06-19 16:11:09.096 584077 WARNING oslo_rootwrap.client [req-431da185-1901-4939-9ba9-8f4b744b4ebb - - - - -] Leaving behind already spawned process with pid 584103, root should kill it if it's still there (I can't)
2017-06-19 16:11:09.198 584077 INFO oslo_rootwrap.client [req-431da185-1901-4939-9ba9-8f4b744b4ebb - - - - -] Spawned new rootwrap daemon process with pid=584402
2017-06-19 16:11:09.214 584077 ERROR neutron.agent.linux.async_process [-] Error received from [ovsdb-client monitor Interface name,ofport,external_ids --format=json]: None
2017-06-19 16:11:09.214 584077 ERROR neutron.agent.linux.async_process [-] Process [ovsdb-client monitor Interface name,ofport,external_ids --format=json] dies due to the error: None
2017-06-19 16:11:09.216 584077 ERROR ryu.lib.hub [-] hub: uncaught exception: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/ryu/base/app_manager.py", line 545, in close
    self.uninstantiate(app_name)
  File "/usr/lib/python2.7/site-packages/ryu/base/app_manager.py", line 528, in uninstantiate
    app = self.applications.pop(name)
KeyError: 'ofctl_service'

2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl [-] Post-commit checks failed
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl Traceback (most recent call last):
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 149, in post_commit
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl     self.do_post_commit(txn)
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 171, in do_post_commit
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl     'timeout': self.timeout})
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl TimeoutException: Commands [SetControllerCommand(bridge=br-int, targets=['tcp:127.0.0.1:6633'])] exceeded timeout 10 seconds post-commit
2017-06-19 16:11:27.385 584449 ERROR neutron.agent.ovsdb.impl_idl
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl [-] Post-commit checks failed
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl Traceback (most recent call last):
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 149, in post_commit
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl     self.do_post_commit(txn)
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_idl.py", line 171, in do_post_commit
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl     'timeout': self.timeout})
2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl TimeoutException: Commands [DbSetCommand(table=Controller, col_values=(('connection_mode', 'out-of-band'),), record=c559694e-2d47-4a33-bfbb-1e2c3690878b)] exceeded timeout 10 seconds post-commit

2017-06-19 16:11:37.396 584449 ERROR neutron.agent.ovsdb.impl_idl
2017-06-19 16:11:37.398 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int has datapath-ID 0000e2f6186fb445
2017-06-19 16:12:07.411 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:12:07.412 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:12:37.425 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:12:37.426 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:13:07.441 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:13:07.443 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:13:37.457 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:13:37.458 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:14:07.471 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:14:07.473 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:14:37.488 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:14:37.489 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:15:07.505 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:15:07.506 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:15:37.520 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:15:37.521 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:16:07.536 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:16:07.537 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445
2017-06-19 16:16:37.552 584449 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-06-19 16:16:37.553 584449 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from e2f6186fb445 to 0000e2f6186fb445

Version-Release number of selected component (if applicable):
puppet-opendaylight-3.7.0-1.b2d8d9dgit.el7ost.noarch
And neutron packages
openstack-neutron-9.1.0-8.el7ost.noarch
openstack-neutron-bigswitch-agent-9.40.0-1.1.el7ost.noarch
openstack-neutron-bigswitch-lldp-9.40.0-1.1.el7ost.noarch
openstack-neutron-common-9.1.0-8.el7ost.noarch
openstack-neutron-lbaas-9.1.0-1.el7ost.noarch
openstack-neutron-metering-agent-9.1.0-8.el7ost.noarch
openstack-neutron-ml2-9.1.0-8.el7ost.noarch
openstack-neutron-openvswitch-9.1.0-8.el7ost.noarch
openstack-neutron-sriov-nic-agent-9.1.0-8.el7ost.noarch
puppet-neutron-9.4.2-1.el7ost.noarch
python-neutron-9.1.0-8.el7ost.noarch
python-neutron-lbaas-9.1.0-1.el7ost.noarch
python-neutron-lib-0.4.0-1.el7ost.noarch
python-neutron-tests-9.1.0-8.el7ost.noarch
python-neutronclient-6.0.0-2.el7ost.noarch

How reproducible:
Customer says he has seen this a couple of times on lab enviroment but is not sure how to reproduce the issue.
Comment 2 Nir Yechiel 2017-07-20 04:35:35 EDT
Not sure why this was filed under puppet-opendaylight. In any case, I assume that this was fixed by now. Please repoen if it's not the case.

Note You need to log in before you can comment on or make changes to this bug.