Bug 1125207

Summary: neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: "ip6tables-restore v1.4.7: ip6tables-restore: unable to initialize table 'filter'\n\nError occurred at line: 2\n
Product: Red Hat OpenStack Reporter: Eduard Barrera <ebarrera>
Component: openstack-neutronAssignee: Jakub Libosvar <jlibosva>
Status: CLOSED ERRATA QA Contact: Roey Dekel <rdekel>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.0CC: chrisw, ihrachys, jlibosva, nyechiel, oblaut, sputhenp, tfreger, yeylon
Target Milestone: z2Keywords: ZStream
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-2014.1.2-5.el6ost openstack-neutron-2014.1.2-3.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1150898 (view as bug list) Environment:
Last Closed: 2014-11-03 08:38:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1150898    

Description Eduard Barrera 2014-07-31 10:26:18 UTC 
Description of problem:

I completly disabled IPv6 after installing packstack. And now I'm having this error flooding the audit.log and openvswitch-agent.log 

modules/ipv6.conf
------------------
options ipv6 disable=1
install ipv6 /bin/true
blacklist ipv6
alias ipv6 off


openvswitch-agent.log 
---------------------
2014-07-31 11:45:16.039 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 
2014-07-31 11:45:16.040 1891 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
2014-07-31 11:45:16.041 1891 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of sync with plugin!
2014-07-31 11:45:17.933 1891 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'085562f6-de05-4294-984c-bdd5b8448b88', u'aca468f0-7518-4ec0-bcba-dc819d961248', u'c8338884-adc6-477f-bbe6-1f459a401083'])
2014-07-31 11:45:18.445 1891 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error in agent event loop
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1081, in rpc_loop
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     sync = self.process_network_ports(port_info)
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1011, in process_network_ports
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     resync_a = self.treat_devices_added(port_info['added'])
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 912, in treat_devices_added
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.sg_agent.prepare_devices_filter(devices)
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 123, in prepare_devices_filter
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.firewall.prepare_port_filter(device)
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.gen.next()
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/firewall.py", line 107, in defer_apply
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.filter_defer_apply_off()
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_firewall.py", line 353, in filter_defer_apply_off
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self.iptables.defer_apply_off()
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 346, in defer_apply_off
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     self._apply()
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/lockutils.py", line 233, in inner
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     retval = f(*args, **kwargs)
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/iptables_manager.py", line 382, in _apply
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     root_helper=self.root_helper)
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py", line 75, in execute
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent     raise RuntimeError(m)
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RuntimeError: 
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip6tables-restore', '-c']
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Exit code: 2
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stdout: ''
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Stderr: "ip6tables-restore v1.4.7: ip6tables-restore: unable to initialize table 'filter'\n\nError occurred at line: 2\nTry `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.\n"
2014-07-31 11:45:18.445 1891 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent 
2014-07-31 11:45:18.446 1891 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
2014-07-31 11:45:18.446 1891 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent tunnel out of sync with plugin!



This is how the file grows in 10 seconds

# > openvswitch-agent.log; ll openvswitch-agent.log ; sleep 10; ll openvswitch-agent.log
-rw-r--r--. 1 neutron neutron 0 Jul 31 11:42 openvswitch-agent.log
-rw-r--r--. 1 neutron neutron 15482 Jul 31 11:42 openvswitch-agent.log


Version-Release number of selected component (if applicable):

openstack-nova-conductor-2013.2.2-2.el6ost.noarch
openstack-nova-cert-2013.2.2-2.el6ost.noarch
openstack-neutron-openvswitch-2013.2.2-1.el6ost.noarch
redhat-access-plugin-openstack-4.0.0-0.el6ost.noarch
openstack-keystone-2013.2.2-1.el6ost.noarch
openstack-cinder-2013.2.2-1.el6ost.noarch
openstack-utils-2013.2-3.el6ost.noarch
openstack-nova-api-2013.2.2-2.el6ost.noarch
openstack-nova-compute-2013.2.2-2.el6ost.noarch
openstack-nova-scheduler-2013.2.2-2.el6ost.noarch
openstack-dashboard-2013.2.2-1.el6ost.noarch
openstack-ceilometer-central-2013.2.2-1.el6ost.noarch
openstack-ceilometer-collector-2013.2.2-1.el6ost.noarch
openstack-ceilometer-api-2013.2.2-1.el6ost.noarch
openstack-ceilometer-compute-2013.2.2-1.el6ost.noarch
openstack-nova-novncproxy-2013.2.2-2.el6ost.noarch
openstack-neutron-2013.2.2-1.el6ost.noarch
openstack-packstack-2013.2.1-0.25.dev987.el6ost.noarch
python-django-openstack-auth-1.1.2-2.el6ost.noarch
openstack-dashboard-theme-2013.2.2-1.el6ost.noarch
openstack-selinux-0.1.3-2.el6ost.noarch
openstack-ceilometer-alarm-2013.2.2-1.el6ost.noarch
sos-plugins-openstack-2013.2-5.el6ost.noarch
openstack-glance-2013.2.2-2.el6ost.noarch
openstack-nova-common-2013.2.2-2.el6ost.noarch
openstack-ceilometer-common-2013.2.2-1.el6ost.noarch
openstack-nova-console-2013.2.2-2.el6ost.noarch


commenting the entries on modules/ipv6.conf and executing modprobe ipv6 fixes the issue.

I didn't found any way to disable the ipv6 behavior in neutron by any conf option


This is the cpu (vm 1 core) usage during the issue:

Tasks: 142 total,   2 running, 140 sleeping,   0 stopped,   0 zombie
Cpu(s): 78.7%us, 20.7%sy,  0.0%ni,  0.3%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st

This is cpu usage after modprobe ipv6

Tasks: 142 total,   1 running, 141 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.1%us,  1.4%sy,  0.0%ni, 85.0%id,  9.5%wa,  0.0%hi,  0.0%si,  0.0%st
Comment 2 Jakub Libosvar 2014-08-18 15:55:45 UTC 
*** Bug 1002569 has been marked as a duplicate of this bug. ***
Comment 6 Roey Dekel 2014-10-14 12:49:14 UTC 
Verified on Icehouse with:

Version-Release number of selected component:
---------------------------------------------
Red Hat Enterprise Linux Server release 7.0 (Maipo)
puddle: 2014-10-10.4
openstack-neutron-ml2-2014.1.3-4.el7ost.noarch

Reproduce Steps:
----------------
1. Edit file /etc/sysctl.conf, enter the following command:
    1.1	[root@devops ~]# vim /etc/sysctl.conf

2. Adding the below lines into that file:
    2.1	net.ipv6.conf.all.disable_ipv6 = 1
    2.2	net.ipv6.conf.default.disable_ipv6 = 1
    2.3	net.ipv6.conf.lo.disable_ipv6 = 1

3. Save and close that file, then restart sysctl with the following command:
    3.1	[root@devops ~]# sysctl -p

4. check “disable_ipv6″ file to check if IPv6 is disabled,enter:
    4.1	[root@devops ~]# cat /proc/sys/net/ipv6/conf/all/disable_ipv6
    4.2	1

Results:
--------
No errors in audit.log nor openvswitch-agent.log
Comment 8 errata-xmlrpc 2014-11-03 08:38:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2014-1786.html