Created attachment 957171 [details] logs I have an HA+Neutron VXLAN deployment. After some time,I get a lot of messaging errors and openswitch agent stops working. Restart openswitch agent solves the problem. openswitch-agent.log attached ,neutron-server logs form the controllers and rabbitmq logs. ----------------------- openstack-puppet-modules-2014.1-24.el6ost.noarch rhel-osp-installer-0.4.7-1.el6ost.noarch ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el6ost.noarch openstack-foreman-installer-2.0.32-1.el6ost.noarch ---------------------------- python-neutronclient-2.3.4-3.el7ost.noarch openstack-neutron-2014.1.3-7.el7ost.noarch openstack-neutron-openvswitch-2014.1.3-7.el7ost.noarch python-neutron-2014.1.3-7.el7ost.noarch ------------------------------------------------ 2014-11-13 07:17:48.704 2313 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin! 2014-11-13 07:17:48.952 2313 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'78b0999e-a4b2-4e4c-ae38-37e43c48bdd7', u'36ccd710-ab1a-485d-b70b-6f57009562f2', u'139f0ef f-c4e6-43ca-88e2-e91a233c41f3', u'1bd4464c-6e26-452b-abed-51512e9fd2c9']) 2014-11-13 07:18:48.969 2313 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error while processing VIF ports 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last): 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1335, i n rpc_loop 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent ovs_restarted) 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1139, i n process_network_ports 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set())) 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 247, in setup_port_filters 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.prepare_devices_filter(new_devices) 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 161, in prepare_devices_fi lter 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.context, list(device_ids)) 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.py", line 86, in security_group_rule s_for_devices 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent topic=self.topic) 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/proxy.py", line 129, in call 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent exc.info, real_topic, msg.get('method')) 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Timeout: Timeout while waiting on RPC response - topic: "q-plugin", RPC method: "security_group_rules_for_devi ces" info: "<unknown>" 2014-11-13 07:18:48.969 2313 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent
I wonder if this is related either to: 1) several active neutron-servers 2) a messaging/HA problem, agent tries to attach to a fanout queue it's not there in previous messages (not posted in this log part) 3) a problem with neutron-server I will examine the logs ASAP.
Also seeing the same problem in our deployment HA + VXLAN
Closing list of bugs for RHEL OSP Installer since its support cycle has already ended [0]. If there is some bug closed by mistake, feel free to re-open. For new deployments, please, use RHOSP director (starting with version 7). -- Jaromir Coufal -- Sr. Product Manager -- Red Hat OpenStack Platform [0] https://access.redhat.com/support/policy/updates/openstack/platform