Created attachment 902597 [details] iptables Description of problem: no port opened for neutron API for communication between nodes Result - VMs unable to boot VMs boot with error state: 500 - Connection to neutron failed: Maximum attempts reached notes RHOS-5 on RHEL-7 with VXLAN workaround (on controller): iptables -I INPUT 1 -s 10.35.0.0/16 -j ACCEPT [root@puma06 ~(keystone_admin)]# rpm -qa | grep openstack openstack-ceilometer-central-2014.1-2.1.el7ost.noarch openstack-dashboard-theme-2014.1-7.el7ost.noarch openstack-swift-object-1.13.1-1.el7ost.noarch openstack-ceilometer-common-2014.1-2.1.el7ost.noarch openstack-nova-common-2014.1-3.el7ost.noarch openstack-nova-cert-2014.1-3.el7ost.noarch openstack-neutron-openvswitch-2014.1-26.el7ost.noarch openstack-heat-common-2014.1-1.4.el7ost.noarch openstack-puppet-modules-2014.1-13.el7ost.noarch openstack-glance-2014.1-2.el7ost.noarch openstack-nova-console-2014.1-3.el7ost.noarch openstack-nova-scheduler-2014.1-3.el7ost.noarch openstack-neutron-2014.1-26.el7ost.noarch python-django-openstack-auth-1.1.5-2.el7ost.noarch redhat-access-plugin-openstack-5.0.0-3.el7ost.noarch openstack-swift-account-1.13.1-1.el7ost.noarch openstack-swift-proxy-1.13.1-1.el7ost.noarch openstack-heat-api-cloudwatch-2014.1-1.4.el7ost.noarch openstack-ceilometer-notification-2014.1-2.1.el7ost.noarch openstack-ceilometer-alarm-2014.1-2.1.el7ost.noarch openstack-packstack-puppet-2014.1.1-0.20.dev1109.el7ost.noarch openstack-keystone-2014.1-2.el7ost.noarch openstack-utils-2014.1-1.el7ost.noarch openstack-nova-novncproxy-2014.1-3.el7ost.noarch openstack-dashboard-2014.1-7.el7ost.noarch openstack-swift-1.13.1-1.el7ost.noarch openstack-swift-plugin-swift3-1.7-3.el7ost.noarch openstack-heat-api-2014.1-1.4.el7ost.noarch openstack-ceilometer-collector-2014.1-2.1.el7ost.noarch openstack-packstack-2014.1.1-0.20.dev1109.el7ost.noarch openstack-nova-api-2014.1-3.el7ost.noarch openstack-heat-engine-2014.1-1.4.el7ost.noarch openstack-ceilometer-api-2014.1-2.1.el7ost.noarch openstack-nova-objectstore-2014.1-3.el7ost.noarch openstack-cinder-2014.1-4.el7ost.noarch openstack-nova-conductor-2014.1-3.el7ost.noarch openstack-swift-container-1.13.1-1.el7ost.noarch openstack-heat-api-cfn-2014.1-1.4.el7ost.noarch [root@puma06 ~(keystone_admin)]# nova show 736b5f51-70f4-41da-ad4c-9f746abc39f7 +--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | puma09.scl.lab.tlv.redhat.com | | OS-EXT-SRV-ATTR:hypervisor_hostname | puma09.scl.lab.tlv.redhat.com | | OS-EXT-SRV-ATTR:instance_name | instance-0000005b | | OS-EXT-STS:power_state | 0 | | OS-EXT-STS:task_state | deleting | | OS-EXT-STS:vm_state | error | | OS-SRV-USG:launched_at | - | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2014-06-05T14:20:54Z | | fault | {"message": "Connection to neutron failed: Maximum attempts reached", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 296, in decorated_function | | | return function(self, context, *args, **kwargs) | | | File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2262, in terminate_instance | | | do_terminate_instance(instance, bdms) | | | File \"/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py\", line 249, in inner | | | return f(*args, **kwargs) | | | File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2260, in do_terminate_instance | | | self._set_instance_error_state(context, instance['uuid']) | | | File \"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py\", line 68, in __exit__ | | | six.reraise(self.type_, self.value, self.tb) | | | File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2250, in do_terminate_instance | | | reservations=reservations) | | | File \"/usr/lib/python2.7/site-packages/nova/hooks.py\", line 103, in inner | | | rv = f(*args, **kwargs) | | | File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2220, in _delete_instance | | | user_id=user_id) | | | File \"/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py\", line 68, in __exit__ | | | six.reraise(self.type_, self.value, self.tb) | | | File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2190, in _delete_instance | | | self._shutdown_instance(context, db_inst, bdms) | | | File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2101, in _shutdown_instance | | | network_info = self._get_instance_nw_info(context, instance) | | | File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 1118, in _get_instance_nw_info | | | instance) | | | File \"/usr/lib/python2.7/site-packages/nova/network/api.py\", line 48, in wrapper | | | res = f(self, context, *args, **kwargs) | | | File \"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py\", line 473, in get_instance_nw_info | | | port_ids) | | | File \"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py\", line 482, in _get_instance_nw_info | | | port_ids) | | | File \"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py\", line 1113, in _build_network_info_model | | | data = client.list_ports(**search_opts) | | | File \"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 111, in with_params | | | ret = self.function(instance, *args, **kwargs) | | | File \"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 306, in list_ports | | | **_params) | | | File \"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1250, in list | | | for r in self._pagination(collection, path, **params): | | | File \"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1263, in _pagination | | | res = self.get(path, params=params) | | | File \"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1236, in get | | | headers=headers, params=params) | | | File \"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py\", line 1228, in retry_request | | | raise exceptions.ConnectionFailed(reason=_(\"Maximum attempts reached\")) | | | ", "created": "2014-06-05T15:56:56Z"} | | flavor | m1.nano (42) | | hostId | 13bbae95eff97723169ef0e1b401ac00c7870922cd7b2538c77b1fd8 | | id | 736b5f51-70f4-41da-ad4c-9f746abc39f7 | | image | cirros-0.3.2-x86_64-uec (9540307c-9df5-4539-be90-11d59f3c875c) | | key_name | - | | metadata | {} | | name | test_server_-1789686399 | | os-extended-volumes:volumes_attached | [] | | status | ERROR | | tenant_id | cdea88e645544d5f8ee355a53c4d1e04 | | updated | 2014-06-05T15:56:56Z | | user_id | 7b1b8228d99f46c6a566c5cc9db20e7f | +--------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
This is affects Multi-Host setup where compute nodes are not on the same host as neutron server
Can you post (or attach) the output of "iptables -S" on your controller?
I've been able to confirm this. Working on a fix right now.
Change proposed in: https://review.openstack.org/#/c/98438/
RHOS-5.0 on RHEL-7.0 [root@puma45 ~]# rpm -qa | grep "packstack\|neutron" openstack-packstack-2014.1.1-0.22.dev1117.el7ost.noarch openstack-neutron-openvswitch-2014.1-26.el7ost.noarch python-neutronclient-2.3.4-2.el7ost.noarch openstack-neutron-2014.1-26.el7ost.noarch python-neutron-2014.1-26.el7ost.noarch openstack-packstack-puppet-2014.1.1-0.22.dev1117.el7ost.noarch [root@puma45 ~]# iptables -nL | grep 9696 ACCEPT tcp -- 10.35.160.171 0.0.0.0/0 multiport dports 9696 /* 001 neutron server incoming neutron_server_10.35.160.171_10.35.160.171 */ ACCEPT tcp -- 10.35.160.175 0.0.0.0/0 multiport dports 9696 /* 001 neutron server incoming neutron_server_10.35.160.171_10.35.160.175 */ ACCEPT tcp -- 10.35.160.193 0.0.0.0/0 multiport dports 9696 /* 001 neutron server incoming neutron_server_10.35.160.171_10.35.160.193 */
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-0846.html