Hide Forgot
Description of problem: After apply patch "Creates multiple worker processes for API server"(https://review.openstack.org/#/c/60082/1), nova-compute cannot connect to neutron-server. Version-Release number of selected component (if applicable): neutron-server node: kernel 2.6.32-358.114.1.openstack.el6.x86_64 openstack-neutron 2013.2-1.el6 compute node: kernel 2.6.32-358.123.2.openstack.el6.x86_64 openstack-nova-compute 2013.2-2.el6 How reproducible: Happens every time. Steps to Reproduce: 1. apply patch "Creates multiple worker processes for API server" (https://review.openstack.org/#/c/60082/1) 2. set api_workers = 32 in /etc/neutron/neutron.conf 3. restart neutron-server 4. run command "nova boot --flavor m1.tiny --image ${img_id} --nic net-id=${net_id} test" Actual results: 1. instance entet error status. 2. error message in compute node: 2013-12-10 16:26:36.387 8639 ERROR nova.network.neutronv2.api [-] [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] Neutron error creating port on network 8232d0b8-4204-4751-ba42-04345c2dbe5a 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] Traceback (most recent call last): 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 182, in _create_port 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] port_id = port_client.create_port(port_req_body)['port']['id'] 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 108, in with_params 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] ret = self.function(instance, *args, **kwargs) 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 308, in create_port 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] return self.post(self.ports_path, body=body) 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1188, in post 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] headers=headers, params=params) 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1103, in do_request 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] resp, replybody = self.httpclient.do_request(action, method, body=body) 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 185, in do_request 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] **kwargs) 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 152, in _cs_request 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] raise exceptions.ConnectionFailed(reason=e) 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] ConnectionFailed: Connection to neutron failed: timed out 2013-12-10 16:26:36.387 8639 TRACE nova.network.neutronv2.api [instance: 0b3e7924-83cb-4726-a5e8-13735c5e88f5] 2013-12-10 16:26:36.410 8639 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s) 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager Traceback (most recent call last): 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1228, in _allocate_network_async 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager dhcp_options=dhcp_options) 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/nova/network/api.py", line 49, in wrapper 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager res = f(self, context, *args, **kwargs) 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 358, in allocate_for_instance 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager LOG.exception(msg, port_id) 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 335, in allocate_for_instance 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager security_group_ids, available_macs, dhcp_opts)) 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py", line 182, in _create_port 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager port_id = port_client.create_port(port_req_body)['port']['id'] 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 108, in with_params 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager ret = self.function(instance, *args, **kwargs) 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 308, in create_port 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager return self.post(self.ports_path, body=body) 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1188, in post 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager headers=headers, params=params) 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py", line 1103, in do_request 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager resp, replybody = self.httpclient.do_request(action, method, body=body) 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 185, in do_request 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager **kwargs) 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager File "/usr/lib/python2.6/site-packages/neutronclient/client.py", line 152, in _cs_request 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager raise exceptions.ConnectionFailed(reason=e) 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager ConnectionFailed: Connection to neutron failed: timed out 2013-12-10 16:26:36.410 8639 TRACE nova.compute.manager ........ Expected results: instance enter active status, and can be network accessed!
The Nova log is insufficient to diagnose this problem. Can you please respond with errors appearing in the Neutron log? If you are seeing db connection errors, as I suspect you might, you will want to check that the maximum number of mysql connections is not being exceeded. Each additional worker process will add to the number of active connections. In any case, I recommend you use a more conservative value for api_workers (e.g. 4). While using 1 or more workers increases REST scalability, rpc processing is still single-process while we work to ensure that concurrent processing of rpc is safe. This means that increasing the number of worker processes can result in a bottleneck in rpc processing depending on the deployment's workload.
There is no error log in neutron. What I can only know is that I can apply the patch under a Ubuntu system, but not CentOS. I have test set api_workers=4 as well, the issue remains. We have a guess this might caused by python2.6, but only a guess currently.
What problem are you trying to solve by applying the patch in question? It is only one of many being worked on upstream to try to improve Neutron reliability and it would be helpful to know why you believe it is necessary to apply this one in isolation.
We're doing a test it will boot new instances in a high rate in our cloud. When the boot rate is high enough ,we found the main reason instances will boot failed is due to neutron connection time out. While, the reason neutron server can't be connected is because it is too busy,single thread can't finished all requests. So,we hope neutron-server can have multi-process,such as nova.
This patch is part f havana stable u/s branch, we need to make sure we get that on our next rebase or manually backport it along with other patches as Maru mentioned in comment #4
The patch was merged u/s to stable havana branch - https://review.openstack.org/#/c/60082/ and is included in the first async release thanks to the rebase- Bug 1045419 - Rebase openstack-neutron to 2013.2.1