Bug 1575817
| Summary: | Instance failed to spawn due to "ConnectTimeout: Request to http://XXX/v2.0/ports timed out" | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Chen <cchen> |
| Component: | openstack-neutron | Assignee: | Assaf Muller <amuller> |
| Status: | CLOSED NOTABUG | QA Contact: | Roee Agiman <ragiman> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 12.0 (Pike) | CC: | amuller, bhaley, cchen, chrisw, dalvarez, dhill, eglynn, fpercoco, mfuruta, nalmond, njohnston, nyechiel, pgrist, ragiman, skaplons, srevivo |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-10-09 13:25:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Chen
2018-05-08 02:43:09 UTC
Hi, In nova-compute.log we got timeout error 2018-05-17 10:00:05.953 1 ERROR nova.compute.manager [instance: 93941bb3-f6b4-46a6-8ad7-63e75378b37b] ConnectTimeout: Request to http://172.17.0.11:9696/v2.0/ports timed out In neutron server.log, we have 2018-05-17 10:00:37.425 57220 INFO neutron.wsgi [req-d1e7309b-da4a-4cc9-9418-5c151dbc751b 577b8e030640426490c49dad0c03a238 9a6a7fdf37e5455b9c53a8d782d938a9 - default default] 172.17.0.21 "POST /v2.0/ports HTTP/1.1" status: 201 len: 1027 time: 61.4733062 which means, the POST took 61 seconds to finish. And here I pasted the log where it took long time to finish: 1. Quota related work seemed fine. But after that the next job waited 30 seconds: 09:59:36 -> 10:00:06 2018-05-17 09:59:36.011 57220 DEBUG neutron.pecan_wsgi.hooks.quota_enforcement [req-d1e7309b-da4a-4cc9-9418-5c151dbc751b 577b8e030640426490c49dad0c03a238 9a6a7fdf37e5455b9c53a8d782d938a9 - default default] Made reservation on behalf of 9a6a7fdf37e5455b9c53a8d782d938a9 for: {'port': 1} before /usr/lib/python2.7/site-packages/neutron/pecan_wsgi/hooks/quota_enforcement.py:55 2018-05-17 10:00:06.048 57220 DEBUG neutron_lib.callbacks.manager [req-d1e7309b-da4a-4cc9-9418-5c151dbc751b 577b8e030640426490c49dad0c03a238 9a6a7fdf37e5455b9c53a8d782d938a9 - default default] Notify callbacks ['neutron.plugins.ml2.plugin.Ml2Plugin._ensure_default_security_group_handler--9223372036853858547'] for port, before_create _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167 2. event-dispatch wasn't released until 30 seconds:10:00:07 -> 10:00:37 2018-05-17 10:00:07.370 57220 DEBUG neutron.scheduler.dhcp_agent_scheduler [req-d1e7309b-da4a-4cc9-9418-5c151dbc751b 577b8e030640426490c49dad0c03a238 9a6a7fdf37e5455b9c53a8d782d938a9 - default default] Network f5c68d5c-e52d-4fda-b21d-0f0c7f38057d is already hosted by enough agents. _get_dhcp_agents_hosting_network /usr/lib/python2.7/site-packages/neutron/scheduler/dhcp_agent_scheduler.py:243 2018-05-17 10:00:37.347 57220 DEBUG oslo_concurrency.lockutils [req-d1e7309b-da4a-4cc9-9418-5c151dbc751b 577b8e030640426490c49dad0c03a238 9a6a7fdf37e5455b9c53a8d782d938a9 - - -] Lock "event-dispatch" released by "neutron.plugins.ml2.ovo_rpc.dispatch_events" :: held 30.196s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282 I asked the customer to increase "timeout" option to 90 and the instance can be created successfully. But what was costing so much time ? Best Regards, Chen Hi, The customer redeployed the environment with 3 controllers and the customer found it is necessary to enlarge the timeout in compute node or the instance creation would fail all the time. Best Regards, Chen The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |