Bug 1399374

Summary:	API response times are slow
Product:	[Community] RDO	Reporter:	Christopher Brown <chris.brown>
Component:	python-keystoneclient	Assignee:	Nathan Kinder <nkinder>
Status:	CLOSED ERRATA	QA Contact:	Shai Revivo <srevivo>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	Mitaka	CC:	ayoung, srevivo
Target Milestone:	---
Target Release:	trunk
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-02-15 17:20:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Christopher Brown 2016-11-28 22:26:27 UTC

Description of problem:

On a deployed cloud, API response times are slow - up to 45 seconds requesting a listing of endpoints.

Version-Release number of selected component (if applicable):

Both Mitaka and Newton show this.

How reproducible:

Always, though seems to improve after a few hours

Steps to Reproduce:
1. Deploy a one controller, one compute node cloud
2. Run "openstack endpoint list --timing"
3. Observe poor speeds



Actual results:

[stack@undercloud ~]$ openstack endpoint list --timing
+----------------------------------+-----------+--------------+----------------+
| ID                               | Region    | Service Name | Service Type   |
+----------------------------------+-----------+--------------+----------------+
| 01d1f6098a9e4c4c874ac662e185f0b8 | regionOne | gnocchi      | metric         |
| 63ec0d3d574444da96537769a447a70a | regionOne | heat         | orchestration  |
| 376239edd44e4a5a9d08acf9b2034ada | regionOne | aodh         | alarming       |
| cadc28d3dd1d4772a395ac12be61c7dc | regionOne | heat-cfn     | cloudformation |
| d8721e9e6e464463aacd35b39fbb5637 | regionOne | neutron      | network        |
| d84984b938cf4ebbb481f9213faaac32 | regionOne | swift        | object-store   |
| 71a53a84dc914ba6bea13366b00cceb5 | regionOne | cinderv2     | volumev2       |
| 1c3dc22bb1a1478d8e8ae69216d908d0 | regionOne | cinderv3     | volumev3       |
| aec55ba83cdd48509c7ec588259407e0 | regionOne | glance       | image          |
| 660f840e9a874ab08a1bf1c14856570a | regionOne | ceilometer   | metering       |
| baf1881f32ce467bbc8423abf0459481 | regionOne | nova         | compute        |
| 54a8aed0cf9d4f9689d130370b9635de | regionOne | cinder       | volume         |
| a0d74959916a4570af3a5e9258cf3c3e | regionOne | keystone     | identity       |
+----------------------------------+-----------+--------------+----------------+

+-----------------------------------------------------------------------------------+-----------+
| URL                                                                               |   Seconds |
+-----------------------------------------------------------------------------------+-----------+
| GET http://10.0.0.8:5000/v2.0                                                     |  2.194681 |
| POST http://10.0.0.8:5000/v2.0/tokens                                             |   2.44672 |
| POST http://10.0.0.8:5000/v2.0/tokens                                             |  2.460463 |
| GET http://10.0.0.8:35357/                                                        |  0.007647 |
| GET http://10.0.0.8:35357/v2.0/endpoints                                          |  0.037036 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/6e57018bd8ba44d7b85a40bb29d792d7 |  0.025002 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/70d296346051444e987f29452e3d56b9 |  0.118714 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/d50df18f2db94df9bd613e27fc57e6cc |   0.03642 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/549f7a4c1b13420fb7c08c2af9a07fe9 |    2.3991 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/ae944096e3664e92a7d5adf0c9b79878 |  0.118197 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/45a031be8ddf43e1b9add5c4ca624a64 |    0.0449 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/d1beac8dab464f6992de9d3bf483b5a2 |  0.034414 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/4c0d3e812329468e9ebcbbb6f8958858 |  0.026568 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/d6be081615a247a3a5c09f94b6062030 |  0.032789 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/42858357f6e340029bbd7eeabc9771fe |  0.140968 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/12883d88d9be4a6ca546f7134e9855db |  0.130114 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/de0d004e184f462c87ad45ff77d610e8 |   2.36879 |
| GET http://10.0.0.8:35357/v2.0/OS-KSADM/services/de543a84e1184a45bce76786d4cf4916 |  0.026789 |
| Total                                                                             | 12.649312 |
+-----------------------------------------------------------------------------------+-----------+


Expected results:

Should be under a second

Additional info:

Note that the delay is around 2 seconds each time. With RHEL OSP 9 this is around 1 second.

We have updated switch firmware, changed STP, changed MTU however this is still the case when running commands on a single controller and therefore hitting the local VIP directly.

Comment 1 Christopher Brown 2017-02-15 17:20:27 UTC

Never got to the bottom of this, assuming local issue or bug.