Description of problem: ======================= keystone-all process reaches 100% CPU consumption. I created a new user group and tried to move all users to that group (via Horizon). The action got stuck and I had to refresh the page. From that point I started to noticed that each action takes a long time (around 10 sec at minimum) and that includes the simplest CLI commands such as "nova list" "cinder list" etc. 'top' shows that each time I perform an action - the keystone-all process CPU consumption climbs to 100% for ~10 seconds. I removed all users from the user group I created and even restarted keystone - but the issue still reproduces. Version-Release number of selected component (if applicable): ============================================================= RDO Havana --> http://repos.fedorapeople.org/repos/openstack/openstack-havana/rdo-release-havana-6.noarch.rpm openstack-packstack-2013.2.1-0.9.dev756.el6.noarch openstack-keystone-2013.2-0.11.b3.el6.noarch How reproducible: ================= Reproduced several times with my setup. Does not reproduce with every setup. Steps to Reproduce: =================== See problem description. Actual results: =============== keystone-all process reaches 100% CPU consumption for a least 10 sec, which causes actions to take a lot longer than usual. Expected results: ================= keystone-all process should not consume that amount of CPU. Additional info: ================ Keystone log attached with debug and verbose enabled.
Created attachment 798678 [details] packstack answer file
Created attachment 798679 [details] keystone.log Keystone log attached with debug and verbose enabled.
Ran mysql query: SELECT token.id AS token_id, token.expires AS token_expires, token.extra AS token_extra, token.valid AS token_valid, token.user_id AS token_user_id, token.trust_id AS token_trust_id FROM token WHERE token.expires > NOW() AND token.valid = 0 copy & paste from mysql: 1294 rows in set (0.55 sec) it took 55 seconds to get the full response (not a typo).
So how do you measure 55s ?
You can also trying profiling: http://dev.mysql.com/doc/refman/5.1/en/show-profile.html
Please see upstream bug related to MySQL setup: https://bugs.launchpad.net/keystone/+bug/1182481 And confirm that you have set up mysql approapriatly. If that fixes it, please close this issue.
(In reply to Adam Young from comment #6) > Please see upstream bug related to MySQL setup: > > https://bugs.launchpad.net/keystone/+bug/1182481 > > And confirm that you have set up mysql approapriatly. If that fixes it, > please close this issue. Adam, I followed your comment[1] and I still see high CPU utilization by keystone-all. [1] https://bugs.launchpad.net/keystone/+bug/1182481/comments/3
This should have been fixed by the upstream change to create the index on the token table.
Hello All, We are using Juno on RHEL 7 and we are also facing the same issue :( All requests takes long time... Please help us :) This is from OS controller node: KiB Mem : 26376526+total, 23464988+free, 19374300 used, 9741064 buff/cache KiB Swap: 4194300 total, 4194300 free, 0 used. 24385766+avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5631 keystone 20 0 2838596 684948 7804 S 115.7 0.3 57:34.22 httpd keystone is consuming 100% CPU
Please see comment 8. If there is still a problem, please open a new bug with the context, logs, and steps to replicate/