Bug 1451122

Hi,
I am cloning this bug (Eng needs to fix the token_flush issue anyway) and routing to Cloudforms.
It appears that our token flush issue on OSP10 is being caused by Cloudforms generating a -lot- of tokens.
Here's an example:
$ sudo mysql keystone -e 'select count(token.id) as count,name from token,local_user where token.user_id=local_user.user_id group by name order by count desc;'
+---------+-------------------------+
| count   | name                    |
+---------+-------------------------+
| 3359889 | cloudforms              |
|  115602 | neutron                 |
|   87496 | ceilometer              |
|   55042 | nova                    |
|   27849 | glance                  |
|   27281 | gnocchi                 |
|   26843 | heat                    |
|   24704 | cinder                  |
|   21809 | sevone                  |
|    7472 | admin                   |
|    3270 | swift                   |
|     928 | foobar                  |
|     366 | evillal                 |
|      59 | heat_stack_domain_admin |
|      13 | aodh                    |
|      10 | heat-cfn                |
|       8 | coombsu                 |
+---------+-------------------------+

How come that cloudforms generated this many tokens?
Is this normal behaviour?

The only reason I can think for generating so many tokens is a CloudForms worker dying and restarting repeatedly.  Each time the worker restarts, it might try to reconnect to OSP to gather data.

Could we also get some information as to which version of CFME is being used?

https://github.com/ManageIQ/manageiq-gems-pending/pull/160

I've tested and merged Marek's PR:

https://github.com/ManageIQ/manageiq-providers-openstack/pull/45

The other one (https://github.com/ManageIQ/manageiq-gems-pending/pull/160) should not be considered part of this BZ.

By my testing Marek's PR reduces token generation to a quarter of what it was before, which is hopefully enough to avoid the token storm issue.  Marek, are there any other optimizations that you wanted to look into?

https://github.com/ManageIQ/manageiq-providers-openstack/pull/45 should be the fix on CF side.

https://github.com/ManageIQ/manageiq-gems-pending/pull/160 is kind of enhancement and refactoring with just a minor effect on this issue, so I don't think we need backport it.

Other change which could decrease amount of Keystone Auth tokens significatly is merging EventWorkers (Cloud, Network, Storage if will be present) into one worker and distribute captured Events inside CF. This would be a bigger change in CF codebase and providers architecture. Which looks to me to be outside of scope of this BZ.

the version is 5.7.0.17 if you still need it.

Ah, yep - do you have any information around the question I have in https://bugzilla.redhat.com/show_bug.cgi?id=1451122#c25?

i don't see comment 25

Oh, whoops:

Thanks for the information! Just for clarification - NEC1 seems to be the environment where token generation is most under control, while MSC1, MSC2, and NEC2 seem to have unsustainable growth - does that match what you're seeing? What's the difference between these four environments?

Usually if a change is made on one environment we make the change accross all 4 environments. All 4 environments were setup using the same proceedure and were done within a month of each other. There are 3 controllers on all 4 environments and 14 or 32 computes, nec or msc respectively. 

nec1 and nec2 are the same hardware and network setup. There are newer packages on nec2 due to an update we performed for a security vendor operating in this environment. I believe these will be pushed to all the environments eventually. We have been collecting data on nec1 for about a month. We've seen it go up to a little over 3k tokens.  Nec1 is the least used of the 4 environments by outside vendors. 

nec2 is being used to check for voulnerabilities and has satellite puppet modules installed for openscap scans. 

msc1 and msc2 are identical hardware/network/software. The only difference i know of is different vendors are on each environments. They work on hosting networks for wireless communications.

Thanks for the info! Looking at the logs, on MSC2 it looks like refreshes cause a small spike in tokens - maybe 9 or so - while seven tokens are generated every 45 seconds. The high frequency indicates that this has something to do with the event worker. We'll focus our efforts there.

It looks like there was an issue where certain kinds of API requests - including those for events - would result in ManageIQ looping over every single tenant and making a connection for each. https://github.com/ManageIQ/manageiq-providers-openstack/pull/62 should fix this issue.

Verified
========
CFME 5.9.0.19 + RHOS11

(In reply to Ido Ovadia from comment #40)
> Verified
> ========
> CFME 5.9.0.19 + RHOS11

typo fix the verification made on RHOS 10