Description of problem: The sql database grows with no limit. There should be a cron job to flush the expired tokens every minute: */1 * * * * /usr/bin/keystone-manage token_flush >/dev/null 2>&1 How reproducible: 100% Steps to Reproduce: 1. Install tripleo in a virtual environment according to the instructions in: https://openstack.redhat.com/Deploying_RDO_on_a_Virtual_Machine_Environment_using_Instack 2. Deploy an undercloud according to: https://openstack.redhat.com/Deploying_an_RDO_Undercloud_with_Instack 3. Deploy an overcloud with instack-deploy-overcloud according to: https://openstack.redhat.com/Deploying_an_RDO_Overcloud_with_Instack 4. ssh to the controller machine, for example: ssh heat-admin.2.9 5. Become root: sudo -i 6. Check the token table: mysql use keystone; select count(*) from token; select id, expires from token; 7. Now log out of sql, run the command "keystone-manage token_flush", and repeat step 6... Actual results: Running the token_flush manually reduces the size of the table by flushing all expired tokens. Expected results: There should be a cron job that flushes expired tokens every minute: */1 * * * * /usr/bin/keystone-manage token_flush >/dev/null 2>&1
We had the same problem in Puppet, it's maybe because Keystone user does not have a shell. We fixed it by enforcing a shell when we run the crontab. Hope that helps, let me know.
See also bug 1212126
There is nothing in the cron tables of heat-admin, keystone and root. I also waited for the tokens table to fill up and saw that expired tokens don't get flushed. Only when I ran the flush job manually, did all the expired tokens finally get flushed - the bugs is still valid.
After thinking about this, I really think this should be part of keystone packaging and not the responsibility of the installer/deployment tool. Bug 1212126 referenced in comment 5 asks to add this to keystone packaging which strikes me as the right place.
Hi James, Do you see this as a blocker? Thanks, Steve
(In reply to Mike Burns from comment #9) > After thinking about this, I really think this should be part of keystone > packaging and not the responsibility of the installer/deployment tool. Bug > 1212126 referenced in comment 5 asks to add this to keystone packaging which > strikes me as the right place. This has been brought up before. The problem is, the openstack-keystone package can not determine what token format soemone is going to use/configure. Multiple token formats are supported, and newer formats (like fernet tokens) do not need a flush job since tokens are not kept in a table in the database. It's going to need to be dealt with by the deployment tool since that is where knowledge will exist about configuring keystone.
*** Bug 1243332 has been marked as a duplicate of this bug. ***
Done upstream. Can be backported once we get acks.
*** Bug 1267952 has been marked as a duplicate of this bug. ***
Hello, We (RCIP delivery team) are deploying OSP7 with director for customers. We just hit an issue on the OSP7 cloud we had deployed for RHQE (RedHat QE). Everything had become very slow over the course of the last few weeks/days. I had to flush the tokens manually to solve their issue. I am concerned we will hit this on -all- OSP7 deployments and this will most likely not look good for the customers, this needs prioritization. There seems to be some provision for doing this in: /etc/puppet/modules/keystone/manifests/cron/token_flush.pp: class keystone::cron::token_flush ( $ensure = present, $minute = 1, $hour = 0, $monthday = '*', $month = '*', $weekday = '*', $maxdelay = 0, ) { But this file/class is never getting used in the base templates: [stack@instack-prv ~]$ grep -r keystone::cron::token_flush /usr/share/openstack-tripleo-heat-templates [stack@instack-prv ~]$ rpm -qf /usr/share/openstack-tripleo-heat-templates openstack-tripleo-heat-templates-0.8.6-71.el7ost.noarch Can we please get this prioritized? Thank you
*** Bug 1283388 has been marked as a duplicate of this bug. ***
I ran "sudo crontab -l -u keystone" and got: 1 0 * * * sleep `expr ${RANDOM} \% 3600`; keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1 This means that the job will run only daily, at 0:01 (one minute past midnight), after waiting a random number of seconds up to 3600... We know from experience with customers that there could be such a large number of expired tokens to flush, and the job locks the database for a very long time (and takes the cloud offline) if there are too many deletions needed. Therefor, we recommend running the job every minute to avoid the database getting too big. Has that recommendation changed?
We hit the same problem with Keystone tokens never purged on undercloud node. Not sure if I have to open a new bug about this.
Yes please, if you can report a separate bug for the undercloud, that would be great. The overcloud fix is already backported and tested, and the undercloud fix would go into a different repo/package.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2015:2651