Bug 1289614 - Keystone tokens never purged in undercloud MySQL database
Summary: Keystone tokens never purged in undercloud MySQL database
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
Target Milestone: beta
: 10.0 (Newton)
Assignee: James Slagle
QA Contact: Shai Revivo
: 1293274 1328180 1347359 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2015-12-08 15:03 UTC by Nicolas Auvray
Modified: 2016-12-14 15:19 UTC (History)
15 users (show)

Fixed In Version: instack-undercloud-5.0.0-0.20160907134010.649dc3f.el7ost
Doc Type: Bug Fix
Doc Text:
Prior to this update, there was no automated process for periodically purging expired tokens from the Identity Service (keystone) database. Consequently, the keystone database could potentially continue to grow, resulting in a large database size and the possible consumption of all available disk space. With this update, a crontab entry was added to periodically query and delete expired tokens in the keystone database, running once per day. As a result, the keystone database will no longer face unlimited growth due to expired tokens.
Clone Of:
Last Closed: 2016-12-14 15:19:25 UTC

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC

Description Nicolas Auvray 2015-12-08 15:03:39 UTC
Description of problem:

Same as referenced in https://bugzilla.redhat.com/show_bug.cgi?id=1173970 , but this BZ is specific to undercloud node.

How reproducible:

Steps to Reproduce:
1. Install an undercloud
2. Do some openstack-related stuff and see your MySQL database growing indefinitely.

Actual results:
Running the token_flush manually reduces the size of the table by flushing all expired tokens.

Expected results:
There should be a cron job that flushes expired tokens every minute:
*/1 * * * * /usr/bin/keystone-manage token_flush >/dev/null 2>&1

Comment 2 Mike Burns 2015-12-21 13:31:53 UTC
*** Bug 1293274 has been marked as a duplicate of this bug. ***

Comment 3 Mike Burns 2016-04-07 21:00:12 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 6 David Juran 2016-04-18 13:56:24 UTC
I'd even consider changing the component to openstack-keystone and having the rpm drop the cron-job into /etc/cron.d as a default.
Or is there a reason a customer would _not_ want the tokens flushed?

Comment 7 Mike Burns 2016-04-18 15:56:49 UTC
*** Bug 1328180 has been marked as a duplicate of this bug. ***

Comment 8 Mike Burns 2016-04-18 16:13:06 UTC
(In reply to David Juran from comment #6)
> I'd even consider changing the component to openstack-keystone and having
> the rpm drop the cron-job into /etc/cron.d as a default.
> Or is there a reason a customer would _not_ want the tokens flushed?

I've asked for that for a long time and the consistent answer has been that the keystone folks won't do that in the rpm.  They have a few reasons:

* they're trying to move to a model where they don't need the flush (fernet tokens, iirc) so having it is extra if that option is in use
* The flushing can be db intensive and the operator really needs to determine how often they want/need the flush.  Some people might prefer 1 really large flush weekly at a (for them) off-peak time.  Others might prefer an extremely aggressive flush ever hour or 2.

Comment 9 Udi 2016-04-19 08:27:56 UTC
I think this is an urgent blocker, I changed the severity accordingly. We keep getting hit by this issue and it's easy for a user to not be aware of this problem, even if the user is a very knowledgeable one.

Comment 10 Mike Burns 2016-06-16 16:09:21 UTC
*** Bug 1347359 has been marked as a duplicate of this bug. ***

Comment 11 Emilien Macchi 2016-09-16 15:13:11 UTC
The default is set to "purge tokens every 24h". Do you want to reduce the frequency to 1 min? Or something else?

Comment 12 Udi 2016-09-18 10:16:42 UTC
As far as I can recall, the frequency of once every 24 hours is what was decided on, and it can stay the way it is. The problem is that we don't see this cron job created at all on the undercloud. Need to check it on the overcloud as well. What is responsible for creating it?

Comment 13 Emilien Macchi 2016-09-19 13:09:15 UTC
Puppet is responsible of creating the crontab. Could you verify OSP10 have the cron exist on both undercloud and overcloud? I checked the puppet code and it looks good to me.

Comment 14 James Slagle 2016-09-20 15:54:29 UTC
to check if the cron job is defined, as root you can run:

sudo -u keystone crontab -l

should see something similar output to:

[root@instack ~]# sudo -u keystone crontab -l
# HEADER: This file was autogenerated at 2016-08-15 12:10:54 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: keystone-manage token_flush
PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
1 0 * * * keystone-manage token_flush >>/dev/null 2>&1

you can also check the /var/spool/cron/keystone file

Comment 15 Udi 2016-09-21 11:13:50 UTC
Verified that it's fixed for the undercloud in OSP10 (puddle 2016-09-20.1).

Comment 19 errata-xmlrpc 2016-12-14 15:19:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.