Bug 1289614

Summary:	Keystone tokens never purged in undercloud MySQL database
Product:	Red Hat OpenStack	Reporter:	Nicolas Auvray <nauvray>
Component:	instack-undercloud	Assignee:	James Slagle <jslagle>
Status:	CLOSED ERRATA	QA Contact:	Shai Revivo <srevivo>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	7.0 (Kilo)	CC:	afazekas, djuran, emacchi, imouzann, jcoufal, jschluet, mburns, mcornea, mlopes, mtanino, pcaruana, rhel-osp-director-maint, sclewis, ukalifon, vcojot
Target Milestone:	beta	Keywords:	TestOnly, Triaged
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	instack-undercloud-5.0.0-0.20160907134010.649dc3f.el7ost	Doc Type:	Bug Fix
Doc Text:	Prior to this update, there was no automated process for periodically purging expired tokens from the Identity Service (keystone) database. Consequently, the keystone database could potentially continue to grow, resulting in a large database size and the possible consumption of all available disk space. With this update, a crontab entry was added to periodically query and delete expired tokens in the keystone database, running once per day. As a result, the keystone database will no longer face unlimited growth due to expired tokens.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-12-14 15:19:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Nicolas Auvray 2015-12-08 15:03:39 UTC

Description of problem:

Same as referenced in https://bugzilla.redhat.com/show_bug.cgi?id=1173970 , but this BZ is specific to undercloud node.


How reproducible:
100%


Steps to Reproduce:
1. Install an undercloud
2. Do some openstack-related stuff and see your MySQL database growing indefinitely.


Actual results:
Running the token_flush manually reduces the size of the table by flushing all expired tokens.


Expected results:
There should be a cron job that flushes expired tokens every minute:
*/1 * * * * /usr/bin/keystone-manage token_flush >/dev/null 2>&1

Comment 2 Mike Burns 2015-12-21 13:31:53 UTC

*** Bug 1293274 has been marked as a duplicate of this bug. ***

Comment 3 Mike Burns 2016-04-07 21:00:12 UTC

This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 6 David Juran 2016-04-18 13:56:24 UTC

I'd even consider changing the component to openstack-keystone and having the rpm drop the cron-job into /etc/cron.d as a default.
Or is there a reason a customer would _not_ want the tokens flushed?

Comment 7 Mike Burns 2016-04-18 15:56:49 UTC

*** Bug 1328180 has been marked as a duplicate of this bug. ***

Comment 8 Mike Burns 2016-04-18 16:13:06 UTC

(In reply to David Juran from comment #6)
> I'd even consider changing the component to openstack-keystone and having
> the rpm drop the cron-job into /etc/cron.d as a default.
> Or is there a reason a customer would _not_ want the tokens flushed?

I've asked for that for a long time and the consistent answer has been that the keystone folks won't do that in the rpm.  They have a few reasons:

* they're trying to move to a model where they don't need the flush (fernet tokens, iirc) so having it is extra if that option is in use
* The flushing can be db intensive and the operator really needs to determine how often they want/need the flush.  Some people might prefer 1 really large flush weekly at a (for them) off-peak time.  Others might prefer an extremely aggressive flush ever hour or 2.

Comment 9 Udi Kalifon 2016-04-19 08:27:56 UTC

I think this is an urgent blocker, I changed the severity accordingly. We keep getting hit by this issue and it's easy for a user to not be aware of this problem, even if the user is a very knowledgeable one.

Comment 10 Mike Burns 2016-06-16 16:09:21 UTC

*** Bug 1347359 has been marked as a duplicate of this bug. ***

Comment 11 Emilien Macchi 2016-09-16 15:13:11 UTC

The default is set to "purge tokens every 24h". Do you want to reduce the frequency to 1 min? Or something else?

Comment 12 Udi Kalifon 2016-09-18 10:16:42 UTC

As far as I can recall, the frequency of once every 24 hours is what was decided on, and it can stay the way it is. The problem is that we don't see this cron job created at all on the undercloud. Need to check it on the overcloud as well. What is responsible for creating it?

Comment 13 Emilien Macchi 2016-09-19 13:09:15 UTC

Puppet is responsible of creating the crontab. Could you verify OSP10 have the cron exist on both undercloud and overcloud? I checked the puppet code and it looks good to me.

Comment 14 James Slagle 2016-09-20 15:54:29 UTC

to check if the cron job is defined, as root you can run:

sudo -u keystone crontab -l

should see something similar output to:

[root@instack ~]# sudo -u keystone crontab -l
# HEADER: This file was autogenerated at 2016-08-15 12:10:54 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: keystone-manage token_flush
PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
1 0 * * * keystone-manage token_flush >>/dev/null 2>&1


you can also check the /var/spool/cron/keystone file

Comment 15 Udi Kalifon 2016-09-21 11:13:50 UTC

Verified that it's fixed for the undercloud in OSP10 (puddle 2016-09-20.1).

Comment 19 errata-xmlrpc 2016-12-14 15:19:25 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html