Description of problem: keystone-manage token_flush cron job fails with: expr: division by zero sleep: missing operand: [root@controller-0 heat-admin]# cat /var/spool/cron/keystone # HEADER: This file was autogenerated at 2017-07-31 11:50:18 +0000 by puppet. # HEADER: While it can still be managed manually, it is definitely not recommended. # HEADER: Note particularly that the comments starting with 'Puppet Name' should # HEADER: not be deleted, as doing so could cause duplicate cron jobs. # Puppet Name: keystone-manage token_flush PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh 1 * * * * sleep `expr ${RANDOM} \% 0`; keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1 # Puppet Name: cinder-manage db purge PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh 1 0 * * * cinder-manage db purge 0 >>/var/log/cinder/cinder-rowsflush.log 2>&1 Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-6.1.0-2.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy OSP11 2. Check /var/spool/cron/keystone Actual results: # Puppet Name: keystone-manage token_flush PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh 1 * * * * sleep `expr ${RANDOM} \% 0`; keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1 Expected results: No errors. Additional info: [root@controller-0 heat-admin]# expr ${RANDOM} \% 0 expr: division by zero
I think this may be a problem related to the tripleo-heat-template related to the token flush cronjob (though I am admittedly not an expert on puppet or t-h-t). The puppet-keystone code is only supposed to add a sleep to the token flush cronjob is maxdelay is a positive integer. The code that checks for this is here: https://github.com/openstack/puppet-keystone/blob/b479685940e27dea5fa86155e3a2ba6878865abc/manifests/cron/token_flush.pp#L70-L74 As you can see in the above code, a maxdelay of 0 should result in $sleep being set to an empty string. The crontab shown in this bug clearly shows that maxdelay must have been 0, yet $sleep was set. If you look at the t-h-t code for this, it uses a maxdelay default of 0, but the type is 'string': https://github.com/openstack/tripleo-heat-templates/blob/617e5f1022da5309328a1d9c96399c1fedd257ec/puppet/services/keystone.yaml#L180-L184 My theory is that the type is incorrect in t-h-t. Because it is a string, the evaluation of "if $maxdelay == 0" in puppet-keystone ends up being false since we are comparing the number 0 to the string "0". I think that the type for KeystoneCronTokenFlushMaxDelay in t-h-t should be 'number'. I don't have an install currently to personally test this theory, so it would be great if you could try it out to prove/disprove it.
The fix to tripleo-heat-templates has merged upstream for stable/ocata: https://review.openstack.org/#/c/492023/ There is a related change to puppet-keystone to perform better validation which is still under review, but that isn't really needed to fix this issue. We can consider this fixed with just the t-h-t patch. For reference, the puppet-keystone review is here: https://review.openstack.org/#/c/493304
This fix did not make it into the upstream 6.2.0 release of tripleo-heat-templates, which is the latest 6.x release. We should cherry-pick this downstream for now: https://github.com/openstack/tripleo-heat-templates/commit/564e3feeb9c518851afde9ed778360e4032e9213
Verified as follows - Sleep is not added to the keystone token-flush cron job when maxdelay is 0. ***** LOGS ***** [stack@undercloud-0 ~]$ rpm -qa openstack-tripleo-heat-templates openstack-tripleo-heat-templates-6.2.0-3.el7ost.noarch [heat-admin@controller-0 ~]$ sudo cat /var/spool/cron/keystone # HEADER: This file was autogenerated at 2017-08-31 13:54:59 +0000 by puppet. # HEADER: While it can still be managed manually, it is definitely not recommended. # HEADER: Note particularly that the comments starting with 'Puppet Name' should # HEADER: not be deleted, as doing so could cause duplicate cron jobs. # Puppet Name: keystone-manage token_flush PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh 1 * * * * keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1 # Puppet Name: cinder-manage db purge PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh 1 0 * * * cinder-manage db purge 0 >>/var/log/cinder/cinder-rowsflush.log 2>&1 [heat-admin@controller-0 ~]$
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2721