Bug 1476762
| Summary: | keystone-manage token_flush cron job fails with: expr: division by zero sleep: missing operand | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Marius Cornea <mcornea> | |
| Component: | openstack-tripleo-heat-templates | Assignee: | Emilien Macchi <emacchi> | |
| Status: | CLOSED ERRATA | QA Contact: | Prasanth Anbalagan <panbalag> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 11.0 (Ocata) | CC: | acanan, aschultz, dbecker, josorior, mburns, mcornea, morazi, nkinder, panbalag, rhel-osp-director-maint | |
| Target Milestone: | z2 | Keywords: | Triaged, ZStream | |
| Target Release: | 11.0 (Ocata) | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-tripleo-heat-templates-6.2.0-3.el7ost | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1478993 (view as bug list) | Environment: | ||
| Last Closed: | 2017-09-13 21:43:17 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1473713, 1478993 | |||
I think this may be a problem related to the tripleo-heat-template related to the token flush cronjob (though I am admittedly not an expert on puppet or t-h-t).
The puppet-keystone code is only supposed to add a sleep to the token flush cronjob is maxdelay is a positive integer. The code that checks for this is here:
https://github.com/openstack/puppet-keystone/blob/b479685940e27dea5fa86155e3a2ba6878865abc/manifests/cron/token_flush.pp#L70-L74
As you can see in the above code, a maxdelay of 0 should result in $sleep being set to an empty string. The crontab shown in this bug clearly shows that maxdelay must have been 0, yet $sleep was set.
If you look at the t-h-t code for this, it uses a maxdelay default of 0, but the type is 'string':
https://github.com/openstack/tripleo-heat-templates/blob/617e5f1022da5309328a1d9c96399c1fedd257ec/puppet/services/keystone.yaml#L180-L184
My theory is that the type is incorrect in t-h-t. Because it is a string, the evaluation of "if $maxdelay == 0" in puppet-keystone ends up being false since we are comparing the number 0 to the string "0". I think that the type for KeystoneCronTokenFlushMaxDelay in t-h-t should be 'number'. I don't have an install currently to personally test this theory, so it would be great if you could try it out to prove/disprove it.
The fix to tripleo-heat-templates has merged upstream for stable/ocata: https://review.openstack.org/#/c/492023/ There is a related change to puppet-keystone to perform better validation which is still under review, but that isn't really needed to fix this issue. We can consider this fixed with just the t-h-t patch. For reference, the puppet-keystone review is here: https://review.openstack.org/#/c/493304 This fix did not make it into the upstream 6.2.0 release of tripleo-heat-templates, which is the latest 6.x release. We should cherry-pick this downstream for now: https://github.com/openstack/tripleo-heat-templates/commit/564e3feeb9c518851afde9ed778360e4032e9213 Verified as follows - Sleep is not added to the keystone token-flush cron job when maxdelay is 0. ***** LOGS ***** [stack@undercloud-0 ~]$ rpm -qa openstack-tripleo-heat-templates openstack-tripleo-heat-templates-6.2.0-3.el7ost.noarch [heat-admin@controller-0 ~]$ sudo cat /var/spool/cron/keystone # HEADER: This file was autogenerated at 2017-08-31 13:54:59 +0000 by puppet. # HEADER: While it can still be managed manually, it is definitely not recommended. # HEADER: Note particularly that the comments starting with 'Puppet Name' should # HEADER: not be deleted, as doing so could cause duplicate cron jobs. # Puppet Name: keystone-manage token_flush PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh 1 * * * * keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1 # Puppet Name: cinder-manage db purge PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh 1 0 * * * cinder-manage db purge 0 >>/var/log/cinder/cinder-rowsflush.log 2>&1 [heat-admin@controller-0 ~]$ Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2721 |
Description of problem: keystone-manage token_flush cron job fails with: expr: division by zero sleep: missing operand: [root@controller-0 heat-admin]# cat /var/spool/cron/keystone # HEADER: This file was autogenerated at 2017-07-31 11:50:18 +0000 by puppet. # HEADER: While it can still be managed manually, it is definitely not recommended. # HEADER: Note particularly that the comments starting with 'Puppet Name' should # HEADER: not be deleted, as doing so could cause duplicate cron jobs. # Puppet Name: keystone-manage token_flush PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh 1 * * * * sleep `expr ${RANDOM} \% 0`; keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1 # Puppet Name: cinder-manage db purge PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh 1 0 * * * cinder-manage db purge 0 >>/var/log/cinder/cinder-rowsflush.log 2>&1 Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-6.1.0-2.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy OSP11 2. Check /var/spool/cron/keystone Actual results: # Puppet Name: keystone-manage token_flush PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh 1 * * * * sleep `expr ${RANDOM} \% 0`; keystone-manage token_flush >>/var/log/keystone/keystone-tokenflush.log 2>&1 Expected results: No errors. Additional info: [root@controller-0 heat-admin]# expr ${RANDOM} \% 0 expr: division by zero