Bug 1706774

Summary: [OSP14] cinder cron job definition should have randomized sleep command like puppet-heat for decreasing DB spike or lock contention.
Product: Red Hat OpenStack Reporter: Pablo Caruana <pcaruana>
Component: openstack-tripleo-heat-templatesAssignee: Pablo Caruana <pcaruana>
Status: CLOSED ERRATA QA Contact: Sasha Smolyak <ssmolyak>
Severity: low Docs Contact:
Priority: medium    
Version: 14.0 (Rocky)CC: abishop, cschwede, eharney, jjoyce, jschluet, knoha, lmarsh, mburns, mfuruta, mowens, pcaruana, pgrist, rheslop, slinaber, tkajinam, tshefi, tvignaud
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-9.3.1-0.20190314162766.d0a6cb1.el7ost puppet-cinder-13.3.3-0.20190118031243.0d80a9f.el7ost Doc Type: Bug Fix
Doc Text:
This patch introduces two new parameters: * CinderCronDbPurgeMaxDelay - Configure maxdelay parameter in the Cinder database purge cron job. * NovaCronDBArchiveMaxDelay - Configure maxdelay for the Nova database archive cron job.
 Story Points: ---
Clone Of: 1603014 Environment:
Last Closed: 2019-07-02 20:08:44 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1603014    

Description Pablo Caruana 2019-05-06 09:50:27 UTC 
Description of problem:
cinder cron job definition should have randomized sleep command like puppet-heat for decreasing DB spike or lock contention.

Version-Release number of selected component (if applicable):
Current puppet-cinder.

Additional info:
Current puppet-cinder configures the cron job like
~~~
# HEADER: This file was autogenerated at 2018-07-13 06:00:08 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: cinder-manage db purge
PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
1 0 * * * cinder-manage db purge 30 >>/var/log/cinder/cinder-rowsflush.log 2>&1
~~~

If many DB cron jobs are executed at the same time, it will trigger lock contention or DB spike.

We should have randomized sleep in puppet-heat[0].

[0] https://github.com/openstack/puppet-heat/blob/master/manifests/cron/purge_deleted.pp#L70-L75
Comment 14 errata-xmlrpc 2019-07-02 20:08:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1672