Bug 1603015

Summary:	nova cron job definition should have randomized sleep command like puppet-heat for decreasing DB spike or lock contention.
Product:	Red Hat OpenStack	Reporter:	Keigo Noha <knoha>
Component:	puppet-nova	Assignee:	Rajesh Tailor <ratailor>
Status:	CLOSED ERRATA	QA Contact:	OSP DFG:Compute <osp-dfg-compute>
Severity:	low	Docs Contact:
Priority:	low
Version:	13.0 (Queens)	CC:	jjoyce, jschluet, lmarsh, mbooth, nlevinki, ratailor, slinaber, tvignaud
Target Milestone:	z3	Keywords:	Triaged, ZStream
Target Release:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	puppet-nova-12.4.0-9.el7ost	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-11-13 22:27:47 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Keigo Noha 2018-07-19 02:04:44 UTC

Description of problem:
nova cron job definition should have randomized sleep command like puppet-heat for decreasing DB spike or lock contention.

Version-Release number of selected component (if applicable):
Current puppet-nova.

Additional info:
Current puppet-nova configures the cron job like
~~~
# HEADER: This file was autogenerated at 2018-07-13 05:59:35 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: nova-manage db archive_deleted_rows
PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
1 0 * * * nova-manage db archive_deleted_rows --max_rows 100  >>/var/log/nova/nova-rowsflush.log 2>&1
~~~

If many DB cron jobs are executed at the same time, it will trigger lock contention or DB spike.

We should have randomized sleep in puppet-heat[0].

[0] https://github.com/openstack/puppet-heat/blob/master/manifests/cron/purge_deleted.pp#L70-L75

Comment 1 Shelley Dunne 2018-10-12 15:36:31 UTC

Updating Target Milestone to z3 for all Modified low bugs

Comment 8 errata-xmlrpc 2018-11-13 22:27:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3587