Bug 1603015 - nova cron job definition should have randomized sleep command like puppet-heat for decreasing DB spike or lock contention.
Summary: nova cron job definition should have randomized sleep command like puppet-hea...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-nova
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: z3
: 13.0 (Queens)
Assignee: Rajesh Tailor
QA Contact: OSP DFG:Compute
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-19 02:04 UTC by Keigo Noha
Modified: 2023-03-24 14:09 UTC (History)
8 users (show)

Fixed In Version: puppet-nova-12.4.0-9.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-13 22:27:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 584339 0 None None None 2018-07-20 12:22:44 UTC
OpenStack gerrit 588473 0 None None None 2018-08-16 07:02:28 UTC
Red Hat Issue Tracker OSP-23367 0 None None None 2023-03-21 18:59:50 UTC
Red Hat Product Errata RHBA-2018:3587 0 None None None 2018-11-13 22:28:27 UTC

Description Keigo Noha 2018-07-19 02:04:44 UTC
Description of problem:
nova cron job definition should have randomized sleep command like puppet-heat for decreasing DB spike or lock contention.

Version-Release number of selected component (if applicable):
Current puppet-nova.

Additional info:
Current puppet-nova configures the cron job like
~~~
# HEADER: This file was autogenerated at 2018-07-13 05:59:35 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: nova-manage db archive_deleted_rows
PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
1 0 * * * nova-manage db archive_deleted_rows --max_rows 100  >>/var/log/nova/nova-rowsflush.log 2>&1
~~~

If many DB cron jobs are executed at the same time, it will trigger lock contention or DB spike.

We should have randomized sleep in puppet-heat[0].

[0] https://github.com/openstack/puppet-heat/blob/master/manifests/cron/purge_deleted.pp#L70-L75

Comment 1 Shelley Dunne 2018-10-12 15:36:31 UTC
Updating Target Milestone to z3 for all Modified low bugs

Comment 8 errata-xmlrpc 2018-11-13 22:27:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3587


Note You need to log in before you can comment on or make changes to this bug.