Bug 1603014 - [OSP13] cinder cron job definition should have randomized sleep command like puppet-heat for decreasing DB spike or lock contention.
Summary: [OSP13] cinder cron job definition should have randomized sleep command like ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Pablo Caruana
QA Contact: Tzach Shefi
Laura Marsh
URL:
Whiteboard:
Depends On: 1706774
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-19 02:02 UTC by Keigo Noha
Modified: 2023-03-24 14:09 UTC (History)
14 users (show)

Fixed In Version: openstack-tripleo-heat-templates-8.3.1-20.el7ost puppet-cinder-12.4.1-5.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, you configured maxdelay in heat cron job with the HeatCronPurgeDeletedMaxDelay parameter. With this update, you can also configure maxdelay in cinder database purge cron job and nova database archive cron job with the new parameters, CinderCronDbPurgeMaxDelay and NovaCronDBArchivedMaxDelay.
Clone Of:
: 1706774 (view as bug list)
Environment:
Last Closed: 2019-07-10 13:03:16 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1823274 0 None None None 2019-05-06 09:51:40 UTC
OpenStack gerrit 644714 0 None MERGED Add sleep in cinder cron jobs 2020-02-13 18:15:41 UTC
OpenStack gerrit 656759 0 None MERGED Add parameter to configure maxdelay in db purge/archive job 2020-02-13 18:15:41 UTC
Red Hat Product Errata RHBA-2019:1738 0 None None None 2019-07-10 13:04:57 UTC

Description Keigo Noha 2018-07-19 02:02:14 UTC
Description of problem:
cinder cron job definition should have randomized sleep command like puppet-heat for decreasing DB spike or lock contention.

Version-Release number of selected component (if applicable):
Current puppet-cinder.

Additional info:
Current puppet-cinder configures the cron job like
~~~
# HEADER: This file was autogenerated at 2018-07-13 06:00:08 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: cinder-manage db purge
PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
1 0 * * * cinder-manage db purge 30 >>/var/log/cinder/cinder-rowsflush.log 2>&1
~~~

If many DB cron jobs are executed at the same time, it will trigger lock contention or DB spike.

We should have randomized sleep in puppet-heat[0].

[0] https://github.com/openstack/puppet-heat/blob/master/manifests/cron/purge_deleted.pp#L70-L75

Comment 1 Christian Schwede (cschwede) 2018-08-01 13:30:34 UTC
Looks like an improvement request, not an actual bug - marking as possible feature request for next releases.

Comment 2 Keigo Noha 2018-10-09 00:26:56 UTC
Hello team,

A bugzilla for nova[1] is MODIFIED status. Could you consider to implement it until RHOSP13z3?

[1] https://bugzilla.redhat.com/1603015

Best Regards,
Keigo Noha

Comment 3 Keigo Noha 2018-10-23 00:11:25 UTC
Hello Christian,

Is it possible to implement this request in RHOSP13z3. A request for nova is already MODIFIED status.
This request is fixed at the same time as nova.

Comment 4 Christian Schwede (cschwede) 2018-10-23 13:56:01 UTC
Hello Keigo,

> Is it possible to implement this request in RHOSP13z3. 

unfortunately this won't be possible, there is no patch proposed upstream on master yet. As mentioned in comment #1, we see this as possible RFE for a future release, but so far this hasn't been planned for any specific release.

Comment 8 Alan Bishop 2019-03-20 13:04:52 UTC
The puppet-cinder patch is currently under review on upstream stable/rocky, after which it will need to be submitted to stable/queens. There is also a THT patch to enable the randomized delay. Assigning to Pablo to track the upstream patches, and manage the downstream backports.

Comment 9 Keigo Noha 2019-04-03 00:52:14 UTC
Hi Pablo,

The backport to stable/queens was done in upstream. Would you please proceed the backport process into RHOSP13?

Thanks,
Keigo Noha

Comment 11 Alan Bishop 2019-04-04 16:45:37 UTC
This bz requires patches in puppet-cinder, which have merged on stable/queens, and THT. The THT is still under review on upstream master, and will need to be backported.

Comment 12 Keigo Noha 2019-04-24 01:00:56 UTC
Hi Alan,

Thank you for your review in upstream.
https://review.opendev.org/#/c/641563/ doen't have any progress for 2 weeks. 
Would you be able to ask someone to review the patch and proceed it if it looks good to him/her?

Best Regards,
Keigo Noha

Comment 13 Pablo Caruana 2019-05-06 08:48:39 UTC
Hello  Keigo.
As finally https://review.opendev.org/641563/ was merged, I will continue preparing the equivalents at product level.

Comment 23 errata-xmlrpc 2019-07-10 13:03:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1738


Note You need to log in before you can comment on or make changes to this bug.