Bug 1598001 - Failed to expired reports when the reports table grow too large
Summary: Failed to expired reports when the reports table grow too large
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Reporting
Version: 6.3.2
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: 6.4.0
Assignee: Lukas Zapletal
QA Contact: Jan Hutař
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-04 06:38 UTC by Hao Chang Yu
Modified: 2021-08-03 19:14 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Release Note
Doc Text:
Starting from Satellite 6.4, the cron job to delete old reports is reconfigured to delete reports in batches of 1000 records with a fractional delay between tasks. This reduces the likelihood of updating workers becoming blocked. After the upgrade, monitor the number of reports in the database and the output of the report expiration tasks. In case of concurrency problems, update the check-in time for both the Puppet client, which is 30 minutes by default, and RHSM, which is four hours by default. This decreases the load on Satellite Server.
Clone Of:
Environment:
Last Closed: 2018-10-16 18:55:27 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 23623 0 Normal Closed Break report expiration into batches 2020-03-12 06:42:59 UTC

Comment 2 Satellite Program 2018-07-04 08:29:38 UTC 
Upstream bug assigned to lzap
Comment 3 Satellite Program 2018-07-04 08:29:41 UTC 
Upstream bug assigned to lzap
Comment 5 Satellite Program 2018-07-12 08:29:31 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/23623 has been resolved.
Comment 15 Lukas Zapletal 2018-08-16 13:59:37 UTC 
For googlers, this bug reports database transaction deadlocks. This is because on higher-loaded Satellite 6 servers incoming reports are being saved into database while rake task is attempting to acquire exclusive locks on three tables to delete data. One or another process (Satellite 6 request or rake task) is usually kicked out.

We changed to rake task to delete data in smaller batches (configurable, by default 1k reports) and put a sleep (0.2 second) in between batches so SQL server can process incoming requests. This should lower amount of deadlocks from Satellite 6 requests.

This also effectively makes expiration task SLOWER, it can also still error out with deadlock, this BZ does not aim to completely fix it as it is technically not possible. This kind of data (high-volume of non-critical data - reports) does not belong to SQL database in my opinion and the real solution would be to store them outside of the relation database or at least in different form (normal form is subideal).
Comment 21 Bryan Kearney 2018-10-16 18:55:27 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927
Note You need to log in before you can comment on or make changes to this bug.