Bug 1598001 – Failed to expired reports when the reports table grow too large

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1598001 - Failed to expired reports when the reports table grow too large

Summary:	Failed to expired reports when the reports table grow too large

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat Satellite 6
Classification:	Red Hat
Component:	Reporting (Show other bugs)
Sub Component:
Version:	6.3.2
Hardware:	Unspecified Unspecified

Priority	unspecified Severity high (vote)
Target Milestone:	GA
Target Release:	Unused
Assigned To:	Lukas Zapletal
QA Contact:	Jan Hutař
Docs Contact:

URL:
Whiteboard:
Keywords:	Triaged

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2018-07-04 02:38 EDT by Hao Chang Yu
Modified:	2018-10-16 14:55 EDT (History)
CC List:	7 users (show)

See Also:
Fixed In Version:
Doc Type:	Release Note
Doc Text:	Starting from Satellite 6.4, the cron job to delete old reports is reconfigured to delete reports in batches of 1000 records with a fractional delay between tasks. This reduces the likelihood of updating workers becoming blocked. After the upgrade, monitor the number of reports in the database and the output of the report expiration tasks. In case of concurrency problems, update the check-in time for both the Puppet client, which is 30 minutes by default, and RHSM, which is four hours by default. This decreases the load on Satellite Server.
Story Points:	---
Clone Of:
Environment:
Last Closed:	2018-10-16 14:55:27 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Foreman Issue Tracker	23623	None	None	None	2018-07-04 02:40 EDT

Groups: None (edit)

Comment 2 pm-sat@redhat.com 2018-07-04 04:29:38 EDT

Upstream bug assigned to lzap@redhat.com

Comment 3 pm-sat@redhat.com 2018-07-04 04:29:41 EDT

Upstream bug assigned to lzap@redhat.com

Comment 5 pm-sat@redhat.com 2018-07-12 04:29:31 EDT

Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/23623 has been resolved.

Comment 15 Lukas Zapletal 2018-08-16 09:59:37 EDT

For googlers, this bug reports database transaction deadlocks. This is because on higher-loaded Satellite 6 servers incoming reports are being saved into database while rake task is attempting to acquire exclusive locks on three tables to delete data. One or another process (Satellite 6 request or rake task) is usually kicked out.

We changed to rake task to delete data in smaller batches (configurable, by default 1k reports) and put a sleep (0.2 second) in between batches so SQL server can process incoming requests. This should lower amount of deadlocks from Satellite 6 requests.

This also effectively makes expiration task SLOWER, it can also still error out with deadlock, this BZ does not aim to completely fix it as it is technically not possible. This kind of data (high-volume of non-critical data - reports) does not belong to SQL database in my opinion and the real solution would be to store them outside of the relation database or at least in different form (normal form is subideal).

Comment 21 Bryan Kearney 2018-10-16 14:55:27 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927

Note You need to log in before you can comment on or make changes to this bug.