Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1250570 - A rake trends:clean on the server consumes all of the memory and some swap.
Summary: A rake trends:clean on the server consumes all of the memory and some swap.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Uncategorized
Version: 6.0.8
Hardware: All
OS: Linux
high
high
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: sthirugn@redhat.com
URL: http://projects.theforeman.org/issues...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-08-05 13:24 UTC by Fotios Tsiadimos
Modified: 2019-08-15 05:03 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-09-15 07:22:30 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:1786 0 normal SHIPPED_LIVE Satellite 6.1.2 bug fix update 2015-09-15 11:20:04 UTC

Description Fotios Tsiadimos 2015-08-05 13:24:13 UTC 
Description of problem:
Every morning around 8:00am CST a satellite process starts , it is running a rake trends:clean on the server. It consumes all of the memory and some swap.  Every few days it actually causes the VM to hang and we have to forcefully reboot it.

Version-Release number of selected component (if applicable):
Satellite 6.0.8

How reproducible:

Customer is hitting this upstream bug:
http://projects.theforeman.org/issues/5568


Customer tried the patch(upstream) and is issue is fixed, do we know if that patch will included in the Satellite 6.1 version?
Comment 1 Corey Welton 2015-08-18 17:21:48 UTC 
QE notes: will need details on how much trends data is being stored and/or reasonable steps to repro
Comment 4 Tazim Kolhar 2015-08-31 07:32:12 UTC 
hi,

   please provide verification steps
   thanks

thanks and regards,
Tazim
Comment 7 sthirugn@redhat.com 2015-09-10 19:00:14 UTC 
Verified in Sat 6.1.2 Compose 3.  I used a satellite which had more than a million rows.

# time foreman-rake trends:counter
real	0m26.776s
user	0m25.678s
sys	0m0.798s
Comment 8 Mike McCune 2015-09-10 19:09:42 UTC 
Query to generate a lot of rows and you need existing trends:

# select * from trends;
 id | trendable_type | trendable_id |          name           |
----+----------------+--------------+-------------------------+-
  1 | FactName       |           20 | hostname trend          |
  2 | FactName       |           20 | ibm-x3250m4-05          |
  3 | FactName       |           20 | libvirt-rhel-6-7-01     | 
  4 | FactName       |           81 | Facts-selinux           | 
  5 | FactName       |           81 | true                    | 
  6 | FactName       |            1 | Facts-architecture      | 
  7 | FactName       |            1 | x86_64                  | 
  8 | FactName       |           11 | Facts-domain            | 
  9 | FactName       |           11 | lab.eng.rdu2.redhat.com | 

.. trimmed

note the ID for a specific trend, in this example 'Facts-domain'

now insert:

# insert into trend_counters (trend_id, count, created_at, updated_at) 
select 9, generate_series(1,100) AS count,  '2015-9-1', '2015-9-1';

This will insert 100 rows. If this runs OK, repeat with 1mil:

# insert into trend_counters (trend_id, count, created_at, updated_at) 
select 9, generate_series(1,1000000) AS count,  '2015-9-1', '2015-9-1';


now you have 1mil rows in a trend.
Comment 9 sthirugn@redhat.com 2015-09-10 19:53:17 UTC 
Another test with two million rows in trend_counters table:

foreman=# select count(*) from trend_counters;  count  
---------
 2011394
(1 row)


[root@satt6host ~]# time foreman-rake trends:counter

real	0m26.884s
user	0m25.742s
sys	0m0.747s
Comment 10 sthirugn@redhat.com 2015-09-10 20:00:16 UTC 
3 mill records:

foreman=# select count(*) from trend_counters;  count  
---------
 3011404
(1 row)

[root@sat6host ~]# time foreman-rake trends:counter

real	0m27.178s
user	0m25.854s
sys	0m0.741s



4 mill records:
foreman=# select count(*) from trend_counters;  count  
---------
 4011414
(1 row)

[root@sat6host ~]# time foreman-rake trends:counter

real	0m27.586s
user	0m26.138s
sys	0m0.841s


5 mill records:
foreman=# select count(*) from trend_counters;  count  
---------
 5011414
(1 row)

[root@sat6host ~]# time foreman-rake trends:counter

real	0m27.313s
user	0m25.815s
sys	0m0.801s
Comment 12 errata-xmlrpc 2015-09-15 07:22:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1786
Note You need to log in before you can comment on or make changes to this bug.