Red Hat Bugzilla – Bug 1250570
A rake trends:clean on the server consumes all of the memory and some swap.
Last modified: 2017-02-23 14:48:27 EST
Description of problem: Every morning around 8:00am CST a satellite process starts , it is running a rake trends:clean on the server. It consumes all of the memory and some swap. Every few days it actually causes the VM to hang and we have to forcefully reboot it. Version-Release number of selected component (if applicable): Satellite 6.0.8 How reproducible: Customer is hitting this upstream bug: http://projects.theforeman.org/issues/5568 Customer tried the patch(upstream) and is issue is fixed, do we know if that patch will included in the Satellite 6.1 version?
QE notes: will need details on how much trends data is being stored and/or reasonable steps to repro
hi, please provide verification steps thanks thanks and regards, Tazim
Verified in Sat 6.1.2 Compose 3. I used a satellite which had more than a million rows. # time foreman-rake trends:counter real 0m26.776s user 0m25.678s sys 0m0.798s
Query to generate a lot of rows and you need existing trends: # select * from trends; id | trendable_type | trendable_id | name | ----+----------------+--------------+-------------------------+- 1 | FactName | 20 | hostname trend | 2 | FactName | 20 | ibm-x3250m4-05 | 3 | FactName | 20 | libvirt-rhel-6-7-01 | 4 | FactName | 81 | Facts-selinux | 5 | FactName | 81 | true | 6 | FactName | 1 | Facts-architecture | 7 | FactName | 1 | x86_64 | 8 | FactName | 11 | Facts-domain | 9 | FactName | 11 | lab.eng.rdu2.redhat.com | .. trimmed note the ID for a specific trend, in this example 'Facts-domain' now insert: # insert into trend_counters (trend_id, count, created_at, updated_at) select 9, generate_series(1,100) AS count, '2015-9-1', '2015-9-1'; This will insert 100 rows. If this runs OK, repeat with 1mil: # insert into trend_counters (trend_id, count, created_at, updated_at) select 9, generate_series(1,1000000) AS count, '2015-9-1', '2015-9-1'; now you have 1mil rows in a trend.
Another test with two million rows in trend_counters table: foreman=# select count(*) from trend_counters; count --------- 2011394 (1 row) [root@satt6host ~]# time foreman-rake trends:counter real 0m26.884s user 0m25.742s sys 0m0.747s
3 mill records: foreman=# select count(*) from trend_counters; count --------- 3011404 (1 row) [root@sat6host ~]# time foreman-rake trends:counter real 0m27.178s user 0m25.854s sys 0m0.741s 4 mill records: foreman=# select count(*) from trend_counters; count --------- 4011414 (1 row) [root@sat6host ~]# time foreman-rake trends:counter real 0m27.586s user 0m26.138s sys 0m0.841s 5 mill records: foreman=# select count(*) from trend_counters; count --------- 5011414 (1 row) [root@sat6host ~]# time foreman-rake trends:counter real 0m27.313s user 0m25.815s sys 0m0.801s
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2015:1786