Bug 1452772

Summary: [RFE] Passenger graceful killer cron job
Product: Red Hat Satellite Reporter: Satellite Program <pm-sat>
Component: InfrastructureAssignee: Ivan Necas <inecas>
Status: CLOSED ERRATA QA Contact: Peter Ondrejka <pondrejk>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: bkearney, bmidwood, cwelton, inecas, jcallaha, lzap, mmccune, pcreech, stbenjam
Target Milestone: 6.4.0Keywords: FutureFeature, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-foreman_maintain-0.2.9-2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-16 15:27:32 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Satellite Program 2017-05-19 15:15:33 UTC
Passenger open-source version does support process restarting on memory threshold. It looks like passenger have reserved SIGUSR1 for graceful restart. A cron job could do the job easily:

https://gist.github.com/kascote/366441

http://mensfeld.pl/2012/08/simple-rubyrails-passenger-memory-consumption-limit-monitoring/

Comment 1 Satellite Program 2017-05-19 15:15:38 UTC
Created from redmine issue http://projects.theforeman.org/issues/19496

Comment 2 Satellite Program 2017-05-19 15:15:41 UTC
Upstream bug assigned to lzap

Comment 3 Bryan Kearney 2017-05-19 15:16:47 UTC
*** Bug 1452771 has been marked as a duplicate of this bug. ***

Comment 4 Bryan Kearney 2017-05-19 15:17:07 UTC
*** Bug 1452665 has been marked as a duplicate of this bug. ***

Comment 5 Lukas Zapletal 2017-05-22 06:59:06 UTC
Thanks, is this meant for 6.3 or 6.2? This is easily backportable, just a script and packaging change.

Comment 6 Satellite Program 2017-11-02 20:24:00 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/19496 has been resolved.

Comment 7 Matt Pusateri 2018-09-18 21:00:04 UTC
Dev, can we get some assistance on how to reproduce pushing passenger into the failed state.

Comment 8 Lukas Zapletal 2018-09-19 14:31:03 UTC
Dumb approach, set the memory limit in this script to 400 MB, restart Satellite httpd and run

watch passenger-status

Click in the UI randomly, do some actions via CLI/API to increase memory consumption above 400 MB and then wait until the script gets executed from cron (or execute it). It should restart one process per execution.

The config is /etc/passenger-recycler.yaml and value you want to change is MAX_PRIV_RSS_MEMORY. Here is an example file:

https://github.com/swapab/foreman_maintain/blob/302581759bf5bc31de962f6cc5ccecb889c84748/config/passenger-recycler.yaml

It looks like for some reason this YAML file is not being distributed with foreman-maintain, likely a bug.

Comment 10 Lukas Zapletal 2018-09-19 15:19:08 UTC
Correction of the YAML file link:

https://github.com/theforeman/foreman_maintain/blob/master/config/passenger-recycler.yaml

Filed BZ#1630958 for the missing YAML file.

Comment 11 Ivan Necas 2018-09-19 15:32:38 UTC
*** Bug 1630958 has been marked as a duplicate of this bug. ***

Comment 12 Satellite Program 2018-09-19 16:05:52 UTC
Upstream bug assigned to inecas

Comment 13 Satellite Program 2018-09-19 16:05:56 UTC
Upstream bug assigned to inecas

Comment 15 Patrick Creech 2018-09-21 00:40:01 UTC
adding sat-maintenance flag as the above pr deals with foreman-maintain

Comment 16 Peter Ondrejka 2018-09-24 13:04:44 UTC
Verified on Satellite 6.4 snap 23, expected files are present: 

rpm -ql rubygem-foreman_maintain-0.2.10-1.el7sat.noarch | grep passenger
/etc/passenger-recycler.yaml
/usr/bin/passenger-recycler
/usr/share/gems/gems/foreman_maintain-0.2.10/bin/passenger-recycler
/usr/share/gems/gems/foreman_maintain-0.2.10/config/passenger-recycler.yaml
/usr/share/gems/gems/foreman_maintain-0.2.10/definitions/procedures/passenger_recycler.rb

With setup suggested in comment #8, running passenger-recycler:
...
Terminating 15319 (started 08:22:44) with private RSS size of 459 MB
Process successfully 15319 terminated
...

So the custom memory limit is respected. The cron task is not set automatically with rpm installation, however this is intended, as per discussion with inecas

Comment 18 errata-xmlrpc 2018-10-16 15:27:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927