Bug 1452772 - [RFE] Passenger graceful killer cron job
Summary: [RFE] Passenger graceful killer cron job
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Infrastructure
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high vote
Target Milestone: 6.4.0
Assignee: Ivan Necas
QA Contact: Peter Ondrejka
URL:
Whiteboard:
: 1452665 1452771 1630958 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-19 15:15 UTC by pm-sat@redhat.com
Modified: 2019-11-05 23:05 UTC (History)
9 users (show)

Fixed In Version: rubygem-foreman_maintain-0.2.9-2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-10-16 15:27:32 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2927 None None None 2018-10-16 15:28:07 UTC
Foreman Issue Tracker 19496 None None None 2017-05-19 15:15:35 UTC
Foreman Issue Tracker 24984 None None None 2018-09-19 14:46:37 UTC
Red Hat Bugzilla 1478364 None NEW kill passenger threads that have not returned in 60 minutes 2019-02-01 22:17:36 UTC

Internal Links: 1478364

Description pm-sat@redhat.com 2017-05-19 15:15:33 UTC
Passenger open-source version does support process restarting on memory threshold. It looks like passenger have reserved SIGUSR1 for graceful restart. A cron job could do the job easily:

https://gist.github.com/kascote/366441

http://mensfeld.pl/2012/08/simple-rubyrails-passenger-memory-consumption-limit-monitoring/

Comment 1 pm-sat@redhat.com 2017-05-19 15:15:38 UTC
Created from redmine issue http://projects.theforeman.org/issues/19496

Comment 2 pm-sat@redhat.com 2017-05-19 15:15:41 UTC
Upstream bug assigned to lzap@redhat.com

Comment 3 Bryan Kearney 2017-05-19 15:16:47 UTC
*** Bug 1452771 has been marked as a duplicate of this bug. ***

Comment 4 Bryan Kearney 2017-05-19 15:17:07 UTC
*** Bug 1452665 has been marked as a duplicate of this bug. ***

Comment 5 Lukas Zapletal 2017-05-22 06:59:06 UTC
Thanks, is this meant for 6.3 or 6.2? This is easily backportable, just a script and packaging change.

Comment 6 pm-sat@redhat.com 2017-11-02 20:24:00 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/19496 has been resolved.

Comment 7 Matt Pusateri 2018-09-18 21:00:04 UTC
Dev, can we get some assistance on how to reproduce pushing passenger into the failed state.

Comment 8 Lukas Zapletal 2018-09-19 14:31:03 UTC
Dumb approach, set the memory limit in this script to 400 MB, restart Satellite httpd and run

watch passenger-status

Click in the UI randomly, do some actions via CLI/API to increase memory consumption above 400 MB and then wait until the script gets executed from cron (or execute it). It should restart one process per execution.

The config is /etc/passenger-recycler.yaml and value you want to change is MAX_PRIV_RSS_MEMORY. Here is an example file:

https://github.com/swapab/foreman_maintain/blob/302581759bf5bc31de962f6cc5ccecb889c84748/config/passenger-recycler.yaml

It looks like for some reason this YAML file is not being distributed with foreman-maintain, likely a bug.

Comment 10 Lukas Zapletal 2018-09-19 15:19:08 UTC
Correction of the YAML file link:

https://github.com/theforeman/foreman_maintain/blob/master/config/passenger-recycler.yaml

Filed BZ#1630958 for the missing YAML file.

Comment 11 Ivan Necas 2018-09-19 15:32:38 UTC
*** Bug 1630958 has been marked as a duplicate of this bug. ***

Comment 12 pm-sat@redhat.com 2018-09-19 16:05:52 UTC
Upstream bug assigned to inecas@redhat.com

Comment 13 pm-sat@redhat.com 2018-09-19 16:05:56 UTC
Upstream bug assigned to inecas@redhat.com

Comment 15 Patrick Creech 2018-09-21 00:40:01 UTC
adding sat-maintenance flag as the above pr deals with foreman-maintain

Comment 16 Peter Ondrejka 2018-09-24 13:04:44 UTC
Verified on Satellite 6.4 snap 23, expected files are present: 

rpm -ql rubygem-foreman_maintain-0.2.10-1.el7sat.noarch | grep passenger
/etc/passenger-recycler.yaml
/usr/bin/passenger-recycler
/usr/share/gems/gems/foreman_maintain-0.2.10/bin/passenger-recycler
/usr/share/gems/gems/foreman_maintain-0.2.10/config/passenger-recycler.yaml
/usr/share/gems/gems/foreman_maintain-0.2.10/definitions/procedures/passenger_recycler.rb

With setup suggested in comment #8, running passenger-recycler:
...
Terminating 15319 (started 08:22:44) with private RSS size of 459 MB
Process successfully 15319 terminated
...

So the custom memory limit is respected. The cron task is not set automatically with rpm installation, however this is intended, as per discussion with inecas

Comment 18 errata-xmlrpc 2018-10-16 15:27:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927


Note You need to log in before you can comment on or make changes to this bug.