Bug 485532 - Apache process SizeLimit reached quickly and frequently during rhnpush/sat-sync/and other operations on Satellite 5.2/x86_64
Apache process SizeLimit reached quickly and frequently during rhnpush/sat-sy...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Server (Show other bugs)
520
x86_64 Linux
high Severity medium
: ---
: ---
Assigned To: Pradeep Kilambi
Jeff Ortel
:
: 496151 (view as bug list)
Depends On:
Blocks: 465198 456985 487993
  Show dependency treegraph
 
Reported: 2009-02-13 18:12 EST by Xixi
Modified: 2010-10-23 03:43 EDT (History)
20 users (show)

See Also:
Fixed In Version: sat530
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 465796
Environment:
Last Closed: 2009-09-10 16:32:59 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
x86_64 process state after restart (24.88 KB, application/octet-stream)
2009-02-25 23:09 EST, Clifford Perry
no flags Details

  None (edit)
Comment 1 Xixi 2009-02-13 18:17:55 EST
Description of problem:

Satellite 5.2 on x86_64 platform reports the following error in /var/log/httpd/error_log:

Apache2::SizeLimit httpd process too big, exiting at SIZE=400136 KB  SHARE=12560 KB  REQUESTS=153 LIFETIM
E=4013 seconds

The message comes every other second when deploying a new machine. This means that the deployment of servers are extremely slow to the point where the Satellite becomes unusable.

How reproducible:

Install Satellite 5.2 on x86_64 HW and deploy a server.

Steps to Reproduce:

Actual results:

It appears that each time a satellite is installed on a x86_64 box, the default MAX_PROCESS_SIZE is too low and has to be increased in order to avoid apache being killed because it eat too much memory.

Deploying a server takes 30 minutes instead of five. The server load is heavy because httpd-processes are killed, a new is forked which grows to the SizeLimit and then gets killed again.

Sample errors:

[Fri Dec 26 10:01:06 2008] (2022) Apache2::SizeLimit httpd process too big,
exiting at SIZE=526616 KB  SHARE=13580 KB  REQUESTS=2 LIFETIME=0 seconds
[Fri Dec 26 10:01:08 2008] (2034) Apache2::SizeLimit httpd process too big,
exiting at SIZE=526708 KB  SHARE=13548 KB  REQUESTS=2 LIFETIME=0 seconds
[Fri Dec 26 10:01:13 2008] (2044) Apache2::SizeLimit httpd process too big,
exiting at SIZE=526708 KB  SHARE=13548 KB  REQUESTS=2 LIFETIME=0 seconds
...

Expected results:

Deployment of server should be a lot faster and httpd-process should not be killed.

Temporary workaround:

Edit /etc/rhn/satellite-httpd/conf/startup.pl and changing the
MAX_PROCESS_SIZE to a size larger the the processes are reported to getting killed at (ideally stay multiples of 8), after that everything works much quicker and without errors.
Comment 18 Clifford Perry 2009-02-25 23:09:35 EST
Created attachment 333274 [details]
x86_64 process state after restart
Comment 27 Chris Duryee 2009-03-02 11:04:17 EST
Satellite team:

What happens if you remove Apache2::SizeLimit, and rely on the OOM killer? Does it still die?

We had a very similar problem on the www-perl stack, it turned out that Apache2::Sizelimit was measuring memory usage in a way that wasn't quite right on 64 bit boxes, and it ends up killing Apache when it shouldn't be killed. We bumped MAX_PROCESS_SIZE up as a workaround, but we'll likely be removing SizeLimit in the next release.
Comment 45 Brandon Perkins 2009-09-10 16:32:59 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2009-1434.html

Note You need to log in before you can comment on or make changes to this bug.