Bug 485532 - Apache process SizeLimit reached quickly and frequently during rhnpush/sat-sync/and other operations on Satellite 5.2/x86_64
Summary: Apache process SizeLimit reached quickly and frequently during rhnpush/sat-sy...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Server
Version: 520
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: ---
Assignee: Pradeep Kilambi
QA Contact: Jeff Ortel
URL:
Whiteboard:
: 496151 (view as bug list)
Depends On:
Blocks: 456985 465198 487993
TreeView+ depends on / blocked
 
Reported: 2009-02-13 23:12 UTC by Xixi
Modified: 2018-10-20 02:31 UTC (History)
20 users (show)

Fixed In Version: sat530
Doc Type: Bug Fix
Doc Text:
Clone Of: 465796
Environment:
Last Closed: 2009-09-10 20:32:59 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
x86_64 process state after restart (24.88 KB, application/octet-stream)
2009-02-26 04:09 UTC, Clifford Perry
no flags Details

Comment 1 Xixi 2009-02-13 23:17:55 UTC
Description of problem:

Satellite 5.2 on x86_64 platform reports the following error in /var/log/httpd/error_log:

Apache2::SizeLimit httpd process too big, exiting at SIZE=400136 KB  SHARE=12560 KB  REQUESTS=153 LIFETIM
E=4013 seconds

The message comes every other second when deploying a new machine. This means that the deployment of servers are extremely slow to the point where the Satellite becomes unusable.

How reproducible:

Install Satellite 5.2 on x86_64 HW and deploy a server.

Steps to Reproduce:

Actual results:

It appears that each time a satellite is installed on a x86_64 box, the default MAX_PROCESS_SIZE is too low and has to be increased in order to avoid apache being killed because it eat too much memory.

Deploying a server takes 30 minutes instead of five. The server load is heavy because httpd-processes are killed, a new is forked which grows to the SizeLimit and then gets killed again.

Sample errors:

[Fri Dec 26 10:01:06 2008] (2022) Apache2::SizeLimit httpd process too big,
exiting at SIZE=526616 KB  SHARE=13580 KB  REQUESTS=2 LIFETIME=0 seconds
[Fri Dec 26 10:01:08 2008] (2034) Apache2::SizeLimit httpd process too big,
exiting at SIZE=526708 KB  SHARE=13548 KB  REQUESTS=2 LIFETIME=0 seconds
[Fri Dec 26 10:01:13 2008] (2044) Apache2::SizeLimit httpd process too big,
exiting at SIZE=526708 KB  SHARE=13548 KB  REQUESTS=2 LIFETIME=0 seconds
...

Expected results:

Deployment of server should be a lot faster and httpd-process should not be killed.

Temporary workaround:

Edit /etc/rhn/satellite-httpd/conf/startup.pl and changing the
MAX_PROCESS_SIZE to a size larger the the processes are reported to getting killed at (ideally stay multiples of 8), after that everything works much quicker and without errors.

Comment 18 Clifford Perry 2009-02-26 04:09:35 UTC
Created attachment 333274 [details]
x86_64 process state after restart

Comment 27 Chris Duryee 2009-03-02 16:04:17 UTC
Satellite team:

What happens if you remove Apache2::SizeLimit, and rely on the OOM killer? Does it still die?

We had a very similar problem on the www-perl stack, it turned out that Apache2::Sizelimit was measuring memory usage in a way that wasn't quite right on 64 bit boxes, and it ends up killing Apache when it shouldn't be killed. We bumped MAX_PROCESS_SIZE up as a workaround, but we'll likely be removing SizeLimit in the next release.

Comment 45 Brandon Perkins 2009-09-10 20:32:59 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2009-1434.html


Note You need to log in before you can comment on or make changes to this bug.