Bug 503187
| Summary: | RHN Proxy on 64bit bloats httpd until oom-killer triggers | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite Proxy 5 | Reporter: | Martin Poole <mpoole> | ||||||
| Component: | Server | Assignee: | Miroslav Suchý <msuchy> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Preethi Thomas <pthomas> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | low | ||||||||
| Version: | 520 | CC: | bperkins, cperry, mzazrivec, pthomas, tao | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | sat530 | Doc Type: | Bug Fix | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | |||||||||
| : | 504342 (view as bug list) | Environment: | |||||||
| Last Closed: | 2009-09-10 14:39:02 UTC | Type: | --- | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 456999, 504342 | ||||||||
| Attachments: |
|
||||||||
|
Description
Martin Poole
2009-05-29 14:09:30 UTC
Commit 866cfc28694446faea177c3eb2e74a545b658bff for WebUI installer. Commit 501703a251702723c320a9ee46551b171ab28f5b for CLI installer. compose 20090612 moving ON_QA On tag was not closed. Fixed in 65806ffdc604a27c5ef9bbdfc86f3fdf5012ba91 iso 20090616 moving to ON_QA I had RHN Proxy 5.2.0 on x86_64, and indeed, the resident memory of httpd processes grows. I've then installed spacewalk-proxy-installer (after workarounding bug 509522 with yum remove rhns-proxy-tools) and run configure-proxy.sh: [root@xen86 ~]# configure-proxy.sh RHN Parent [rlx-1-18.rhndev.redhat.com]: Proxy version to activate [5.3]: Traceback email []: jpazdziora Use SSL [Y/n]: y CA Chain [/usr/share/rhn/RHNS-CA-CERT]: /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT HTTP Proxy []: Regardless of whether you enabled SSL for the connection to the Spacewalk Parent Server, you will be prompted to generate an SSL certificate. This SSL certificate will allow client systems to connect to this Spacewalk Proxy securely. Refer to the Spacewalk Proxy Installation Guide for more information. Organization []: Organization Unit [xen86.englab.brq.redhat.com]: Common Name [xen86.englab.brq.redhat.com]: City []: B State []: . Country code []: CZ Email [jpazdziora]: API version: 5.3.0 RHN Proxy successfully activated. Loaded plugins: rhnplugin, security redhat-rhn-proxy-5.3-server-x86_64-5 | 871 B 00:00 primary.xml.gz | 33 kB 00:00 redhat-rhn-proxy-5.3-server-x86_64-5 101/101 Setting up Install Process Parsing package install arguments Resolving Dependencies There are unfinished transactions remaining. You mightconsider running yum-complete-transaction first to finish them. --> Running transaction check ---> Package spacewalk-proxy-management.noarch 0:0.5.7-7.el5sat set to be updated --> Processing Dependency: spacewalk-proxy-redirect = 0.5.7 for package: spacewalk-proxy-management --> Processing Dependency: spacewalk-proxy-broker = 0.5.7 for package: spacewalk-proxy-management --> Processing Dependency: spacewalk-proxy-common >= 0.5.7 for package: spacewalk-proxy-management --> Processing Dependency: spacewalk-proxy-selinux for package: spacewalk-proxy-management --> Processing Dependency: spacewalk-proxy-docs for package: spacewalk-proxy-management --> Processing Dependency: sos for package: spacewalk-proxy-management --> Processing Dependency: spacewalk-backend for package: spacewalk-proxy-management --> Processing Dependency: spacewalk-proxy-html for package: spacewalk-proxy-management --> Running transaction check --> Processing Dependency: rhns-proxy-broker >= 3.6.0 for package: rhns-proxy-package-manager ---> Package rhn-proxy-branding.noarch 0:5.3.0.24-1.el5sat set to be updated ---> Package spacewalk-proxy-selinux.noarch 0:0.5.2-6.el5sat set to be updated ---> Package spacewalk-proxy-broker.noarch 0:0.5.7-7.el5sat set to be updated --> Processing Dependency: spacewalk-certs-tools for package: spacewalk-proxy-broker ---> Package spacewalk-backend.noarch 0:0.5.28-24.el5sat set to be updated ---> Package sos.noarch 0:1.7-9.16.el5_3.5 set to be updated ---> Package spacewalk-proxy-common.noarch 0:0.5.7-7.el5sat set to be updated ---> Package spacewalk-proxy-redirect.noarch 0:0.5.7-7.el5sat set to be updated ---> Package spacewalk-proxy-docs.noarch 0:0.4.1-2.el5sat set to be updated --> Running transaction check ---> Package spacewalk-proxy-package-manager.noarch 0:0.5.7-7.el5sat set to be updated ---> Package spacewalk-certs-tools.noarch 0:0.5.5-6.el5sat set to be updated --> Finished Dependency Resolution Dependencies Resolved ================================================================================================================================================================ Package Arch Version Repository Size ================================================================================================================================================================ Installing: spacewalk-backend noarch 0.5.28-24.el5sat redhat-rhn-proxy-5.3-server-x86_64-5 124 k replacing rhns.noarch 5.2.0-15.el5 spacewalk-certs-tools noarch 0.5.5-6.el5sat rhn-tools-rhel-x86_64-server-5 131 k replacing rhns-certs-tools.noarch 5.2.0-4.el5 spacewalk-proxy-broker noarch 0.5.7-7.el5sat redhat-rhn-proxy-5.3-server-x86_64-5 37 k replacing rhns-proxy-broker.noarch 5.2.0-15.el5 spacewalk-proxy-docs noarch 0.4.1-2.el5sat redhat-rhn-proxy-5.3-server-x86_64-5 5.1 M replacing rhns-proxy-docs.noarch 5.2.0-15.el5 spacewalk-proxy-management noarch 0.5.7-7.el5sat redhat-rhn-proxy-5.3-server-x86_64-5 17 k spacewalk-proxy-package-manager noarch 0.5.7-7.el5sat redhat-rhn-proxy-5.3-server-x86_64-5 37 k replacing rhns-proxy-package-manager.noarch 5.2.0-15.el5 spacewalk-proxy-redirect noarch 0.5.7-7.el5sat redhat-rhn-proxy-5.3-server-x86_64-5 26 k replacing rhns-proxy-redirect.noarch 5.2.0-15.el5 Installing for dependencies: rhn-proxy-branding noarch 5.3.0.24-1.el5sat redhat-rhn-proxy-5.3-server-x86_64-5 12 k sos noarch 1.7-9.16.el5_3.5 rhel-x86_64-server-5 116 k spacewalk-proxy-common noarch 0.5.7-7.el5sat redhat-rhn-proxy-5.3-server-x86_64-5 61 k spacewalk-proxy-selinux noarch 0.5.2-6.el5sat redhat-rhn-proxy-5.3-server-x86_64-5 10 k Transaction Summary ================================================================================================================================================================ Install 11 Package(s) Update 0 Package(s) Remove 0 Package(s) Total download size: 5.7 M Is this ok [y/N]: y Downloading Packages: (1/11): spacewalk-proxy-selinux-0.5.2-6.el5sat.noarch.rpm | 10 kB 00:00 (2/11): rhn-proxy-branding-5.3.0.24-1.el5sat.noarch.rpm | 12 kB 00:00 (3/11): spacewalk-proxy-management-0.5.7-7.el5sat.noarch.rpm | 17 kB 00:00 (4/11): spacewalk-proxy-redirect-0.5.7-7.el5sat.noarch.rpm | 26 kB 00:00 (5/11): spacewalk-proxy-package-manager-0.5.7-7.el5sat.noarch.rpm | 37 kB 00:00 (6/11): spacewalk-proxy-broker-0.5.7-7.el5sat.noarch.rpm | 37 kB 00:00 (7/11): spacewalk-proxy-common-0.5.7-7.el5sat.noarch.rpm | 61 kB 00:00 (8/11): sos-1.7-9.16.el5_3.5.noarch.rpm | 116 kB 00:00 (9/11): spacewalk-backend-0.5.28-24.el5sat.noarch.rpm | 124 kB 00:00 (10/11): spacewalk-certs-tools-0.5.5-6.el5sat.noarch.rpm | 131 kB 00:00 (11/11): spacewalk-proxy-docs-0.4.1-2.el5sat.noarch.rpm | 5.1 MB 00:12 ---------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 280 kB/s | 5.7 MB 00:20 Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Installing : spacewalk-certs-tools [ 1/17] Installing : spacewalk-proxy-docs [ 2/17] Installing : sos [ 3/17] Installing : rhn-proxy-branding [ 4/17] Installing : spacewalk-proxy-common [ 5/17] warning: /etc/httpd/conf.d/rhn_proxy.conf created as /etc/httpd/conf.d/rhn_proxy.conf.rpmnew Installing : spacewalk-proxy-package-manager [ 6/17] Installing : spacewalk-proxy-broker [ 7/17] Installing : spacewalk-backend [ 8/17] Installing : spacewalk-proxy-redirect [ 9/17] Installing : spacewalk-proxy-selinux [10/17] /sbin/restorecon reset /var/log/rhn context system_u:object_r:var_log_t:s0->system_u:object_r:spacewalk_log_t:s0 /sbin/restorecon reset /var/log/rhn/rhn_proxy_broker.log context root:object_r:httpd_log_t:s0->system_u:object_r:spacewalk_proxy_httpd_log_t:s0 /sbin/restorecon reset /var/log/rhn/rhn_proxy_redirect.log context root:object_r:httpd_log_t:s0->system_u:object_r:spacewalk_proxy_httpd_log_t:s0 /sbin/restorecon reset /var/cache/rhn/proxy-auth context system_u:object_r:var_t:s0->system_u:object_r:spacewalk_proxy_cache_t:s0 /sbin/restorecon reset /var/cache/rhn/proxy-auth/1000010260 context root:object_r:var_t:s0->system_u:object_r:spacewalk_proxy_cache_t:s0 /sbin/restorecon reset /var/cache/rhn/proxy-auth/p1000010160 context root:object_r:var_t:s0->system_u:object_r:spacewalk_proxy_cache_t:s0 /sbin/restorecon reset /var/spool/rhn-proxy context system_u:object_r:var_spool_t:s0->system_u:object_r:spacewalk_proxy_data_t:s0 Installing : spacewalk-proxy-management [11/17] Erasing : rhns-proxy-broker [12/17] Erasing : rhns-proxy-redirect [13/17] Erasing : rhns [14/17] Erasing : rhns-certs-tools [15/17] Erasing : rhns-proxy-docs [16/17] Erasing : rhns-proxy-package-manager [17/17] Installed: spacewalk-backend.noarch 0:0.5.28-24.el5sat spacewalk-certs-tools.noarch 0:0.5.5-6.el5sat spacewalk-proxy-broker.noarch 0:0.5.7-7.el5sat spacewalk-proxy-docs.noarch 0:0.4.1-2.el5sat spacewalk-proxy-management.noarch 0:0.5.7-7.el5sat spacewalk-proxy-package-manager.noarch 0:0.5.7-7.el5sat spacewalk-proxy-redirect.noarch 0:0.5.7-7.el5sat Dependency Installed: rhn-proxy-branding.noarch 0:5.3.0.24-1.el5sat sos.noarch 0:1.7-9.16.el5_3.5 spacewalk-proxy-common.noarch 0:0.5.7-7.el5sat spacewalk-proxy-selinux.noarch 0:0.5.2-6.el5sat Replaced: rhns.noarch 0:5.2.0-15.el5 rhns-certs-tools.noarch 0:5.2.0-4.el5 rhns-proxy-broker.noarch 0:5.2.0-15.el5 rhns-proxy-docs.noarch 0:5.2.0-15.el5 rhns-proxy-package-manager.noarch 0:5.2.0-15.el5 rhns-proxy-redirect.noarch 0:5.2.0-15.el5 Complete! You do not have monitoring installed. Do you want to install it? Will run 'yum install spacewalk-proxy-monitoring'. [Y/n]: n Using CA key at /root/ssl-build/RHN-ORG-PRIVATE-SSL-KEY. Generating distributable RPM for CA public certificate: Generating SSL key and public certificate: CA password: Installing SSL certificate for Apache and Jabberd: Preparing packages for installation... rhn-org-httpd-ssl-key-pair-xen86.englab.brq-1.0-1 Create and populate configuration channel rhn_proxy_config_1000010160? [Y/n]: Y Using server name rlx-1-18.rhndev.redhat.com Red Hat Network username: admin Password: Creating config channel rhn_proxy_config_1000010160 Config channel rhn_proxy_config_1000010160 already exists Using server name rlx-1-18.rhndev.redhat.com Pushing to channel rhn_proxy_config_1000010160: Local file /etc/httpd/conf.d/ssl.conf -> remote file /etc/httpd/conf.d/ssl.conf Local file /etc/rhn/rhn.conf -> remote file /etc/rhn/rhn.conf Local file /etc/rhn/cluster.ini -> remote file /etc/rhn/cluster.ini Local file /etc/squid/squid.conf -> remote file /etc/squid/squid.conf Local file /etc/httpd/conf.d/cobbler-proxy.conf -> remote file /etc/httpd/conf.d/cobbler-proxy.conf Local file /etc/httpd/conf/httpd.conf -> remote file /etc/httpd/conf/httpd.conf Local file /etc/httpd/conf.d/rhn_proxy.conf -> remote file /etc/httpd/conf.d/rhn_proxy.conf Local file /etc/httpd/conf.d/proxy_broker.conf -> remote file /etc/httpd/conf.d/proxy_broker.conf Local file /etc/httpd/conf.d/proxy_redirect.conf -> remote file /etc/httpd/conf.d/proxy_redirect.conf Local file /etc/jabberd/c2s.xml -> remote file /etc/jabberd/c2s.xml Local file /etc/jabberd/sm.xml -> remote file /etc/jabberd/sm.xml Enabling Spacewalk Proxy. Shutting down rhn-proxy... Shutting down Jabber router: [ OK ] Stopping httpd: [ OK ] Stopping squid: . [ OK ] Done. Starting rhn-proxy... Starting squid: . [ OK ] Starting httpd: [ OK ] Starting Jabber services [ OK ] Done. However, since the /etc/httpd/conf.d/rhn_proxy.conf was created as .rpmnew, then newly started RHn Proxy does not have the MaxRequestsPerChild fix: # diff -u /etc/httpd/conf.d/rhn_proxy.conf /etc/httpd/conf.d/rhn_proxy.conf.rpmnew --- /etc/httpd/conf.d/rhn_proxy.conf 2009-07-02 10:06:13.000000000 +0200 +++ /etc/httpd/conf.d/rhn_proxy.conf.rpmnew 2009-06-15 23:35:29.000000000 +0200 @@ -1,6 +1,11 @@ # ** DO NOT EDIT ** # RHN Proxy handler configuration file +<IfModule prefork.c> + # bug #503187 + MaxRequestsPerChild 200 +</IfModule> + # RHN Proxy Server location <LocationMatch "^/*"> # this stanza contains all the handlers for RHN app code @@ -79,5 +84,5 @@ #Disable SSL2, left only higher SSLProtocol all -SSLv2 - +SSLProxyEngine on I changed %config(noreplace) to plain %config. This way this file will be overwritten during upgrade and old file will be save as .rpmorig Commited as 673955d2f971801c2ebb755c3d537d158c2bcb96 ISO 20090707.0 Mass moving to ON_QA Fails QA. I'm seeing different results. In Stage, I have the SSLProxyEngine on but I don't have the MaxRequestsPerChild modification: /etc/httpd/conf/httpd.conf:# MaxRequestsPerChild: maximum number of requests a server process serves /etc/httpd/conf/httpd.conf:MaxRequestsPerChild 4000 /etc/httpd/conf/httpd.conf:# MaxRequestsPerChild: maximum number of requests a server process serves /etc/httpd/conf/httpd.conf:MaxRequestsPerChild 0 I did my Proxy on RHEL5 instead of RHEL4 like Comment #10 . But I followed the steps in Comment #6 pretty identically. My Proxy was dhcp77-177.rhndev.redhat.com as a guest off of bperkins-64.rhndev.redhat.com and was using test1182.test.redhat.com as a Stage Satellite. [root@dhcp77-177 httpd]# rpm -V spacewalk-proxy-common S.5....T c /etc/httpd/conf.d/rhn_proxy.conf S.5....T c /etc/rhn/rhn.conf missing /var/cache/rhn/proxy-auth missing /var/spool/rhn-proxy/list Hmm I wonder how this can happend. I did: [root@dhcp77-177 tmp]# rpm --force -Uvh spacewalk-proxy-common-0.5.7-8.el4sat.noarch.rpm warning: spacewalk-proxy-common-0.5.7-8.el4sat.noarch.rpm: Header V3 DSA signature: NOKEY, key ID db42a60e Preparing... ########################################### [100%] 1:spacewalk-proxy-common ########################################### [100%] but still: [root@dhcp77-177 tmp]# rpm -V spacewalk-proxy-common S.5....T c /etc/httpd/conf.d/rhn_proxy.conf S.5....T c /etc/rhn/rhn.conf I wonder how your situation been created. Acording to http://www-uxsup.csx.cam.ac.uk/~jw35/docs/rpm_config.html this situation can only happend when the configuration file in rpm is the same as in previous version (which was not this case, so you had to installed it previously) and then edit it manualy. Hmm: [root@dhcp77-177 ~]# rhncfg-client verify /etc/httpd/conf.d/rhn_proxy.conf /etc/httpd/conf.d/rhn_proxy.conf So you have the rhn_proxy.conf the same as in configuration channel. May it be that install previously some old proxy, populate config channel, then install 5.3 and choose to not create/populate config channel and then restore config files from config file? My Steps: Clean RHEL5 Box. Installed 5.2 Proxy with Monitoring via the WebUI and it installed clean: =================================== # rhn_check Loaded plugins: rhnplugin * Adding users -- Prod account nocpulse Locking password for user nocpulse. passwd: Success -- Login account nocops * Finished adding users * Setting passwds -- root Root already has password ($1$PvaCh1kP$/QXbbiJp.FtlRLkG7jHN1.) * Finished setting root passwd * Setting up nocpulse homedir and ssh key pair * Finished setting up nocpulse homedir and ssh key pair =================================== Verified that the Proxy was functioning. Then I did an upgrade per the attached script output text. Created attachment 356910 [details]
Script Output from running 5.3 Upgrade.
OK. I put into configure-proxy.sh part where just after activation we upgrade all packages to recent version. This will pull new code and new config files. Commit bc581333ab6bf49764fccb6bf9982b6017a75b35 typo correction in commit 38a125da4cbc91a8be15cc79cf19ec507d7b4e7a fails_qa satellite: dhcp77-153.rhndev.redhat.com Created attachment 357696 [details]
Script Output for proxy upgrade
Got it. I can confirm it.
It happens in %post section of spacewalk-proxy-management when is called:
if rhncfg-client verify /etc/httpd/conf.d/rhn_proxy.conf | grep -E '(modified|missing)'; then
/sbin/service httpd stop
/usr/bin/rhncfg-client get /etc/httpd/conf.d/rhn_proxy.conf
/sbin/service httpd start
else ...
So the rhn_proxy.conf, but then is correctly installed, but then is overwritten with version from config channel, which is the old one.
This code has been added in:
http://svn.rhndev.redhat.com/viewcvs/trunk/eng/proxy/proxy/proxy.spec?rev=132312&r1=131902&r2=132312
We had following possibilities with these pros and cons:
a) do nothing, leave it as is
Pros: if customer done changes in these file and put it in conf channel, thay will be preserved
Cons: our update (mainly MaxRequestsPerChild) will not be applied.
b) remove this part of code,
Pros: we get our updates
Cons: client updates will be lost - well you had to manualy verify and merge it with content in you config dir (well - with previous version since at the end of install/upgrade will be created new version in config channel).
c) remove this file from configuration channel
Pros: upgrade will be clean
Cons: client updates will be lost and previous version in channel will be lost as well.
As I'm writing this I see that option b) is less painfull.
Removed by commit dce792558b6d97c86a807f53d342f94c3e7d3576 Verified in stage with the same results as in comment #23 -> RELEASE_PENDING. An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHEA-2009-1433.html |