Description of problem: Running # service MonitoringScout start on RHN Proxy 5.3.0 prints # service MonitoringScout start Starting MonitoringScout ... [ OK ] but there is no scout process to be found. Version-Release number of selected component (if applicable): spacewalk-proxy-monitoring-0.4.4-2.el5sat.noarch.rpm, upgraded from spacewalk-proxy-monitoring-0.4.4-1.el5sat.noarch.rpm (no MonitoringScout to be found in 0.4.4-1). How reproducible: Deterministic. Steps to Reproduce: 1. Run configure-proxy.sh, let it install monitoring. 2. Run service MonitoringScout stop 3. Run ps axuw > /tmp/a1 4. Run service MonitoringScout start 5. Run ps axuw > /tmp/a2 6. Run diff -u /tmp/a[12] Actual results: Similar to --- /tmp/a1 2009-05-06 10:56:17.000000000 +0200 +++ /tmp/a2 2009-05-06 10:56:20.000000000 +0200 @@ -90,4 +90,4 @@ jabberd 15710 0.0 0.2 6384 1644 ? S 10:42 0:00 /usr/bin/s2s -c /etc/jabberd/s2s.xml jabberd 15711 0.0 0.3 7248 2232 ? S 10:42 0:00 /usr/bin/c2s -c /etc/jabberd/c2s.xml jabberd 15712 0.0 0.3 7580 2516 ? Ss 10:42 0:00 perl -w -x /usr/bin/jabberd -b -root 15829 0.0 0.1 4252 948 pts/0 R+ 10:56 0:00 ps axuw +root 15839 0.0 0.1 4252 940 pts/0 R+ 10:56 0:00 ps axuw Expected results: There should be some scout-ish processes running. Additional info:
On Satellite 5.3.0, the MonitoringScout says upon startup: # service MonitoringScout start Starting MonitoringScout ... [ FAIL ] Starting NPBootstrap ... [ OK ] Starting SputLite ... [ OK ] Starting Dequeuer ... [ OK ] Starting Dispatcher ... [ OK ] [ OK ] # So I assume the problem is that on RHN Proxy, those "sub-services" do not get started. Plus the master MonitoringScout falsely reports that the service was started.
When I bump the logFileLevel in /etc/rc.d/np.d/SysV.ini to 10, this is what I get in /var/log/SysVStep.log: 2009-05-06 11:06:02 MonitoringScout: Debug level = 0 2009-05-06 11:06:02 MonitoringScout: Switches: start 2009-05-06 11:06:02 MonitoringScout: Non-LibMode reincarnation 2009-05-06 11:06:02 MonitoringScout: Loading self from database 2009-05-06 11:06:02 MonitoringScout: No exceptions specified, operation applies to all steps 2009-05-06 11:06:02 MonitoringScout: RUN: start 2009-05-06 11:06:02 MonitoringScout: Clearing last action errors list 2009-05-06 11:06:02 MonitoringScout: ============ STATUS =============== 2009-05-06 11:06:02 MonitoringScout: ---- Monitoring backend functionality is disabled 2009-05-06 11:06:02 MonitoringScout: ---- Monitoring scout functionality is disabled 2009-05-06 11:06:02 MonitoringScout: Last action: start 2009-05-06 11:06:02 MonitoringScout: ** Installed for SysV startup ** 2009-05-06 11:06:02 MonitoringScout: STARTED and RUNNING 2009-05-06 11:06:02 MonitoringScout: Stop actions: 2009-05-06 11:06:02 MonitoringScout: ===================================
When I change web.is_monitoring_scout to 1 in /etc/rhn/rhn.conf, I now get # /etc/init.d/MonitoringScout stop Stopping MonitoringScout ... [ OK ] [root@vmware145 tmp]# /etc/init.d/MonitoringScout start Starting MonitoringScout ... Starting InstallSoftwareConfig ... [ OK ] Starting NPBootstrap ... [ OK ] Starting SputLite ... [ OK ] Starting Dequeuer ... [ OK ] Starting Dispatcher ... [ OK ] [ OK ] # I wonder why configure-proxy.sh or some rpm %post script did not put that "1" there.
Alas, even if there are now multiple gogo.pl processes on the RHN Proxy, and there are accesses like 10.34.34.145 - - [06/May/2009:10:23:36 +0200] "GET /satconfig/cgi-bin/fetch_nocpulseini.cgi HTTP/1.1" 200 5305 in Satellite's log, the RHN Proxy's scout is not shown on Satellite's https://FQDN/network/monitoring/scout/index.pxt page.
The logs on the Satellite server are: ==> /var/log/httpd/ssl_access_log <== 10.34.34.145 - - [06/May/2009:10:28:41 +0200] "GET /satconfig/cgi-bin/fetch_nocpulseini.cgi HTTP/1.1" 200 5305 ==> /var/log/httpd/ssl_request_log <== [06/May/2009:10:28:41 +0200] 10.34.34.145 TLSv1 DHE-RSA-AES256-SHA "GET /satconfig/cgi-bin/fetch_nocpulseini.cgi HTTP/1.1" 5305 ==> /var/log/httpd/ssl_access_log <== 10.34.34.145 - - [06/May/2009:10:28:42 +0200] "GET /satconfig/cgi-bin/fetch_netsaintid.cgi?ssk=a1985d7dc3d8&publickey=ssh-dss%20AAAAB3NzaC1kc3MAAACBAJZfezeXngL6s81lTFgj74HEWWwukxw2PBncDv%2Fr2BIY7i1p3jedftgF1g93OLnnCTrXMI9BJ%2B0XCvNVWlSopvIKK91ndgnABg8CvLsJ%2FVk%2BQ2ZiwfrjE72JJ8pjAhJwdNP6anehI6XlGka9%2B1JobaAYRQak%2F%2BcW0giShHit%2FI8DAAAAFQCBP%2BKNpZ75C9gydq6w2p1Ena4hPQAAAIEAk%2Fjr1HbbIhnKUFBJFDUQ4%2F8MdDpge%2BdaG3vfJT8UH47%2BGZFDq6mkVJZVTuP81pCRR%2BaXZs90Ay2GpVyPul6UkmBcFHEAWvpW8rTn2Z1MHp25rKk7fIYx6F0N96vXfMbZfR7v6TeQnIn0%2FTo2tEDJCN00fYFZnxzUAdcLb5c0P9cAAACBAJMr%2FUENIUObFhz3I6PePIT1JEVGcg9k9kM%2B2JZKc4f3DQAE8KjhqZEtvjyK2MEJy9aFMM6ogBK0v9VxJi1ujNQlHpxsbFOsrxpsJF2hb2h8NpQv8lUjfmv5VjTOQ49Bfy1msIYTfbLuhAmASwUbnq5buVtrEE706rL5bzAlpVrS%20nocpulse%40vmware145%2Eenglab%2Ebrq%2Eredhat%2Ecom%0A HTTP/1.1" 200 44 ==> /var/log/httpd/ssl_request_log <== [06/May/2009:10:28:42 +0200] 10.34.34.145 TLSv1 DHE-RSA-AES256-SHA "GET /satconfig/cgi-bin/fetch_netsaintid.cgi?ssk=a1985d7dc3d8&publickey=ssh-dss%20AAAAB3NzaC1kc3MAAACBAJZfezeXngL6s81lTFgj74HEWWwukxw2PBncDv%2Fr2BIY7i1p3jedftgF1g93OLnnCTrXMI9BJ%2B0XCvNVWlSopvIKK91ndgnABg8CvLsJ%2FVk%2BQ2ZiwfrjE72JJ8pjAhJwdNP6anehI6XlGka9%2B1JobaAYRQak%2F%2BcW0giShHit%2FI8DAAAAFQCBP%2BKNpZ75C9gydq6w2p1Ena4hPQAAAIEAk%2Fjr1HbbIhnKUFBJFDUQ4%2F8MdDpge%2BdaG3vfJT8UH47%2BGZFDq6mkVJZVTuP81pCRR%2BaXZs90Ay2GpVyPul6UkmBcFHEAWvpW8rTn2Z1MHp25rKk7fIYx6F0N96vXfMbZfR7v6TeQnIn0%2FTo2tEDJCN00fYFZnxzUAdcLb5c0P9cAAACBAJMr%2FUENIUObFhz3I6PePIT1JEVGcg9k9kM%2B2JZKc4f3DQAE8KjhqZEtvjyK2MEJy9aFMM6ogBK0v9VxJi1ujNQlHpxsbFOsrxpsJF2hb2h8NpQv8lUjfmv5VjTOQ49Bfy1msIYTfbLuhAmASwUbnq5buVtrEE706rL5bzAlpVrS%20nocpulse%40vmware145%2Eenglab%2Ebrq%2Eredhat%2Ecom%0A HTTP/1.1" 44 ==> /var/log/httpd/ssl_access_log <== 10.34.34.145 - - [06/May/2009:10:28:43 +0200] "POST /cgi-bin/https_mx.cgi HTTP/1.1" 200 - ==> /var/log/httpd/ssl_request_log <== [06/May/2009:10:28:43 +0200] 10.34.34.145 TLSv1 DHE-RSA-AES256-SHA "POST /cgi-bin/https_mx.cgi HTTP/1.1" - ==> /var/log/httpd/ssl_access_log <== 10.34.34.145 - - [06/May/2009:10:28:46 +0200] "GET /satconfig/cgi-mod-perl/fetch_commands.cgi?cluster_id=a1985d7dc3d8&node_id=2&role=lead&version=1.0 HTTP/1.1" 200 23 ==> /var/log/httpd/ssl_request_log <== [06/May/2009:10:28:46 +0200] 10.34.34.145 TLSv1 DHE-RSA-AES256-SHA "GET /satconfig/cgi-mod-perl/fetch_commands.cgi?cluster_id=a1985d7dc3d8&node_id=2&role=lead&version=1.0 HTTP/1.1" 23 ==> /var/log/httpd/access_log <== 10.34.34.139 - - [06/May/2009:10:28:49 +0200] "POST /cgi-mod-perl/enqueue.cgi HTTP/1.1" 500 3962 "-" "libwww-perl/5.805" ==> /var/log/httpd/error_log <== [Wed May 06 10:28:49 2009] [error] Can't locate object method "from_query" via package "NOCpulse::Notif::Alert" at /var/www/cgi-mod-perl/enqueue.cgi line 189.\n [Wed May 06 10:28:49 2009] [error] [client 10.34.34.139] malformed header from script. Bad header=\xec!\x8e\xdbH$\x81\x94\x12\xad]\xc0\xabm\xfd\xa1W\xdcy\xce\xf9?\xc7\x1c\xfb\x9e\xecc\xc3\xcf: enqueue.cgi [Wed May 06 10:28:49 2009] [warn] /cgi-mod-perl/enqueue.cgi did not send an HTTP header [Wed May 06 10:28:49 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74. [Wed May 06 10:28:49 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74. ==> /var/log/httpd/ssl_error_log <== [Wed May 06 10:28:49 2009] [error] notif: Cannot POST to http://vmware139.englab.brq.redhat.com/cgi-mod-perl/enqueue.cgi: 500 Internal Server Error\n\n\n \n \n\n <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n\n\n \n \n \n \n\n<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">\n \n <head>\n \n <title>RHN Satellite - Install</title>\n <link rel="shortcut icon" href="/img/favicon.ico" />\n <link rel="stylesheet" href="/css/rhn-base.css" type="text/css" />\n <!--[if IE]>\n\t<link rel="stylesheet" type="text/css" href="/css/rhn-iecompat.css" />\n <![endif]-->\n </head>\n <script src="/javascript/prototype-1.6.0.js"></script>\n\n\n\n <body><div id="wrap">\n \n \n\n\n\n\n<div id="utility">\n <div id="utilityLinks">\n<p id="linx"><span class="hide"><strong>Shortcuts:</strong> </span><a\nhref="http://kbase.redhat.com/">Knowledgebase</a> <span\nclass="navPipe">|</span> <a href="/help">Documentation</a></p>\n </div>\n <div id="utilityAccount">\n \n </div>\n\n</div>\n\n<div id="header">\n\t<a href="http://www.redhat.com/" title="Visit redhat.com for more information about Red Hat products and services"><img src="/img/logo_header_rh.gif" alt="Red Hat" width="37" height="31" id="rhLogo" /></a><a href="/" title="Red Hat Network homepage"><img src="/img/rhn_satellite.gif" alt="Red Hat Network" width="235" height="31" id="rhnLogo" /></a>\n <div id="searchbar">\n <div id="searchbarinner">\n <form name="form1" action="/rhn/Search.do" method="get">\n <select name="search_type">\n\n <option value="systems">Systems</option>\n\n <option value="packages">Packages</option>\n <option value="errata">Errata</option>\n <option value="docs">Documentation</option>\n </select><input type="text" name="search_string" maxlength="40"\nsize="20" />\n\n <input type="hidden" name="submitted" value="true"/>\n <input type="submit" class="button" name="image-1" value="Search"\nalign="top" /></form>\n </div>\n </div>\n</div>\n\n<div id="navWrap">\n<div id="mainNavWrap">\n\n\n<ul>\n<li id="mainFirst-active"><a href="/rhn/newlogin/CreateFirstUser.do" class="mainFirstLink">Install</a></li>\n<li><a href="/">Sign In</a></li>\n\n<li id="mainLast"><a href="/help/about.pxt" class="mainLastLink">About</a></li>\n</ul>\n</div><!-- end mainNavWrap -->\n</div><!-- end navWrap -->\n\n \n\n \n \n \n \n\n<div id="bar">\n\n<div id="systembar">\n<div id="systembarinner">\n\n\n\n\n\n\n<div>\n\n</div>\n\n\n</div>\n</div>\n\n</div>\n<!-- close bar -->\n\n\n\n\n <div id="content">\n <table width="100%" cellspacing="0">\n <tr>\n <td class="sidebar">\n\t\n \n<div id="sidenavp"><ul></ul>\n</div>\n\n\t\n\n\t\n\t\n\n\n\n\n\n\n\t</td>\n <td class="page-content" width="100%">\n\n\n\n\n\n\n\n<h1>500 Error - Internal Server Error</h1>\n\n<p>You may have reached this page in one of the following ways:</p>\n <ol>\n <li>You are using an outdated version of Konqueror 3.0, which \n may not handle form variables properly in all cases. Continuing to \n use this outdated version of Konqueror may have unexpected results. \n Please up2date to the latest version of Konqueror (3.03 or \n greater), or if this is not possible, please use another \n browser.</li>\n <li>You've found an error in the site. Please report this error\n to your local administrator with details of how you received this \n message.</li>\n </ol>\n\n \n\n\t</td>\n </tr>\n </table>\n\n \n\n\n\n\n <div id="footer">\n Copyright © 2002-08 Red Hat, Inc. All rights reserved. <a href="https://www.redhat.com/legal/privacy_statement.html">Privacy statement</a> : <a href="http://www.redhat.com/legal/legal_statement.html">Legal statement</a> : <a href="http://www.redhat.com/">redhat.com</a>\n <div style="color: black">\n Red Hat Network release <a href="/rhn/help/release-notes/satellite/index.jsp">5.3.0</a>\n </div>\n <p></p>\n </div>\n\n\n \n\n\n \n\n\n\n </div> <!-- close content -->\n </div> <!-- close wrap -->\n\n </body>\n</html>\n\n\n\n\n\n ==> /var/log/httpd/ssl_access_log <== 10.34.34.145 - - [06/May/2009:10:28:49 +0200] "POST /cgi-bin/eventHandler.cgi HTTP/1.1" 500 - ==> /var/log/httpd/ssl_request_log <== [06/May/2009:10:28:49 +0200] 10.34.34.145 TLSv1 DHE-RSA-AES256-SHA "POST /cgi-bin/eventHandler.cgi HTTP/1.1" -
Problem in #3 fixed by commit 59f0fd3f5588953ab7296c5cb497e96a9b193143 (master) and 590b04ab29f4761497146a639270655a752eaaac (vader).
The problem in #5 is due to missing [notification] config_dir=/etc/notification in /etc/NOCpulse.ini I wonder why it is missing.
(In reply to comment #7) > The problem in #5 is due to missing > > [notification] > config_dir=/etc/notification > > in /etc/NOCpulse.ini > I wonder why it is missing. Added, restarted. It still does not seem to address issue in comment 4.
Hmm the the problem seems to be in non-existence sat_cluster record on satellite. It is created during webui installation, but the api call used in cli do not have it. in webui installer is this step: <activity description="Create SatCluster record for Satellite Monitoring Proxy" acl="session_data_true(enable_monitoring)"> <method-call class="RHN::ProxyInstall" method="create_sat_cluster_record"> <customer-id>${user.org_id}</customer-id> <description>${app.name} ${system.name} (${system.id})</description> <last-update-user>installer</last-update-user> </method-call> </activity> And in perl module is: sub create_sat_cluster_record { my $class = shift; my %params = validate(@_, { session => 1, customer_id => 1, description => 1, last_update_user => 0, }); my $session = $params{session}; my $hostname = $session->param('hostname'); my $vip = RHN::Utils::find_ip_address($hostname); my $sc = new RHN::SatCluster(customer_id => $params{customer_id}, description => $params{description}, last_update_user => $params{last_update_user} || 'installer', server_id => $params{session}->get_server->id, vip => $vip, ); $sc->create_new(); my $key = RHN::SatCluster->fetch_key($sc->recid); $params{session}->param(scout_shared_key => $key); return 0; } I should port this to Java.
New api call proxy.createMonitoringScout commited in 27add04dff3773387c5ce382accecdf7125096dc Now I had to modify cli installer to use this api call.
I did minor fixes in commits: 52a9a0caa50d0b323b11de5bb9cc52641e49bf48 f325fdde299eeb6c056c221570d3f473ec548f0f 0d72257f5df8bcac9f003fff651f8f82922c2d16 6d4ad92dd2cae2149dce48864c9480fb16f1b452 In commit b00e047e7cf61f087945be2a53638c7f0e7614a3 I made configure-proxy.sh to call api proxy.createMonitoringScout In commit 9bd443586e56a63c61fa96739d2799e62cfd17af I change logic of api call proxy.createMonitoringScout to return scout shared key on success In commit 906cb7232b1eb6ce4e65bf2843a63ae4198f220b I say configure-proxy to utilize this returned value and user is not asked for SCOUT_SHARED_KEY any more. Test plan: install proxy using command line installer make sure you enable Monitoring and Monitoring Scout (on satellite should be enabled too) check if new scout is created on satellite after successfull installation and scout is running on proxy.
Moving ON_QA QA Build: 20090529
must be related to proxy, this works w/o it..
err of course QA build 20090605
When the proxy installation tries to start MonitoringScout, I'm getting the following error: Starting rhn-proxy... init_cache_dir /var/spool/squid... Starting squid: ..[ OK ] Starting httpd: [FAILED] Starting Jabber services[ OK ] Starting MonitoringScout ... Starting NPBootstrap ... 2009-06-15 09:51:31 NPBootstrap: !! ERROR FROM SHELL COMMAND: 2009-06-15 09:51:31 NPBootstrap: !! STDOUT: 2009-06-15 09:51:31 NPBootstrap: !! STDERR: Can't call method "get" on an undefined value at /usr/bin/npBootstrap.pl line 15. 2009-06-15 09:51:31 NPBootstrap: !! EXIT: 512 [ FAIL ] 2009-06-15 09:51:31 NPBootstrap: WARNING: STARTED BUT *NOT* RUNNING 2009-06-15 09:51:31 NPBootstrap: ERRORS ENCOUNTERED DURING LAST ACTION: 2009-06-15 09:51:31 NPBootstrap: !! ERROR FROM SHELL COMMAND: 2009-06-15 09:51:31 NPBootstrap: !! STDOUT: 2009-06-15 09:51:31 NPBootstrap: !! STDERR: Can't call method "get" on an undefined value at /usr/bin/npBootstrap.pl line 15. 2009-06-15 09:51:31 NPBootstrap: !! EXIT: 512 Starting SputLite ... 2009-06-15 09:51:32 SputLite: !! ERROR FROM SHELL COMMAND: 2009-06-15 09:51:32 SputLite: !! STDOUT: 2009-06-15 09:51:32 SputLite: !! STDERR: 2009-06-15 09:51:32 SputLite: !! EXIT: 512 [ FAIL ] 2009-06-15 09:51:32 SputLite: WARNING: STARTED BUT *NOT* RUNNING 2009-06-15 09:51:32 SputLite: ERRORS ENCOUNTERED DURING LAST ACTION: 2009-06-15 09:51:32 SputLite: !! ERROR FROM SHELL COMMAND: 2009-06-15 09:51:32 SputLite: !! STDOUT: 2009-06-15 09:51:32 SputLite: !! STDERR: 2009-06-15 09:51:32 SputLite: !! EXIT: 512 Starting Dequeuer ... 2009-06-15 09:51:32 Dequeuer: !! ERROR FROM SHELL COMMAND: 2009-06-15 09:51:32 Dequeuer: !! STDOUT: 2009-06-15 09:51:32 Dequeuer: !! STDERR: 2009-06-15 09:51:32 Dequeuer: !! EXIT: 512 [ FAIL ] 2009-06-15 09:51:32 Dequeuer: WARNING: STARTED BUT *NOT* RUNNING 2009-06-15 09:51:32 Dequeuer: ERRORS ENCOUNTERED DURING LAST ACTION: 2009-06-15 09:51:32 Dequeuer: !! ERROR FROM SHELL COMMAND: 2009-06-15 09:51:32 Dequeuer: !! STDOUT: 2009-06-15 09:51:32 Dequeuer: !! STDERR: 2009-06-15 09:51:32 Dequeuer: !! EXIT: 512 Starting Dispatcher ... 2009-06-15 09:51:32 Dispatcher: !! ERROR FROM SHELL COMMAND: 2009-06-15 09:51:32 Dispatcher: !! STDOUT: 2009-06-15 09:51:32 Dispatcher: !! STDERR: 2009-06-15 09:51:32 Dispatcher: !! EXIT: 512 [ FAIL ] 2009-06-15 09:51:32 Dispatcher: WARNING: STARTED BUT *NOT* RUNNING 2009-06-15 09:51:32 Dispatcher: ERRORS ENCOUNTERED DURING LAST ACTION: 2009-06-15 09:51:32 Dispatcher: !! ERROR FROM SHELL COMMAND: 2009-06-15 09:51:32 Dispatcher: !! STDOUT: 2009-06-15 09:51:32 Dispatcher: !! STDERR: 2009-06-15 09:51:32 Dispatcher: !! EXIT: 512 [ OK ] Done. [root@rhndev1 rhn]#
Satellite for the above was rlx-0-06.rhndev.redhat.com
Ops I forgot to commit one line of code. Commited as 47c27ee2438923fe92c8f5745af58a34427f3c4e
iso 20090616 moving to ON_QA
moving back, it did not get to ISO.
It fail on new composes. When I revert commit from BZ 499564. Appareantly InstallSoftwareConfig is now not running on Proxy.
I modified BZ 499564, so this should work now.
iso 20090619.0 moving ON_QA
With Satellite-5.3.0-RHEL5-re20090625.0 and Proxy # rpm -qa | grep proxy rhn-proxy-branding-5.3.0.24-1.el5sat spacewalk-proxy-redirect-0.5.7-7.el5sat spacewalk-proxy-monitoring-0.4.4-3.el5sat spacewalk-proxy-installer-0.5.25-13.el5sat spacewalk-proxy-docs-0.4.1-2.el5sat spacewalk-proxy-common-0.5.7-7.el5sat spacewalk-proxy-package-manager-0.5.7-7.el5sat spacewalk-proxy-selinux-0.5.2-6.el5sat spacewalk-proxy-broker-0.5.7-7.el5sat spacewalk-proxy-management-0.5.7-7.el5sat the monitoring scout start behaves as expected -- the processes are started, and the scout appears in Satellite's /network/monitoring/scout/index.pxt page. The line config_dir=/etc/notification mentioned in comment 7 is not present in /etc/NOCpulse.ini on the Proxy but Mirek S. says that it's not needed after all. Marking as VERIFIED.