Bug 498211

Summary: Monitoring Not Functional: Missing Stderr.pm and malformed header errors in error_log
Product: Red Hat Satellite 5 Reporter: Devan Goodwin <dgoodwin>
Component: MonitoringAssignee: Miroslav Suchý <msuchy>
Status: CLOSED CURRENTRELEASE QA Contact: Petr Sklenar <psklenar>
Severity: medium Docs Contact:
Priority: low    
Version: 530CC: cperry, psklenar, whayutin
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard: Fixed in the 5.4.0 Release - GA'd 2010-10-27
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-28 15:01:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 608752    
Attachments:
Description Flags
monitoring works none

Description Devan Goodwin 2009-04-29 13:27:45 UTC
Description of problem:

Monitoring appears broken, error HTML appear in rhn-satellite status, pushing scout configs sits at "request pending".

Version-Release number of selected component (if applicable):

Satellite-5.3.0-RHEL5-re20090424.1-i386-embedded-oracle.iso

How reproducible:

Unclear, only have installed one system and it surfaced.

Steps to Reproduce:

As far as I know,

1. Install Satellite.
2. Enable monitoring.
3. rhn-satellite restart
4. Attempt to push scout config.
  
Actual results:

Scout config sits at request pending, errors in rhn-satellite status dumping out HTML indicating there's been an internal server error.

httpd error_log snippit:

[Wed Apr 29 00:52:12 2009] [error] Can't locate IO/Capture/Stderr.pm in @INC (@INC contains: /var/www/lib /usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.7/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.6/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.7/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.6/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib/perl5/5.8.8/i386-linux-thread-multi /usr/lib/perl5/5.8.8 . /etc/httpd) at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Mailer.pm line 10.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Mailer.pm line 10.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/NotifMailer.pm line 12.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/NotifMailer.pm line 12.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/EmailContactMethod.pm line 12.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/EmailContactMethod.pm line 12.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/PagerContactMethod.pm line 11.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/PagerContactMethod.pm line 11.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Send.pm line 12.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Send.pm line 12.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Escalator.pm line 20.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Escalator.pm line 20.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/EscalateStrategy.pm line 5.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/EscalateStrategy.pm line 5.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Alert.pm line 27.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Alert.pm line 27.\nCompilation failed in require at /var/www/cgi-mod-perl/enqueue.cgi line 43.\nBEGIN failed--compilation aborted at /var/www/cgi-mod-perl/enqueue.cgi line 43.\n
[Wed Apr 29 00:52:12 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.
[Wed Apr 29 00:52:12 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.
[Wed Apr 29 00:52:12 2009] [error] Can't locate IO/Capture/Stderr.pm in @INC (@INC contains: /var/www/lib /usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.7/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.6/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.7/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.6/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib/perl5/5.8.8/i386-linux-thread-multi /usr/lib/perl5/5.8.8 . /etc/httpd) at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Mailer.pm line 10.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Mailer.pm line 10.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/NotifMailer.pm line 12.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/NotifMailer.pm line 12.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/EmailContactMethod.pm line 12.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/EmailContactMethod.pm line 12.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/PagerContactMethod.pm line 11.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/PagerContactMethod.pm line 11.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Send.pm line 12.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Send.pm line 12.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Escalator.pm line 20.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Escalator.pm line 20.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/EscalateStrategy.pm line 5.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/EscalateStrategy.pm line 5.\nCompilation failed in require at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Alert.pm line 27.\nBEGIN failed--compilation aborted at /usr/lib/perl5/vendor_perl/5.8.8/NOCpulse/Notif/Alert.pm line 27.\nCompilation failed in require at /var/www/cgi-mod-perl/enqueue.cgi line 43.\nBEGIN failed--compilation aborted at /var/www/cgi-mod-perl/enqueue.cgi line 43.\n
[Wed Apr 29 00:52:13 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.
[Wed Apr 29 00:52:13 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.
[Wed Apr 29 01:00:59 2009] [error] Can't locate object method "from_query" via package "NOCpulse::Notif::Alert" at /var/www/cgi-mod-perl/enqueue.cgi line 189.\n
[Wed Apr 29 01:00:59 2009] [error] [client 127.0.0.1] malformed header from script. Bad header=\xc30\x81\x96\x06\x03U\x1d#\x04\x81\x8e0\x81\x8b\x80\x14\xb8\xcc\xffh\x1eY\xfc\x05\t\xae\xd76\xce: enqueue.cgi
[Wed Apr 29 01:00:59 2009] [warn] /cgi-mod-perl/enqueue.cgi did not send an HTTP header
[Wed Apr 29 01:00:59 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.
[Wed Apr 29 01:01:00 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.
[Wed Apr 29 01:01:55 2009] [error] Can't locate object method "from_query" via package "NOCpulse::Notif::Alert" at /var/www/cgi-mod-perl/enqueue.cgi line 189.\n
[Wed Apr 29 01:01:55 2009] [error] [client 127.0.0.1] malformed header from script. Bad header=/p>: enqueue.cgi
[Wed Apr 29 01:01:55 2009] [warn] /cgi-mod-perl/enqueue.cgi did not send an HTTP header
[Wed Apr 29 01:01:55 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.
[Wed Apr 29 01:01:55 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.
[Wed Apr 29 01:02:06 2009] [error] Can't locate object method "from_query" via package "NOCpulse::Notif::Alert" at /var/www/cgi-mod-perl/enqueue.cgi line 189.\n
[Wed Apr 29 01:02:06 2009] [error] [client 127.0.0.1] malformed header from script. Bad header=\x81\x96\x06\x03U\x1d#\x04\x81\x8e0\x81\x8b\x80\x14\xb8\xcc\xffh\x1eY\xfc\x05\t\xae\xd76\xce\xca\x1a: enqueue.cgi
[Wed Apr 29 01:02:06 2009] [warn] /cgi-mod-perl/enqueue.cgi did not send an HTTP header
[Wed Apr 29 01:02:06 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.
[Wed Apr 29 01:02:06 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.
[Wed Apr 29 01:02:12 2009] [error] Can't locate object method "from_query" via package "NOCpulse::Notif::Alert" at /var/www/cgi-mod-perl/enqueue.cgi line 189.\n
[Wed Apr 29 01:02:12 2009] [error] [client 127.0.0.1] malformed header from script. Bad header=cessor%20exited&type=adhoct=&s: enqueue.cgi
[Wed Apr 29 01:02:12 2009] [warn] /cgi-mod-perl/enqueue.cgi did not send an HTTP header
[Wed Apr 29 01:02:12 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.
[Wed Apr 29 01:02:12 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.
[Wed Apr 29 01:02:13 2009] [error] Can't locate object method "from_query" via package "NOCpulse::Notif::Alert" at /var/www/cgi-mod-perl/enqueue.cgi line 189.\n
[Wed Apr 29 01:02:13 2009] [error] [client 127.0.0.1] malformed header from script. Bad header=\x06\x03U\x1d#\x04\x81\x8e0\x81\x8b\x80\x14\xb8\xcc\xffh\x1eY\xfc\x05\t\xae\xd76\xce\xca\x1a\x0c8: enqueue.cgi
[Wed Apr 29 01:02:13 2009] [warn] /cgi-mod-perl/enqueue.cgi did not send an HTTP header
[Wed Apr 29 01:02:13 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.
[Wed Apr 29 01:02:13 2009] [error] returning 1 at /usr/lib/perl5/vendor_perl/5.8.8/RHN/Access.pm line 74.

Expected results:

Monitoring works. :)

Additional info:

The box has no clients registered, and just RHEL 5 + VT + Tools channels synced, which was done using Inter Satellite sync.

Comment 1 Devan Goodwin 2009-04-29 13:54:29 UTC
Did some more investigation and it looks like part of the problems above (the error HTML using rhn-satellite) was related to bug #492150. Upgrading to eventReceivers-2.20.12-1.noarch.rpm manually corrected this problem, the errors listed above are still appearing in error_log, but Scout Config Push completes, and rhn-satellite status does not dump HTML anymore.

rhn-satellite restart shows the following:

Starting MonitoringScout ...  [ FAIL ]

Comment 2 Miroslav Suchý 2009-04-30 07:56:51 UTC
This should wanted behaviour when MonitoringScout is not enabled. (It is your situation?)
I understand that if it is not enabled it should not write anything (neither OK nor FAIL), but is hard to implement now, since it is currently not made as true service (but one script which start bunch of other scripts which should be standalone services).
I'm reluctant to fix this in 530 since it will break a lot of monitoring code.

Since the first item is fixed, is it ok for you to close this bug as NOTABUG or DUPLICATE?

Comment 3 Devan Goodwin 2009-04-30 13:17:00 UTC
Monitoring scout is definitely enabled, but I'm still seeing that FAIL when starting the service. Monitoring in general is acting very weird for me, I don't know if it's related. I think I have it working (maybe), some probes are checking in at least (others aren't). Something feels unhealthy but to be honest the errors reported above have stopped, I don't know why. :( 

I guess that means the only thing left for this ticket is why MonitoringScout shows as FAIL when starting?

Comment 4 wes hayutin 2009-05-01 19:57:21 UTC
related or same bug?
https://bugzilla.redhat.com/show_bug.cgi?id=498062

Comment 5 wes hayutin 2009-05-01 20:24:03 UTC
the above bug actually breaks monitoring.. so I dont think we can push these off to another release

Comment 6 Devan Goodwin 2009-05-01 20:25:01 UTC
I think different bugs, I wasn't getting that error from NPBootstrap, just the monitoring FAIL above it.

Comment 7 Miroslav Suchý 2009-07-23 09:40:38 UTC
Fixed some time ago with commit 2a9119bb5558f6baf8eac173dc5d26b757a73ddd

Comment 8 Petr Sklenar 2010-09-21 16:24:15 UTC
Created attachment 448740 [details]
monitoring works

it works for me,
tried Satellite-5.4.0-RHEL5-re20100910.2
--
verified

Comment 9 Clifford Perry 2010-10-28 15:01:36 UTC
The 5.4.0 RHN Satellite and RHN Proxy release has occurred. This issue has been resolved with this release. 


RHEA-2010:0801 - RHN Satellite Server 5.4.0 Upgrade
https://rhn.redhat.com/rhn/errata/details/Details.do?eid=10332

RHEA-2010:0803 - RHN Tools enhancement update
https://rhn.redhat.com/rhn/errata/details/Details.do?eid=10333

RHEA-2010:0802 - RHN Proxy Server 5.4.0 bug fix update
https://rhn.redhat.com/rhn/errata/details/Details.do?eid=10334

RHEA-2010:0800 - RHN Satellite Server 5.4.0
https://rhn.redhat.com/rhn/errata/details/Details.do?eid=10335

Docs are available:

http://docs.redhat.com/docs/en-US/Red_Hat_Network_Satellite/index.html 

Regards,
Clifford