Bug 817997

Summary: osa-dispatcher service will not start unless "/var/lib/jabberd/db/" is cleared out beforehand
Product: [Community] Spacewalk Reporter: Dan Burkland <dburkland>
Component: ServerAssignee: Milan Zázrivec <mzazrivec>
Status: CLOSED CURRENTRELEASE QA Contact: Red Hat Satellite QA List <satqe-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 1.7CC: jpazdziora
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-05-08 12:18:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 871344    

Description Dan Burkland 2012-05-02 01:29:41 UTC
Description of problem: The osa-dispatcher service will not start unless "/var/lib/jabberd/db/" is cleared out beforehand


Version-Release number of selected component (if applicable): osa-dispatcher-5.10.41-1.el6.noarch (Spacewalk 1.7)


How reproducible: Everytime


Steps to Reproduce: /etc/init.d/osa-dispatcher restart (or start)
1. /usr/sbin/spacewalk-service restart (or start)
2.
3.
  
Actual results: The init script shows that the osa-dispatcher service starts however a "/etc/init.d/osa-dispatcher status" shows that it is "dead". A simple "ps" shows that indeed the service is no longer running


Expected results: It should start without issue


Additional info: I tailed the "/var/log/rhn/osa-dispatcher.log" file and noticed the following entries after running the "/etc/init.d/osa-dispatcher start" command:

2012/05/01 20:19:58 -05:00 31439 0.0.0.0: osad/jabber_lib.__init__
2012/05/01 20:19:58 -05:00 31439 0.0.0.0: osad/jabber_lib.setup_connection('Connected to jabber server', 'srv03.dburkland.com')
2012/05/01 20:19:58 -05:00 31439 0.0.0.0: osad/jabber_lib.register('ERROR', 'Invalid password')

If I clear out the "/var/lib/jabberd/db/" directory and then start the osa-dispatcher service it works fine.

Comment 1 Jan Pazdziora (Red Hat) 2012-05-03 11:45:30 UTC
(In reply to comment #0)
> 
> If I clear out the "/var/lib/jabberd/db/" directory and then start the
> osa-dispatcher service it works fine.

So by clearing out /var/lib/jabberd/db, were you able to resolve the issue on your installation for good, or does the problem reoccur?

How many clients using osad do you have? Is the auth database in good shape (not corrupted, etc.)?

Comment 2 Dan Burkland 2012-05-04 05:39:57 UTC
Before each restart I must clear /var/lib/jabberd/db otherwise the osa-dispatcher service will fail to start (it reoccurs).

I have about 6 clients via OSAD (just a small home network). As far as I know the auth database is in good shape and not corrupted but wouldn't know how to find out for sure.

Thanks for the reply,

Dan

Comment 3 Milan Zázrivec 2012-05-04 07:10:46 UTC
OK, please connect to your database via sqlplus (if Oracle) or psql
(if PostgreSQL) and see what the following query returns:

    select id, jabber_id, password, hostname from rhnPushDispatcher;

Then, please paste the output from

    tail -f /var/log/messages

while doing

    service jabberd restart

Comment 4 Dan Burkland 2012-05-05 07:59:23 UTC
The posted query returns the following results: http://pastebin.com/kxCcdRZy

"/var/log/messages" output from when I ran the "service jabberd restart" command: http://pastebin.com/Yzz6iqgw

It looks like I have an incorrect hostname somehow in my Spacewalk database, any idea on how to delete that record?

Thanks,

Dan

Comment 5 Milan Zázrivec 2012-05-06 18:18:49 UTC
Your /var/log/messages look all right, how you got to have two entries
in rhnPushDispatcher -- I don't understand.

Nonetheless, try to connect to your database with psql again and delete
the incorrect fqdn with:

    delete from rhnPushDispatcher where id = 1;

restart your Spacewalk instance with:

    spacewalk-service restart

and see whether this helps your situation.

Comment 6 Dan Burkland 2012-05-06 23:55:56 UTC
I ran the suggested commands and while the osa-dispatcher service stayed running, several error log entries would appear each time I restarted the osad service on my Spacewalk clients. I resolved this by:

1) Again clearing out /var/lib/jabberd/db
2) Restarting the Spacewalk services on the server several times to make sure it wasn't a fluke
3) Restarting the osad service on my Spacewalk clients several times

After performing the aforementioned steps no error messages appear in the logs and everything seems to work just fine. 

One thing I forgot to mention was that the hostname of this system recently changed and I'm guessing the removal of the previously mentioned database record was skipped over by the spacewalk-hostname-rename tool.

Thanks again for your help!

Dan

Comment 7 Milan Zázrivec 2012-05-08 12:18:13 UTC
Great, since the problem described here seems to be resolved, I'm closing
this report.