Bug 1419199 - osa-dispatcher fails to start/run following fresh install from nightly
Summary: osa-dispatcher fails to start/run following fresh install from nightly
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Spacewalk
Classification: Community
Component: Clients
Version: 2.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Eric Herget
QA Contact: Red Hat Satellite QA List
URL:
Whiteboard:
Depends On:
Blocks: spacewalk-review space27
TreeView+ depends on / blocked
 
Reported: 2017-02-03 21:16 UTC by Eric Herget
Modified: 2017-09-27 19:31 UTC (History)
1 user (show)

Fixed In Version: osad-5.11.78-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-09-27 19:31:32 UTC


Attachments (Terms of Use)

Description Eric Herget 2017-02-03 21:16:34 UTC
Description of problem:
osa-dispatcher is failing to start/run.  This following a fresh install of spacewalk from nightly.  Note that I'm seeing two different issues happen.  When osa-dispatcher is attempted to start just after jabberd, there is a "Invalid Password" error.  This looks like the issue reported in BZ1071713.  After jabberd has time to start, the error changes to Permission denied /var/run/osa-dispatcher.pid.  I'm opening this ticket for the latter issue.

Version-Release number of selected component (if applicable):
osa-dispatcher-5.11.77-1.el6.noarch
jabberd-2.4.0-4.el6.x86_64

How reproducible:
Always.

Steps to Reproduce:
1. Do fresh install of spacewalk on newly provisioned and updated RHEL 6 system
2. service osa-dispatcher start
3. cat /var/log/rhn/osa-dispatcher.log

Actual results:

osa-dispatcher.log contains...
2017/02/03 14:31:33 -04:00 4852 0.0.0.0: osad/jabber_lib.__init__
2017/02/03 14:31:34 -04:00 4852 0.0.0.0: osad/jabber_lib.setup_connection('Connected to jabber server', '<host-retracted>')
2017/02/03 14:31:34 -04:00 4852 0.0.0.0: osad/jabber_lib.main('ERROR', 'Error caught:')
2017/02/03 14:31:34 -04:00 4852 0.0.0.0: osad/jabber_lib.main('ERROR', 'Traceback (most recent call last):\n  File "/usr/share/rhn/osad/jabber_lib.py", line 126, in main\n    c = self.setup_connection(no_fork=no_fork)\n  File "/usr/share/rhn/osad/jabber_lib.py", line 300, in setup_connection\n    self.push_to_background()\n  File "/usr/share/rhn/osad/jabber_lib.py", line 223, in push_to_background\n    os.unlink(pid_file)\nOSError: [Errno 13] Permission denied: \'/var/run/osa-dispatcher.pid\'\n')


Expected results:

osa-dispatcher starts and connects to jabberd

Additional info:

If osa-dispatcher is started just after jabberd, and jabberd hasn't finished starting up, the osa-dispatcher.log shows...
2017/02/03 14:17:58 -04:00 4348 0.0.0.0: osad/jabber_lib.__init__
2017/02/03 14:17:58 -04:00 4348 0.0.0.0: osad/jabber_lib.setup_connection('Connected to jabber server', '<host-retracted>')
2017/02/03 14:17:58 -04:00 4348 0.0.0.0: osad/jabber_lib.register('ERROR', 'Invalid password')

If osa-dispatcher is started after jabberd is up and running, osa-dispatcher.log shows...
2017/02/03 14:31:33 -04:00 4852 0.0.0.0: osad/jabber_lib.__init__
2017/02/03 14:31:34 -04:00 4852 0.0.0.0: osad/jabber_lib.setup_connection('Connected to jabber server', '<host-retracted>')
2017/02/03 14:31:34 -04:00 4852 0.0.0.0: osad/jabber_lib.main('ERROR', 'Error caught:')
2017/02/03 14:31:34 -04:00 4852 0.0.0.0: osad/jabber_lib.main('ERROR', 'Traceback (most recent call last):\n  File "/usr/share/rhn/osad/jabber_lib.py", line 126, in main\n    c = self.setup_connection(no_fork=no_fork)\n  File "/usr/share/rhn/osad/jabber_lib.py", line 300, in setup_connection\n    self.push_to_background()\n  File "/usr/share/rhn/osad/jabber_lib.py", line 223, in push_to_background\n    os.unlink(pid_file)\nOSError: [Errno 13] Permission denied: \'/var/run/osa-dispatcher.pid\'\n')

Comment 1 Eric Herget 2017-02-07 19:04:30 UTC
Much of the above details were actually red herrings hiding the underlying issue.  The underlying issue was a failure to register the rhn-dispatcher-sat user to jabberd.

As a result, osa-dispatcher could never start because the user had never gotten registered.  This issue can be confirmed by examining the rhnPushDispatcher database table and finding it empty.

Comment 2 Eric Herget 2017-02-07 19:17:41 UTC
spacewalk.github:
4fad52394b6de3b3449a0938790d2a08fa25f1cb

Comment 3 Ales Dujicek 2017-08-02 11:08:28 UTC
On fresh nightly installation osa-dispatcher still does not start correctly:

# service osa-dispatcher status
osa-dispatcher (pid  30526) is running...

but /var/log/rhn/osa-dispatcher.log is full of:
2017/08/02 13:02:02 +02:00 30524 0.0.0.0: osad/jabber_lib.__init__

2017/08/02 13:02:02 +02:00 30524 0.0.0.0: osad/jabber_lib.setup_connection('Connected to jabber server', 'spacewalk.url.com')
2017/08/02 13:02:02 +02:00 30524 0.0.0.0: osad/jabber_lib.main('ERROR', 'Error caught:')
2017/08/02 13:02:02 +02:00 30524 0.0.0.0: osad/jabber_lib.main('ERROR', 'Traceback (most recent call last):\n  File "/usr/share/rhn/osad/jabber_lib.py", line 126, in main\n    c = self.setup_connection(no_fork=no_fork)\n  File "/usr/share/rhn/osad/jabber_lib.py", line 304, in setup_connection\n    resource=self._resource)\n  File "/usr/share/rhn/osad/dispatcher_client.py", line 36, in start\n    self.auth(username, password, resource)\n  File "/usr/share/rhn/osad/jabber_lib.py", line 897, in auth\n    self.register(username, password)\n  File "/usr/share/rhn/osad/jabber_lib.py", line 1143, in register\n    self.sendRegInfo()\n  File "/usr/lib/python2.6/site-packages/jabber/jabber.py", line 644, in sendRegInfo\n    return self.SendAndWaitForResponse(reg_iq)\n  File "/usr/lib/python2.6/site-packages/jabber/jabber.py", line 401, in SendAndWaitForResponse\n    return self.waitForResponse(ID,timeout)\n  File "/usr/share/rhn/osad/jabber_lib.py", line 1213, in waitForResponse\n    raise JabberQualifiedError(self.lastErrCode, self.lastErr)\nJabberQualifiedError: <JabberQualifiedError instance at 25800528; errcode=500; err=>\n')

and osad on clients never pickups events..

fresh installation from nightly
osa-common-5.11.85-1.el6.noarch
osa-dispatcher-5.11.85-1.el6.noarch
osa-dispatcher-selinux-5.11.85-1.el6.noarch
jabberd-2.6.1-2.el6.x86_64

Comment 4 Eric Herget 2017-08-11 17:16:41 UTC
I reproduced this issue and found that the sqlite database had no tables in it.  /var/log/messages had many errors whick lead me to look at the db:
Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: [23] [::ffff:10.13.129.36, port=39054] connect
Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg
Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg
Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg

googling for this error, I found a suggestion to run the db-setup.sqlite sql provided with jabberd package, so I did:

# sqlite3 /var/lib/jabberd/db/sqlite.db
sqlite> .tables
sqlite> .read /usr/share/jabberd/db-setup.sqlite
sqlite> .tables
active                   privacy-items            roster-items           
authreg                  private                  status                 
logout                   published-roster         vacation-settings      
motd-message             published-roster-groups  vcard                  
motd-times               queue                    verify                 
privacy-default          roster-groups          
sqlite> <ctrl-d>
# spacewalk-service restart

and all works as expected

Now I'm on to figure out if we used to run the db-setup.sqlite sql in our install scripts, if jabberd does it, etc, in order to figure out how to fix this.

Comment 5 Eric Herget 2017-08-11 17:47:38 UTC
Since the issue reported in comment 3 is a new issue, I've opened a new BZ 1480697.  I'm setting this back to MODIFIED

Comment 6 Eric Herget 2017-09-27 19:31:32 UTC
Spacewalk 2.7 has been released.

https://github.com/spacewalkproject/spacewalk/wiki/ReleaseNotes27


Note You need to log in before you can comment on or make changes to this bug.