Description of problem: osa-dispatcher is failing to start/run. This following a fresh install of spacewalk from nightly. Note that I'm seeing two different issues happen. When osa-dispatcher is attempted to start just after jabberd, there is a "Invalid Password" error. This looks like the issue reported in BZ1071713. After jabberd has time to start, the error changes to Permission denied /var/run/osa-dispatcher.pid. I'm opening this ticket for the latter issue. Version-Release number of selected component (if applicable): osa-dispatcher-5.11.77-1.el6.noarch jabberd-2.4.0-4.el6.x86_64 How reproducible: Always. Steps to Reproduce: 1. Do fresh install of spacewalk on newly provisioned and updated RHEL 6 system 2. service osa-dispatcher start 3. cat /var/log/rhn/osa-dispatcher.log Actual results: osa-dispatcher.log contains... 2017/02/03 14:31:33 -04:00 4852 0.0.0.0: osad/jabber_lib.__init__ 2017/02/03 14:31:34 -04:00 4852 0.0.0.0: osad/jabber_lib.setup_connection('Connected to jabber server', '<host-retracted>') 2017/02/03 14:31:34 -04:00 4852 0.0.0.0: osad/jabber_lib.main('ERROR', 'Error caught:') 2017/02/03 14:31:34 -04:00 4852 0.0.0.0: osad/jabber_lib.main('ERROR', 'Traceback (most recent call last):\n File "/usr/share/rhn/osad/jabber_lib.py", line 126, in main\n c = self.setup_connection(no_fork=no_fork)\n File "/usr/share/rhn/osad/jabber_lib.py", line 300, in setup_connection\n self.push_to_background()\n File "/usr/share/rhn/osad/jabber_lib.py", line 223, in push_to_background\n os.unlink(pid_file)\nOSError: [Errno 13] Permission denied: \'/var/run/osa-dispatcher.pid\'\n') Expected results: osa-dispatcher starts and connects to jabberd Additional info: If osa-dispatcher is started just after jabberd, and jabberd hasn't finished starting up, the osa-dispatcher.log shows... 2017/02/03 14:17:58 -04:00 4348 0.0.0.0: osad/jabber_lib.__init__ 2017/02/03 14:17:58 -04:00 4348 0.0.0.0: osad/jabber_lib.setup_connection('Connected to jabber server', '<host-retracted>') 2017/02/03 14:17:58 -04:00 4348 0.0.0.0: osad/jabber_lib.register('ERROR', 'Invalid password') If osa-dispatcher is started after jabberd is up and running, osa-dispatcher.log shows... 2017/02/03 14:31:33 -04:00 4852 0.0.0.0: osad/jabber_lib.__init__ 2017/02/03 14:31:34 -04:00 4852 0.0.0.0: osad/jabber_lib.setup_connection('Connected to jabber server', '<host-retracted>') 2017/02/03 14:31:34 -04:00 4852 0.0.0.0: osad/jabber_lib.main('ERROR', 'Error caught:') 2017/02/03 14:31:34 -04:00 4852 0.0.0.0: osad/jabber_lib.main('ERROR', 'Traceback (most recent call last):\n File "/usr/share/rhn/osad/jabber_lib.py", line 126, in main\n c = self.setup_connection(no_fork=no_fork)\n File "/usr/share/rhn/osad/jabber_lib.py", line 300, in setup_connection\n self.push_to_background()\n File "/usr/share/rhn/osad/jabber_lib.py", line 223, in push_to_background\n os.unlink(pid_file)\nOSError: [Errno 13] Permission denied: \'/var/run/osa-dispatcher.pid\'\n')
Much of the above details were actually red herrings hiding the underlying issue. The underlying issue was a failure to register the rhn-dispatcher-sat user to jabberd. As a result, osa-dispatcher could never start because the user had never gotten registered. This issue can be confirmed by examining the rhnPushDispatcher database table and finding it empty.
spacewalk.github: 4fad52394b6de3b3449a0938790d2a08fa25f1cb
On fresh nightly installation osa-dispatcher still does not start correctly: # service osa-dispatcher status osa-dispatcher (pid 30526) is running... but /var/log/rhn/osa-dispatcher.log is full of: 2017/08/02 13:02:02 +02:00 30524 0.0.0.0: osad/jabber_lib.__init__ 2017/08/02 13:02:02 +02:00 30524 0.0.0.0: osad/jabber_lib.setup_connection('Connected to jabber server', 'spacewalk.url.com') 2017/08/02 13:02:02 +02:00 30524 0.0.0.0: osad/jabber_lib.main('ERROR', 'Error caught:') 2017/08/02 13:02:02 +02:00 30524 0.0.0.0: osad/jabber_lib.main('ERROR', 'Traceback (most recent call last):\n File "/usr/share/rhn/osad/jabber_lib.py", line 126, in main\n c = self.setup_connection(no_fork=no_fork)\n File "/usr/share/rhn/osad/jabber_lib.py", line 304, in setup_connection\n resource=self._resource)\n File "/usr/share/rhn/osad/dispatcher_client.py", line 36, in start\n self.auth(username, password, resource)\n File "/usr/share/rhn/osad/jabber_lib.py", line 897, in auth\n self.register(username, password)\n File "/usr/share/rhn/osad/jabber_lib.py", line 1143, in register\n self.sendRegInfo()\n File "/usr/lib/python2.6/site-packages/jabber/jabber.py", line 644, in sendRegInfo\n return self.SendAndWaitForResponse(reg_iq)\n File "/usr/lib/python2.6/site-packages/jabber/jabber.py", line 401, in SendAndWaitForResponse\n return self.waitForResponse(ID,timeout)\n File "/usr/share/rhn/osad/jabber_lib.py", line 1213, in waitForResponse\n raise JabberQualifiedError(self.lastErrCode, self.lastErr)\nJabberQualifiedError: <JabberQualifiedError instance at 25800528; errcode=500; err=>\n') and osad on clients never pickups events.. fresh installation from nightly osa-common-5.11.85-1.el6.noarch osa-dispatcher-5.11.85-1.el6.noarch osa-dispatcher-selinux-5.11.85-1.el6.noarch jabberd-2.6.1-2.el6.x86_64
I reproduced this issue and found that the sqlite database had no tables in it. /var/log/messages had many errors whick lead me to look at the db: Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: [23] [::ffff:10.13.129.36, port=39054] connect Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg googling for this error, I found a suggestion to run the db-setup.sqlite sql provided with jabberd package, so I did: # sqlite3 /var/lib/jabberd/db/sqlite.db sqlite> .tables sqlite> .read /usr/share/jabberd/db-setup.sqlite sqlite> .tables active privacy-items roster-items authreg private status logout published-roster vacation-settings motd-message published-roster-groups vcard motd-times queue verify privacy-default roster-groups sqlite> <ctrl-d> # spacewalk-service restart and all works as expected Now I'm on to figure out if we used to run the db-setup.sqlite sql in our install scripts, if jabberd does it, etc, in order to figure out how to fix this.
Since the issue reported in comment 3 is a new issue, I've opened a new BZ 1480697. I'm setting this back to MODIFIED
Spacewalk 2.7 has been released. https://github.com/spacewalkproject/spacewalk/wiki/ReleaseNotes27