Bug 1480697 - osa-dispatcher fails to start/run following fresh install from nightly - episode 2
osa-dispatcher fails to start/run following fresh install from nightly - epis...
Status: CLOSED CURRENTRELEASE
Product: Spacewalk
Classification: Community
Component: Clients (Show other bugs)
2.6
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Eric Herget
Red Hat Satellite QA List
:
Depends On:
Blocks: spacewalk-review space27
  Show dependency treegraph
 
Reported: 2017-08-11 13:44 EDT by Eric Herget
Modified: 2017-09-27 15:37 EDT (History)
1 user (show)

See Also:
Fixed In Version: spacewalk-setup-jabberd-2.7.3-1
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-09-27 15:37:57 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Eric Herget 2017-08-11 13:44:50 EDT
Description of problem:
osa-dispatcher is failing to start/run.  This following a fresh install of spacewalk from nightly.  The failure is caused by jabberd's sqlite db containing no tables

Version-Release number of selected component (if applicable):
osa-common-5.11.85-1.el6.noarch
osa-dispatcher-5.11.85-1.el6.noarch
osa-dispatcher-selinux-5.11.85-1.el6.noarch
jabberd-2.6.1-2.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Do fresh install of spacewalk on newly provisioned and updated RHEL 6 system
2. service osa-dispatcher status
3. cat /var/log/rhn/osa-dispatcher.log

Actual results:
osa-dispatcher.log is full of...
2017/08/11 13:05:11 -04:00 11723 0.0.0.0: osad/jabber_lib.__init__
2017/08/11 13:05:11 -04:00 11723 0.0.0.0: osad/jabber_lib.setup_connection('Connected to jabber server', 'eherget-sw-rhel6.usersys.redhat.com')
2017/08/11 13:05:11 -04:00 11723 0.0.0.0: osad/jabber_lib.main('ERROR', 'Error caught:')
2017/08/11 13:05:11 -04:00 11723 0.0.0.0: osad/jabber_lib.main('ERROR', 'Traceback (most recent call last):\n  File "/usr/share/rhn/osad/jabber_lib.py", line 126, in main\n    c = self.setup_connection(no_fork=no_fork)\n  File "/usr/share/rhn/osad/jabber_lib.py", line 304, in setup_connection\n    resource=self._resource)\n  File "/usr/share/rhn/osad/dispatcher_client.py", line 36, in start\n    self.auth(username, password, resource)\n  File "/usr/share/rhn/osad/jabber_lib.py", line 897, in auth\n    self.register(username, password)\n  File "/usr/share/rhn/osad/jabber_lib.py", line 1143, in register\n    self.sendRegInfo()\n  File "/usr/lib/python2.6/site-packages/jabber/jabber.py", line 644, in sendRegInfo\n    return self.SendAndWaitForResponse(reg_iq)\n  File "/usr/lib/python2.6/site-packages/jabber/jabber.py", line 401, in SendAndWaitForResponse\n    return self.waitForResponse(ID,timeout)\n  File "/usr/share/rhn/osad/jabber_lib.py", line 1213, in waitForResponse\n    raise JabberQualifiedError(self.lastErrCode, self.lastErr)\nJabberQualifiedError: <JabberQualifiedError instance at 29318192; errcode=500; err=>\n')


and /var/log/messages is full of...
Aug 11 13:05:41 eherget-sw-rhel6 jabberd/c2s[11498]: [32] [::ffff:10.13.129.36, port=39078] connect
Aug 11 13:05:41 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg
Aug 11 13:05:42 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg
Aug 11 13:05:42 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg


Expected results:
osa-dispatcher starts and connects to jabberd

Additional info:
Comment 1 Eric Herget 2017-08-11 14:03:46 EDT
I reproduced this issue and found that the sqlite database had no tables in it.  /var/log/messages had many errors whick lead me to look at the db:
Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: [23] [::ffff:10.13.129.36, port=39054] connect
Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg
Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg
Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg

googling for this error, I found a suggestion to run the db-setup.sqlite sql provided with jabberd package, so I did:

# sqlite3 /var/lib/jabberd/db/sqlite.db
sqlite> .tables
sqlite> .read /usr/share/jabberd/db-setup.sqlite
sqlite> .tables
active                   privacy-items            roster-items           
authreg                  private                  status                 
logout                   published-roster         vacation-settings      
motd-message             published-roster-groups  vcard                  
motd-times               queue                    verify                 
privacy-default          roster-groups          
sqlite> <ctrl-d>
# spacewalk-service restart

and all works as expected

Now I'm on to figure out if we used to run the db-setup.sqlite sql in our install scripts, if jabberd does it, etc, in order to figure out how to fix this.
Comment 2 Eric Herget 2017-08-11 14:05:48 EDT
In trying to figure out what is supposed to populate the sqlite database, I did the following:

spacewalk-service stop
rm -rf /var/lib/jabberd/db/*
spacewalk-service start

Since the rm command wipes the database, this shows the db gets populated successfully by something.  So now the question is, why does the database contain no tables following a fresh install?
Comment 3 Eric Herget 2017-08-14 15:26:16 EDT
Digging into this issue further, I am unable to duplicate what I reported in comment 2 above.  Even doing the above, the issue re-occurs.
Comment 4 Eric Herget 2017-08-14 15:31:06 EDT
I have found the issue.  In May, 2017, we changed spacewalk-setup-jabberd to switch from Berkeley DB to sqlite.  The problem is that there is a step required during installation to initialize the sqlite database.  These steps must be done to initialize the database:

# sqlite3 /var/lib/jabberd/db/sqlite.db < /usr/share/jabberd/db-setup.sqlite
# chown jabber:jabber /var/lib/jabberd/db/sqlite.db
Comment 5 Eric Herget 2017-08-14 15:55:33 EDT
I have updated the spacewalk-setup-jabberd script to initialize the database if it is not already initialized.

spacewalk.github:
926fd3caf4386d579348baf4a3df8f52a00cdf38
Comment 6 Eric Herget 2017-09-27 15:37:57 EDT
Spacewalk 2.7 has been released.

https://github.com/spacewalkproject/spacewalk/wiki/ReleaseNotes27

Note You need to log in before you can comment on or make changes to this bug.