Bug 1480697
| Summary: | osa-dispatcher fails to start/run following fresh install from nightly - episode 2 | ||
|---|---|---|---|
| Product: | [Community] Spacewalk | Reporter: | Eric Herget <eherget> |
| Component: | Clients | Assignee: | Eric Herget <eherget> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Red Hat Satellite QA List <satqe-list> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 2.6 | CC: | adujicek, ethan.bonick |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | spacewalk-setup-jabberd-2.7.3-1 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-09-27 19:37:57 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1310034, 1484117 | ||
I reproduced this issue and found that the sqlite database had no tables in it. /var/log/messages had many errors whick lead me to look at the db: Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: [23] [::ffff:10.13.129.36, port=39054] connect Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg Aug 11 13:04:51 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg googling for this error, I found a suggestion to run the db-setup.sqlite sql provided with jabberd package, so I did: # sqlite3 /var/lib/jabberd/db/sqlite.db sqlite> .tables sqlite> .read /usr/share/jabberd/db-setup.sqlite sqlite> .tables active privacy-items roster-items authreg private status logout published-roster vacation-settings motd-message published-roster-groups vcard motd-times queue verify privacy-default roster-groups sqlite> <ctrl-d> # spacewalk-service restart and all works as expected Now I'm on to figure out if we used to run the db-setup.sqlite sql in our install scripts, if jabberd does it, etc, in order to figure out how to fix this. In trying to figure out what is supposed to populate the sqlite database, I did the following: spacewalk-service stop rm -rf /var/lib/jabberd/db/* spacewalk-service start Since the rm command wipes the database, this shows the db gets populated successfully by something. So now the question is, why does the database contain no tables following a fresh install? Digging into this issue further, I am unable to duplicate what I reported in comment 2 above. Even doing the above, the issue re-occurs. I have found the issue. In May, 2017, we changed spacewalk-setup-jabberd to switch from Berkeley DB to sqlite. The problem is that there is a step required during installation to initialize the sqlite database. These steps must be done to initialize the database: # sqlite3 /var/lib/jabberd/db/sqlite.db < /usr/share/jabberd/db-setup.sqlite # chown jabber:jabber /var/lib/jabberd/db/sqlite.db I have updated the spacewalk-setup-jabberd script to initialize the database if it is not already initialized. spacewalk.github: 926fd3caf4386d579348baf4a3df8f52a00cdf38 Spacewalk 2.7 has been released. https://github.com/spacewalkproject/spacewalk/wiki/ReleaseNotes27 I ran into this same issue on Spacewalk 2.8. I can duplicate the issue by removing the /var/lib/jabberd/db/* files and then restarting jabberd and osa-dispatcher. Once I run the commands in comment #4 everything works again. This instance has been upgraded from 2.6 to 2.7 to 2.8. The osa-dispatcher has been an issue on and off for a while and until recently the removing the sqlite db files worked. Below is my upgrade path from the yum logs. 2018 Jul 17 09:40:37 Installed: python2-spacewalk-usix-2.8.3-1.el7.centos.noarch Jul 17 09:40:41 Updated: spacewalk-base-minimal-2.8.7-1.el7.centos.noarch Jul 17 09:40:41 Updated: spacewalk-backend-libs-2.8.60-1.el7.noarch Jul 17 09:40:42 Updated: spacewalk-java-postgresql-2.8.79-1.el7.noarch Jul 17 09:40:42 Updated: spacewalk-backend-sql-postgresql-2.8.60-1.el7.noarch Jul 17 09:40:43 Updated: spacewalk-config-2.8.5-1.el7.centos.noarch Jul 17 09:40:43 Updated: spacewalk-backend-2.8.60-1.el7.noarch Jul 17 09:40:43 Updated: spacewalk-backend-sql-2.8.60-1.el7.noarch Jul 17 09:40:44 Updated: spacewalk-backend-server-2.8.60-1.el7.noarch Jul 17 09:40:45 Updated: spacewalk-backend-xml-export-libs-2.8.60-1.el7.noarch Jul 17 09:40:45 Updated: spacewalk-backend-config-files-common-2.8.60-1.el7.noarch Jul 17 09:40:45 Updated: spacewalk-backend-app-2.8.60-1.el7.noarch Jul 17 09:40:46 Updated: spacewalk-backend-xmlrpc-2.8.60-1.el7.noarch Jul 17 09:40:47 Updated: spacewalk-java-lib-2.8.79-1.el7.noarch Jul 17 09:40:48 Updated: spacewalk-java-config-2.8.79-1.el7.noarch Jul 17 09:40:48 Updated: spacewalk-base-2.8.7-1.el7.centos.noarch Jul 17 09:40:48 Updated: spacewalk-admin-2.8.4-1.el7.centos.noarch Jul 17 09:40:48 Updated: spacewalk-backend-config-files-2.8.60-1.el7.noarch Jul 17 09:40:48 Updated: spacewalk-backend-config-files-tool-2.8.60-1.el7.noarch Jul 17 09:40:48 Updated: spacewalk-backend-iss-export-2.8.60-1.el7.noarch Jul 17 09:40:49 Updated: spacewalk-backend-iss-2.8.60-1.el7.noarch Jul 17 09:40:49 Updated: spacewalk-backend-applet-2.8.60-1.el7.noarch Jul 17 09:40:49 Updated: spacewalk-backend-package-push-server-2.8.60-1.el7.noarch Jul 17 09:40:50 Updated: spacewalk-certs-tools-2.8.8-1.el7.centos.noarch Jul 17 09:40:50 Installed: python2-spacewalk-certs-tools-2.8.8-1.el7.centos.noarch Jul 17 09:40:50 Updated: spacewalk-dobby-2.8.7-1.el7.centos.noarch Jul 17 09:40:51 Updated: spacewalk-base-minimal-config-2.8.7-1.el7.centos.noarch Jul 17 09:40:51 Updated: spacewalk-usix-2.8.3-1.el7.centos.noarch Jul 17 09:40:56 Updated: spacewalk-doc-indexes-2.8.4-1.el7.centos.noarch Jul 17 09:40:56 Updated: spacewalk-search-2.8.3-1.el7.centos.noarch Jul 17 09:41:01 Updated: spacewalk-taskomatic-2.8.79-1.el7.noarch Jul 17 09:41:03 Updated: spacewalk-backend-tools-2.8.60-1.el7.noarch Jul 17 09:41:08 Updated: spacewalk-branding-2.8.5-1.el7.centos.noarch Jul 17 09:41:13 Updated: spacewalk-java-2.8.79-1.el7.noarch Jul 17 09:41:13 Updated: spacewalk-html-2.8.7-1.el7.centos.noarch Jul 17 09:41:13 Updated: spacewalk-setup-jabberd-2.8.5-1.el7.centos.noarch Jul 17 09:41:19 Updated: spacewalk-schema-2.8.25-1.el7.noarch Jul 17 09:41:19 Updated: spacewalk-setup-2.8.7-1.el7.centos.noarch Jul 17 09:42:13 Updated: spacewalk-selinux-2.8.6-1.el7.centos.noarch Jul 17 09:42:43 Updated: spacewalk-common-2.8.2-1.el7.centos.noarch Jul 17 09:42:43 Updated: spacewalk-postgresql-2.8.2-1.el7.centos.noarch Jul 17 09:42:43 Updated: spacewalk-setup-postgresql-2.8.4-1.el7.centos.noarch 2017 Feb 16 15:15:10 Installed: spacewalk-backend-usix-2.6.75-1.el7.noarch Feb 16 15:15:22 Installed: spacewalk-backend-libs-2.6.75-1.el7.noarch Feb 16 15:15:28 Installed: spacewalk-java-lib-2.6.49-1.el7.noarch Feb 16 15:15:28 Installed: spacewalk-java-config-2.6.49-1.el7.noarch Feb 16 15:15:53 Installed: spacewalk-backend-sql-postgresql-2.6.75-1.el7.noarch Feb 16 15:16:08 Installed: spacewalk-config-2.6.5-1.el7.noarch Feb 16 15:16:08 Installed: spacewalk-backend-2.6.75-1.el7.noarch Feb 16 15:16:08 Installed: spacewalk-backend-sql-2.6.75-1.el7.noarch Feb 16 15:16:08 Installed: spacewalk-backend-server-2.6.75-1.el7.noarch Feb 16 15:16:09 Installed: spacewalk-backend-config-files-common-2.6.75-1.el7.noarch Feb 16 15:16:09 Installed: spacewalk-backend-xml-export-libs-2.6.75-1.el7.noarch Feb 16 15:16:09 Installed: spacewalk-backend-xmlrpc-2.6.75-1.el7.noarch Feb 16 15:16:09 Installed: spacewalk-backend-app-2.6.75-1.el7.noarch Feb 16 15:16:13 Installed: spacewalk-backend-iss-export-2.6.75-1.el7.noarch Feb 16 15:16:13 Installed: spacewalk-backend-config-files-tool-2.6.75-1.el7.noarch Feb 16 15:16:13 Installed: spacewalk-backend-config-files-2.6.75-1.el7.noarch Feb 16 15:16:13 Installed: spacewalk-backend-package-push-server-2.6.75-1.el7.noarch Feb 16 15:16:14 Installed: spacewalk-backend-iss-2.6.75-1.el7.noarch Feb 16 15:16:14 Installed: spacewalk-backend-applet-2.6.75-1.el7.noarch Feb 16 15:16:20 Installed: spacewalk-base-2.6.6-1.el7.noarch Feb 16 15:16:20 Installed: spacewalk-admin-2.6.1-1.el7.noarch Feb 16 15:16:21 Installed: spacewalk-base-minimal-2.6.6-1.el7.noarch Feb 16 15:16:21 Installed: spacewalk-base-minimal-config-2.6.6-1.el7.noarch Feb 16 15:16:42 Installed: spacewalk-branding-2.5.3-1.el7.noarch Feb 16 15:16:42 Installed: spacewalk-setup-jabberd-2.3.2-1.el7.noarch Feb 16 15:16:43 Installed: spacewalk-html-2.6.6-1.el7.noarch Feb 16 15:17:12 Installed: spacewalk-doc-indexes-2.6.2-1.el7.noarch Feb 16 15:17:16 Installed: spacewalk-jpp-workaround-2.3.5-1.el7.noarch Feb 16 15:17:19 Installed: spacewalk-java-postgresql-2.6.49-1.el7.noarch Feb 16 15:17:19 Installed: spacewalk-taskomatic-2.6.49-1.el7.noarch Feb 16 15:17:22 Installed: spacewalk-java-2.6.49-1.el7.noarch Feb 16 15:17:23 Installed: spacewalk-search-2.6.1-1.el7.noarch Feb 16 15:17:25 Installed: spacewalk-certs-tools-2.5.3-1.el7.noarch Feb 16 15:17:26 Installed: spacewalk-backend-tools-2.6.75-1.el7.noarch Feb 16 15:17:28 Installed: spacewalk-schema-2.6.16-1.el7.noarch Feb 16 15:17:28 Installed: spacewalk-setup-2.6.2-1.el7.noarch Feb 16 15:17:28 Installed: spacewalk-selinux-2.3.2-1.el7.noarch Feb 16 15:17:30 Installed: spacewalk-common-2.6.1-1.el7.noarch Feb 16 15:17:31 Installed: spacewalk-postgresql-2.6.1-1.el7.noarch Feb 16 15:21:15 Installed: spacewalk-setup-postgresql-2.6.2-1.el7.noarch Apr 26 14:13:07 Updated: spacewalk-backend-usix-2.6.77-1.el7.noarch Apr 26 14:13:12 Updated: spacewalk-backend-sql-postgresql-2.6.77-1.el7.noarch Apr 26 14:13:12 Updated: spacewalk-backend-libs-2.6.77-1.el7.noarch Apr 26 14:16:01 Updated: spacewalk-backend-2.6.77-1.el7.noarch Apr 26 14:16:02 Updated: spacewalk-backend-sql-2.6.77-1.el7.noarch Apr 26 14:16:02 Updated: spacewalk-backend-server-2.6.77-1.el7.noarch Apr 26 14:16:03 Updated: spacewalk-backend-config-files-common-2.6.77-1.el7.noarch Apr 26 14:16:03 Updated: spacewalk-backend-xml-export-libs-2.6.77-1.el7.noarch Apr 26 14:16:03 Updated: spacewalk-backend-app-2.6.77-1.el7.noarch Apr 26 14:16:04 Updated: spacewalk-backend-xmlrpc-2.6.77-1.el7.noarch Apr 26 14:16:12 Updated: spacewalk-backend-tools-2.6.77-1.el7.noarch Apr 26 14:16:12 Updated: spacewalk-backend-iss-export-2.6.77-1.el7.noarch Apr 26 14:16:12 Updated: spacewalk-backend-config-files-tool-2.6.77-1.el7.noarch Apr 26 14:16:12 Updated: spacewalk-backend-config-files-2.6.77-1.el7.noarch Apr 26 14:16:13 Updated: spacewalk-backend-iss-2.6.77-1.el7.noarch Apr 26 14:16:13 Updated: spacewalk-backend-package-push-server-2.6.77-1.el7.noarch Apr 26 14:16:13 Updated: spacewalk-backend-applet-2.6.77-1.el7.noarch Apr 26 14:17:13 Updated: spacewalk-schema-2.6.17-1.el7.noarch Aug 28 10:05:30 Updated: spacewalk-backend-usix-2.6.78-1.el7.noarch Aug 28 10:06:20 Updated: spacewalk-backend-libs-2.6.78-1.el7.noarch Aug 28 10:06:20 Updated: spacewalk-backend-2.6.78-1.el7.noarch Aug 28 10:06:20 Updated: spacewalk-backend-sql-postgresql-2.6.78-1.el7.noarch Aug 28 10:06:20 Updated: spacewalk-backend-sql-2.6.78-1.el7.noarch Aug 28 10:06:21 Updated: spacewalk-backend-server-2.6.78-1.el7.noarch Aug 28 10:06:22 Updated: spacewalk-backend-xml-export-libs-2.6.78-1.el7.noarch Aug 28 10:06:22 Updated: spacewalk-backend-config-files-common-2.6.78-1.el7.noarch Aug 28 10:06:22 Updated: spacewalk-backend-xmlrpc-2.6.78-1.el7.noarch Aug 28 10:06:22 Updated: spacewalk-backend-app-2.6.78-1.el7.noarch Aug 28 10:06:26 Updated: spacewalk-backend-tools-2.6.78-1.el7.noarch Aug 28 10:06:26 Updated: spacewalk-backend-config-files-2.6.78-1.el7.noarch Aug 28 10:06:26 Updated: spacewalk-backend-config-files-tool-2.6.78-1.el7.noarch Aug 28 10:06:26 Updated: spacewalk-backend-iss-export-2.6.78-1.el7.noarch Aug 28 10:06:26 Updated: spacewalk-backend-applet-2.6.78-1.el7.noarch Aug 28 10:06:26 Updated: spacewalk-backend-iss-2.6.78-1.el7.noarch Aug 28 10:06:26 Updated: spacewalk-backend-package-push-server-2.6.78-1.el7.noarch Oct 23 13:18:12 Installed: spacewalk-usix-2.7.8-1.el7.noarch Oct 23 13:18:15 Updated: spacewalk-base-minimal-2.7.6-1.el7.noarch Oct 23 13:18:16 Updated: spacewalk-backend-libs-2.7.139-1.el7.noarch Oct 23 13:18:16 Updated: spacewalk-certs-tools-2.7.3-1.el7.noarch Oct 23 13:18:17 Updated: spacewalk-java-postgresql-2.7.116-1.el7.noarch Oct 23 13:18:18 Updated: spacewalk-backend-sql-postgresql-2.7.139-1.el7.noarch Oct 23 13:18:20 Updated: spacewalk-java-config-2.7.116-1.el7.noarch Oct 23 13:18:21 Updated: spacewalk-java-lib-2.7.116-1.el7.noarch Oct 23 13:18:35 Updated: spacewalk-doc-indexes-2.7.3-1.el7.noarch Oct 23 13:18:39 Updated: spacewalk-search-2.7.6-1.el7.noarch Oct 23 13:18:59 Installed: spacewalk-dobby-2.7.6-1.el7.noarch Oct 23 13:19:18 Updated: spacewalk-branding-2.7.5-1.el7.noarch Oct 23 13:19:18 Updated: spacewalk-config-2.7.4-1.el7.noarch Oct 23 13:19:18 Updated: spacewalk-backend-2.7.139-1.el7.noarch Oct 23 13:19:18 Updated: spacewalk-backend-sql-2.7.139-1.el7.noarch Oct 23 13:19:19 Updated: spacewalk-backend-server-2.7.139-1.el7.noarch Oct 23 13:19:19 Updated: spacewalk-backend-xml-export-libs-2.7.139-1.el7.noarch Oct 23 13:19:19 Updated: spacewalk-backend-config-files-common-2.7.139-1.el7.noarch Oct 23 13:19:19 Updated: spacewalk-backend-app-2.7.139-1.el7.noarch Oct 23 13:19:19 Updated: spacewalk-backend-xmlrpc-2.7.139-1.el7.noarch Oct 23 13:19:19 Updated: spacewalk-base-2.7.6-1.el7.noarch Oct 23 13:19:20 Updated: spacewalk-admin-2.7.1-1.el7.noarch Oct 23 13:19:20 Updated: spacewalk-backend-config-files-tool-2.7.139-1.el7.noarch Oct 23 13:19:20 Updated: spacewalk-backend-config-files-2.7.139-1.el7.noarch Oct 23 13:19:20 Updated: spacewalk-backend-iss-export-2.7.139-1.el7.noarch Oct 23 13:19:20 Updated: spacewalk-backend-iss-2.7.139-1.el7.noarch Oct 23 13:19:20 Updated: spacewalk-backend-applet-2.7.139-1.el7.noarch Oct 23 13:19:20 Updated: spacewalk-backend-package-push-server-2.7.139-1.el7.noarch Oct 23 13:19:20 Updated: spacewalk-setup-jabberd-2.7.4-1.el7.noarch Oct 23 13:19:20 Updated: spacewalk-html-2.7.6-1.el7.noarch Oct 23 13:19:21 Updated: spacewalk-taskomatic-2.7.116-1.el7.noarch Oct 23 13:19:21 Updated: spacewalk-base-minimal-config-2.7.6-1.el7.noarch Oct 23 13:19:21 Updated: spacewalk-backend-tools-2.7.139-1.el7.noarch Oct 23 13:19:24 Updated: spacewalk-schema-2.7.28-1.el7.noarch Oct 23 13:19:24 Updated: spacewalk-setup-2.7.12-1.el7.noarch Oct 23 13:19:56 Updated: spacewalk-selinux-2.7.6-1.el7.noarch Oct 23 13:20:22 Installed: spacewalk-java-2.7.116-1.el7.noarch Oct 23 13:20:22 Updated: spacewalk-common-2.7.4-1.el7.noarch Oct 23 13:20:24 Updated: spacewalk-postgresql-2.7.4-1.el7.noarch Oct 23 13:20:40 Updated: spacewalk-setup-postgresql-2.7.3-1.el7.noarch Oct 23 13:21:04 Erased: spacewalk-backend-usix-2.6.78-1.el7.noarch Oct 23 13:21:07 Erased: spacewalk-jpp-workaround-2.3.5-1.el7.noarch |
Description of problem: osa-dispatcher is failing to start/run. This following a fresh install of spacewalk from nightly. The failure is caused by jabberd's sqlite db containing no tables Version-Release number of selected component (if applicable): osa-common-5.11.85-1.el6.noarch osa-dispatcher-5.11.85-1.el6.noarch osa-dispatcher-selinux-5.11.85-1.el6.noarch jabberd-2.6.1-2.el6.x86_64 How reproducible: Always Steps to Reproduce: 1. Do fresh install of spacewalk on newly provisioned and updated RHEL 6 system 2. service osa-dispatcher status 3. cat /var/log/rhn/osa-dispatcher.log Actual results: osa-dispatcher.log is full of... 2017/08/11 13:05:11 -04:00 11723 0.0.0.0: osad/jabber_lib.__init__ 2017/08/11 13:05:11 -04:00 11723 0.0.0.0: osad/jabber_lib.setup_connection('Connected to jabber server', 'eherget-sw-rhel6.usersys.redhat.com') 2017/08/11 13:05:11 -04:00 11723 0.0.0.0: osad/jabber_lib.main('ERROR', 'Error caught:') 2017/08/11 13:05:11 -04:00 11723 0.0.0.0: osad/jabber_lib.main('ERROR', 'Traceback (most recent call last):\n File "/usr/share/rhn/osad/jabber_lib.py", line 126, in main\n c = self.setup_connection(no_fork=no_fork)\n File "/usr/share/rhn/osad/jabber_lib.py", line 304, in setup_connection\n resource=self._resource)\n File "/usr/share/rhn/osad/dispatcher_client.py", line 36, in start\n self.auth(username, password, resource)\n File "/usr/share/rhn/osad/jabber_lib.py", line 897, in auth\n self.register(username, password)\n File "/usr/share/rhn/osad/jabber_lib.py", line 1143, in register\n self.sendRegInfo()\n File "/usr/lib/python2.6/site-packages/jabber/jabber.py", line 644, in sendRegInfo\n return self.SendAndWaitForResponse(reg_iq)\n File "/usr/lib/python2.6/site-packages/jabber/jabber.py", line 401, in SendAndWaitForResponse\n return self.waitForResponse(ID,timeout)\n File "/usr/share/rhn/osad/jabber_lib.py", line 1213, in waitForResponse\n raise JabberQualifiedError(self.lastErrCode, self.lastErr)\nJabberQualifiedError: <JabberQualifiedError instance at 29318192; errcode=500; err=>\n') and /var/log/messages is full of... Aug 11 13:05:41 eherget-sw-rhel6 jabberd/c2s[11498]: [32] [::ffff:10.13.129.36, port=39078] connect Aug 11 13:05:41 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg Aug 11 13:05:42 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg Aug 11 13:05:42 eherget-sw-rhel6 jabberd/c2s[11498]: sqlite (authreg): no such table: authreg Expected results: osa-dispatcher starts and connects to jabberd Additional info: