Description of problem: Cannot get either osad nor osa connecting to jabber on spacewalk-server. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Follow https://fedorahosted.org/spacewalk/wiki/HowToInstall 2. /etc/init.d/osa-dispatcher restart 3. Actual results: From /var/log/messages May 27 03:34:00 Servername jabberd/c2s[1302]: [7] [::ffff:xx.xx.xx.xx, port=40774] disconnect jid=unbound, packets: 0 May 27 03:34:11 Servername jabberd/c2s[1302]: [7] [::ffff:xx.xx.xx.xx, port=40780] connect May 27 03:34:11 Servername jabberd/c2s[1302]: [7] [::ffff:xx.xx.xx.xx, port=40780] disconnect jid=unbound, packets: 0 ...Goes on forever iterating the port. On /etc/init.d/osa-dispatcher restart: Shutting down osa-dispatcher: [ OK ] Starting osa-dispatcher: RHN 8681 2012/05/29 09:55:42 +02:00: ('Not able to reconnect',) RHN 8681 2012/05/29 09:55:42 +02:00: ('Traceback (most recent call last):\n File "/usr/share/rhn/osad/jabber_lib.py", line 252, in setup_connection\n c = self._get_jabber_client(js)\n File "/usr/share/rhn/osad/jabber_lib.py", line 309, in _get_jabber_client\n c.connect()\n File "/usr/share/rhn/osad/jabber_lib.py", line 589, in connect\n raise SSLDisabledError\nSSLDisabledError\n',) [ OK ] Client will print out same lines on stdout on service osad restart. Expected results: Working osad. Additional info: # grep osa /etc/rhn/rhn.conf osa-dispatcher.jabber_server = FQDN osa-dispatcher.osa_ssl_cert = /var/www/html/pub/RHN-ORG-TRUSTED-SSL-CERT The Guide did not mention configuring the jabber server. Do I have to setup one myself? Found https://fedorahosted.org/spacewalk/wiki/OsadHowTo. But this does not state how to setup jabber in the first place.
When I attempt to remove a package from a registered client, the job will fail: This action's status is: Failed. The client picked up this action on 05/29/12 3:55:43 AM EDT. The client completed this action on 05/29/12 3:55:44 AM EDT. Client execution returned "Fatal error in Python code occured [[6]]" (code -1) I assume the Python error is caused by osad?
https://fedorahosted.org/spacewalk/wiki/OSADSetup I did all of this.
Am I correct to assume that the 'jabber server' should be the spacewalk server itself? As far as I read the spacewalk-setup should have configured the jabber configuration. Or do I have to do this? I did not find any page in the Spacewalk-Wiki about jabber-server configuration.
What do your jabberd config files look like? I'm interested in: /etc/jabberd/c2s.xml /etc/jabberd/s2s.xml /etc/jabberd/sm.xml /etc/jabberd/router.xml
Created attachment 588395 [details] The Jabber Logs requested
The config files look OK. When you start the jabberd service on your Spacewalk, do you see all jabberd daemons (c2s, s2s, sm, router) running? Do you see any SELinux denials? Is /etc/pki/spacewalk/jabberd/server.pem readable by jabber? I.e it should look like: # ls -lZ /etc/pki/spacewalk/jabberd/server.pem -rw------- jabber jabber system_u:object_r:cert_t /etc/pki/spacewalk/jabberd/server.pem Are the jabberd config files readable by jabber? I.e.: # ls -lZ /etc/jabberd/*.xml -rw-r----- jabber jabber root:object_r:etc_t /etc/jabberd/c2s.xml -rw-r----- jabber jabber root:object_r:etc_t /etc/jabberd/router-filter.xml -rw-r----- jabber jabber root:object_r:etc_t /etc/jabberd/router-users.xml -rw-r----- jabber jabber root:object_r:etc_t /etc/jabberd/router.xml -rw-r----- jabber jabber root:object_r:etc_t /etc/jabberd/s2s.xml -rw-r----- jabber jabber root:object_r:etc_t /etc/jabberd/sm.xml
Hello, thanks fpr checking my config files. >When you start the jabberd service on your Spacewalk, do you see all >jabberd daemons (c2s, s2s, sm, router) running? Restarting spacewalk services gives me this output # spacewalk-service restart Shutting down spacewalk services... Stopping RHN Taskomatic... Stopped RHN Taskomatic. Stopping cobbler daemon: [ OK ] Stopping rhn-search... Stopped rhn-search. Stopping MonitoringScout ... [ OK ] Stopping Monitoring ... [ OK ] Stopping httpd: [ OK ] Stopping tomcat6: [ OK ] Shutting down osa-dispatcher: [ OK ] Terminating jabberd processes ... Stopping router: [ OK ] Stopping sm: [ OK ] Stopping c2s: [ OK ] Stopping s2s: [ OK ] Done. Starting spacewalk services... Initializing jabberd processes ... Starting router: [ OK ] Starting sm: [ OK ] Starting c2s: [ OK ] Starting s2s: [ OK ] Starting osa-dispatcher: RHN 12084 2012/06/04 07:44:58 +02:00: ('Not able to reconnect',) RHN 12084 2012/06/04 07:44:58 +02:00: ('Traceback (most recent call last):\n File "/usr/share/rhn/osad/jabber_lib.py", line 252, in setup_connection\n c = self._get_jabber_client(js)\n File "/usr/share/rhn/osad/jabber_lib.py", line 309, in _get_jabber_client\n c.connect()\n File "/usr/share/rhn/osad/jabber_lib.py", line 589, in connect\n raise SSLDisabledError\nSSLDisabledError\n',) [ OK ] Starting tomcat6: [ OK ] Waiting for tomcat to be ready ... Starting httpd: [ OK ] Starting Monitoring ... [ OK ] Starting MonitoringScout ... [ OK ] Starting rhn-search... Starting cobbler daemon: [ OK ] Starting RHN Taskomatic... Done. ------------------------------------------------------------------------- After restarting all four services are listed in 'ps -A'. >Do you see any SELinux denials? Where can I see those? >Is /etc/pki/spacewalk/jabberd/server.pem readable by jabber? I.e it should >look like: They look different: -rw-------. jabber jabber system_u:object_r:cert_t:s0 /etc/pki/spacewalk/jabberd/server.pem -rw-r-----. jabber jabber system_u:object_r:etc_t:s0 /etc/jabberd/c2s.xml -rw-r-----. jabber jabber system_u:object_r:etc_t:s0 /etc/jabberd/router-filter.xml -rw-r-----. jabber jabber system_u:object_r:etc_t:s0 /etc/jabberd/router-users.xml -rw-r-----. jabber jabber system_u:object_r:etc_t:s0 /etc/jabberd/router.xml -rw-r-----. jabber jabber system_u:object_r:etc_t:s0 /etc/jabberd/s2s.xml -rw-r-----. jabber jabber system_u:object_r:etc_t:s0 /etc/jabberd/sm.xml What can I do to fix this?
# grep "SELinux is preventing" /var/log/messages returns nothing # grep "denied" /var/log/audit/audit.log returns some messages about ntpd: type=APPARMOR_DENIED msg=audit(1334141972.102:12): operation="open" pid=3678 parent=1 profile="/usr/sbin/ntpd" requested_mask="r::" denied_mask="r::" fsuid=0 ouid=0 name="/var/lib/ntp/proc/sys/kernel/ngroups_max"
type=AVC msg=audit(1338788183.618:7113): avc: denied { sendto } for pid=15358 comm="c2s" path="/dev/log" scontext=unconfined_u:system_r:jabberd_router_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=unix_dgram_socket Could this be relevant?
type=AVC msg=audit(1338788694.553:7130): avc: denied { write } for pid=12056 comm="sm" name="log" dev=devtmpfs ino=1594877 scontext=unconfined_u:system_r:jabberd_t:s0 tcontext=unconfined_u:object_r:device_t:s0 tclass=sock_file type=AVC msg=audit(1338788708.313:7149): avc: denied { sendto } for pid=12063 comm="c2s" path="/dev/log" scontext=unconfined_u:system_r:jabberd_router_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=unix_dgram_socket type=AVC msg=audit(1338788825.637:7150): avc: denied { sendto } for pid=12063 comm="c2s" path="/dev/log" scontext=unconfined_u:system_r:jabberd_router_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=unix_dgram_socket
OK, does the described problem show with SELinux permissive? Would relabeling the file system (SELinux labels) help?
Hello, I just disabled SELinux temporarily by echo 0 >/selinux/enforce and tried spacewalk-service restart: The error was still present. Then I tried relabeling the SELinux labels by touch /.autorelabel reboot and tried spacewalk-service restart: The error was still present. Also tried switching off SELinux after relabeling but the error is still present. regards
(In reply to comment #12) > Hello, > > I just disabled SELinux temporarily by > echo 0 >/selinux/enforce > and tried spacewalk-service restart: The error was still present. > > Then I tried relabeling the SELinux labels by > touch /.autorelabel > reboot > > and tried spacewalk-service restart: The error was still present. > > Also tried switching off SELinux after relabeling but the error is still > present. Which error is still present after relabeling? The SELinux denials? If so, are they new denials?
Hello, I was referring to the error on restarting the spacewalk-services: Starting osa-dispatcher: RHN 3688 2012/06/05 08:43:10 +02:00: ('Not able to reconnect',) RHN 3688 2012/06/05 08:43:10 +02:00: ('Traceback (most recent call last):\n File "/usr/share/rhn/osad/jabber_lib.py", line 252, in setup_connection\n c = self._get_jabber_client(js)\n File "/usr/share/rhn/osad/jabber_lib.py", line 309, in _get_jabber_client\n c.connect()\n File "/usr/share/rhn/osad/jabber_lib.py", line 589, in connect\n raise SSLDisabledError\nSSLDisabledError\n',) [ OK ]
Do you think a reinstallation could solve this issue? Currently we are running CentOS. Could a change to fedora help? Are there any Appliances for Spacewalk?
What you're experiencing seems like an SSL configuration issue (incorrect certificate, common name, etc). Reinstallation could help the problem, although I don't have a good sense on what would be better for you -- Fedora or CentOS. Both should work just fine (although CentOS / RHEL is certainly more tested & stable).
I just redid spacewalk-setup --disconnected with no success. When it asks for a CA password during SSL cert creation, should I leave that empty? Could that be the issue?
(In reply to comment #17) > I just redid spacewalk-setup --disconnected with no success. When it asks > for a CA password during SSL cert creation, should I leave that empty? Could > that be the issue? No, don't leave that empty. You need to have your CA password protected.
Is there a log where I may find clues about what is wrong with SSL? The error provided by osa-dispatcher seems very generic as I've found a lot of other users getting the same error, but their solutions won't fix mine. Could I disable SSL in osa-dispatcher for testing?
Not really a log. You just need to make sure your hostname / fqdn is configured correctly and that it matches CN found in your SSL certificate (that server.pem file that jabberd is pointed to). For some reason it seems like your jabberd configuration doesn't work with SSL / TLS. You can check /var/log/messages for errors from jabberd processes (c2s in particular) when starting jabberd or you can start c2s manually (when jabberd is off) and look for errors with: # . /etc/init.d/functions # daemon --user jabber "/usr/bin/c2s -D -c /etc/jabberd/c2s.xml"
The CN entry is the correct FQDN. There seems to be no output to messages. This is the output of the command provided (spacewalk services down): # daemon --user jabber "/usr/bin/c2s -D -c /etc/jabberd/c2s.xml" Mon Jun 11 08:09:05 2012 [notice] starting up Mon Jun 11 08:09:05 2012 [info] process id is 17639, written to /var/lib/jabberd/pid/c2s.pid Mon Jun 11 08:09:05 2012 [notice] modules search path: /usr/lib64/jabberd Mon Jun 11 08:09:05 2012 [info] loading 'db' authreg module Mon Jun 11 08:09:05 2012 authreg.c:73 preloaded module 'db' (not initialized yet) Mon Jun 11 08:09:05 2012 [notice] initialized auth module 'db' Mon Jun 11 08:09:05 2012 ack.c:104 initialising stanza acknowledgements sx plugin sx (env.c:75) plugin initialised (index 0) Mon Jun 11 08:09:05 2012 address.c:43 initialising address sx plugin sx (env.c:75) plugin initialised (index 1) sx (sasl_gsasl.c:913) initialising sasl plugin sx (sasl_gsasl.c:941) sasl context initialised sx (env.c:75) plugin initialised (index 2) Mon Jun 11 08:09:05 2012 bind.c:72 initialising resource bind sx plugin sx (env.c:75) plugin initialised (index 3) sx (ssl.c:818) initialising ssl plugin sx (ssl.c:911) No CA chain specified. Loading SSL default CA certs: /etc/ssl/certs sx (ssl.c:943) setting ssl context '' verify mode to 00 sx (ssl.c:911) No CA chain specified. Loading SSL default CA certs: /etc/ssl/certs sx (ssl.c:943) setting ssl context '*' verify mode to 00 sx (ssl.c:964) ssl context '*' initialised; certificate and key loaded from /etc/pki/spacewalk/jabberd/server.pem sx (ssl.c:964) ssl context '' initialised; certificate and key loaded from /etc/pki/spacewalk/jabberd/server.pem sx (env.c:75) plugin initialised (index 4) Mon Jun 11 08:09:05 2012 [notice] [spm01.hsz-bw.de] configured; realm=, registration enabled Mon Jun 11 08:09:05 2012 [notice] attempting connection to router at ::1, port=5347 sx (sx.c:61) allocated new sx for 4 sx (client.c:122) doing client init for sx 4 sx (client.c:138) stream request: ns (null) to (null) from (null) version 1.0 Mon Jun 11 08:09:05 2012 ack.c:30 hacking ack namespace decl onto stream header sx (client.c:168) prepared stream header: <?xml version='1.0'?><stream:stream xmlns:stream='http://etherx.jabber.org/streams' version='1.0' xmlns:ack='http://www.xmpp.org/extensions/xep-0198.html#ns'> sx (client.c:175) tag 4 event 1 data 0x0 Mon Jun 11 08:09:05 2012 c2s.c:694 want write Mon Jun 11 08:09:05 2012 c2s.c:1325 write action on fd 4 sx (io.c:328) 4 ready for writing sx (io.c:286) encoding 158 bytes for writing: <?xml version='1.0'?><stream:stream xmlns:stream='http://etherx.jabber.org/streams' version='1.0' xmlns:ack='http://www.xmpp.org/extensions/xep-0198.html#ns'> sx (chain.c:79) calling io write chain sx (io.c:349) handing app 158 bytes to write sx (io.c:350) tag 4 event 3 data 0xcf26b0 Mon Jun 11 08:09:05 2012 c2s.c:731 writing to 4 Mon Jun 11 08:09:05 2012 [notice] [4] [router] write error: Connection refused (111) sx (io.c:498) 4 state change from 0 to 6 sx (io.c:499) tag 4 event 7 data 0x0 Mon Jun 11 08:09:05 2012 c2s.c:1329 close action on fd 4 Mon Jun 11 08:09:05 2012 [notice] connection to router closed Mon Jun 11 08:09:05 2012 [notice] attempting reconnect (3 left) sx (sx.c:78) freeing sx for 4 sx (sx.c:111) freeing 5 env plugins Mon Jun 11 08:09:07 2012 [notice] attempting connection to router at ::1, port=5347 sx (sx.c:61) allocated new sx for 4 sx (client.c:122) doing client init for sx 4 sx (client.c:138) stream request: ns (null) to (null) from (null) version 1.0 Mon Jun 11 08:09:07 2012 ack.c:30 hacking ack namespace decl onto stream header sx (client.c:168) prepared stream header: <?xml version='1.0'?><stream:stream xmlns:stream='http://etherx.jabber.org/streams' version='1.0' xmlns:ack='http://www.xmpp.org/extensions/xep-0198.html#ns'> sx (client.c:175) tag 4 event 1 data 0x0 Mon Jun 11 08:09:07 2012 c2s.c:694 want write Mon Jun 11 08:09:07 2012 main.c:795 running time checks Mon Jun 11 08:09:07 2012 main.c:800 next time check at 1339395007 Mon Jun 11 08:09:07 2012 c2s.c:1325 write action on fd 4 sx (io.c:328) 4 ready for writing sx (io.c:286) encoding 158 bytes for writing: <?xml version='1.0'?><stream:stream xmlns:stream='http://etherx.jabber.org/streams' version='1.0' xmlns:ack='http://www.xmpp.org/extensions/xep-0198.html#ns'> sx (chain.c:79) calling io write chain sx (io.c:349) handing app 158 bytes to write sx (io.c:350) tag 4 event 3 data 0xd1d870 Mon Jun 11 08:09:07 2012 c2s.c:731 writing to 4 Mon Jun 11 08:09:07 2012 [notice] [4] [router] write error: Connection refused (111) sx (io.c:498) 4 state change from 0 to 6 sx (io.c:499) tag 4 event 7 data 0x0 Mon Jun 11 08:09:07 2012 c2s.c:1329 close action on fd 4 Mon Jun 11 08:09:07 2012 [notice] connection to router closed Mon Jun 11 08:09:07 2012 [notice] attempting reconnect (2 left) sx (sx.c:78) freeing sx for 4 sx (sx.c:111) freeing 5 env plugins Mon Jun 11 08:09:09 2012 [notice] attempting connection to router at ::1, port=5347 sx (sx.c:61) allocated new sx for 4 sx (client.c:122) doing client init for sx 4 sx (client.c:138) stream request: ns (null) to (null) from (null) version 1.0 Mon Jun 11 08:09:09 2012 ack.c:30 hacking ack namespace decl onto stream header sx (client.c:168) prepared stream header: <?xml version='1.0'?><stream:stream xmlns:stream='http://etherx.jabber.org/streams' version='1.0' xmlns:ack='http://www.xmpp.org/extensions/xep-0198.html#ns'> sx (client.c:175) tag 4 event 1 data 0x0 Mon Jun 11 08:09:09 2012 c2s.c:694 want write Mon Jun 11 08:09:09 2012 c2s.c:1325 write action on fd 4 sx (io.c:328) 4 ready for writing sx (io.c:286) encoding 158 bytes for writing: <?xml version='1.0'?><stream:stream xmlns:stream='http://etherx.jabber.org/streams' version='1.0' xmlns:ack='http://www.xmpp.org/extensions/xep-0198.html#ns'> sx (chain.c:79) calling io write chain sx (io.c:349) handing app 158 bytes to write sx (io.c:350) tag 4 event 3 data 0xd047b0 Mon Jun 11 08:09:09 2012 c2s.c:731 writing to 4 Mon Jun 11 08:09:09 2012 [notice] [4] [router] write error: Connection refused (111) sx (io.c:498) 4 state change from 0 to 6 sx (io.c:499) tag 4 event 7 data 0x0 Mon Jun 11 08:09:09 2012 c2s.c:1329 close action on fd 4 Mon Jun 11 08:09:09 2012 [notice] connection to router closed Mon Jun 11 08:09:09 2012 [notice] attempting reconnect (1 left) sx (sx.c:78) freeing sx for 4 sx (sx.c:111) freeing 5 env plugins Mon Jun 11 08:09:11 2012 [notice] attempting connection to router at ::1, port=5347 sx (sx.c:61) allocated new sx for 4 sx (client.c:122) doing client init for sx 4 sx (client.c:138) stream request: ns (null) to (null) from (null) version 1.0 Mon Jun 11 08:09:11 2012 ack.c:30 hacking ack namespace decl onto stream header sx (client.c:168) prepared stream header: <?xml version='1.0'?><stream:stream xmlns:stream='http://etherx.jabber.org/streams' version='1.0' xmlns:ack='http://www.xmpp.org/extensions/xep-0198.html#ns'> sx (client.c:175) tag 4 event 1 data 0x0 Mon Jun 11 08:09:11 2012 c2s.c:694 want write Mon Jun 11 08:09:11 2012 c2s.c:1325 write action on fd 4 sx (io.c:328) 4 ready for writing sx (io.c:286) encoding 158 bytes for writing: <?xml version='1.0'?><stream:stream xmlns:stream='http://etherx.jabber.org/streams' version='1.0' xmlns:ack='http://www.xmpp.org/extensions/xep-0198.html#ns'> sx (chain.c:79) calling io write chain sx (io.c:349) handing app 158 bytes to write sx (io.c:350) tag 4 event 3 data 0xd047e0 Mon Jun 11 08:09:11 2012 c2s.c:731 writing to 4 Mon Jun 11 08:09:11 2012 [notice] [4] [router] write error: Connection refused (111) sx (io.c:498) 4 state change from 0 to 6 sx (io.c:499) tag 4 event 7 data 0x0 Mon Jun 11 08:09:11 2012 c2s.c:1329 close action on fd 4 Mon Jun 11 08:09:11 2012 [notice] connection to router closed Mon Jun 11 08:09:11 2012 [notice] shutting down sx (sx.c:78) freeing sx for 4 sx (sx.c:111) freeing 5 env plugins Mon Jun 11 08:09:11 2012 authreg_db.c:260 db module shutting down
Hej Milan, could the above mentioned bug be caused by a domain name with capital letters? The given config files do contain a capitalized domain name, and the Red Hat Network Satellite installation documentation states in chapter 2.4 (additional requirements), that this can cause problems: >It is important that the hostname of a Satellite contains no uppercase > letters. A hostname that includes uppercase letters can cause jabberd > to fail. Cheers, Roland
Hmm, could be. In truth, I was not aware of this requirement.
Hello, the problem was solved by renaming the server's hostname so it does not include capital letters. Afterwards I had to rerun spacewalk-setup as the spacewalk-rename script (from spacewalk-utils package) does not work with the postgres backend (Oracle only). For the setup to be able to overwrite the certificate I had to remove the rpm identified by # rpm -qa|grep rhn-org-httpd I would like to suggest to add this information ( about the capital letters incompatibility ) to https://fedorahosted.org/spacewalk/wiki/OsadHowTo or to add a check to the spacewalk-setup in order to provide some a warning at least.