Description of problem: Spacewalk-Web-Page gives back error 404 for all URL even /rhn/Login.do HTTP Status 404 - type Status report message description The requested resource () is not available. Apache Tomcat/6.0.24 Version-Release number of selected component (if applicable): How reproducible: Unknown. Steps to Reproduce: 1. 2. 3. Actual results: going to URL https://spacewalk.<domain>/rhn/Login.do -> Error 404 Expected results: going to URL https://spacewalk.<domain>/rhn/Login.do -> Spacewalk login page Additional info: Spacewalk ran for about six months. Package Versions now are: # rpm -qa | grep 'space\|rhn' | sort perl-XML-NamespaceSupport-1.10-3.el6.noarch rhncfg-5.10.99-1.el6.noarch rhncfg-actions-5.10.99-1.el6.noarch rhncfg-client-5.10.99-1.el6.noarch rhn-check-2.6.8-1.el6.noarch rhn-client-tools-2.6.8-1.el6.noarch rhnlib-2.6.3-1.el6.noarch rhn-org-httpd-ssl-key-pair-spacewalk-dmz-1.0-1.noarch rhnpush-5.5.101-1.el6.noarch rhnsd-5.0.25-1.el6.x86_64 rhn-setup-2.6.8-1.el6.noarch spacecmd-2.6.16-1.el6.noarch spacewalk-admin-2.6.1-1.el6.noarch spacewalk-backend-2.6.77-1.el6.noarch spacewalk-backend-app-2.6.77-1.el6.noarch spacewalk-backend-applet-2.6.77-1.el6.noarch spacewalk-backend-config-files-2.6.77-1.el6.noarch spacewalk-backend-config-files-common-2.6.77-1.el6.noarch spacewalk-backend-config-files-tool-2.6.77-1.el6.noarch spacewalk-backend-iss-2.6.77-1.el6.noarch spacewalk-backend-iss-export-2.6.77-1.el6.noarch spacewalk-backend-libs-2.6.77-1.el6.noarch spacewalk-backend-package-push-server-2.6.77-1.el6.noarch spacewalk-backend-server-2.6.77-1.el6.noarch spacewalk-backend-sql-2.6.77-1.el6.noarch spacewalk-backend-sql-postgresql-2.6.77-1.el6.noarch spacewalk-backend-tools-2.6.77-1.el6.noarch spacewalk-backend-usix-2.6.77-1.el6.noarch spacewalk-backend-xml-export-libs-2.6.77-1.el6.noarch spacewalk-backend-xmlrpc-2.6.77-1.el6.noarch spacewalk-base-2.6.6-1.el6.noarch spacewalk-base-minimal-2.6.6-1.el6.noarch spacewalk-base-minimal-config-2.6.6-1.el6.noarch spacewalk-branding-2.5.3-1.el6.noarch spacewalk-certs-tools-2.5.3-1.el6.noarch spacewalk-common-2.6.1-1.el6.noarch spacewalk-config-2.6.5-1.el6.noarch spacewalk-doc-indexes-2.6.2-1.el6.noarch spacewalk-html-2.6.6-1.el6.noarch spacewalk-java-2.6.49-1.el6.noarch spacewalk-java-config-2.6.49-1.el6.noarch spacewalk-java-lib-2.6.49-1.el6.noarch spacewalk-java-postgresql-2.6.49-1.el6.noarch spacewalk-jpp-workaround-2.3.5-1.el6.noarch spacewalk-repo-2.6-0.el6.noarch spacewalk-schema-2.6.17-1.el6.noarch spacewalk-search-2.6.1-1.el6.noarch spacewalk-selinux-2.3.2-1.el6.noarch spacewalk-setup-2.6.2-1.el6.noarch spacewalk-setup-jabberd-2.3.2-1.el6.noarch spacewalk-setup-postgresql-2.6.2-1.el6.noarch spacewalk-slf4j-1.6.1-6.el6.noarch spacewalk-taskomatic-2.6.49-1.el6.noarch yum-rhn-plugin-2.6.3-1.el6.noarch # rpm -qa | grep python | sort audit-libs-python-2.4.5-6.el6.x86_64 dbus-python-0.83.0-6.1.el6.x86_64 gnome-python2-2.28.0-3.el6.x86_64 gnome-python2-bonobo-2.28.0-3.el6.x86_64 gnome-python2-canvas-2.28.0-3.el6.x86_64 gnome-python2-gnome-2.28.0-3.el6.x86_64 gnome-python2-gnomevfs-2.28.0-3.el6.x86_64 libproxy-python-0.3.0-10.el6.x86_64 libreport-python-2.0.9-33.0.1.el6.x86_64 libselinux-python-2.0.94-7.el6.x86_64 libsemanage-python-2.0.43-5.1.el6.x86_64 libxml2-python-2.7.6-21.0.1.el6_8.1.x86_64 newt-python-0.52.11-4.el6.x86_64 policycoreutils-python-2.0.83-30.1.0.1.el6_8.x86_64 postgresql96-plpython-9.6.3-1PGDG.rhel6.x86_64 python-2.6.6-66.0.1.el6_8.x86_64 python-argparse-1.2.1-3.rhel6.noarch python-backports-1.0-5.el6.x86_64 python-backports-ssl_match_hostname-3.4.0.2-5.el6.noarch python-chardet-2.2.1-1.el6.noarch python-cheetah-2.4.1-1.el6.x86_64 python-debian-0.1.27-3.el6.noarch python-deltarpm-3.5-0.5.20090913git.el6.x86_64 python-devel-2.6.6-66.0.1.el6_8.x86_64 python-dmidecode-3.10.15-1.el6.x86_64 python-ethtool-0.6-6.el6.x86_64 python-gudev-147.1-4.el6_0.1.x86_64 python-gzipstream-2.3.3-1.el6.noarch python-hwdata-1.7.3-1.el6.noarch python-iniparse-0.3.1-2.1.el6.noarch python-iwlib-0.1-1.2.el6.x86_64 python-libs-2.6.6-66.0.1.el6_8.x86_64 python-markdown-2.0.1-4.el6.noarch python-netaddr-0.7.5-4.el6.noarch python-psycopg2-2.7.1-1.rhel6.x86_64 python-pycurl-7.19.0-9.el6.x86_64 python-pygments-1.1.1-2.el6.noarch python-requests-2.6.0-4.el6.noarch python-setuptools-0.6.10-3.el6.noarch python-simplejson-3.8.2-1.rhel6.x86_64 python-six-1.9.0-2.el6.noarch python-sss-1.13.3-56.el6.x86_64 python-sssdconfig-1.13.3-56.el6.noarch python-urlgrabber-3.9.1-11.0.1.el6.noarch python-urllib3-1.10.2-3.el6.noarch rpm-python-4.8.0-55.el6.x86_64 setools-libs-python-3.3.7-4.el6.x86_64 # java -version openjdk version "1.8.0_131" OpenJDK Runtime Environment (build 1.8.0_131-b11) OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode) # rpm -qa | grep tom | sort apache-tomcat-apis-0.1-1.el6.noarch jquery-ui-1.10.4.custom-3.el6.noarch tomcat5-jsp-2.0-api-5.5.27-7.jpp5.noarch tomcat5-servlet-2.4-api-5.5.27-7.jpp5.noarch tomcat6-6.0.24-105.el6_8.noarch tomcat6-el-2.1-api-6.0.24-105.el6_8.noarch tomcat6-jsp-2.1-api-6.0.24-105.el6_8.noarch tomcat6-lib-6.0.24-105.el6_8.noarch tomcat6-servlet-2.5-api-6.0.24-105.el6_8.noarch # rpm -qa | grep postgres | sort postgresql96-9.6.3-1PGDG.rhel6.x86_64 postgresql96-contrib-9.6.3-1PGDG.rhel6.x86_64 postgresql96-libs-9.6.3-1PGDG.rhel6.x86_64 postgresql96-plperl-9.6.3-1PGDG.rhel6.x86_64 postgresql96-plpython-9.6.3-1PGDG.rhel6.x86_64 postgresql96-pltcl-9.6.3-1PGDG.rhel6.x86_64 postgresql96-server-9.6.3-1PGDG.rhel6.x86_64 postgresql-jdbc-42.1.1-1.rhel6.noarch spacewalk-backend-sql-postgresql-2.6.77-1.el6.noarch spacewalk-java-postgresql-2.6.49-1.el6.noarch spacewalk-setup-postgresql-2.6.2-1.el6.noarch for postgresql 9.6 I had to patch spacewalk within /usr/share/rhn/server/handlers/xmlrpc/queue.py addin "as" (http://git.net/ml/spacewalk-devel/2016-11/msg00004.html and https://github.com/spacewalkproject/spacewalk/blob/master/backend/server/handlers/xmlrpc/queue.py Further things I found within httpd logs: access_log: [17/May/2017:16:27:21 +0200] "POST /cobbler_api HTTP/1.1" 200 144 "-" "Java/1.8.0_131" [17/May/2017:16:27:21 +0200] "POST /rpc/api HTTP/1.0" 404 952 "-" "xmlrpclib.py/1.0.1 (by www.pythonware.com)" [17/May/2017:16:27:21 +0200] "POST /cobbler_api HTTP/1.1" 200 326 "-" "Java/1.8.0_131" [17/May/2017:16:28:00 +0200] "POST /cobbler_api HTTP/1.1" 200 144 "-" "Java/1.8.0_131" [17/May/2017:16:28:00 +0200] "POST /rpc/api HTTP/1.0" 404 952 "-" "xmlrpclib.py/1.0.1 (by www.pythonware.com)" Since 200 is OK, 404 is not. Apache seems to have problems with /rpc/api! Within error_log there is nothing remarkable. Python 2.6.6 is OK or does spacewalk rely on Python 3.x? [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0 [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [notice] Digest: generating secret for digest authentication ... [notice] Digest: done [notice] Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips mod_wsgi/3.2 Python/2.6.6 mod_apreq2-20090110/2.8.0 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations Client request seem to be handled OK: [17/May/2017:16:36:04 +0200] "POST /XMLRPC HTTP/1.1" 200 112276 [17/May/2017:16:36:06 +0200] "POST /XMLRPC HTTP/1.1" 200 915 [17/May/2017:16:36:07 +0200] "POST /XMLRPC HTTP/1.1" 200 121 [17/May/2017:16:36:07 +0200] "POST /XMLRPC HTTP/1.1" 200 126 [17/May/2017:16:36:08 +0200] "POST /XMLRPC HTTP/1.1" 200 1082 [17/May/2017:16:36:08 +0200] "GET /XMLRPC/GET-REQ/ol6-imisterm/repodata/repomd.xml HTTP/1.1" 200 1302 While web-requests are not: [17/May/2017:16:36:34 +0200] "GET /rhn/schedule/CompletedActions.do HTTP/1.1" 404 952 [17/May/2017:16:36:50 +0200] "GET /rhn/Login.do HTTP/1.1" 404 952 [17/May/2017:16:37:34 +0200] "GET /rhn/schedule/CompletedActions.do HTTP/1.1" 404 952 [17/May/2017:16:37:50 +0200] "GET /rhn/Login.do HTTP/1.1" 404 952 ssl_error_log is empty ssl_request_log seem good too: TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/schedule/CompletedActions.do HTTP/1.1" 952 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/Login.do HTTP/1.1" 952 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/schedule/CompletedActions.do HTTP/1.1" 952 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/Login.do HTTP/1.1" 952 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/schedule/CompletedActions.do HTTP/1.1" 952 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/Login.do HTTP/1.1" 952 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/schedule/CompletedActions.do HTTP/1.1" 952 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /rhn/Login.do HTTP/1.1" 952 Apache loaded Modules: mod_wsgi.c, mod_ssl.c, mod_perl.c, mod_apreq2.c, mod_version.c, mod_cgi.c, mod_disk_cache.c, mod_suexec.c, mod_cache.c, mod_proxy_connect.c, mod_proxy_ajp.c, mod_proxy_http.c, mod_proxy_ftp.c, mod_proxy_balancer.c, mod_proxy.c, mod_rewrite.c, mod_substitute.c, mod_alias.c, mod_userdir.c, mod_speling.c, mod_actions.c, mod_dir.c, mod_negotiation.c, mod_vhost_alias.c, mod_dav_fs.c, mod_info.c, mod_autoindex.c, mod_status.c, mod_dav.c, mod_mime.c, mod_setenvif.c, mod_usertrack.c, mod_headers.c, mod_deflate.c, mod_expires.c, mod_mime_magic.c, mod_ext_filter.c, mod_env.c, mod_logio.c, mod_log_config.c, mod_include.c, mod_authnz_ldap.c, util_ldap.c, mod_authz_default.c, mod_authz_dbm.c, mod_authz_groupfile.c, mod_authz_owner.c, mod_authz_user.c, mod_authz_host.c, mod_authn_default.c, mod_authn_dbm.c, mod_authn_anon.c, mod_authn_alias.c, mod_authn_file.c, mod_auth_digest.c, mod_auth_basic.c, mod_so.c, http_core.c, prefork.c, core.c Redirections seem OK too. Within /var/log/rhn/rhn_server_xmlrpc.log I cant find any query from any host accessing web-pages. Logs seem OK for spacewalk clients: 2017/05/17 19:32:15 +02:00 9156 134.92.47.22: xmlrpc/queue.get(1000010086, 2, 'checkins enabled') 2017/05/17 19:32:16 +02:00 10725 134.92.47.22: xmlrpc/up2date.login(1000010086,) 2017/05/17 19:32:16 +02:00 9049 134.92.47.22: xmlrpc/up2date.listChannels(1000010086,) 2017/05/17 19:32:17 +02:00 9055 134.92.47.22: xmlrpc/registration.welcome_message('lang: None',) 2017/05/17 19:32:41 +02:00 9154 134.92.46.130: xmlrpc/registration.welcome_message('lang: None',) 2017/05/17 19:33:58 +02:00 9055 134.92.40.131: xmlrpc/queue.get(1000010055, 2, 'checkins enabled') 2017/05/17 19:33:59 +02:00 9048 134.92.40.131: xmlrpc/up2date.login(1000010055,) 2017/05/17 19:34:00 +02:00 9154 134.92.40.131: xmlrpc/queue.submit(1000010055, 3395, 0) 2017/05/17 19:34:00 +02:00 9052 134.92.40.131: xmlrpc/queue.get(1000010055, 2, 'checkins enabled') 2017/05/17 19:34:00 +02:00 9157 134.92.40.131: xmlrpc/up2date.listChannels(1000010055,) 2017/05/17 19:36:04 +02:00 9054 134.92.39.131: xmlrpc/queue.get(1000010133, 2, 'checkins enabled') 2017/05/17 19:36:05 +02:00 9154 134.92.39.131: xmlrpc/up2date.login(1000010133,) 2017/05/17 19:36:05 +02:00 9051 134.92.39.131: xmlrpc/up2date.listChannels(1000010133,) 2017/05/17 19:36:06 +02:00 9050 134.92.39.131: xmlrpc/registration.welcome_message('lang: None',) 2017/05/17 19:41:39 +02:00 9050 134.92.74.130: xmlrpc/registration.welcome_message('lang: None',) 2017/05/17 19:47:54 +02:00 9054 134.92.60.131: xmlrpc/registration.welcome_message('lang: None',) 2017/05/17 19:48:01 +02:00 9048 134.92.60.131: xmlrpc/registration.update_packages(1000010060, 'packages: 1282') 2017/05/17 19:48:04 +02:00 9053 134.92.60.131: xmlrpc/queue.submit(1000010060, 2800, 0) 2017/05/17 19:48:04 +02:00 9052 134.92.60.131: xmlrpc/queue.get(1000010060, 2, 'checkins enabled') 2017/05/17 19:48:05 +02:00 9157 134.92.60.131: xmlrpc/up2date.login(1000010060,) 2017/05/17 19:48:19 +02:00 9156 134.92.60.131: xmlrpc/registration.welcome_message('lang: None',) 2017/05/17 19:48:25 +02:00 9054 134.92.60.131: xmlrpc/registration.update_packages(1000010060, 'packages: 1283') 2017/05/17 19:48:27 +02:00 9050 134.92.60.131: xmlrpc/queue.submit(1000010060, 2801, 0) 2017/05/17 19:48:28 +02:00 9048 134.92.60.131: xmlrpc/queue.get(1000010060, 2, 'checkins enabled') 2017/05/17 19:48:28 +02:00 9053 134.92.60.131: xmlrpc/up2date.login(1000010060,) 2017/05/17 19:48:29 +02:00 9052 134.92.60.131: xmlrpc/queue.submit(1000010060, 2802, 0) 2017/05/17 19:48:29 +02:00 9157 134.92.60.131: xmlrpc/queue.get(1000010060, 2, 'checkins enabled') 2017/05/17 19:48:30 +02:00 10629 134.92.60.131: xmlrpc/up2date.login(1000010060,) 2017/05/17 19:48:31 +02:00 9049 134.92.60.131: xmlrpc/queue.submit(1000010060, 3016, 0) 2017/05/17 19:48:31 +02:00 9055 134.92.60.131: xmlrpc/queue.get(1000010060, 2, 'checkins enabled') 2017/05/17 19:48:32 +02:00 9156 134.92.60.131: xmlrpc/up2date.login(1000010060,) 2017/05/17 19:48:44 +02:00 9051 134.92.60.131: xmlrpc/registration.welcome_message('lang: None',) 2017/05/17 19:48:51 +02:00 9054 134.92.60.131: xmlrpc/registration.update_packages(1000010060, 'packages: 1284') 2017/05/17 19:48:53 +02:00 9050 134.92.60.131: xmlrpc/queue.submit(1000010060, 3047, 0) 2017/05/17 19:48:53 +02:00 9048 134.92.60.131: xmlrpc/queue.get(1000010060, 2, 'checkins enabled') 2017/05/17 19:48:54 +02:00 9053 134.92.60.131: xmlrpc/up2date.login(1000010060,) Within /var/log/rhn/rhn_taskomatic.log: STATUS | wrapper | 2017/05/17 16:27:17 | --> Wrapper Started as Daemon STATUS | wrapper | 2017/05/17 16:27:18 | Launching a JVM... INFO | jvm 1 | 2017/05/17 16:27:18 | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org INFO | jvm 1 | 2017/05/17 16:27:18 | Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved. INFO | jvm 1 | 2017/05/17 16:27:18 | INFO | jvm 1 | 2017/05/17 16:27:19 | May 17, 2017 4:27:19 PM com.mchange.v2.log.MLog <clinit> INFO | jvm 1 | 2017/05/17 16:27:19 | INFO: MLog clients using java 1.4+ standard logging. INFO | jvm 1 | 2017/05/17 16:27:19 | May 17, 2017 4:27:19 PM com.mchange.v2.c3p0.C3P0Registry banner INFO | jvm 1 | 2017/05/17 16:27:19 | INFO: Initializing c3p0-0.9.1.2 [built 06-August-2008 15:35:00; debug? false; trace: 5] INFO | jvm 1 | 2017/05/17 16:27:19 | May 17, 2017 4:27:19 PM com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource getPoolManager INFO | jvm 1 | 2017/05/17 16:27:19 | INFO: Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@ce111814 [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@d261991e [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> com.redhat.rhn.common.db.RhnConnectionCustomizer, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, debugUnreturnedConnectionStackTraces -> false, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2yhf1j9o6bsw311wql4ix|552667aa, idleConnectionTestPeriod -> 300, initialPoolSize -> 5, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 300, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 20, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 5, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@d73a4630 [ description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 2yhf1j9o6bsw311wql4ix|4f849ee, jdbcUrl -> jdbc:postgresql:rhnschema, properties -> {user=******, password=******, driver_proto=jdbc:postgresql} ], preferredTestQuery -> select 'c3p0 ping' from dual, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> true, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false; userOverrides: {} ], dataSourceName -> null, factoryClassLocation -> null, identityToken -> 2yhf1j9o6bsw311wql4ix|3cabc3c1, numHelperThreads -> 3 ] INFO | jvm 1 | 2017/05/17 16:27:21 | 2017-05-17 16:27:21,774 [DefaultQuartzScheduler_Worker-1] ERROR com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand - XmlRpcFault while logging in. most likely user doesn't have permissions. INFO | jvm 1 | 2017/05/17 16:27:21 | redstone.xmlrpc.XmlRpcFault: <class 'cobbler.cexceptions.CX'>:'login failed (taskomatic_user)' INFO | jvm 1 | 2017/05/17 16:27:21 | at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:443) INFO | jvm 1 | 2017/05/17 16:27:21 | at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376) INFO | jvm 1 | 2017/05/17 16:27:21 | at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165) INFO | jvm 1 | 2017/05/17 16:27:21 | at com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMethod(CobblerXMLRPCHelper.java:69) INFO | jvm 1 | 2017/05/17 16:27:21 | at com.redhat.rhn.manager.kickstart.cobbler.CobblerLoginCommand.login(CobblerLoginCommand.java:52) INFO | jvm 1 | 2017/05/17 16:27:21 | at com.redhat.rhn.frontend.integration.IntegrationService.authorize(IntegrationService.java:114) INFO | jvm 1 | 2017/05/17 16:27:21 | at com.redhat.rhn.frontend.integration.IntegrationService.getAuthToken(IntegrationService.java:67) INFO | jvm 1 | 2017/05/17 16:27:21 | at com.redhat.rhn.manager.kickstart.cobbler.CobblerCommand.<init>(CobblerCommand.java:68) INFO | jvm 1 | 2017/05/17 16:27:21 | at com.redhat.rhn.manager.kickstart.cobbler.CobblerDistroSyncCommand.<init>(CobblerDistroSyncCommand.java:52) INFO | jvm 1 | 2017/05/17 16:27:21 | at com.redhat.rhn.taskomatic.task.CobblerSyncTask.execute(CobblerSyncTask.java:78) INFO | jvm 1 | 2017/05/17 16:27:21 | at com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:88) INFO | jvm 1 | 2017/05/17 16:27:21 | at com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:186) INFO | jvm 1 | 2017/05/17 16:27:21 | at org.quartz.core.JobRunShell.run(JobRunShell.java:216) INFO | jvm 1 | 2017/05/17 16:27:21 | at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549) INFO | jvm 1 | 2017/05/17 16:27:21 | 2017-05-17 16:27:21,781 [DefaultQuartzScheduler_Worker-1] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - RuntimeExceptionError trying to sync to cobbler: We had an error trying to login. Varieties of java errors are thrown. But if I try to login to the database with given credentials, this works: # psql -h localhost -p 5432 -U rhnuser -W rhnschema Password for user rhnuser: psql (9.6.3) Type "help" for help. rhnschema=> This I'll assume something missing from java since May, 12th 2017. Java was upgraded to a new version then.
Created attachment 1280011 [details] spacewalk config and logs
I could walk that problem down to various libraries not linking to correct jar-files in /var/lib/tomcat6/webapps/rhn/WEB-INF/lib. After reinstalling packages the wrong links belonged to these links where corrected. I could not track down, how and why these links where not updated. Second thing: After upgrading OpenJDK the default was switched to OpenJDK 1.7 instead of the installed OpenJDK 1.8. After correcting this to OpenJDK 1.8 the java part of spacewalk runs now! I think these errors do not have any relation to spacewalk, but to installation routines: something must have badly gone wrong while installing updates. Maybe it is a yum/dnf- or rpm-problem.
"After reinstalling packages the wrong links belonged to these links where corrected." Can you list what packages you reinstalled? We are facing the same issue with Spacewalk 2.4 (CentOS 7) and OpenJDK 8. -Olli
This BZ closed some time during 2.5, 2.6 or 2.7. Adding to 2.7 tracking bug.
Solved meanwhile. Fixed by later updates.
>Just reinstalled all java, tomcat related stuff, after removing anything including deployed applications. Then reinstalled everything from scratch, copied configs from backup and redeployed applications. -- I's a lot of work, but made the whole thing run again. I've noticed some packages leaving stuff behind, after removing them. These are configs never touched again, even if they do not match the system later on. Most of them are generated "on the fly" some hold java functions leading to crashes later on if loaded.