Bug 1522939 - Internal Server Error - Syncing Repos to Channel
Internal Server Error - Syncing Repos to Channel
Status: CLOSED CURRENTRELEASE
Product: Spacewalk
Classification: Community
Component: WebUI (Show other bugs)
2.7
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Grant Gainey
Red Hat Satellite QA List
:
Depends On:
Blocks: space28
  Show dependency treegraph
 
Reported: 2017-12-06 14:10 EST by Taylor Strange
Modified: 2018-04-20 08:32 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-04-20 08:32:02 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Taylor Strange 2017-12-06 14:10:27 EST
Description of problem:
I receive the following error when syncing a channel for the first time. 

" Internal Server Error

The server experienced a problem which prevented your request from being filled out. It may not be possible to execute this action at this time.

Please help us correct this problem by contacting us with details of how you received this message."


Version-Release number of selected component (if applicable):
2.7

How reproducible:
Reliably reproducible.

Steps to Reproduce:
1.Go to Channels > Manage Software Channels
2. Select Channel > Add / Remove Repos > Update Repos
3. Go to Sync Tab > Sync Now
4. Refresh page and see the error.

Actual results:


Expected results:


Additional info:
Comment 1 Jiří Dostál 2017-12-07 06:06:58 EST
I've tried your steps but I wasn't able to reproduce the issue. Could you please provide us traceback from /var/log/tomcat*/localhost? 
Thank you
Comment 2 Toni Feric 2018-01-13 19:29:08 EST
I had the same issue in Spacewalk 2.7 on CentOS 7.4
This seems to be an issue with selinux.
After getting the problem, I found that setting selinux to "permissive", makes the issue go away.

I found these lines in the audit.log:
type=AVC msg=audit(1515888593.542:208): avc:  denied  { read } for  pid=994 comm="java" name="epel_centos7-x64.log" dev="dm-0" ino=25603119 scontext=system_u:system_r:tomcat_t:s0 tcontext=unconfined_u:object_r:spacewalk_log_t:s0 tclass=file
type=AVC msg=audit(1515888593.542:208): avc:  denied  { open } for  pid=994 comm="java" path="/var/log/rhn/reposync/epel_centos7-x64.log" dev="dm-0" ino=25603119 scontext=system_u:system_r:tomcat_t:s0 tcontext=unconfined_u:object_r:spacewalk_log_t:s0 tclass=file


The click-stream in my UI looks slightly different than the one from Taylor:
1. Go to Channels > Manage Software Channels
2. Select Channel
3. In that Channel, select the "Repositories" tab
3. Go to Sync Tab > Sync Now
4. Refresh page and see the error.

Once this has happened, the error will be persistent. Even after restarting Spacewalk, the "Sync" page will not display anymore:
1. Go to Channels > Manage Software Channels
2. Select Channel
3. In that Channel, select the "Repositories" tab
3. Go to Sync Tab and see the error

Traceback from /var/log/tomcat/localhost.2018-01-14.logJan 14, 2018 1:06:17 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [action] in context with path [/rhn] threw exception [java.lang.RuntimeException: File not found: /var/log/rhn/reposync/epel_centos7-x64.log] with root cause
java.lang.RuntimeException: File not found: /var/log/rhn/reposync/epel_centos7-x64.log
	at com.redhat.rhn.common.util.FileUtils.readStringFromFile(FileUtils.java:101)
	at com.redhat.rhn.frontend.action.channel.manage.SyncRepositoriesAction.getLastSyncLog(SyncRepositoriesAction.java:215)
	at com.redhat.rhn.frontend.action.channel.manage.SyncRepositoriesAction.parseSyncLog(SyncRepositoriesAction.java:227)
	at com.redhat.rhn.frontend.action.channel.manage.SyncRepositoriesAction.execute(SyncRepositoriesAction.java:84)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:229)
	at com.redhat.rhn.frontend.struts.RhnRequestProcessor.process(RhnRequestProcessor.java:105)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1926)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:451)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.AuthFilter.doFilter(AuthFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:57)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)


But the file "/var/log/rhn/reposync/epel_centos7-x64.log is there and readable:# ls -l /var/log/rhn/reposync/epel_centos7-x64.log
-rw-rw----. 1 apache apache 2371085 Jan 14 01:17 /var/log/rhn/reposync/epel_centos7-x64.log
Comment 3 Christian Stankowic 2018-01-24 17:08:00 EST
Hello,

I'm having the same issue on a fresh CentOS 7.4 and Spacewalk 2.7 installation.

Like Taylor Strange and Toni Feric, I was able to reproduce it simply via the web interface by selecting a repository and sync. Setting SELinux to permissive made the issue disappear.

For my installation it figured out, that the tomcat_t SELinux type lacks permissions to read files with the spacewalk_log_t type. Creating a SELinux module with the following code did the trick for me:

# cat reposync_tomcat.te

module reposync_tomcat 1.0;

require {
        type tomcat_t;
        type spacewalk_log_t;
        class file read;
}


So it looks like the Spacewalk SELinux configuration/modules are missing some flags.

My full troubleshooting is documented here: https://cstan.io/?p=11264&lang=en



With kind regards,
Christian Stankowic.
Comment 4 Andres Ofner 2018-03-02 10:17:04 EST
Hi,

I had the very same issue, but not only needed read but also open permissions.
Full policy thus here:

--------------------------------------------------------
module reposync_tomcat 1.0;

require {
        type tomcat_t;
        type spacewalk_log_t;
        class file { open read };
}

#============= tomcat_t ==============
allow tomcat_t spacewalk_log_t:file open;
allow tomcat_t spacewalk_log_t:file read;
--------------------------------------------------------

Considering the packages are already ~ 6 months old, I am surprised that has not yet been noticed or fixed upstream, especially as the spacewalk-setup process takes quite some time to get SELinux right in the setup phase (I did an upgrade from 2.4, actually).


regards,
Andres
Comment 5 Tomáš Kašpárek 2018-03-26 04:04:28 EDT
spacewalk.git(master): daf37e6008e2ddea13bc193c1d36a66dd88a87e8
Comment 6 Jiří Dostál 2018-03-26 08:16:50 EDT
Moving ON_QA
Comment 7 Jiří Dostál 2018-04-20 08:32:02 EDT
Spacewalk 2.8 has been released.
https://github.com/spacewalkproject/spacewalk/wiki/ReleaseNotes28

Note You need to log in before you can comment on or make changes to this bug.