Bug 1522939

Summary: Internal Server Error - Syncing Repos to Channel
Product: [Community] Spacewalk Reporter: Taylor Strange <taylorstrange>
Component: WebUIAssignee: Grant Gainey <ggainey>
Status: CLOSED CURRENTRELEASE QA Contact: Red Hat Satellite QA List <satqe-list>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 2.7CC: andres.ofner, angystardust, info, jdostal, tkasparek, Toni.Feric
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-20 12:32:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1564160    

Description Taylor Strange 2017-12-06 19:10:27 UTC
Description of problem:
I receive the following error when syncing a channel for the first time. 

" Internal Server Error

The server experienced a problem which prevented your request from being filled out. It may not be possible to execute this action at this time.

Please help us correct this problem by contacting us with details of how you received this message."


Version-Release number of selected component (if applicable):
2.7

How reproducible:
Reliably reproducible.

Steps to Reproduce:
1.Go to Channels > Manage Software Channels
2. Select Channel > Add / Remove Repos > Update Repos
3. Go to Sync Tab > Sync Now
4. Refresh page and see the error.

Actual results:


Expected results:


Additional info:

Comment 1 Jiří Dostál 2017-12-07 11:06:58 UTC
I've tried your steps but I wasn't able to reproduce the issue. Could you please provide us traceback from /var/log/tomcat*/localhost? 
Thank you

Comment 2 Toni Feric 2018-01-14 00:29:08 UTC
I had the same issue in Spacewalk 2.7 on CentOS 7.4
This seems to be an issue with selinux.
After getting the problem, I found that setting selinux to "permissive", makes the issue go away.

I found these lines in the audit.log:
type=AVC msg=audit(1515888593.542:208): avc:  denied  { read } for  pid=994 comm="java" name="epel_centos7-x64.log" dev="dm-0" ino=25603119 scontext=system_u:system_r:tomcat_t:s0 tcontext=unconfined_u:object_r:spacewalk_log_t:s0 tclass=file
type=AVC msg=audit(1515888593.542:208): avc:  denied  { open } for  pid=994 comm="java" path="/var/log/rhn/reposync/epel_centos7-x64.log" dev="dm-0" ino=25603119 scontext=system_u:system_r:tomcat_t:s0 tcontext=unconfined_u:object_r:spacewalk_log_t:s0 tclass=file


The click-stream in my UI looks slightly different than the one from Taylor:
1. Go to Channels > Manage Software Channels
2. Select Channel
3. In that Channel, select the "Repositories" tab
3. Go to Sync Tab > Sync Now
4. Refresh page and see the error.

Once this has happened, the error will be persistent. Even after restarting Spacewalk, the "Sync" page will not display anymore:
1. Go to Channels > Manage Software Channels
2. Select Channel
3. In that Channel, select the "Repositories" tab
3. Go to Sync Tab and see the error

Traceback from /var/log/tomcat/localhost.2018-01-14.logJan 14, 2018 1:06:17 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [action] in context with path [/rhn] threw exception [java.lang.RuntimeException: File not found: /var/log/rhn/reposync/epel_centos7-x64.log] with root cause
java.lang.RuntimeException: File not found: /var/log/rhn/reposync/epel_centos7-x64.log
	at com.redhat.rhn.common.util.FileUtils.readStringFromFile(FileUtils.java:101)
	at com.redhat.rhn.frontend.action.channel.manage.SyncRepositoriesAction.getLastSyncLog(SyncRepositoriesAction.java:215)
	at com.redhat.rhn.frontend.action.channel.manage.SyncRepositoriesAction.parseSyncLog(SyncRepositoriesAction.java:227)
	at com.redhat.rhn.frontend.action.channel.manage.SyncRepositoriesAction.execute(SyncRepositoriesAction.java:84)
	at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
	at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:229)
	at com.redhat.rhn.frontend.struts.RhnRequestProcessor.process(RhnRequestProcessor.java:105)
	at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1926)
	at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:451)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.AuthFilter.doFilter(AuthFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:57)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)


But the file "/var/log/rhn/reposync/epel_centos7-x64.log is there and readable:# ls -l /var/log/rhn/reposync/epel_centos7-x64.log
-rw-rw----. 1 apache apache 2371085 Jan 14 01:17 /var/log/rhn/reposync/epel_centos7-x64.log

Comment 3 Christian Stankowic 2018-01-24 22:08:00 UTC
Hello,

I'm having the same issue on a fresh CentOS 7.4 and Spacewalk 2.7 installation.

Like Taylor Strange and Toni Feric, I was able to reproduce it simply via the web interface by selecting a repository and sync. Setting SELinux to permissive made the issue disappear.

For my installation it figured out, that the tomcat_t SELinux type lacks permissions to read files with the spacewalk_log_t type. Creating a SELinux module with the following code did the trick for me:

# cat reposync_tomcat.te

module reposync_tomcat 1.0;

require {
        type tomcat_t;
        type spacewalk_log_t;
        class file read;
}


So it looks like the Spacewalk SELinux configuration/modules are missing some flags.

My full troubleshooting is documented here: https://cstan.io/?p=11264&lang=en



With kind regards,
Christian Stankowic.

Comment 4 Andres Ofner 2018-03-02 15:17:04 UTC
Hi,

I had the very same issue, but not only needed read but also open permissions.
Full policy thus here:

--------------------------------------------------------
module reposync_tomcat 1.0;

require {
        type tomcat_t;
        type spacewalk_log_t;
        class file { open read };
}

#============= tomcat_t ==============
allow tomcat_t spacewalk_log_t:file open;
allow tomcat_t spacewalk_log_t:file read;
--------------------------------------------------------

Considering the packages are already ~ 6 months old, I am surprised that has not yet been noticed or fixed upstream, especially as the spacewalk-setup process takes quite some time to get SELinux right in the setup phase (I did an upgrade from 2.4, actually).


regards,
Andres

Comment 5 Tomáš Kašpárek 2018-03-26 08:04:28 UTC
spacewalk.git(master): daf37e6008e2ddea13bc193c1d36a66dd88a87e8

Comment 6 Jiří Dostál 2018-03-26 12:16:50 UTC
Moving ON_QA

Comment 7 Jiří Dostál 2018-04-20 12:32:02 UTC
Spacewalk 2.8 has been released.
https://github.com/spacewalkproject/spacewalk/wiki/ReleaseNotes28