+++ This bug was initially created as a clone of Bug #874273 +++ Description of problem: API method errata.findByCve() returns java exception java.util.ConcurrentModificationException() when listing erratas ==> /var/log/tomcat5/catalina.out <== 2012-11-07 16:49:26,252 [TP-Processor2] ERROR com.redhat.rhn.frontend.xmlrpc.BaseHandler - Error calling method: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:600) at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:108) at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123) at com.redhat.rhn.frontend.xmlrpc.RhnXmlRpcServer.execute(RhnXmlRpcServer.java:54) at com.redhat.rhn.frontend.xmlrpc.XmlRpcServlet.doPost(XmlRpcServlet.java:162) at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:142) at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:108) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685) at java.lang.Thread.run(Thread.java:736) Caused by: java.util.ConcurrentModificationException at java.util.LinkedList$LinkIterator.next(LinkedList.java:119) at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427) ... 41 more 2012-11-07 16:49:26,253 [TP-Processor2] ERROR com.redhat.rhn.frontend.xmlrpc.BaseHandler - Caused by: java.util.ConcurrentModificationException at java.util.LinkedList$LinkIterator.next(LinkedList.java:119) at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:600) at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:108) at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123) at com.redhat.rhn.frontend.xmlrpc.RhnXmlRpcServer.execute(RhnXmlRpcServer.java:54) at com.redhat.rhn.frontend.xmlrpc.XmlRpcServlet.doPost(XmlRpcServlet.java:162) at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:142) at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:108) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:55) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685) at java.lang.Thread.run(Thread.java:736) Nov 7, 2012 4:49:26 PM redstone.xmlrpc.XmlRpcDispatcher writeError WARNING: redstone.xmlrpc.XmlRpcFault: unhandled internal exception: null Version-Release number of selected component (if applicable): spacewalk-java-1.7.54-100 How reproducible: 100% (using a large DB) Steps to Reproduce: 1. execute the reproducer (test.py) attached on the case Actual results: $ ./test.py CVE # CVE-2012-0547 Traceback (most recent call last): File "./test.py", line 16, in <module> cveData = client.errata.findByCve(token,cve) File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1297, in single_request return self.parse_response(response) File "/usr/lib64/python2.7/xmlrpclib.py", line 1473, in parse_response return u.close() File "/usr/lib64/python2.7/xmlrpclib.py", line 793, in close raise Fault(**self._stack[0]) xmlrpclib.Fault: <Fault -1: 'redstone.xmlrpc.XmlRpcFault: unhandled internal exception: null'> Expected results: API works as expected --- Additional comment from mmello on 2012-11-07 18:23:26 BRST --- Created attachment 640333 [details] Reproducer Reproducer is available internally --- Additional comment from mmello on 2012-11-07 18:26:07 BRST --- After importing the customer's DB we were able to hit the same issue initially reported by customer: Investigating the code we have: java/code/src/com/redhat/rhn/frontend/xmlrpc/errata/ErrataHandler.java --------------------------------- /** * Lookup the details for errata associated with the given CVE. * @param sessionKey session of the logged in user * @param cveName name of the CVE * @return List of Errata objects * * @xmlrpc.doc Lookup the details for errata associated with the given CVE * (e.g. CVE-2008-3270) * @xmlrpc.param #session_key() * @xmlrpc.param #param("string", "cveName") * @xmlrpc.returntype * #array() * $ErrataSerializer * #array_end() */ public List<Errata> findByCve(String sessionKey, String cveName) { // Get the logged in user. We don't care what roles this user has, we // just want to make sure the caller is logged in. User loggedInUser = getLoggedInUser(sessionKey); List<Errata> erratas = ErrataManager.lookupByCVE(cveName); for (Errata errata : erratas) { // Remove errata that do not apply to the user's org if (errata.getOrg() != null && !errata.getOrg().equals(loggedInUser.getOrg())) { erratas.remove(errata); } } return erratas; } } java/code/src/com/redhat/rhn/manager/errata/ErrataManager.java --------------------------------- /** * Looks up errata by CVE string * @param cve errata's CVE string * @return Errata if found, otherwise null */ public static List lookupByCVE(String cve) { return ErrataFactory.lookupByCVE(cve); } java/code/src/com/redhat/rhn/domain/errata/ErrataFactory.java --------------------------------- /** * Finds errata based on CVE string * @param cve cve text * @return Errata if found, otherwise null */ public static List lookupByCVE(String cve) { List retval = new LinkedList(); SelectMode mode = ModeFactory.getMode("Errata_queries", "erratas_for_cve"); Map params = new HashMap(); params.put("cve", cve); List result = mode.execute(params); Session session = HibernateFactory.getSession(); for (Iterator iter = result.iterator(); iter.hasNext();) { Map row = (Map) iter.next(); Long rawId = (Long) row.get("id"); retval.add(session.load(PublishedErrata.class, rawId)); } return retval; } java/code/src/com/redhat/rhn/common/db/datasource/xml/Errata_queries.xml --------------------------------- <mode name="erratas_for_cve"> <query params="cve"> SELECT E.id FROM rhnErrata E, rhnCVE C, rhnErrataCVE ec WHERE C.name = :cve AND EC.cve_id = C.id AND EC.errata_id = E.id </query> </mode> Executing the SQL manually we can see: SQL> SELECT E.id FROM rhnErrata E, rhnCVE C, rhnErrataCVE ec WHERE C.name = 'CVE-2012-0547' AND EC.cve_id = C.id AND EC.errata_id = E.id; ID ---------- 37388 37390 44738 44745 15353 15355 15356 15373 49465 27856 10 rows selected. SQL> select ID, ADVISORY_NAME from rhnerrata WHERE ID IN ( SELECT E.id FROM rhnErrata E, rhnCVE C, rhnErrataCVE ec WHERE C.name = 'CVE-2012-0547' AND EC.cve_id = C.id AND EC.errata_id = E.id ); 2 3 4 5 6 ID ADVISORY_NAME ---------- ------------------------------ 37390 CLA-2012:1223 49465 CLA-2012:1222 27856 RHSA-2012:1289 15353 RHSA-2012:1222 15355 RHSA-2012:1221 37388 CLA-2012:1221 44745 CLA-2012:1289 44738 CLA-2012:1225 15356 RHSA-2012:1223 15373 RHSA-2012:1225 10 rows selected. Since database seems to be functional, enabling the rkbloom driver we could see: $ ./test.py CVE # CVE-2012-0547 Traceback (most recent call last): File "./test.py", line 16, in <module> cveData = client.errata.findByCve(token,cve) File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1297, in single_request return self.parse_response(response) File "/usr/lib64/python2.7/xmlrpclib.py", line 1473, in parse_response return u.close() File "/usr/lib64/python2.7/xmlrpclib.py", line 793, in close raise Fault(**self._stack[0]) xmlrpclib.Fault: <Fault -1: 'redstone.xmlrpc.XmlRpcFault: unhandled internal exception: null'> /var/log/tomcat5/catalina.out --------------------------------- 2012-11-07 17:15:02,355 [TP-Processor3] DEBUG net.rkbloom.logdriver.LogPreparedStatement - executing PreparedStatement: 'SELECT E.id FROM rhnErrata E, rhnCVE C, rhnErrataCVE ec WHERE C.name = ? AND EC.cve_id = C.id AND EC.errata_id = E.id' with bind parameters: {1=CVE-2012-0547} 2012-11-07 17:15:02,385 [TP-Processor3] ERROR com.redhat.rhn.frontend.xmlrpc.BaseHandler - Error calling method: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:600) at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:108) at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123) { .. SNIP .. } at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685) at java.lang.Thread.run(Thread.java:736) Caused by: java.util.ConcurrentModificationException at java.util.LinkedList$LinkIterator.next(LinkedList.java:119) at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427) ... 41 more 2012-11-07 17:15:02,387 [TP-Processor3] ERROR com.redhat.rhn.frontend.xmlrpc.BaseHandler - Caused by: java.util.ConcurrentModificationException at java.util.LinkedList$LinkIterator.next(LinkedList.java:119) at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) { ... SNIP ... } at java.lang.Thread.run(Thread.java:736) Due the error highlighted below, it seems we are hitting an issue due a ListConcorrency: ======= Caused by: java.util.ConcurrentModificationException at java.util.LinkedList$LinkIterator.next(LinkedList.java:119) at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427) =======
Created attachment 640478 [details] Patch proposed Upstream patch proposed at https://www.redhat.com/archives/spacewalk-devel/2012-November/msg00012.html
Hello Marcelo, using your reproducer script I am not able to reproduce the Exception on latest Spacewalk nightly. Can you reproduce it also on a different machine?
Hello Tomas, Initially I got the same results as you. I was able to reproduce the issue into 2 different systems, but only after importing some DB dumps from customers (was able to reproduce using DB dumps from 2 different customers). I guess this happened due the database size. Best, mmello
Patch applied as: spacewalk.git: e5dd56a4f3cdcdd67b8332f428082536d0859761 Stephen, that's correct. However I'd like either to see the reproducer or to investigate the issue on to the reproducer machine. I'm not gonna switch the BZ to MODIFIED until we understand the problem. Marcelo, can I have the access to the reproducer machine?
Marking bug as ON_QA since tonight's build of Spacewalk nightly is a release candidate for Spacewalk 1.9.
Spacewalk 1.9 has been released. https://fedorahosted.org/spacewalk/wiki/ReleaseNotes19