Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 874278

Summary: API method errata.findByCve() returns java exception java.util.ConcurrentModificationException() when listing erratas
Product: [Community] Spacewalk Reporter: Marcelo Moreira de Mello <mmello>
Component: APIAssignee: Tomas Lestach <tlestach>
Status: CLOSED CURRENTRELEASE QA Contact: Red Hat Satellite QA List <satqe-list>
Severity: medium Docs Contact:
Priority: medium    
Version: 1.8CC: mmello
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 874273 Environment:
Last Closed: 2013-03-06 18:35:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 917805    
Attachments:
Description Flags
Patch proposed none

Description Marcelo Moreira de Mello 2012-11-07 20:34:16 UTC 
+++ This bug was initially created as a clone of Bug #874273 +++

Description of problem:

API method errata.findByCve() returns java exception  java.util.ConcurrentModificationException() when listing erratas

 ==> /var/log/tomcat5/catalina.out <==
2012-11-07 16:49:26,252 [TP-Processor2] ERROR com.redhat.rhn.frontend.xmlrpc.BaseHandler - Error calling method: 
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:600)
	at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:108)
	at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123)
	at com.redhat.rhn.frontend.xmlrpc.RhnXmlRpcServer.execute(RhnXmlRpcServer.java:54)
	at com.redhat.rhn.frontend.xmlrpc.XmlRpcServlet.doPost(XmlRpcServlet.java:162)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:142)
	at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:58)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:108)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:55)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
	at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
	at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
	at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
	at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
	at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
	at java.lang.Thread.run(Thread.java:736)
Caused by: 
java.util.ConcurrentModificationException
	at java.util.LinkedList$LinkIterator.next(LinkedList.java:119)
	at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427)
	... 41 more
2012-11-07 16:49:26,253 [TP-Processor2] ERROR com.redhat.rhn.frontend.xmlrpc.BaseHandler - Caused by: 
java.util.ConcurrentModificationException
	at java.util.LinkedList$LinkIterator.next(LinkedList.java:119)
	at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:600)
	at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:108)
	at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123)
	at com.redhat.rhn.frontend.xmlrpc.RhnXmlRpcServer.execute(RhnXmlRpcServer.java:54)
	at com.redhat.rhn.frontend.xmlrpc.XmlRpcServlet.doPost(XmlRpcServlet.java:162)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:142)
	at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:58)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:108)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:55)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
	at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
	at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
	at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
	at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
	at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
	at java.lang.Thread.run(Thread.java:736)
Nov 7, 2012 4:49:26 PM redstone.xmlrpc.XmlRpcDispatcher writeError
WARNING: redstone.xmlrpc.XmlRpcFault: unhandled internal exception: null




Version-Release number of selected component (if applicable):

spacewalk-java-1.7.54-100

How reproducible:
100% (using a large DB)


Steps to Reproduce:
1. execute the reproducer (test.py) attached on the case

Actual results:
$ ./test.py 
CVE # CVE-2012-0547
Traceback (most recent call last):
  File "./test.py", line 16, in <module>
    cveData = client.errata.findByCve(token,cve)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1297, in single_request
    return self.parse_response(response)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1473, in parse_response
    return u.close()
  File "/usr/lib64/python2.7/xmlrpclib.py", line 793, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault -1: 'redstone.xmlrpc.XmlRpcFault: unhandled internal exception: null'>


Expected results:


   API works as expected

--- Additional comment from mmello on 2012-11-07 18:23:26 BRST ---

Created attachment 640333 [details]
Reproducer


  Reproducer is available internally

--- Additional comment from mmello on 2012-11-07 18:26:07 BRST ---

   After importing the customer's DB we were able to hit the same issue initially reported by customer:
     
  Investigating the code we have:
   
java/code/src/com/redhat/rhn/frontend/xmlrpc/errata/ErrataHandler.java
---------------------------------
    /**  
     * Lookup the details for errata associated with the given CVE.
     * @param sessionKey session of the logged in user
     * @param cveName name of the CVE
     * @return List of Errata objects
     *
     * @xmlrpc.doc Lookup the details for errata associated with the given CVE
     * (e.g. CVE-2008-3270)
     * @xmlrpc.param #session_key()
     * @xmlrpc.param #param("string", "cveName")
     * @xmlrpc.returntype
     *          #array()
     *              $ErrataSerializer
     *          #array_end()
     */
    public List<Errata> findByCve(String sessionKey, String cveName) {
        // Get the logged in user. We don't care what roles this user has, we
        // just want to make sure the caller is logged in.
        User loggedInUser = getLoggedInUser(sessionKey);

        List<Errata> erratas = ErrataManager.lookupByCVE(cveName);
        for (Errata errata : erratas) {
            // Remove errata that do not apply to the user's org
            if (errata.getOrg() != null &&
                    !errata.getOrg().equals(loggedInUser.getOrg())) {
                erratas.remove(errata);
            }
        }
        return erratas;
    }    

}

java/code/src/com/redhat/rhn/manager/errata/ErrataManager.java
---------------------------------
    /**  
     * Looks up errata by CVE string
     * @param cve errata's CVE string
     * @return Errata if found, otherwise null
     */
    public static List lookupByCVE(String cve) {
        return ErrataFactory.lookupByCVE(cve);
    } 
    
java/code/src/com/redhat/rhn/domain/errata/ErrataFactory.java
---------------------------------
    /**  
     * Finds errata based on CVE string
     * @param cve cve text
     * @return Errata if found, otherwise null
     */
    public static List lookupByCVE(String cve) {
        List retval = new LinkedList();
        SelectMode mode = ModeFactory.getMode("Errata_queries", "erratas_for_cve");
        Map params = new HashMap();
        params.put("cve", cve);
        List result = mode.execute(params);
        Session session = HibernateFactory.getSession();
        for (Iterator iter = result.iterator(); iter.hasNext();) {
            Map row = (Map) iter.next();
            Long rawId = (Long) row.get("id");
            retval.add(session.load(PublishedErrata.class, rawId));
        }
        return retval;
    }  

java/code/src/com/redhat/rhn/common/db/datasource/xml/Errata_queries.xml
---------------------------------
<mode name="erratas_for_cve">
   <query params="cve">
SELECT E.id 
FROM rhnErrata E, rhnCVE C, rhnErrataCVE ec
WHERE C.name = :cve 
AND EC.cve_id = C.id 
AND EC.errata_id = E.id 
   </query>
</mode>


   Executing the SQL manually we can see:

SQL> SELECT E.id 
FROM rhnErrata E, rhnCVE C, rhnErrataCVE ec
WHERE C.name = 'CVE-2012-0547'
AND EC.cve_id = C.id 
AND EC.errata_id = E.id;

	ID
----------
     37388
     37390
     44738
     44745
     15353
     15355
     15356
     15373
     49465
     27856

10 rows selected.

SQL> select ID, ADVISORY_NAME from rhnerrata 
    WHERE ID IN
        ( SELECT E.id FROM rhnErrata E, rhnCVE C, rhnErrataCVE ec
            WHERE C.name = 'CVE-2012-0547'
              AND EC.cve_id = C.id 
              AND EC.errata_id = E.id );  2    3    4    5    6  

	ID ADVISORY_NAME
---------- ------------------------------
     37390 CLA-2012:1223
     49465 CLA-2012:1222
     27856 RHSA-2012:1289
     15353 RHSA-2012:1222
     15355 RHSA-2012:1221
     37388 CLA-2012:1221
     44745 CLA-2012:1289
     44738 CLA-2012:1225
     15356 RHSA-2012:1223
     15373 RHSA-2012:1225

10 rows selected.


   Since database seems to be functional, enabling the rkbloom driver we could see:
   
$ ./test.py 
CVE # CVE-2012-0547
Traceback (most recent call last):
  File "./test.py", line 16, in <module>
    cveData = client.errata.findByCve(token,cve)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1297, in single_request
    return self.parse_response(response)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1473, in parse_response
    return u.close()
  File "/usr/lib64/python2.7/xmlrpclib.py", line 793, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault -1: 'redstone.xmlrpc.XmlRpcFault: unhandled internal exception: null'>

/var/log/tomcat5/catalina.out
---------------------------------
2012-11-07 17:15:02,355 [TP-Processor3] DEBUG net.rkbloom.logdriver.LogPreparedStatement - executing PreparedStatement: 'SELECT E.id
FROM rhnErrata E, rhnCVE C, rhnErrataCVE ec
WHERE C.name = ?
AND EC.cve_id = C.id
AND EC.errata_id = E.id' with bind parameters: {1=CVE-2012-0547}
2012-11-07 17:15:02,385 [TP-Processor3] ERROR com.redhat.rhn.frontend.xmlrpc.BaseHandler - Error calling method: 
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:600)
	at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:108)
	at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123)
{ .. SNIP .. }
	at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
	at java.lang.Thread.run(Thread.java:736)
Caused by: 
java.util.ConcurrentModificationException
	at java.util.LinkedList$LinkIterator.next(LinkedList.java:119)
	at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427)
	... 41 more
2012-11-07 17:15:02,387 [TP-Processor3] ERROR com.redhat.rhn.frontend.xmlrpc.BaseHandler - Caused by: 
java.util.ConcurrentModificationException
	at java.util.LinkedList$LinkIterator.next(LinkedList.java:119)
	at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{ ... SNIP ... }
	at java.lang.Thread.run(Thread.java:736)
	


   Due the error highlighted below, it seems we are hitting an issue due a ListConcorrency: 
=======
   Caused by: 
java.util.ConcurrentModificationException
	at java.util.LinkedList$LinkIterator.next(LinkedList.java:119)
	at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427)
=======
Comment 1 Marcelo Moreira de Mello 2012-11-07 23:56:21 UTC 
Created attachment 640478 [details]
Patch proposed


  Upstream patch proposed at https://www.redhat.com/archives/spacewalk-devel/2012-November/msg00012.html
Comment 2 Tomas Lestach 2012-11-08 10:05:05 UTC 
Hello Marcelo, using your reproducer script I am not able to reproduce the Exception on latest Spacewalk nightly. Can you reproduce it also on a different machine?
Comment 3 Marcelo Moreira de Mello 2012-11-08 19:13:53 UTC 
  Hello Tomas, 
    
    Initially I got the same results as you. I was able to reproduce the issue into 2 different systems, but only after importing some DB dumps from customers (was able to reproduce using DB dumps from 2 different customers). I guess this happened due the database size. 

Best,
mmello
Comment 6 Tomas Lestach 2012-11-12 16:52:45 UTC 
Patch applied as:

spacewalk.git: e5dd56a4f3cdcdd67b8332f428082536d0859761


Stephen, that's correct. However I'd like either to see the reproducer or to investigate the issue on to the reproducer machine. I'm not gonna switch the BZ to MODIFIED until we understand the problem. Marcelo, can I have the access to the reproducer machine?
Comment 9 Stephen Herr 2013-03-01 17:07:41 UTC 
Marking bug as ON_QA since tonight's build of Spacewalk nightly is a release candidate for Spacewalk 1.9.
Comment 10 Stephen Herr 2013-03-06 18:35:12 UTC 
Spacewalk 1.9 has been released.

https://fedorahosted.org/spacewalk/wiki/ReleaseNotes19