This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 874278 - API method errata.findByCve() returns java exception java.util.ConcurrentModificationException() when listing erratas
API method errata.findByCve() returns java exception java.util.ConcurrentMod...
Status: CLOSED CURRENTRELEASE
Product: Spacewalk
Classification: Community
Component: API (Show other bugs)
1.8
All Linux
medium Severity medium
: ---
: ---
Assigned To: Tomas Lestach
Red Hat Satellite QA List
:
Depends On:
Blocks: space19
  Show dependency treegraph
 
Reported: 2012-11-07 15:34 EST by Marcelo Moreira de Mello
Modified: 2013-03-06 13:35 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 874273
Environment:
Last Closed: 2013-03-06 13:35:12 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Patch proposed (1.49 KB, patch)
2012-11-07 18:56 EST, Marcelo Moreira de Mello
no flags Details | Diff

  None (edit)
Description Marcelo Moreira de Mello 2012-11-07 15:34:16 EST
+++ This bug was initially created as a clone of Bug #874273 +++

Description of problem:

API method errata.findByCve() returns java exception  java.util.ConcurrentModificationException() when listing erratas

 ==> /var/log/tomcat5/catalina.out <==
2012-11-07 16:49:26,252 [TP-Processor2] ERROR com.redhat.rhn.frontend.xmlrpc.BaseHandler - Error calling method: 
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:600)
	at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:108)
	at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123)
	at com.redhat.rhn.frontend.xmlrpc.RhnXmlRpcServer.execute(RhnXmlRpcServer.java:54)
	at com.redhat.rhn.frontend.xmlrpc.XmlRpcServlet.doPost(XmlRpcServlet.java:162)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:142)
	at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:58)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:108)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:55)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
	at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
	at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
	at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
	at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
	at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
	at java.lang.Thread.run(Thread.java:736)
Caused by: 
java.util.ConcurrentModificationException
	at java.util.LinkedList$LinkIterator.next(LinkedList.java:119)
	at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427)
	... 41 more
2012-11-07 16:49:26,253 [TP-Processor2] ERROR com.redhat.rhn.frontend.xmlrpc.BaseHandler - Caused by: 
java.util.ConcurrentModificationException
	at java.util.LinkedList$LinkIterator.next(LinkedList.java:119)
	at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:600)
	at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:108)
	at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123)
	at com.redhat.rhn.frontend.xmlrpc.RhnXmlRpcServer.execute(RhnXmlRpcServer.java:54)
	at com.redhat.rhn.frontend.xmlrpc.XmlRpcServlet.doPost(XmlRpcServlet.java:162)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:142)
	at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:58)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.LocalizedEnvironmentFilter.doFilter(LocalizedEnvironmentFilter.java:67)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.EnvironmentFilter.doFilter(EnvironmentFilter.java:108)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.SessionFilter.doFilter(SessionFilter.java:55)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at com.redhat.rhn.frontend.servlets.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:97)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
	at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
	at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
	at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
	at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
	at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
	at java.lang.Thread.run(Thread.java:736)
Nov 7, 2012 4:49:26 PM redstone.xmlrpc.XmlRpcDispatcher writeError
WARNING: redstone.xmlrpc.XmlRpcFault: unhandled internal exception: null




Version-Release number of selected component (if applicable):

spacewalk-java-1.7.54-100

How reproducible:
100% (using a large DB)


Steps to Reproduce:
1. execute the reproducer (test.py) attached on the case

Actual results:
$ ./test.py 
CVE # CVE-2012-0547
Traceback (most recent call last):
  File "./test.py", line 16, in <module>
    cveData = client.errata.findByCve(token,cve)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1297, in single_request
    return self.parse_response(response)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1473, in parse_response
    return u.close()
  File "/usr/lib64/python2.7/xmlrpclib.py", line 793, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault -1: 'redstone.xmlrpc.XmlRpcFault: unhandled internal exception: null'>


Expected results:


   API works as expected

--- Additional comment from mmello@redhat.com on 2012-11-07 18:23:26 BRST ---

Created attachment 640333 [details]
Reproducer


  Reproducer is available internally

--- Additional comment from mmello@redhat.com on 2012-11-07 18:26:07 BRST ---

   After importing the customer's DB we were able to hit the same issue initially reported by customer:
     
  Investigating the code we have:
   
java/code/src/com/redhat/rhn/frontend/xmlrpc/errata/ErrataHandler.java
---------------------------------
    /**  
     * Lookup the details for errata associated with the given CVE.
     * @param sessionKey session of the logged in user
     * @param cveName name of the CVE
     * @return List of Errata objects
     *
     * @xmlrpc.doc Lookup the details for errata associated with the given CVE
     * (e.g. CVE-2008-3270)
     * @xmlrpc.param #session_key()
     * @xmlrpc.param #param("string", "cveName")
     * @xmlrpc.returntype
     *          #array()
     *              $ErrataSerializer
     *          #array_end()
     */
    public List<Errata> findByCve(String sessionKey, String cveName) {
        // Get the logged in user. We don't care what roles this user has, we
        // just want to make sure the caller is logged in.
        User loggedInUser = getLoggedInUser(sessionKey);

        List<Errata> erratas = ErrataManager.lookupByCVE(cveName);
        for (Errata errata : erratas) {
            // Remove errata that do not apply to the user's org
            if (errata.getOrg() != null &&
                    !errata.getOrg().equals(loggedInUser.getOrg())) {
                erratas.remove(errata);
            }
        }
        return erratas;
    }    

}

java/code/src/com/redhat/rhn/manager/errata/ErrataManager.java
---------------------------------
    /**  
     * Looks up errata by CVE string
     * @param cve errata's CVE string
     * @return Errata if found, otherwise null
     */
    public static List lookupByCVE(String cve) {
        return ErrataFactory.lookupByCVE(cve);
    } 
    
java/code/src/com/redhat/rhn/domain/errata/ErrataFactory.java
---------------------------------
    /**  
     * Finds errata based on CVE string
     * @param cve cve text
     * @return Errata if found, otherwise null
     */
    public static List lookupByCVE(String cve) {
        List retval = new LinkedList();
        SelectMode mode = ModeFactory.getMode("Errata_queries", "erratas_for_cve");
        Map params = new HashMap();
        params.put("cve", cve);
        List result = mode.execute(params);
        Session session = HibernateFactory.getSession();
        for (Iterator iter = result.iterator(); iter.hasNext();) {
            Map row = (Map) iter.next();
            Long rawId = (Long) row.get("id");
            retval.add(session.load(PublishedErrata.class, rawId));
        }
        return retval;
    }  

java/code/src/com/redhat/rhn/common/db/datasource/xml/Errata_queries.xml
---------------------------------
<mode name="erratas_for_cve">
   <query params="cve">
SELECT E.id 
FROM rhnErrata E, rhnCVE C, rhnErrataCVE ec
WHERE C.name = :cve 
AND EC.cve_id = C.id 
AND EC.errata_id = E.id 
   </query>
</mode>


   Executing the SQL manually we can see:

SQL> SELECT E.id 
FROM rhnErrata E, rhnCVE C, rhnErrataCVE ec
WHERE C.name = 'CVE-2012-0547'
AND EC.cve_id = C.id 
AND EC.errata_id = E.id;

	ID
----------
     37388
     37390
     44738
     44745
     15353
     15355
     15356
     15373
     49465
     27856

10 rows selected.

SQL> select ID, ADVISORY_NAME from rhnerrata 
    WHERE ID IN
        ( SELECT E.id FROM rhnErrata E, rhnCVE C, rhnErrataCVE ec
            WHERE C.name = 'CVE-2012-0547'
              AND EC.cve_id = C.id 
              AND EC.errata_id = E.id );  2    3    4    5    6  

	ID ADVISORY_NAME
---------- ------------------------------
     37390 CLA-2012:1223
     49465 CLA-2012:1222
     27856 RHSA-2012:1289
     15353 RHSA-2012:1222
     15355 RHSA-2012:1221
     37388 CLA-2012:1221
     44745 CLA-2012:1289
     44738 CLA-2012:1225
     15356 RHSA-2012:1223
     15373 RHSA-2012:1225

10 rows selected.


   Since database seems to be functional, enabling the rkbloom driver we could see:
   
$ ./test.py 
CVE # CVE-2012-0547
Traceback (most recent call last):
  File "./test.py", line 16, in <module>
    cveData = client.errata.findByCve(token,cve)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1297, in single_request
    return self.parse_response(response)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1473, in parse_response
    return u.close()
  File "/usr/lib64/python2.7/xmlrpclib.py", line 793, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault -1: 'redstone.xmlrpc.XmlRpcFault: unhandled internal exception: null'>

/var/log/tomcat5/catalina.out
---------------------------------
2012-11-07 17:15:02,355 [TP-Processor3] DEBUG net.rkbloom.logdriver.LogPreparedStatement - executing PreparedStatement: 'SELECT E.id
FROM rhnErrata E, rhnCVE C, rhnErrataCVE ec
WHERE C.name = ?
AND EC.cve_id = C.id
AND EC.errata_id = E.id' with bind parameters: {1=CVE-2012-0547}
2012-11-07 17:15:02,385 [TP-Processor3] ERROR com.redhat.rhn.frontend.xmlrpc.BaseHandler - Error calling method: 
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:600)
	at com.redhat.rhn.frontend.xmlrpc.BaseHandler.invoke(BaseHandler.java:108)
	at redstone.xmlrpc.XmlRpcDispatcher.dispatch(XmlRpcDispatcher.java:123)
{ .. SNIP .. }
	at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
	at java.lang.Thread.run(Thread.java:736)
Caused by: 
java.util.ConcurrentModificationException
	at java.util.LinkedList$LinkIterator.next(LinkedList.java:119)
	at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427)
	... 41 more
2012-11-07 17:15:02,387 [TP-Processor3] ERROR com.redhat.rhn.frontend.xmlrpc.BaseHandler - Caused by: 
java.util.ConcurrentModificationException
	at java.util.LinkedList$LinkIterator.next(LinkedList.java:119)
	at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{ ... SNIP ... }
	at java.lang.Thread.run(Thread.java:736)
	


   Due the error highlighted below, it seems we are hitting an issue due a ListConcorrency: 
=======
   Caused by: 
java.util.ConcurrentModificationException
	at java.util.LinkedList$LinkIterator.next(LinkedList.java:119)
	at com.redhat.rhn.frontend.xmlrpc.errata.ErrataHandler.findByCve(ErrataHandler.java:1427)
=======
Comment 1 Marcelo Moreira de Mello 2012-11-07 18:56:21 EST
Created attachment 640478 [details]
Patch proposed


  Upstream patch proposed at https://www.redhat.com/archives/spacewalk-devel/2012-November/msg00012.html
Comment 2 Tomas Lestach 2012-11-08 05:05:05 EST
Hello Marcelo, using your reproducer script I am not able to reproduce the Exception on latest Spacewalk nightly. Can you reproduce it also on a different machine?
Comment 3 Marcelo Moreira de Mello 2012-11-08 14:13:53 EST
  Hello Tomas, 
    
    Initially I got the same results as you. I was able to reproduce the issue into 2 different systems, but only after importing some DB dumps from customers (was able to reproduce using DB dumps from 2 different customers). I guess this happened due the database size. 

Best,
mmello
Comment 6 Tomas Lestach 2012-11-12 11:52:45 EST
Patch applied as:

spacewalk.git: e5dd56a4f3cdcdd67b8332f428082536d0859761


Stephen, that's correct. However I'd like either to see the reproducer or to investigate the issue on to the reproducer machine. I'm not gonna switch the BZ to MODIFIED until we understand the problem. Marcelo, can I have the access to the reproducer machine?
Comment 9 Stephen Herr 2013-03-01 12:07:41 EST
Marking bug as ON_QA since tonight's build of Spacewalk nightly is a release candidate for Spacewalk 1.9.
Comment 10 Stephen Herr 2013-03-06 13:35:12 EST
Spacewalk 1.9 has been released.

https://fedorahosted.org/spacewalk/wiki/ReleaseNotes19

Note You need to log in before you can comment on or make changes to this bug.