Bug 1113485 - Using POST in /api resources results in HTTP 500
Summary: Using POST in /api resources results in HTTP 500
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: oVirt
Classification: Retired
Component: ovirt-engine-api
Version: 3.5
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 3.5.0
Assignee: Juan Hernández
QA Contact: Petr Beňas
URL:
Whiteboard: infra
: 1114955 1116764 (view as bug list)
Depends On:
Blocks: 1073943
TreeView+ depends on / blocked
 
Reported: 2014-06-26 09:29 UTC by Idan Shaby
Modified: 2016-02-10 19:30 UTC (History)
11 users (show)

Fixed In Version: ovirt-3.5.0-beta1.1
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-10-17 12:44:49 UTC
oVirt Team: Infra


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 29474 master MERGED restapi: Deploy using modules Never
oVirt gerrit 29578 ovirt-engine-3.5 ABANDONED restapi: Deploy using modules Never

Description Idan Shaby 2014-06-26 09:29:54 UTC
Description of problem:
Any POST request on /api (and not in /ovirt-engine/api) will result in HTTP 500 error due to ClassCastException.

Version-Release number of selected component (if applicable):
oVirt Engine 3.5 compiled from source (commit hash 28ae234)

How reproducible:
100%

Steps to Reproduce:
This bug reproduces with any POST request to /api, here is an example where I first encountered it:
1. Create a local dc.
2. Create a cluster on that dc.
3. Add a host to that cluster.
4. Attempt to add a local storage domain through the REST api:
Send a POST request to http://localhost:8080/api/storagedomains with a body of:
<storage_domain>
  <name>local_sd</name>
  <type>data</type>
  <storage>
    <type>localfs</type>
    <path>/storage/sd</path>
  </storage>
  <host>
    <name>vdshost</name>
  </host>
</storage_domain>

Actual results:
500 Internal Server Error

Exception in server.log:
2014-06-26 12:21:39,405 SEVERE [org.jboss.resteasy.core.SynchronousDispatcher] (http--0.0.0.0-8080-2) Failed executing POST /storagedomains: org.jboss.resteasy.spi.InternalServerErrorException: Bad arguments passed to public abstract javax.ws.rs.core.Response org.ovirt.engine.api.resource.StorageDomainsResource.add(org.ovirt.engine.api.model.StorageDomain)  ( org.ovirt.engine.api.model.StorageDomain org.ovirt.engine.api.model.StorageDomain@5b79202d )
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:196) [resteasy-jaxrs-2.3.2.Final.jar:]
	at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:]
	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:]
	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:]
	at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:]
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:]
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:]
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:]
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:]
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
	at org.jboss.weld.servlet.ConversationPropagationFilter.doFilter(ConversationPropagationFilter.java:62) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
	at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
	at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
	at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
	at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:104) [aaa.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
	at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
	at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
	at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:]
	at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
	at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
	at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]
Caused by: java.lang.IllegalArgumentException: java.lang.ClassCastException@5f607c9
	at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source) [:1.7.0_55]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55]
	at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55]
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:]
	... 49 more

Expected results:
Storage domain should be added.

Additional info:
Dubugging showed a ClassCastException with the message "org.ovirt.engine.api.model.StorageDomain cannot be cast to org.ovirt.engine.api.model.StorageDomain".
Perhaps there are different class loaders for /api and /ovirt-engine/api?

Comment 1 Juan Hernández 2014-06-26 09:56:30 UTC
The RESTAPI application is deployed twice, once for /api and another time for /ovirt-engine/api. This means that classes like o.o.e.a.m.StorageDomain are loaded twice. In theory this shouldn't result in a class cast exception, as the class loaders for WEB-INF/lib of both applications should be isolated, but this is always tricky.

Comment 2 Juan Hernández 2014-06-26 10:05:51 UTC
What version of the application server are you exactly using?

Comment 3 Sandro Bonazzola 2014-06-26 10:19:56 UTC
Juan, looks like a blocker. If you agree, please add this to bug #1073943, thanks.

Comment 4 Juan Hernández 2014-06-26 10:25:40 UTC
I already added it as a blocker.

Comment 5 Juan Hernández 2014-06-26 11:24:48 UTC
This is what is happening:

1. The RESTAPI application is loaded twice, once from the legacy_restapi.war file in order to serve /api, and another time named restapi.war file in order to serve /ovirt-engine/api.

2. The dependencies of the RESTAPI are .jar files inside WEB-INF/lib, in particular the .jar file containing the StorageDomain class is restapi-definition.jar. This file is loaded once for each instance of the application, so there are two class loaders that can provide this class. In theory these class loaders are isolated.

3. The classes of Resteasy are loaded only once, as they are part of the application server.

4. When a resource is requested Resteasy lazily loads the required providers, using the Java SPI mechanism. These providers are stored in a global cache implemented as a singleton by Resteasy.

5. Our own providers (JsonProvider and JAXBProvider) are deployed twice, as a result of the double deployment of the RESTAPI application.

6. Depending on the order of requests Resteasy will load our own providers in different orders, so the content of the global cache of providers depends on the order of requests. For example, if a request to /ovirt-engine/api is made first the global cache will contain the provider from restapi.war. A later request to /api will not load the provider from legacy_restapi.war, as it is already loaded.

7. When the provider requests a class like StorageDomain it will get it from its own class loader. So the provider loaded from legacy_restapi.war will get the class from legacy_restapi.war, regardless of the order of requests.

8. Resteasy doesn't store resources in a global cache, but separated by application. So when Resteasy invokes the StorageDomainResource.add() method on a resource loaded from legacy_restapi.war it may be passing a StorageDomain instance that was created by a provider loaded from restapi.war. So the expected and passed classes are actually different, as they have been loaded by different class loaders.

All in all, the class loader used to load classes like StorageDomain depends on the order of invocation.

Actually, if you restart the application server, and close the webadmin page, things will work correctly, because the first request to /api will load the provider provider from legacy_restapi.war. If you start webadmin then it will send requests to /ovirt-engine/api, and load the provider from restapi.war, triggering this issue.

The solution to this problem is to make sure that RESTAPI classes are loaded only once, which isn't trivial.

This may affect 3.4 as well.

Comment 6 Juan Hernández 2014-07-01 11:35:13 UTC
*** Bug 1114955 has been marked as a duplicate of this bug. ***

Comment 7 Juan Hernández 2014-07-07 12:59:24 UTC
The 3.5 branch will be removed and re-created from master, so moving to MODIFIED.

Comment 8 Juan Hernández 2014-07-14 11:36:20 UTC
*** Bug 1116764 has been marked as a duplicate of this bug. ***

Comment 9 Petr Beňas 2014-08-07 09:19:45 UTC
Verified in ovirt-engine-3.5.0-0.0.master.20140804172041.git23b558e.el6.noarch. The operation is synchronous, but succeeded.

Comment 10 Sandro Bonazzola 2014-10-17 12:44:49 UTC
oVirt 3.5 has been released and should include the fix for this issue.


Note You need to log in before you can comment on or make changes to this bug.