Description of problem: Any POST request on /api (and not in /ovirt-engine/api) will result in HTTP 500 error due to ClassCastException. Version-Release number of selected component (if applicable): oVirt Engine 3.5 compiled from source (commit hash 28ae234) How reproducible: 100% Steps to Reproduce: This bug reproduces with any POST request to /api, here is an example where I first encountered it: 1. Create a local dc. 2. Create a cluster on that dc. 3. Add a host to that cluster. 4. Attempt to add a local storage domain through the REST api: Send a POST request to http://localhost:8080/api/storagedomains with a body of: <storage_domain> <name>local_sd</name> <type>data</type> <storage> <type>localfs</type> <path>/storage/sd</path> </storage> <host> <name>vdshost</name> </host> </storage_domain> Actual results: 500 Internal Server Error Exception in server.log: 2014-06-26 12:21:39,405 SEVERE [org.jboss.resteasy.core.SynchronousDispatcher] (http--0.0.0.0-8080-2) Failed executing POST /storagedomains: org.jboss.resteasy.spi.InternalServerErrorException: Bad arguments passed to public abstract javax.ws.rs.core.Response org.ovirt.engine.api.resource.StorageDomainsResource.add(org.ovirt.engine.api.model.StorageDomain) ( org.ovirt.engine.api.model.StorageDomain org.ovirt.engine.api.model.StorageDomain@5b79202d ) at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:196) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:] at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.jboss.weld.servlet.ConversationPropagationFilter.doFilter(ConversationPropagationFilter.java:62) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.RestApiSessionMgmtFilter.doFilter(RestApiSessionMgmtFilter.java:69) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.EnforceAuthFilter.doFilter(EnforceAuthFilter.java:39) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.LoginFilter.doFilter(LoginFilter.java:73) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.NegotiationFilter.doFilter(NegotiationFilter.java:104) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:75) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.SessionValidationFilter.doFilter(SessionValidationFilter.java:63) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.ovirt.engine.core.aaa.filters.RestApiSessionValidationFilter.doFilter(RestApiSessionValidationFilter.java:31) [aaa.jar:] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:489) [jbossweb-7.0.13.Final.jar:] at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:] at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:] at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55] Caused by: java.lang.IllegalArgumentException: java.lang.ClassCastException@5f607c9 at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source) [:1.7.0_55] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_55] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_55] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:] ... 49 more Expected results: Storage domain should be added. Additional info: Dubugging showed a ClassCastException with the message "org.ovirt.engine.api.model.StorageDomain cannot be cast to org.ovirt.engine.api.model.StorageDomain". Perhaps there are different class loaders for /api and /ovirt-engine/api?
The RESTAPI application is deployed twice, once for /api and another time for /ovirt-engine/api. This means that classes like o.o.e.a.m.StorageDomain are loaded twice. In theory this shouldn't result in a class cast exception, as the class loaders for WEB-INF/lib of both applications should be isolated, but this is always tricky.
What version of the application server are you exactly using?
Juan, looks like a blocker. If you agree, please add this to bug #1073943, thanks.
I already added it as a blocker.
This is what is happening: 1. The RESTAPI application is loaded twice, once from the legacy_restapi.war file in order to serve /api, and another time named restapi.war file in order to serve /ovirt-engine/api. 2. The dependencies of the RESTAPI are .jar files inside WEB-INF/lib, in particular the .jar file containing the StorageDomain class is restapi-definition.jar. This file is loaded once for each instance of the application, so there are two class loaders that can provide this class. In theory these class loaders are isolated. 3. The classes of Resteasy are loaded only once, as they are part of the application server. 4. When a resource is requested Resteasy lazily loads the required providers, using the Java SPI mechanism. These providers are stored in a global cache implemented as a singleton by Resteasy. 5. Our own providers (JsonProvider and JAXBProvider) are deployed twice, as a result of the double deployment of the RESTAPI application. 6. Depending on the order of requests Resteasy will load our own providers in different orders, so the content of the global cache of providers depends on the order of requests. For example, if a request to /ovirt-engine/api is made first the global cache will contain the provider from restapi.war. A later request to /api will not load the provider from legacy_restapi.war, as it is already loaded. 7. When the provider requests a class like StorageDomain it will get it from its own class loader. So the provider loaded from legacy_restapi.war will get the class from legacy_restapi.war, regardless of the order of requests. 8. Resteasy doesn't store resources in a global cache, but separated by application. So when Resteasy invokes the StorageDomainResource.add() method on a resource loaded from legacy_restapi.war it may be passing a StorageDomain instance that was created by a provider loaded from restapi.war. So the expected and passed classes are actually different, as they have been loaded by different class loaders. All in all, the class loader used to load classes like StorageDomain depends on the order of invocation. Actually, if you restart the application server, and close the webadmin page, things will work correctly, because the first request to /api will load the provider provider from legacy_restapi.war. If you start webadmin then it will send requests to /ovirt-engine/api, and load the provider from restapi.war, triggering this issue. The solution to this problem is to make sure that RESTAPI classes are loaded only once, which isn't trivial. This may affect 3.4 as well.
*** Bug 1114955 has been marked as a duplicate of this bug. ***
The 3.5 branch will be removed and re-created from master, so moving to MODIFIED.
*** Bug 1116764 has been marked as a duplicate of this bug. ***
Verified in ovirt-engine-3.5.0-0.0.master.20140804172041.git23b558e.el6.noarch. The operation is synchronous, but succeeded.
oVirt 3.5 has been released and should include the fix for this issue.