+++ This bug was initially created as a clone of Bug #1580319 +++
Description of problem:
With latest 3.6.13, can't reach the engine web APIs. server.log:
2018-05-21 12:15:17,210 INFO [org.quartz.core.JobRunShell] (DefaultQuartzScheduler_Worker-16) Job DEFAULT.org.ovirt.engine.core.bll.tasks.CommandExecutor.invokeCallbackMethods#-9223372036854775787 threw a JobExecutionException: : org.quartz.JobExecutionException: failed to execute job
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) [:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121]
at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81) [scheduler.jar:]
at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52) [scheduler.jar:]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:]
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:]
Version-Release number of selected component (if applicable):
[root@jenkins-vm-03 ~]# rpm -q rhevm
rhevm-3.6.13.1-0.1.el6.noarch
[root@jenkins-vm-03 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.9 (Santiago)
How reproducible:
1/1
Actual results:
Web APIs don't function, we get 404 http error
Additional info:
httpd and ovirt-engine are running:
[root@jenkins-vm-03 ~]# service httpd status
httpd (pid 2077) is running...
[root@jenkins-vm-03 ~]# service ovirt-engine status
ovirt-engine (pid 13255) is running...
The setup is alive, please contact me for its details
--- Additional comment from Martin Perina on 2018-05-21 06:35:25 EDT ---
Below error started the issue:
2018-05-19 23:26:31,255 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 51) MSC000001: Failed to start service jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START: java.lang.IllegalStateException: JBAS011048: Failed to construct component instance
at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:57) [jboss-as-ee.jar:7.5.20.Final-redhat-1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121]
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Caused by: java.lang.IllegalStateException: JBAS011048: Failed to construct component instance
at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:164) [jboss-as-ee.jar:7.5.20.Final-redhat-1]
at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:135) [jboss-as-ee.jar:7.5.20.Final-redhat-1]
at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:90) [jboss-as-ee.jar:7.5.20.Final-redhat-1]
at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:122)
at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:137)
at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54) [jboss-as-ee.jar:7.5.20.Final-redhat-1]
... 6 more
Caused by: javax.ejb.EJBException: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: Illegal type (org.ovirt.engine.core.common.action.AddImageFromScratchParameters) to deserialize: prevented for security reasons
at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:191)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:281)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:353)
at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:67)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:70) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:162) [jboss-as-ee.jar:7.5.20.Final-redhat-1]
... 11 more
Caused by: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: Illegal type (org.ovirt.engine.core.common.action.AddImageFromScratchParameters) to deserialize: prevented for security reasons
at org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer.readJsonString(JsonObjectDeserializer.java:101)
at org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer.deserialize(JsonObjectDeserializer.java:57)
at org.ovirt.engine.core.dao.CommandEntityDaoImpl.deserializeParameters(CommandEntityDaoImpl.java:150)
at org.ovirt.engine.core.dao.CommandEntityDaoImpl.access$000(CommandEntityDaoImpl.java:32)
at org.ovirt.engine.core.dao.CommandEntityDaoImpl$1.mapRow(CommandEntityDaoImpl.java:71)
at org.ovirt.engine.core.dao.CommandEntityDaoImpl$1.mapRow(CommandEntityDaoImpl.java:58)
at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92)
at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:1)
at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:649)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:587)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706)
at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154)
at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120)
at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181)
at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:147)
at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:109)
at org.ovirt.engine.core.dao.DefaultReadDao.getAll(DefaultReadDao.java:77)
at org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.initializeCache(CommandsCacheImpl.java:33)
at org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.get(CommandsCacheImpl.java:50)
at org.ovirt.engine.core.bll.tasks.CommandCoordinatorImpl.getCommandEntity(CommandCoordinatorImpl.java:132)
at org.ovirt.engine.core.bll.tasks.CoCoAsyncTaskHelper.getCommandEntity(CoCoAsyncTaskHelper.java:160)
at org.ovirt.engine.core.bll.tasks.CoCoAsyncTaskHelper.getAllAsyncTasksFromDb(CoCoAsyncTaskHelper.java:153)
at org.ovirt.engine.core.bll.tasks.CommandCoordinatorImpl.getAllAsyncTasksFromDb(CommandCoordinatorImpl.java:292)
at org.ovirt.engine.core.bll.tasks.AsyncTaskManager.initAsyncTaskManager(AsyncTaskManager.java:113)
at org.ovirt.engine.core.bll.tasks.CommandCoordinatorUtil.initAsyncTaskManager(CommandCoordinatorUtil.java:49)
at org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean.create(InitBackendServicesOnStartupBean.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121]
at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96) [jboss-as-ee.jar:7.5.20.Final-redhat-1]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:116)
at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:107)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:93)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:67)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.as.ee.component.ComponentInstantiatorInterceptor.processInvocation(ComponentInstantiatorInterceptor.java:76) [jboss-as-ee.jar:7.5.20.Final-redhat-1]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:92)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.5.20.Final-redhat-1]
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1]
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279)
... 22 more
Caused by: org.codehaus.jackson.map.JsonMappingException: Illegal type (org.ovirt.engine.core.common.action.AddImageFromScratchParameters) to deserialize: prevented for security reasons
at org.codehaus.jackson.map.deser.BeanDeserializerFactory.checkLegalTypes(BeanDeserializerFactory.java:1521)
at org.codehaus.jackson.map.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:655)
at org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:401)
at org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:310)
at org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:290)
at org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:159)
at org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:180)
at org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2829)
at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2728)
at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863)
at org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer.readJsonString(JsonObjectDeserializer.java:99)
... 73 more
2018-05-19 23:26:31,266 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014612: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"JBAS014671: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.InitBackendServicesOnStartupBean.START" => "org.jboss.msc.service.StartException in service jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.InitBackendServicesOnStartupBean.START: java.lang.IllegalStateException: JBAS011048: Failed to construct component instance
Caused by: java.lang.IllegalStateException: JBAS011048: Failed to construct component instance
Caused by: javax.ejb.EJBException: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: Illegal type (org.ovirt.engine.core.common.action.AddImageFromScratchParameters) to deserialize: prevented for security reasons
Caused by: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: Illegal type (org.ovirt.engine.core.common.action.AddImageFromScratchParameters) to deserialize: prevented for security reasons
Caused by: org.codehaus.jackson.map.JsonMappingException: Illegal type (org.ovirt.engine.core.common.action.AddImageFromScratchParameters) to deserialize: prevented for security reasons"}}
This is similar error as seen in BZ1577407, could you please try with EAP 6.4.19?
--- Additional comment from Martin Perina on 2018-05-21 07:00:28 EDT ---
I've just verified that downgrading jackson library included in EAP 6.4.19 fixed the issue, I will open EAP bug and attach it to this one and BZ1577407
--- Additional comment from Martin Perina on 2018-05-21 07:24:23 EDT ---
Workaround: use EAP 6.4.19 from http://download-ipv4.eng.brq.redhat.com/released/jboss/eap6/6.4.19/composes/JBEAP-6.4.19-RHEL-7/Server/x86_64/os/