Created attachment 1439506 [details] /var/log/ from the engine Description of problem: With latest 3.6.13, can't reach the engine web APIs. server.log: 2018-05-21 12:15:17,210 INFO [org.quartz.core.JobRunShell] (DefaultQuartzScheduler_Worker-16) Job DEFAULT.org.ovirt.engine.core.bll.tasks.CommandExecutor.invokeCallbackMethods#-9223372036854775787 threw a JobExecutionException: : org.quartz.JobExecutionException: failed to execute job at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) [:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121] at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:81) [scheduler.jar:] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:52) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] Version-Release number of selected component (if applicable): [root@jenkins-vm-03 ~]# rpm -q rhevm rhevm-3.6.13.1-0.1.el6.noarch [root@jenkins-vm-03 ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.9 (Santiago) How reproducible: 1/1 Actual results: Web APIs don't function, we get 404 http error Additional info: httpd and ovirt-engine are running: [root@jenkins-vm-03 ~]# service httpd status httpd (pid 2077) is running... [root@jenkins-vm-03 ~]# service ovirt-engine status ovirt-engine (pid 13255) is running... The setup is alive, please contact me for its details
Below error started the issue: 2018-05-19 23:26:31,255 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 51) MSC000001: Failed to start service jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START: java.lang.IllegalStateException: JBAS011048: Failed to construct component instance at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:57) [jboss-as-ee.jar:7.5.20.Final-redhat-1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_121] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121] at org.jboss.threads.JBossThread.run(JBossThread.java:122) Caused by: java.lang.IllegalStateException: JBAS011048: Failed to construct component instance at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:164) [jboss-as-ee.jar:7.5.20.Final-redhat-1] at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:135) [jboss-as-ee.jar:7.5.20.Final-redhat-1] at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:90) [jboss-as-ee.jar:7.5.20.Final-redhat-1] at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:122) at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:137) at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54) [jboss-as-ee.jar:7.5.20.Final-redhat-1] ... 6 more Caused by: javax.ejb.EJBException: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: Illegal type (org.ovirt.engine.core.common.action.AddImageFromScratchParameters) to deserialize: prevented for security reasons at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:191) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:281) at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:353) at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:67) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:70) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:162) [jboss-as-ee.jar:7.5.20.Final-redhat-1] ... 11 more Caused by: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: Illegal type (org.ovirt.engine.core.common.action.AddImageFromScratchParameters) to deserialize: prevented for security reasons at org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer.readJsonString(JsonObjectDeserializer.java:101) at org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer.deserialize(JsonObjectDeserializer.java:57) at org.ovirt.engine.core.dao.CommandEntityDaoImpl.deserializeParameters(CommandEntityDaoImpl.java:150) at org.ovirt.engine.core.dao.CommandEntityDaoImpl.access$000(CommandEntityDaoImpl.java:32) at org.ovirt.engine.core.dao.CommandEntityDaoImpl$1.mapRow(CommandEntityDaoImpl.java:71) at org.ovirt.engine.core.dao.CommandEntityDaoImpl$1.mapRow(CommandEntityDaoImpl.java:58) at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:92) at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:1) at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:649) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:587) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:147) at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:109) at org.ovirt.engine.core.dao.DefaultReadDao.getAll(DefaultReadDao.java:77) at org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.initializeCache(CommandsCacheImpl.java:33) at org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.get(CommandsCacheImpl.java:50) at org.ovirt.engine.core.bll.tasks.CommandCoordinatorImpl.getCommandEntity(CommandCoordinatorImpl.java:132) at org.ovirt.engine.core.bll.tasks.CoCoAsyncTaskHelper.getCommandEntity(CoCoAsyncTaskHelper.java:160) at org.ovirt.engine.core.bll.tasks.CoCoAsyncTaskHelper.getAllAsyncTasksFromDb(CoCoAsyncTaskHelper.java:153) at org.ovirt.engine.core.bll.tasks.CommandCoordinatorImpl.getAllAsyncTasksFromDb(CommandCoordinatorImpl.java:292) at org.ovirt.engine.core.bll.tasks.AsyncTaskManager.initAsyncTaskManager(AsyncTaskManager.java:113) at org.ovirt.engine.core.bll.tasks.CommandCoordinatorUtil.initAsyncTaskManager(CommandCoordinatorUtil.java:49) at org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean.create(InitBackendServicesOnStartupBean.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_121] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_121] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_121] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_121] at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96) [jboss-as-ee.jar:7.5.20.Final-redhat-1] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:116) at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:107) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:93) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:67) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.as.ee.component.ComponentInstantiatorInterceptor.processInvocation(ComponentInstantiatorInterceptor.java:76) [jboss-as-ee.jar:7.5.20.Final-redhat-1] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:92) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.5.20.Final-redhat-1] at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.3.Final-redhat-1] at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279) ... 22 more Caused by: org.codehaus.jackson.map.JsonMappingException: Illegal type (org.ovirt.engine.core.common.action.AddImageFromScratchParameters) to deserialize: prevented for security reasons at org.codehaus.jackson.map.deser.BeanDeserializerFactory.checkLegalTypes(BeanDeserializerFactory.java:1521) at org.codehaus.jackson.map.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:655) at org.codehaus.jackson.map.deser.StdDeserializerProvider._createDeserializer(StdDeserializerProvider.java:401) at org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCache2(StdDeserializerProvider.java:310) at org.codehaus.jackson.map.deser.StdDeserializerProvider._createAndCacheValueDeserializer(StdDeserializerProvider.java:290) at org.codehaus.jackson.map.deser.StdDeserializerProvider.findValueDeserializer(StdDeserializerProvider.java:159) at org.codehaus.jackson.map.deser.StdDeserializerProvider.findTypedValueDeserializer(StdDeserializerProvider.java:180) at org.codehaus.jackson.map.ObjectMapper._findRootDeserializer(ObjectMapper.java:2829) at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2728) at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863) at org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer.readJsonString(JsonObjectDeserializer.java:99) ... 73 more 2018-05-19 23:26:31,266 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014612: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"JBAS014671: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.InitBackendServicesOnStartupBean.START" => "org.jboss.msc.service.StartException in service jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.InitBackendServicesOnStartupBean.START: java.lang.IllegalStateException: JBAS011048: Failed to construct component instance Caused by: java.lang.IllegalStateException: JBAS011048: Failed to construct component instance Caused by: javax.ejb.EJBException: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: Illegal type (org.ovirt.engine.core.common.action.AddImageFromScratchParameters) to deserialize: prevented for security reasons Caused by: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: Illegal type (org.ovirt.engine.core.common.action.AddImageFromScratchParameters) to deserialize: prevented for security reasons Caused by: org.codehaus.jackson.map.JsonMappingException: Illegal type (org.ovirt.engine.core.common.action.AddImageFromScratchParameters) to deserialize: prevented for security reasons"}} This is similar error as seen in BZ1577407, could you please try with EAP 6.4.19?
I've just verified that downgrading jackson library included in EAP 6.4.19 fixed the issue, I will open EAP bug and attach it to this one and BZ1577407
Verified on rhevm-3.6.13.4-0.1.el6.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1720
BZ<2>Jira Resync