Created attachment 1681015 [details] logs Description of problem: Administration Portal is not accessible after engine-service restart. I restarted engine-service in the middle of exporting VM to data domain and got the following errors: Engine Log: 2020-04-22 16:48:51,063+03 ERROR [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 61) [] Failed to initialize backend: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke public void org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl.initContextsMap() on org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl@3863a404 at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:85) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.postConstruct(DefaultLifecycleCallbackInvoker.java:66) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.injection.producer.BasicInjectionTarget.postConstruct(BasicInjectionTarget.java:122) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:164) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.contexts.AbstractContext.get(AbstractContext.java:96) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.bean.ContextualInstanceStrategy$ApplicationScopedContextualInstanceStrategy.get(ContextualInstanceStrategy.java:140) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:694) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:794) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.injection.ParameterInjectionPointImpl.getValueToInject(ParameterInjectionPointImpl.java:76) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.injection.ConstructorInjectionPoint.getParameterValues(ConstructorInjectionPoint.java:150) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.injection.ConstructorInjectionPoint.newInstance(ConstructorInjectionPoint.java:75) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.injection.producer.AbstractInstantiator.newInstance(AbstractInstantiator.java:28) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.injection.producer.BasicInjectionTarget.produce(BasicInjectionTarget.java:112) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.injection.producer.BeanInjectionTarget.produce(BeanInjectionTarget.java:186) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:160) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.contexts.AbstractContext.get(AbstractContext.java:96) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.bean.ContextualInstanceStrategy$ApplicationScopedContextualInstanceStrategy.get(ContextualInstanceStrategy.java:140) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:694) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:252) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:114) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean.create(InitBackendServicesOnStartupBean.java:108) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.as.ee.0.GA-redhat-00004//org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.common.0.GA-redhat-00004//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:79) at org.jboss.as.weld.common.0.GA-redhat-00004//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:126) at org.jboss.as.weld.common.0.GA-redhat-00004//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:112) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) at org.jboss.as.weld.common.0.GA-redhat-00004//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.weld.0.GA-redhat-00004//org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:53) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.0.GA-redhat-00004//org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.weld.0.GA-redhat-00004//org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.weld.0.GA-redhat-00004//org.jboss.as.weld.interceptors.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:111) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.0.GA-redhat-00004//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.0.GA-redhat-00004//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:249) at org.jboss.as.ejb3.0.GA-redhat-00004//org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:400) at org.jboss.as.ejb3.0.GA-redhat-00004//org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.weld.0.GA-redhat-00004//org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.0.GA-redhat-00004//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee.0.GA-redhat-00004//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3.0.GA-redhat-00004//org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation.2.Final-redhat-00001//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee.0.GA-redhat-00004//org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161) at org.jboss.as.ee.0.GA-redhat-00004//org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134) at org.jboss.as.ee.0.GA-redhat-00004//org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88) at org.jboss.as.ejb3.0.GA-redhat-00004//org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:127) at org.jboss.as.ejb3.0.GA-redhat-00004//org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:141) at org.jboss.as.ee.0.GA-redhat-00004//org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at org.jboss.threads.3.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at java.base/java.lang.Thread.run(Thread.java:834) at org.jboss.threads.3.Final-redhat-00001//org.jboss.threads.JBossThread.run(JBossThread.java:485) Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.weld.core.2.Final-redhat-00001//org.jboss.weld.injection.producer.DefaultLifecycleCallbackInvoker.invokeMethods(DefaultLifecycleCallbackInvoker.java:83) ... 78 more Caused by: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: No default constructor for [collection type; class java.util.ImmutableCollections$List12, contains [simple type, class org.ovirt.engine.core.compat.Guid]] (through reference chain: org.ovirt.engine.core.common.action.MoveOrCopyImageGroupParameters["destImages"]->org.ovirt.engine.core.common.businessentities.storage.DiskImage["storageIds"]) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer.readJsonString(JsonObjectDeserializer.java:110) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer.deserialize(JsonObjectDeserializer.java:59) at org.ovirt.engine.core.dal//org.ovirt.engine.core.dao.CommandEntityDaoImpl.deserializeParameters(CommandEntityDaoImpl.java:131) at org.ovirt.engine.core.dal//org.ovirt.engine.core.dao.CommandEntityDaoImpl.lambda$static$2(CommandEntityDaoImpl.java:55) at org.springframework.4.RELEASE//org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:93) at org.springframework.4.RELEASE//org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:60) at org.springframework.4.RELEASE//org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:667) at org.springframework.4.RELEASE//org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:605) at org.springframework.4.RELEASE//org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:657) at org.springframework.4.RELEASE//org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:688) at org.springframework.4.RELEASE//org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:741) at org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:152) at org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:118) at org.springframework.4.RELEASE//org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:198) at org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:135) at org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:105) at org.ovirt.engine.core.dal//org.ovirt.engine.core.dao.DefaultReadDao.getAll(DefaultReadDao.java:81) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.initializeCache(CommandsCacheImpl.java:40) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.keySet(CommandsCacheImpl.java:52) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl.initContextsMap(CommandContextsCacheImpl.java:33) ... 83 more Caused by: org.codehaus.jackson.map.JsonMappingException: No default constructor for [collection type; class java.util.ImmutableCollections$List12, contains [simple type, class org.ovirt.engine.core.compat.Guid]] (through reference chain: org.ovirt.engine.core.common.action.MoveOrCopyImageGroupParameters["destImages"]->org.ovirt.engine.core.common.businessentities.storage.DiskImage["storageIds"]) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:218) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:183) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.BeanDeserializer.wrapAndThrow(BeanDeserializer.java:1472) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:699) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:600) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:92) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.BeanDeserializer.deserializeWithType(BeanDeserializer.java:664) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:219) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:194) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:30) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.jsontype.impl.AsArrayTypeDeserializer._deserialize(AsArrayTypeDeserializer.java:88) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.jsontype.impl.AsArrayTypeDeserializer.deserializeTypedFromArray(AsArrayTypeDeserializer.java:45) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserializeWithType(CollectionDeserializer.java:232) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:297) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.SettableBeanProperty$MethodProperty.deserializeAndSet(SettableBeanProperty.java:414) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:697) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:600) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:92) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.BeanDeserializer.deserializeWithType(BeanDeserializer.java:664) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.StdDeserializerProvider$WrappedDeserializer.deserialize(StdDeserializerProvider.java:461) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2732) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.serialization.json.JsonObjectDeserializer.readJsonString(JsonObjectDeserializer.java:108) ... 102 more Caused by: java.lang.IllegalStateException: No default constructor for [collection type; class java.util.ImmutableCollections$List12, contains [simple type, class org.ovirt.engine.core.compat.Guid]] at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.std.StdValueInstantiator.createUsingDefault(StdValueInstantiator.java:209) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:194) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:30) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.jsontype.impl.AsArrayTypeDeserializer._deserialize(AsArrayTypeDeserializer.java:88) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.jsontype.impl.AsArrayTypeDeserializer.deserializeTypedFromArray(AsArrayTypeDeserializer.java:45) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.std.CollectionDeserializer.deserializeWithType(CollectionDeserializer.java:232) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:297) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.SettableBeanProperty$MethodProperty.deserializeAndSet(SettableBeanProperty.java:414) at org.codehaus.jackson.jackson-mapper-asl.13.redhat-00006//org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:697) ... 121 more 2020-04-22 16:48:59,974+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-27) [300b770f] Could not finish afterRefreshTreatment host=host_mixed_2(5ae8607a-4c92-4794-abb6-ea385f00a9f3): org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke public void org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl.initContextsMap() on org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl@3c24b781 2020-04-22 16:48:59,995+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-36) [37b55de0] Could not finish afterRefreshTreatment host=host_mixed_1(65c2af74-77c8-4f08-a478-158ea29d6840): org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke public void org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl.initContextsMap() on org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl@50d8c59b 2020-04-22 16:49:00,013+03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-29) [6240cb61] Could not finish afterRefreshTreatment host=host_mixed_3(bc45abe0-cdb7-47b4-8ef2-41fe4c169f13): org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke public void org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl.initContextsMap() on org.ovirt.engine.core.bll.tasks.CommandContextsCacheImpl@5707b38e 2020-04-22 16:49:19,846+03 ERROR [org.ovirt.engine.core.aaa.servlet.SsoPostLoginServlet] (default task-2) [] java.lang.reflect.UndeclaredThrowableException 2020-04-22 16:49:48,550+03 INFO [org.ovirt.engine.core.bll.quota.QuotaManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-42) [] Quota Cache updated. (37 msec) 2020-04-22 16:54:57,865+03 ERROR [org.ovirt.engine.core.aaa.servlet.SsoPostLoginServlet] (default task-2) [] java.lang.reflect.UndeclaredThrowableException 2020-04-22 Version-Release number of selected component (if applicable): ovirt-engine-4.4.0-0.33.master.el8ev.noarch How reproducible: once (didn't try again) Steps to Reproduce: 1.Create VM with disk 2.Export VM to data domain 3.While exporting, restart engine service Actual results: Administration Portal is not accessible after engine-service restart - java.lang.reflect.UndeclaredThrowableException Expected results: Operation should succeed. Additional info: Logs are attached.
Additional info: Didn't find WA for this, tried to restart it again several times and still nothing.
The real issue can be found in server.log: Caused by: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: No default constructor for [collection type; class java.util.ImmutableCollections$List12, contains [simple type, class org.ovirt.engine.core.compat.Guid]] (through reference chain: org.ovirt.engine.core.common.action.MoveOrCopyImageGroupParameters["destImages"]->org.ovirt.engine.core.common.businessentities.storage.DiskImage["storageIds"]) The problem is, that engine was restarted while one of commands using this parameter file was running. So after restart engine tries to restore that command parameters, which were serialized to JSON as a part of command infrastructure. The problem is that as a part of https://gerrit.ovirt.org/102338 we have added a new property: List<DiskImage> destImages = new ArrayList<>(); But Jackson is unable to deserialize collections. So there are 2 options to fix it: 1. Either write a MixIn class, which marks that field as ignored during deserialization. As an example please take a look at https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/utils/src/main/java/org/ovirt/engine/core/utils/serialization/json/JsonVmMixIn.java 2. Or change declaration into static array: DiskImage[] destImage; As a workaround to be able to start engine correct, you can use /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh to remove serialized commands
This bug report has Keywords: Regression or TestBlocker. Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.
so still a test blocker?
(In reply to Evelina Shames from comment #3) > Additional info: > Didn't find WA for this, tried to restart it again several times and still > nothing. Evelina, please use the suggested workaround and see if this unblock you. (In reply to Michal Skrivanek from comment #6) > so still a test blocker? removing test blocker as this looks to have a WA and a much rare bug than we though initially.
(In reply to Martin Perina from comment #4) > The real issue can be found in server.log: > > Caused by: org.apache.commons.lang.SerializationException: > org.codehaus.jackson.map.JsonMappingException: No default constructor for > [collection type; class java.util.ImmutableCollections$List12, contains > [simple type, class org.ovirt.engine.core.compat.Guid]] (through reference > chain: > org.ovirt.engine.core.common.action. > MoveOrCopyImageGroupParameters["destImages"]->org.ovirt.engine.core.common. > businessentities.storage.DiskImage["storageIds"]) > > The problem is, that engine was restarted while one of commands using this > parameter file was running. So after restart engine tries to restore that > command parameters, which were serialized to JSON as a part of command > infrastructure. The problem is that as a part of > https://gerrit.ovirt.org/102338 we have added a new property: > > List<DiskImage> destImages = new ArrayList<>(); > > But Jackson is unable to deserialize collections. So there are 2 options to > fix it: > > 1. Either write a MixIn class, which marks that field as ignored during > deserialization. As an example please take a look at > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/ > utils/src/main/java/org/ovirt/engine/core/utils/serialization/json/ > JsonVmMixIn.java > > 2. Or change declaration into static array: > > DiskImage[] destImage; > > As a workaround to be able to start engine correct, you can use > /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh to remove serialized > commands I checked and it's actually simpler. From the logs it looks like what bothered Jackson is storageIds, but it's just a list of Guid within DiskImage (for which there is a mixin already), and then I found the culprit[1], trying to be hip I used List.of() which creates an immutable list and can't be deserialized (because ImmutableCollections has a private default constructor), changing it to Arrays.asList() resolves the issue https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/storage/disk/image/ImagesHandler.java#L1062
(In reply to Martin Perina from comment #4) > The real issue can be found in server.log: > > Caused by: org.apache.commons.lang.SerializationException: > org.codehaus.jackson.map.JsonMappingException: No default constructor for > [collection type; class java.util.ImmutableCollections$List12, contains > [simple type, class org.ovirt.engine.core.compat.Guid]] (through reference > chain: > org.ovirt.engine.core.common.action. > MoveOrCopyImageGroupParameters["destImages"]->org.ovirt.engine.core.common. > businessentities.storage.DiskImage["storageIds"]) > > The problem is, that engine was restarted while one of commands using this > parameter file was running. So after restart engine tries to restore that > command parameters, which were serialized to JSON as a part of command > infrastructure. The problem is that as a part of > https://gerrit.ovirt.org/102338 we have added a new property: > > List<DiskImage> destImages = new ArrayList<>(); > > But Jackson is unable to deserialize collections. So there are 2 options to > fix it: > > 1. Either write a MixIn class, which marks that field as ignored during > deserialization. As an example please take a look at > https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/ > utils/src/main/java/org/ovirt/engine/core/utils/serialization/json/ > JsonVmMixIn.java > > 2. Or change declaration into static array: > > DiskImage[] destImage; > > As a workaround to be able to start engine correct, you can use > /usr/share/ovirt-engine/setup/dbutils/taskcleaner.sh to remove serialized > commands What is the full command? (I tried with several flags and it didn't work...)
Please try below: taskcleaner.sh -R -o -A
(In reply to Martin Perina from comment #10) > Please try below: > > taskcleaner.sh -R -o -A Thanks, it worked.
Saw similar errors when restarting vdsm while exporting VM: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: No default constructor for [collection type; class java.util.ImmutableCollections$List12, contains [simple type, class org.ovirt.engine.core.compat.Guid]] (through reference chain: org.ovirt.engine.core.common.action.CopyImageGroupVolumesDataCommandParameters["destImages"]->org.ovirt.engine.core.common.businessentities.storage.DiskImage["storageIds"])
Verified in 4.4.1.1-0.5.el8ev Exporting continues after restarting ovirt-engine restart, it is possible to log in to webadmin after service restart.
This bugzilla is included in oVirt 4.4.1 release, published on July 8th 2020. Since the problem described in this bug report should be resolved in oVirt 4.4.1 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.