Hide Forgot
Created attachment 1197601 [details] server.log Description of problem: The engine fails to start after it was upgraded to 7.3 Version-Release number of selected component (if applicable): engine 4.0.4-1 How reproducible: only tried it on one env Steps to Reproduce: 1. add el7.3 repos 2. yum update 3. run some automated tests 4. restart engine (part of the tests) Actual results: 2016-09-03 15:43:21,361 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 2) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'undeploy' at address '[("deployment" => "engine.ear")]' 2016-09-03 15:43:26,372 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 2) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$5@489da2eb for operation {"operation" => "undeploy","address" => [("deployment" => "engine.ear")],"owner" => [("subsystem" => "deployment-scanner"),("scanner" => "default")]} at address [("deployment" => "engine.ear")] failed handling operation rollback -- java.lang.IllegalStateException: WFLYCTL0345: Timeout after 5 seconds waiting for existing service service jboss.deployment.unit."engine.ear".contents to be removed so a new instance can be installed.: java.lang.IllegalStateException: WFLYCTL0345: Timeout after 5 seconds waiting for existing service service jboss.deployment.unit."engine.ear".contents to be removed so a new instance can be installed. Expected results: the engine should start Additional info:
Created attachment 1197602 [details] engine log
Juan - can you take a look at this one?
Not sure yet about what is the reason of this failure, but it seems related to the storage of JSON objects in the database. The relevant log message from server.log is the following: 2016-09-03 15:38:20,117 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.InitBackendServicesOnStartupBean.START" => "org.jboss.msc.service.StartException in service jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.InitBackendServicesOnStartupBean.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance Caused by: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance Caused by: javax.ejb.EJBException: WFLYEJB0442: Unexpected Error Caused by: java.lang.ExceptionInInitializerError Caused by: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke private void org.ovirt.engine.core.bll.tasks.CommandsRepository.handleUnmanagedCommands() on org.ovirt.engine.core.bll.tasks.CommandsRepository@3725357f Caused by: java.lang.reflect.InvocationTargetException Caused by: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: No default constructor for [collection type; class java.util.Collections$SingletonSet, contains [simple type, class org.ovirt.engine.core.common.businessentities.network.NetworkCluster]] (through reference chain: org.ovirt.engine.core.common.action.ManageNetworkClustersParameters[\"attachments\"]) Caused by: org.codehaus.jackson.map.JsonMappingException: No default constructor for [collection type; class java.util.Collections$SingletonSet, contains [simple type, class org.ovirt.engine.core.common.businessentities.network.NetworkCluster]] (through reference chain: org.ovirt.engine.core.common.action.ManageNetworkClustersParameters[\"attachments\"]) Caused by: java.lang.IllegalStateException: No default constructor for [collection type; class java.util.Collections$SingletonSet, contains [simple type, class org.ovirt.engine.core.common.businessentities.network.NetworkCluster]]"}} 2016-09-03 15:38:20,267 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "rhev.ear" (runtime-name : "rhev.ear") 2016-09-03 15:38:20,268 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "restapi.war" (runtime-name : "restapi.war") 2016-09-03 15:38:20,268 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "engine.ear" (runtime-name : "engine.ear") 2016-09-03 15:38:20,274 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report WFLYCTL0186: Services which failed to start: service jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
This patch adds a test that reproduces this failure: [WIP] Test to reproduce bug #1372952 https://gerrit.ovirt.org/63302
*** This bug has been marked as a duplicate of bug 1372950 ***
Juan, would you reconsider the closure of this bug? I believe that the problem at hand is much bigger than the network-specific report of bug 372950. We can easily stop using singeltonSet. But there should be a system-wide mechanism (could be a unit test) making sure that every serialized object can also be deserialized. Such object must never have elemets with no default constructor.
Another bug was opened on the serialization issue (Bug 1373242). *** This bug has been marked as a duplicate of bug 1372950 ***