Bug 1372952

Summary: Engine fail to start after attempting to deserialize an object with no default constructor
Product: [oVirt] ovirt-engine Reporter: Nelly Credi <ncredi>
Component: BLL.InfraAssignee: Juan Hernández <juan.hernandez>
Status: CLOSED DUPLICATE QA Contact: Pavel Stehlik <pstehlik>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.0.4CC: bugs, danken, juan.hernandez, oourfali
Target Milestone: ---Keywords: AutomationBlocker, Regression, Reopened
Target Release: ---Flags: gklein: ovirt-4.0.z?
gklein: blocker?
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-06 07:04:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
server.log
none
engine log none

Description Nelly Credi 2016-09-04 09:28:44 UTC
Created attachment 1197601 [details]
server.log

Description of problem:
The engine fails to start after it was upgraded to 7.3

Version-Release number of selected component (if applicable):
engine 4.0.4-1

How reproducible:
only tried it on one env

Steps to Reproduce:
1. add el7.3 repos 
2. yum update 
3. run some automated tests
4. restart engine (part of the tests)

Actual results:
2016-09-03 15:43:21,361 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 2) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'undeploy' at address '[("deployment" => "engine.ear")]'
2016-09-03 15:43:26,372 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 2) WFLYCTL0190: Step handler org.jboss.as.server.deployment.DeploymentHandlerUtil$5@489da2eb for operation {"operation" => "undeploy","address" => [("deployment" => "engine.ear")],"owner" => [("subsystem" => "deployment-scanner"),("scanner" => "default")]} at address [("deployment" => "engine.ear")] failed handling operation rollback -- java.lang.IllegalStateException: WFLYCTL0345: Timeout after 5 seconds waiting for existing service service jboss.deployment.unit."engine.ear".contents to be removed so a new instance can be installed.: java.lang.IllegalStateException: WFLYCTL0345: Timeout after 5 seconds waiting for existing service service jboss.deployment.unit."engine.ear".contents to be removed so a new instance can be installed.


Expected results:
the engine should start

Additional info:

Comment 1 Nelly Credi 2016-09-04 09:31:33 UTC
Created attachment 1197602 [details]
engine log

Comment 2 Oved Ourfali 2016-09-05 06:04:00 UTC
Juan - can you take a look at this one?

Comment 3 Juan Hernández 2016-09-05 09:13:22 UTC
Not sure yet about what is the reason of this failure, but it seems related to the storage of JSON objects in the database. The relevant log message from server.log is the following:

2016-09-03 15:38:20,117 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.InitBackendServicesOnStartupBean.START" => "org.jboss.msc.service.StartException in service jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.InitBackendServicesOnStartupBean.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
    Caused by: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
    Caused by: javax.ejb.EJBException: WFLYEJB0442: Unexpected Error
    Caused by: java.lang.ExceptionInInitializerError
    Caused by: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke private void org.ovirt.engine.core.bll.tasks.CommandsRepository.handleUnmanagedCommands() on org.ovirt.engine.core.bll.tasks.CommandsRepository@3725357f
    Caused by: java.lang.reflect.InvocationTargetException
    Caused by: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: No default constructor for [collection type; class java.util.Collections$SingletonSet, contains [simple type, class org.ovirt.engine.core.common.businessentities.network.NetworkCluster]] (through reference chain: org.ovirt.engine.core.common.action.ManageNetworkClustersParameters[\"attachments\"])
    Caused by: org.codehaus.jackson.map.JsonMappingException: No default constructor for [collection type; class java.util.Collections$SingletonSet, contains [simple type, class org.ovirt.engine.core.common.businessentities.network.NetworkCluster]] (through reference chain: org.ovirt.engine.core.common.action.ManageNetworkClustersParameters[\"attachments\"])
    Caused by: java.lang.IllegalStateException: No default constructor for [collection type; class java.util.Collections$SingletonSet, contains [simple type, class org.ovirt.engine.core.common.businessentities.network.NetworkCluster]]"}}
2016-09-03 15:38:20,267 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "rhev.ear" (runtime-name : "rhev.ear")
2016-09-03 15:38:20,268 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "restapi.war" (runtime-name : "restapi.war")
2016-09-03 15:38:20,268 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "engine.ear" (runtime-name : "engine.ear")
2016-09-03 15:38:20,274 INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0186:   Services which failed to start:      service jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.InitBackendServicesOnStartupBean.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance

Comment 4 Juan Hernández 2016-09-05 13:46:23 UTC
This patch adds a test that reproduces this failure:

  [WIP] Test to reproduce bug #1372952
  https://gerrit.ovirt.org/63302

Comment 5 Juan Hernández 2016-09-05 14:38:12 UTC

*** This bug has been marked as a duplicate of bug 1372950 ***

Comment 6 Dan Kenigsberg 2016-09-06 06:59:46 UTC
Juan, would you reconsider the closure of this bug? I believe that the problem at hand is much bigger than the network-specific report of bug 372950.

We can easily stop using singeltonSet. But there should be a system-wide mechanism (could be a unit test) making sure that every serialized object can also be deserialized. Such object must never have elemets with no default constructor.

Comment 7 Oved Ourfali 2016-09-06 07:04:07 UTC
Another bug was opened on the serialization issue (Bug 1373242).

*** This bug has been marked as a duplicate of bug 1372950 ***