Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1694888

Summary:	During live merge, engine was waiting for deleteVolume which already completed.
Product:	Red Hat Enterprise Virtualization Manager	Reporter:	Germano Veit Michel <gveitmic>
Component:	ovirt-engine	Assignee:	Eyal Shenitzky <eshenitz>
Status:	CLOSED DUPLICATE	QA Contact:	Lukas Svaty <lsvaty>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.2.7	CC:	Rhev-m-bugs, tnisan
Target Milestone:	ovirt-4.3.5	Flags:	lsvaty: testing_plan_complete-
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-05 09:13:49 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Storage	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Germano Veit Michel 2019-04-02 00:35:35 UTC

Description of problem:

After the MERGE step when deleting a snapshot for a VM with 4 disks, the deleteVolume command was executed fine for 4 volumes on the SPM, but the engine was waiting forever for the execution of 2 of them.

This is similar to BZ1555116, but with never versions and here there were no storage problems at the time. This "waiting forever" was also not handled on that BZ, which seems to aim to fix the NPE.

Version-Release number of selected component (if applicable):
rhvm-4.2.7.5-0.1.el7ev.noarch
vdsm-4.20.47-1.el7ev.x86_64

How reproducible:
Unknown

Comment 5 Eyal Shenitzky 2019-06-05 09:13:49 UTC

We can see in the engine log the following exception when the engine tries to end the 2 delete image commands:

b9e11047-1e0c-40bb-b1c6-0a0c70fe96fa] [within thread]: endAction for action type DestroyImage threw an exception.: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.ovirt.engine.core.common.action.DestroyImageParameters["liveMerge"])
	at org.ovirt.engine.core.utils.serialization.json.JsonObjectSerializer.writeJsonAsString(JsonObjectSerializer.java:81) [utils.jar:]
	at org.ovirt.engine.core.utils.serialization.json.JsonObjectSerializer.serialize(JsonObjectSerializer.java:64) [utils.jar:]
	at org.ovirt.engine.core.dao.CommandEntityDaoImpl.serializeParameters(CommandEntityDaoImpl.java:112) [dal.jar:]
	at org.ovirt.engine.core.dao.CommandEntityDaoImpl.createFullParametersMapper(CommandEntityDaoImpl.java:96) [dal.jar:]
	at org.ovirt.engine.core.dao.CommandEntityDaoImpl.saveOrUpdate(CommandEntityDaoImpl.java:147) [dal.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.saveOrUpdateWithoutTransaction(CommandsCacheImpl.java:112) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.updateCommandStatus(CommandsCacheImpl.java:87) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandsRepository.updateCommandStatus(CommandsRepository.java:158) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandCoordinatorImpl.updateCommandStatus(CommandCoordinatorImpl.java:160) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandCoordinatorUtil.updateCommandStatus(CommandCoordinatorUtil.java:341) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.setCommandStatus(CommandBase.java:2294) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.setCommandStatus(CommandBase.java:2286) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.handleCommandExecutionEnded(CommandBase.java:623) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:500) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.DecoratedCommand.endAction(DecoratedCommand.java:17) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CoCoAsyncTaskHelper.endAction(CoCoAsyncTaskHelper.java:353) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandCoordinatorImpl.endAction(CommandCoordinatorImpl.java:347) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandAsyncTask.endCommandAction(CommandAsyncTask.java:160) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandAsyncTask.lambda$endActionIfNecessary$0(CommandAsyncTask.java:112) [bll.jar:]
	at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:96) [utils.jar:]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_191]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_191]
	at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_191]
	at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent.jar:1.0.0.redhat-1]
	at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78)
Caused by: org.codehaus.jackson.map.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.ovirt.engine.core.common.action.DestroyImageParameters["liveMerge"])
	at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:218) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:183) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.SerializerBase.wrapAndThrow(SerializerBase.java:140) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:158) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:125) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.StdSerializerProvider$WrappedSerializer.serialize(StdSerializerProvider.java:835) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:610) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(ObjectMapper.java:2575) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ObjectMapper.writeValueAsString(ObjectMapper.java:2097) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.ovirt.engine.core.utils.serialization.json.JsonObjectSerializer.writeJsonAsString(JsonObjectSerializer.java:79) [utils.jar:]
	... 26 more
Caused by: java.lang.NullPointerException
	at org.ovirt.engine.core.common.action.DestroyImageParameters.isLiveMerge(DestroyImageParameters.java:57) [common.jar:]
	at sun.reflect.GeneratedMethodAccessor871.invoke(Unknown Source) [:1.8.0_191]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_191]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_191]
	at org.codehaus.jackson.map.ser.BeanPropertyWriter.get(BeanPropertyWriter.java:483) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:418) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	... 33 more

2019-03-28 03:35:37,053+11 ERROR [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-124) [b9e11047-1e0c-40bb-b1c6-0a0c70fe96fa] [within thread]: endAction for action type DestroyImage threw an exception.: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.ovirt.engine.core.common.action.DestroyImageParameters["liveMerge"])
	at org.ovirt.engine.core.utils.serialization.json.JsonObjectSerializer.writeJsonAsString(JsonObjectSerializer.java:81) [utils.jar:]
	at org.ovirt.engine.core.utils.serialization.json.JsonObjectSerializer.serialize(JsonObjectSerializer.java:64) [utils.jar:]
	at org.ovirt.engine.core.dao.CommandEntityDaoImpl.serializeParameters(CommandEntityDaoImpl.java:112) [dal.jar:]
	at org.ovirt.engine.core.dao.CommandEntityDaoImpl.createFullParametersMapper(CommandEntityDaoImpl.java:96) [dal.jar:]
	at org.ovirt.engine.core.dao.CommandEntityDaoImpl.saveOrUpdate(CommandEntityDaoImpl.java:147) [dal.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.saveOrUpdateWithoutTransaction(CommandsCacheImpl.java:112) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.updateCommandStatus(CommandsCacheImpl.java:87) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandsRepository.updateCommandStatus(CommandsRepository.java:158) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandCoordinatorImpl.updateCommandStatus(CommandCoordinatorImpl.java:160) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandCoordinatorUtil.updateCommandStatus(CommandCoordinatorUtil.java:341) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.setCommandStatus(CommandBase.java:2294) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.setCommandStatus(CommandBase.java:2286) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.handleCommandExecutionEnded(CommandBase.java:623) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:500) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.DecoratedCommand.endAction(DecoratedCommand.java:17) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CoCoAsyncTaskHelper.endAction(CoCoAsyncTaskHelper.java:353) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandCoordinatorImpl.endAction(CommandCoordinatorImpl.java:347) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandAsyncTask.endCommandAction(CommandAsyncTask.java:160) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandAsyncTask.lambda$endActionIfNecessary$0(CommandAsyncTask.java:112) [bll.jar:]
	at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:96) [utils.jar:]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_191]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_191]
	at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_191]
	at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent.jar:1.0.0.redhat-1]
	at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78)
Caused by: org.codehaus.jackson.map.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.ovirt.engine.core.common.action.DestroyImageParameters["liveMerge"])
	at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:218) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:183) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.SerializerBase.wrapAndThrow(SerializerBase.java:140) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:158) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:125) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.StdSerializerProvider$WrappedSerializer.serialize(StdSerializerProvider.java:835) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:610) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(ObjectMapper.java:2575) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ObjectMapper.writeValueAsString(ObjectMapper.java:2097) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.ovirt.engine.core.utils.serialization.json.JsonObjectSerializer.writeJsonAsString(JsonObjectSerializer.java:79) [utils.jar:]
	... 26 more
Caused by: java.lang.NullPointerException
	at org.ovirt.engine.core.common.action.DestroyImageParameters.isLiveMerge(DestroyImageParameters.java:57) [common.jar:]
	at sun.reflect.GeneratedMethodAccessor871.invoke(Unknown Source) [:1.8.0_191]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_191]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_191]
	at org.codehaus.jackson.map.ser.BeanPropertyWriter.get(BeanPropertyWriter.java:483) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:418) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	... 33 more


This is the same exception as bug 1651874.
Closing as duplication.

*** This bug has been marked as a duplicate of bug 1651874 ***