Bug 1694888 - During live merge, engine was waiting for deleteVolume which already completed.
Summary: During live merge, engine was waiting for deleteVolume which already completed.
Keywords:
Status: CLOSED DUPLICATE of bug 1651874
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.2.7
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ovirt-4.3.5
: ---
Assignee: Eyal Shenitzky
QA Contact: Lukas Svaty
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-02 00:35 UTC by Germano Veit Michel
Modified: 2020-08-03 15:29 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-05 09:13:49 UTC
oVirt Team: Storage
Target Upstream Version:
Embargoed:
lsvaty: testing_plan_complete-

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3380701 0 Troubleshoot None RHV: Retrying a failed snapshot deletion fails with "General command validation failure" 2019-04-02 00:44:04 UTC

Description Germano Veit Michel 2019-04-02 00:35:35 UTC 
Description of problem:

After the MERGE step when deleting a snapshot for a VM with 4 disks, the deleteVolume command was executed fine for 4 volumes on the SPM, but the engine was waiting forever for the execution of 2 of them.

This is similar to BZ1555116, but with never versions and here there were no storage problems at the time. This "waiting forever" was also not handled on that BZ, which seems to aim to fix the NPE.

Version-Release number of selected component (if applicable):
rhvm-4.2.7.5-0.1.el7ev.noarch
vdsm-4.20.47-1.el7ev.x86_64

How reproducible:
Unknown
Comment 5 Eyal Shenitzky 2019-06-05 09:13:49 UTC 
We can see in the engine log the following exception when the engine tries to end the 2 delete image commands:

b9e11047-1e0c-40bb-b1c6-0a0c70fe96fa] [within thread]: endAction for action type DestroyImage threw an exception.: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.ovirt.engine.core.common.action.DestroyImageParameters["liveMerge"])
	at org.ovirt.engine.core.utils.serialization.json.JsonObjectSerializer.writeJsonAsString(JsonObjectSerializer.java:81) [utils.jar:]
	at org.ovirt.engine.core.utils.serialization.json.JsonObjectSerializer.serialize(JsonObjectSerializer.java:64) [utils.jar:]
	at org.ovirt.engine.core.dao.CommandEntityDaoImpl.serializeParameters(CommandEntityDaoImpl.java:112) [dal.jar:]
	at org.ovirt.engine.core.dao.CommandEntityDaoImpl.createFullParametersMapper(CommandEntityDaoImpl.java:96) [dal.jar:]
	at org.ovirt.engine.core.dao.CommandEntityDaoImpl.saveOrUpdate(CommandEntityDaoImpl.java:147) [dal.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.saveOrUpdateWithoutTransaction(CommandsCacheImpl.java:112) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.updateCommandStatus(CommandsCacheImpl.java:87) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandsRepository.updateCommandStatus(CommandsRepository.java:158) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandCoordinatorImpl.updateCommandStatus(CommandCoordinatorImpl.java:160) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandCoordinatorUtil.updateCommandStatus(CommandCoordinatorUtil.java:341) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.setCommandStatus(CommandBase.java:2294) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.setCommandStatus(CommandBase.java:2286) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.handleCommandExecutionEnded(CommandBase.java:623) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:500) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.DecoratedCommand.endAction(DecoratedCommand.java:17) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CoCoAsyncTaskHelper.endAction(CoCoAsyncTaskHelper.java:353) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandCoordinatorImpl.endAction(CommandCoordinatorImpl.java:347) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandAsyncTask.endCommandAction(CommandAsyncTask.java:160) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandAsyncTask.lambda$endActionIfNecessary$0(CommandAsyncTask.java:112) [bll.jar:]
	at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:96) [utils.jar:]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_191]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_191]
	at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_191]
	at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent.jar:1.0.0.redhat-1]
	at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78)
Caused by: org.codehaus.jackson.map.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.ovirt.engine.core.common.action.DestroyImageParameters["liveMerge"])
	at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:218) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:183) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.SerializerBase.wrapAndThrow(SerializerBase.java:140) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:158) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:125) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.StdSerializerProvider$WrappedSerializer.serialize(StdSerializerProvider.java:835) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:610) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(ObjectMapper.java:2575) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ObjectMapper.writeValueAsString(ObjectMapper.java:2097) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.ovirt.engine.core.utils.serialization.json.JsonObjectSerializer.writeJsonAsString(JsonObjectSerializer.java:79) [utils.jar:]
	... 26 more
Caused by: java.lang.NullPointerException
	at org.ovirt.engine.core.common.action.DestroyImageParameters.isLiveMerge(DestroyImageParameters.java:57) [common.jar:]
	at sun.reflect.GeneratedMethodAccessor871.invoke(Unknown Source) [:1.8.0_191]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_191]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_191]
	at org.codehaus.jackson.map.ser.BeanPropertyWriter.get(BeanPropertyWriter.java:483) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:418) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	... 33 more

2019-03-28 03:35:37,053+11 ERROR [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-124) [b9e11047-1e0c-40bb-b1c6-0a0c70fe96fa] [within thread]: endAction for action type DestroyImage threw an exception.: org.apache.commons.lang.SerializationException: org.codehaus.jackson.map.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.ovirt.engine.core.common.action.DestroyImageParameters["liveMerge"])
	at org.ovirt.engine.core.utils.serialization.json.JsonObjectSerializer.writeJsonAsString(JsonObjectSerializer.java:81) [utils.jar:]
	at org.ovirt.engine.core.utils.serialization.json.JsonObjectSerializer.serialize(JsonObjectSerializer.java:64) [utils.jar:]
	at org.ovirt.engine.core.dao.CommandEntityDaoImpl.serializeParameters(CommandEntityDaoImpl.java:112) [dal.jar:]
	at org.ovirt.engine.core.dao.CommandEntityDaoImpl.createFullParametersMapper(CommandEntityDaoImpl.java:96) [dal.jar:]
	at org.ovirt.engine.core.dao.CommandEntityDaoImpl.saveOrUpdate(CommandEntityDaoImpl.java:147) [dal.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.saveOrUpdateWithoutTransaction(CommandsCacheImpl.java:112) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandsCacheImpl.updateCommandStatus(CommandsCacheImpl.java:87) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandsRepository.updateCommandStatus(CommandsRepository.java:158) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandCoordinatorImpl.updateCommandStatus(CommandCoordinatorImpl.java:160) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandCoordinatorUtil.updateCommandStatus(CommandCoordinatorUtil.java:341) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.setCommandStatus(CommandBase.java:2294) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.setCommandStatus(CommandBase.java:2286) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.handleCommandExecutionEnded(CommandBase.java:623) [bll.jar:]
	at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:500) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.DecoratedCommand.endAction(DecoratedCommand.java:17) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CoCoAsyncTaskHelper.endAction(CoCoAsyncTaskHelper.java:353) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandCoordinatorImpl.endAction(CommandCoordinatorImpl.java:347) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandAsyncTask.endCommandAction(CommandAsyncTask.java:160) [bll.jar:]
	at org.ovirt.engine.core.bll.tasks.CommandAsyncTask.lambda$endActionIfNecessary$0(CommandAsyncTask.java:112) [bll.jar:]
	at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:96) [utils.jar:]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_191]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_191]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_191]
	at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_191]
	at org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:250) [javax.enterprise.concurrent.jar:1.0.0.redhat-1]
	at org.jboss.as.ee.concurrent.service.ElytronManagedThreadFactory$ElytronManagedThread.run(ElytronManagedThreadFactory.java:78)
Caused by: org.codehaus.jackson.map.JsonMappingException: (was java.lang.NullPointerException) (through reference chain: org.ovirt.engine.core.common.action.DestroyImageParameters["liveMerge"])
	at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:218) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.JsonMappingException.wrapWithPath(JsonMappingException.java:183) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.SerializerBase.wrapAndThrow(SerializerBase.java:140) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:158) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeWithType(BeanSerializerBase.java:125) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.StdSerializerProvider$WrappedSerializer.serialize(StdSerializerProvider.java:835) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:610) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ObjectMapper._configAndWriteValue(ObjectMapper.java:2575) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ObjectMapper.writeValueAsString(ObjectMapper.java:2097) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.ovirt.engine.core.utils.serialization.json.JsonObjectSerializer.writeJsonAsString(JsonObjectSerializer.java:79) [utils.jar:]
	... 26 more
Caused by: java.lang.NullPointerException
	at org.ovirt.engine.core.common.action.DestroyImageParameters.isLiveMerge(DestroyImageParameters.java:57) [common.jar:]
	at sun.reflect.GeneratedMethodAccessor871.invoke(Unknown Source) [:1.8.0_191]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_191]
	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_191]
	at org.codehaus.jackson.map.ser.BeanPropertyWriter.get(BeanPropertyWriter.java:483) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:418) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	at org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150) [jackson-mapper-asl.jar:1.9.13.redhat-4]
	... 33 more


This is the same exception as bug 1651874.
Closing as duplication.

*** This bug has been marked as a duplicate of bug 1651874 ***
Note You need to log in before you can comment on or make changes to this bug.