Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 828192

Summary: ovirt-engine-backend: remove disk will not remove the disk from the snapshot after merge action on multiple disk vm resulting with NPE when trying to preview snapshot
Product: Red Hat Enterprise Virtualization Manager Reporter: Dafna Ron <dron>
Component: ovirt-engineAssignee: Liron Aravot <laravot>
Status: CLOSED CURRENTRELEASE QA Contact: Dafna Ron <dron>
Severity: high Docs Contact:
Priority: medium    
Version: 3.1.0CC: abaron, amureini, dyasny, gcase, hateya, iheim, lpeer, mkenneth, Rhev-m-bugs, yeylon, ykaul
Target Milestone: ---Keywords: Regression
Target Release: 3.1.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: storage
Fixed In Version: SI17 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-12-04 20:06:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
logs
none
log and screen shot
none
log none

Description Dafna Ron 2012-06-04 12:39:49 UTC
Description of problem:

I deleted a snapshot on a vm with multiple disks (which caused a collaps action on teh chain) and then removed one of the disks. 
the disk I removed remains in the snapshot which causes a NPE and failure when trying to preview the snapshot



Version-Release number of selected component (if applicable):

si4

How reproducible:

100%

Steps to Reproduce:
1. create a vm with several disks
2. create 2 snapshots
3. delete the first snapshot (merge)
4. remove one of the disks

Actual results:

if we expend the snapshot to see the images we can see that we still have 3 disks in the snapshot -> if we try to preview the snapshot we will fail with NPE

Expected results:

we should remove the disk image from the snapshot

Additional info: vdsm and backend logs

2012-06-01 15:31:56,088 ERROR [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-10-thread-43) EntityAsyncTask::EndCommandAction [within thread]: EndAction for action type TryBackToAllSnapshotsOfVm threw an exception: java.lang.NullPointe
rException
        at org.ovirt.engine.core.bll.ImagesHandler.getAllImageSnapshots(ImagesHandler.java:332) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.VmCommand.UpdateVmInSpm(VmCommand.java:198) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.VmCommand.UpdateVmInSpm(VmCommand.java:179) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.TryBackToAllSnapshotsOfVmCommand.EndSuccessfully(TryBackToAllSnapshotsOfVmCommand.java:76) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.InternalEndSuccessfully(CommandBase.java:465) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:420) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1157) [engine-bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:204) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:142) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:109) [engine-utils.jar:]
        at org.ovirt.engine.core.bll.CommandBase.EndAction(CommandBase.java:366) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.Backend.endAction(Backend.java:355) [engine-bll.jar:]
        at sun.reflect.GeneratedMethodAccessor360.invoke(Unknown Source) [:1.6.0_24]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.6.0_24]
        at java.lang.reflect.Method.invoke(Method.java:616) [rt.jar:1.6.0_24]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.1.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.ovirt.engine.core.utils.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:11) [engine-utils.jar:]
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) [:1.6.0_24]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.6.0_24]
        at java.lang.reflect.Method.invoke(Method.java:616) [rt.jar:1.6.0_24]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee.jar:7.1.1.Final-
redhat-1]

Comment 1 Dafna Ron 2012-06-04 12:42:10 UTC
Created attachment 589136 [details]
logs

Comment 2 Dafna Ron 2012-06-04 12:50:17 UTC
after preview fails we also cannot remove vm/disks/snapshots because snapshot remains in image locked.

Comment 4 Allon Mureinik 2012-06-19 09:30:01 UTC
Agree the test fails, but this should not be a test blocker - please specify what flows it blocks or remove TestBlocker comment.

Also, we need to discuss the correct behavior here.
Removing a disk across all the snapshots does not sound like the behavior we'd want.

Comment 8 Allon Mureinik 2012-07-02 14:28:37 UTC
patch uploaded to gerrit: http://gerrit.ovirt.org/#/c/5831/

Changing the overall behavior of removing a disk may be too difficult/risky at this stage, but that's no reason you shouldn't be able to run a VM :-)

Comment 11 Allon Mureinik 2012-07-03 13:09:55 UTC
reopening - after further review, this is not how we want to handle this issue.

Comment 13 Allon Mureinik 2012-07-08 18:22:06 UTC
After discussions, we decided to remove the disk from all the snapshots, instead of leaving illegals.

patch in gerrit: http://gerrit.ovirt.org/#/c/6025/

Comment 14 Allon Mureinik 2012-07-23 16:05:55 UTC
Patch was merged.

Steps to Reproduce:
1. create a vm with several disks
2. create 2 snapshots
3. delete the first snapshot (merge)
4. remove one of the disks

You will see the disk /removed/ from the snapshot.

Comment 15 Dafna Ron 2012-07-30 13:56:18 UTC
disks were removed from snapshot only when located on the same domain. 
when I created a vm with disks located on different domains one of the disks in snapshot became illegal but was not removed 

snapshot remains in image locked. 

log attached + screen shot


reproduce: 

1. create a vm with several disks located on two different SD
2. create 2 snapshots
3. delete the first snapshot
4. remove disks from both domains

Comment 16 Dafna Ron 2012-07-30 13:57:34 UTC
Created attachment 601280 [details]
log and screen shot

Comment 17 Dafna Ron 2012-07-30 14:10:23 UTC
Created attachment 601285 [details]
log

operation finally ended with failure with same exception: 

2012-07-30 17:13:42,095 ERROR [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-4-thread-41) EntityAsyncTask::EndCommandAction [within thread]: EndAction for action type TryBackToAllSnapshotsOfVm threw an exception: java.lang.NullPointe
rException
        at org.ovirt.engine.core.bll.ImagesHandler.getAllImageSnapshots(ImagesHandler.java:344) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.VmCommand.UpdateVmInSpm(VmCommand.java:182) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.VmCommand.UpdateVmInSpm(VmCommand.java:164) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.TryBackToAllSnapshotsOfVmCommand.restoreVmConfigFromSnapshot(TryBackToAllSnapshotsOfVmCommand.java:96) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.TryBackToAllSnapshotsOfVmCommand.EndSuccessfully(TryBackToAllSnapshotsOfVmCommand.java:76) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.InternalEndSuccessfully(CommandBase.java:476) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:431) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1243) [engine-bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:204) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:142) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:109) [engine-utils.jar:]
        at org.ovirt.engine.core.bll.CommandBase.EndAction(CommandBase.java:377) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.Backend.endAction(Backend.java:355) [engine-bll.jar:]
        at sun.reflect.GeneratedMethodAccessor423.invoke(Unknown Source) [:1.7.0_05-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_05-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_05-icedtea]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.1.2.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.1.Final-redhat-1]
        at org.ovirt.engine.core.utils.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:11) [engine-utils.jar:]
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) [:1.7.0_05-icedtea]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_05-icedtea]
        at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_05-icedtea]
        at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee.jar:7.1.2.Final
-redhat-1]


new log attached

Comment 18 Allon Mureinik 2012-08-01 08:50:26 UTC
Tested with commit hash cb6eea48bc31cd63caeb840eaec6f0abf133df93, was unable to recreate the issue.

Need QE's help to reproduce, or at least undestand how my reproduction is wrong.

Comment 19 Dafna Ron 2012-08-01 09:14:20 UTC
I wrote the reproduction in comment 15. 
we are working with si12 and vdsm-4.9.6-24.0.el6_3.x86_64

Comment 20 Haim 2012-08-03 13:25:32 UTC
(In reply to comment #19)
> I wrote the reproduction in comment 15. 
> we are working with si12 and vdsm-4.9.6-24.0.el6_3.x86_64

moving back to assigned per comment 19 - there was no fix provided by R&D and bug was failed QA.
please resolve.

Comment 21 Allon Mureinik 2012-08-04 16:29:43 UTC
Error was reproduced.
I believe the problem reproducing (see comment #18) was that I did not attempt to remove the disks simoltaniously, thus not encountering the race condition in snapshot updating.

Working on a solution.

Comment 24 Liron Aravot 2012-09-02 14:21:06 UTC
submitted the following patch as fix : http://gerrit.ovirt.org/#/c/7482/

Comment 25 Allon Mureinik 2012-09-03 07:28:42 UTC
merged change id Iccb44f1aa9d204477955343167133849a4146753

Comment 26 Dafna Ron 2012-09-09 10:48:40 UTC
verified on si17