Bug 844440 - [engine-core] locking mechanism: able to perform live snapshot during hot-plug disk (also, clone vm from snapshot)
[engine-core] locking mechanism: able to perform live snapshot during hot-plu...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.1.0
x86_64 Linux
unspecified Severity medium
: ---
: 3.1.0
Assigned To: Tal Nisan
Dafna Ron
storage
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-30 12:04 EDT by Haim
Modified: 2016-02-10 12:04 EST (History)
14 users (show)

See Also:
Fixed In Version: si20
Doc Type: Release Note
Doc Text:
Previously when hot plugging and unplugging disks the Manager did not lock the virtual machine being modified. As a result it was possible to initiate snapshot and clone operations while the state of the virtual machine was undefined. This is no longer the case.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-04 15:06:41 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Storage
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
engine.log (150.49 KB, application/x-gzip)
2012-07-30 12:07 EDT, Haim
no flags Details
logs (1.86 MB, application/x-gzip)
2012-09-21 04:39 EDT, Dafna Ron
no flags Details

  None (edit)
Description Haim 2012-07-30 12:04:03 EDT
Description of problem:

the above case should be handled as a part of our locking mechanism (or some can_do_action).

a case of running vm, I can run hot-unplug of my disk and create snapshot (snapshot creation would file) or do this permutation with clone vm from snapshot. 

attached backend log - very easy to reproduce.
Comment 1 Haim 2012-07-30 12:07:29 EDT
Created attachment 601306 [details]
engine.log
Comment 4 mkublin 2012-08-01 02:59:07 EDT
The issue is simple, hot plug can be done when vm is up, a create snaphsot can be run when vm is up . These is only one common limitation on both operations.
You shoud add a read lock on vm at hotPlug and write lock on vm at CreateAllSnapshots.
Good example to some one from storage team to start to use a locking mechanism at storage scenario when locking mechanism was developed by infra team
Comment 5 Stephen Gordon 2012-08-09 11:59:24 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
When hot plugging and unplugging disks the Manager does not correctly lock the virtual machine being modified. As a result it is currently possible to initiation snapshot and clone operations while the virtual machines state is undefined. This will be rectified in a future update.
Comment 6 Stephen Gordon 2012-08-09 12:01:08 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-When hot plugging and unplugging disks the Manager does not correctly lock the virtual machine being modified. As a result it is currently possible to initiation snapshot and clone operations while the virtual machines state is undefined. This will be rectified in a future update.+When hot plugging and unplugging disks the Manager does not lock the virtual machine being modified. As a result it is currently possible to initiation snapshot and clone operations while the virtual machines state is undefined. This will be rectified in a future update.
Comment 7 Stephen Gordon 2012-08-09 12:01:26 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-When hot plugging and unplugging disks the Manager does not lock the virtual machine being modified. As a result it is currently possible to initiation snapshot and clone operations while the virtual machines state is undefined. This will be rectified in a future update.+When hot plugging and unplugging disks the Manager does not lock the virtual machine being modified. As a result it is currently possible to initiate snapshot and clone operations while the virtual machines state is undefined. This will be rectified in a future update.
Comment 8 Stephen Gordon 2012-08-09 12:01:39 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-When hot plugging and unplugging disks the Manager does not lock the virtual machine being modified. As a result it is currently possible to initiate snapshot and clone operations while the virtual machines state is undefined. This will be rectified in a future update.+When hot plugging and unplugging disks the Manager does not lock the virtual machine being modified. As a result it is currently possible to initiate snapshot and clone operations while the virtual machine's state is undefined. This will be rectified in a future update.
Comment 9 Tal Nisan 2012-09-10 10:45:33 EDT
http://gerrit.ovirt.org/#/c/7805/
Comment 11 Dafna Ron 2012-09-21 04:38:48 EDT
tested on si18
i was able to send hotplug command to vdsm during snapshot create (it failed with unexpected exception in vdsm)
but the command was sent so the locking mechanism is not preventing both actions from running (will attach both vdsm and backend logs). 

reproduce: 
1. open two sessions and run a vm with OS installed with virtIO disk
2. in one session create a snapshot
3. in second session deactivate the disk

results: 

2012-09-21 11:29:19,475 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-45) [45b9ec9c] Failed in HotPlugDiskVDS method
2012-09-21 11:29:19,475 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-45) [45b9ec9c] Error code unexpected and error message VDSGenericExce
ption: VDSErrorException: Failed to HotPlugDiskVDS, error = Unexpected exception
2012-09-21 11:29:19,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-45) [45b9ec9c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HotP
lugDiskVDSCommand return value 
 Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc
mStatus                       Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc
mCode                         16
mMessage                      Unexpected exception


2012-09-21 11:29:19,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-4-thread-45) [45b9ec9c] Vds: gold-vdsd
2012-09-21 11:29:19,475 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-4-thread-45) [45b9ec9c] Command HotPlugDiskVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to HotPlugDiskVDS, error = Unexpected exception
2012-09-21 11:29:19,475 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HotPlugDiskVDSCommand] (pool-4-thread-45) [45b9ec9c] FINISH, HotPlugDiskVDSCommand, log id: abcaf65
2012-09-21 11:29:19,476 ERROR [org.ovirt.engine.core.bll.HotPlugDiskToVmCommand] (pool-4-thread-45) [45b9ec9c] Command org.ovirt.engine.core.bll.HotPlugDiskToVmCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HotPlugDiskVDS, error = Unexpected exception
2012-09-21 11:29:19,478 INFO  [org.ovirt.engine.core.bll.HotPlugDiskToVmCommand] (pool-4-thread-45) [45b9ec9c] Lock freed to object EngineLock [exclusiveLocks= , sharedLocks= key: a5e725b4-262d-407b-8071-41f5a4ff4794 value: VM
]
Comment 12 Dafna Ron 2012-09-21 04:39:57 EDT
Created attachment 615306 [details]
logs
Comment 13 Tal Nisan 2012-09-27 14:00:29 EDT
The bug was about being able to create a snapshot during hot plug/unplug of a disk, you are talking about the reverse scenario, I think another bug should be opened instead of reopening this one
Comment 14 Tal Nisan 2012-10-02 07:18:53 EDT
http://gerrit.ovirt.org/#/c/8298/
Comment 16 Allon Mureinik 2012-10-02 13:51:20 EDT
merged change id I4980c8cff7aec9bd10bba344f9592bbbf1c20b8d
Comment 17 Dafna Ron 2012-10-07 06:53:20 EDT
verified on si12
we are getting CanDoAction on deactivate disk:

2012-10-07 12:47:54,657 WARN  [org.ovirt.engine.core.bll.HotUnPlugDiskFromVmCommand] (ajp-/127.0.0.1:8702-9) CanDoAction of action HotUnPlugDiskFromVm failed. Reasons:VAR__ACTION__HOT_UNPLUG,VAR__TYPE__VM_DISK,ACTION_TYPE_FAILED_VM_IS_DURING_SNAPSHOT

Note You need to log in before you can comment on or make changes to this bug.