Bug 1508560 - Delete snapshot after live migration fails at reduce stage - BlockStorageDomainManifest' object has no attribute '_extendlock'
Summary: Delete snapshot after live migration fails at reduce stage - BlockStorageDoma...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ovirt-4.2.0
: ---
Assignee: Ala Hino
QA Contact: Avihai
URL:
Whiteboard:
Depends On:
Blocks: 1470055
TreeView+ depends on / blocked
 
Reported: 2017-11-01 16:39 UTC by Avihai
Modified: 2017-12-20 11:21 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2017-12-20 11:21:03 UTC
oVirt Team: Storage
Embargoed:
rule-engine: ovirt-4.2+
rule-engine: blocker+


Attachments (Terms of Use)
engine , vdsm log (614.47 KB, application/x-gzip)
2017-11-01 16:39 UTC, Avihai
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 83505 0 master MERGED blockSD: Rename _extendlock to metadata 2020-05-26 04:13:40 UTC

Description Avihai 2017-11-01 16:39:03 UTC
Created attachment 1346633 [details]
engine , vdsm  log

Description of problem:
Delete snapshot after live migration fails at reduce stage - BlockStorageDomainManifest' object has no attribute '_extendlock'

Version-Release number of selected component (if applicable):
Engine: ovirt-engine-4.2.0-0.0.master.20171030210714.gitef6bb9c
VDSM:   4.20.5-5.git3d47978

How reproducible:
tried it once so far

Steps to Reproduce:
1.Create VM +os disk +additional disk+ 2 snapshots (first one s1 , second one s2)
2.Move the additional disk between 2 storage domains (I moved it between 2 ISCSI domains)
3.After live migration is done delete the oldest snapshot s1

Actual results:
Snapshot deletion fails.

Audit log Events:
Nov 1, 2017, 6:20:27 PM VDSM host_mixed_3 command HSMGetAllTasksStatusesVDS failed: 'BlockStorageDomainManifest' object has no attribute '_extendlock'

Nov 1, 2017, 6:20:40 PM Failed to delete snapshot 's1' for VM 'golden_env_mixed_virtio_1_0'.


Expected results:


Additional info:
Engine:
2017-11-01 18:20:38,302+02 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-4) [df1d29ff-e5fb-4287-8fe1-0fe9254daf91] Successfully m
erged snapshot '41d65b29-13fc-406b-bd41-db66878e7863' images 'efa867a0-8c38-4f05-be36-557219961833'..'7d0e0217-042d-45ee-b21a-92be192f8793'
2017-11-01 18:20:38,323+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-4) [df1d29ff-e5fb-4287-8fe1-0fe9254daf91] Ending command
 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand' with failure.
2017-11-01 18:20:39,353+02 INFO  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-35) [df1d29ff-e5fb-4287-8fe1-0fe9254daf91] Command 'RemoveSna
pshot' id: '4e6a567b-2ac1-4e03-b7cb-302f82a13f4d' child commands '[01850f1d-a6c3-4f54-bdcb-8691d2bbf83a, f7e03f2c-2e5c-48cb-8e4f-5a4f9cd55bfa]' executions were completed, status 'FAILED'
2017-11-01 18:20:40,396+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-2) [df1d29ff-e5fb-4287-8fe1-0fe9254daf91] Ending command 'org.ovirt.en
gine.core.bll.snapshots.RemoveSnapshotCommand' with failure.
2017-11-01 18:20:40,429+02 WARN  [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (EE-ManagedThreadFactory-engineScheduled-Thread-2) [df1d29ff-e5fb-4287-8fe1-0fe9254daf91] Trying to release exclusive lock wh
ich does not exist, lock key: '082e69ee-c075-4445-91c0-6d188a87ed81VM'
2017-11-01 18:20:40,430+02 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-2) [df1d29ff-e5fb-4287-8fe1-0fe9254daf91] Lock freed to object 'Engine
Lock:{exclusiveLocks='[082e69ee-c075-4445-91c0-6d188a87ed81=VM]', sharedLocks=''}'
2017-11-01 18:20:40,461+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-2) [df1d29ff-e5fb-4287-8fe1-0fe9254daf91] EVENT_ID: USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Failed to delete snapshot 's1' for VM 'golden_env_mixed_virtio_1_0'.


VDSM
2017-11-01 18:20:16,821+0200 DEBUG (tasks/8) [storage.TaskManager.Task] (Task='b727339a-9cca-45aa-87b7-768c587b8b3e') Job.run: running reduceVolume: <bound method StoragePool.reduceVolume of <vdsm.storage.sp.Sto
ragePool object at 0x3a0e190>> (args: ('3b727bff-66ce-42b8-ab19-104e3929cd22', '9eab03f9-355f-4e6d-9d3d-20965767e077', '7d0e0217-042d-45ee-b21a-92be192f8793', True) kwargs: {}) callback None (task:333)
2017-11-01 18:20:16,822+0200 ERROR (tasks/8) [storage.TaskManager.Task] (Task='b727339a-9cca-45aa-87b7-768c587b8b3e') Unexpected error (task:875)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run
    return fn(*args, **kargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run
    return self.cmd(*self.argslist, **self.argsdict)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py", line 79, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1418, in reduceVolume
    dom.reduceVolume(imgUUID, volUUID, allowActive=allowActive)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/blockSD.py", line 1539, in reduceVolume
    with self._manifest._extendlock:
AttributeError: 'BlockStorageDomainManifest' object has no attribute '_extendlock'

Comment 1 Red Hat Bugzilla Rules Engine 2017-11-01 16:53:06 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 2 Raz Tamir 2017-11-01 18:07:00 UTC
Raising severity as this is a regression

Comment 3 Avihai 2017-11-07 12:17:51 UTC
verified .

Engine:
ovirt-engine-4.2.0-0.0.master.20171106202508.gitf5140b9.el7.centos.noarch

VDSM:
4.20.6-33.git54a784e

Comment 4 Sandro Bonazzola 2017-12-20 11:21:03 UTC
This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017.

Since the problem described in this bug report should be
resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.