Bug 1236758 - Engine: Live merge fails after a disk containing a snapshot has been extended
Summary: Engine: Live merge fails after a disk containing a snapshot has been extended
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.5.1
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ovirt-3.6.0-rc
: 3.6.0
Assignee: Greg Padgett
QA Contact: Kevin Alon Goldblatt
URL:
Whiteboard:
Depends On:
Blocks: 1241433
TreeView+ depends on / blocked
 
Reported: 2015-06-29 22:59 UTC by Greg Padgett
Modified: 2016-04-18 00:28 UTC (History)
19 users (show)

Fixed In Version: 3.6.0-4 alpha3
Doc Type: Bug Fix
Doc Text:
Cause: A raw, preallocated disk may have snapshots that have been extended beyond the original disk capacity. If one of these snapshots whose parent is the smaller base disk is removed via live merge, the contents of the snapshot will not be able to be merged into the base disk resulting in a failure. Consequence: Live merge to remove the specified snapshot will fail. Fix: Live merge will now automatically extend the base volume in such a case. Result: Live merge can succeed in such cases without user intervention.
Clone Of: 1232481
: 1241433 (view as bug list)
Environment:
Last Closed:
oVirt Team: Storage
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 1488453 0 None None None 2016-04-18 00:28:48 UTC
oVirt gerrit 43025 0 master MERGED core: handle Live Merge when top volume is larger than base Never

Comment 2 Allon Mureinik 2015-07-09 08:52:40 UTC
Greg, could you please add a couple of lines of doctext

Comment 3 Greg Padgett 2015-07-09 23:22:55 UTC
(In reply to Allon Mureinik from comment #2)
> Greg, could you please add a couple of lines of doctext

Draft text added

Comment 4 Elad 2015-08-16 09:22:34 UTC
Executed the scenario mentioned in the bug description using:
ovirt-3.6.0-5
vdsm-4.17.0-1239.git6575e3f.el7.noarch
libvirt-daemon-1.2.8-16.el7_1.3.x86_64
qemu-kvm-ev-2.1.2-23.el7_1.6.1.x86_64
sanlock-3.2.2-2.el7.x86_64
selinux-policy-3.13.1-23.el7_1.13.noarch
python-2.7.5-18.el7_1.1.x86_64

Live merge fails and I don't get the mentioned libvirt error. Instead, I got this one in vdsm:

1169f9ef-1560-4d5d-aa72-03da356e1985::ERROR::2015-08-16 11:42:49,512::task::866::Storage.TaskManager.Task::(_setError) Task=`1169f9ef-1560-4d5d-aa72-03da356e1985`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 873, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/task.py", line 332, in run
    return self.cmd(*self.argslist, **self.argsdict)
  File "/usr/share/vdsm/storage/securable.py", line 77, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/share/vdsm/storage/sp.py", line 1304, in extendVolumeSize
    .produceVolume(imgUUID, volUUID).extendSize(int(newSize))
  File "/usr/share/vdsm/storage/volume.py", line 552, in extendSize
    raise se.VolumeNonWritable(self.volUUID)
VolumeNonWritable: Volume cannot be access to writes: ["u'b7ee712c-d029-452e-bbb4-1ee3b451f97b'"]


I opened a new bug for it: https://bugzilla.redhat.com/show_bug.cgi?id=1253974
And also one for the fact that the task is not cleared on VDSM: https://bugzilla.redhat.com/show_bug.cgi?id=1253975


Greg, should I move the bug back to ASSIGNED?

Comment 5 Yaniv Lavi 2015-08-16 12:34:41 UTC
This bug is also targeted to 3.5.5 therefore should be addressed soon.

Comment 6 Greg Padgett 2015-08-17 18:11:26 UTC
(In reply to Elad from comment #4)
> Executed the scenario mentioned in the bug description using:
> ovirt-3.6.0-5
> vdsm-4.17.0-1239.git6575e3f.el7.noarch
> libvirt-daemon-1.2.8-16.el7_1.3.x86_64
> qemu-kvm-ev-2.1.2-23.el7_1.6.1.x86_64
> sanlock-3.2.2-2.el7.x86_64
> selinux-policy-3.13.1-23.el7_1.13.noarch
> python-2.7.5-18.el7_1.1.x86_64
> 
> Live merge fails and I don't get the mentioned libvirt error. Instead, I got
> this one in vdsm:
> 
> 1169f9ef-1560-4d5d-aa72-03da356e1985::ERROR::2015-08-16
> 11:42:49,512::task::866::Storage.TaskManager.Task::(_setError)
> Task=`1169f9ef-1560-4d5d-aa72-03da356e1985`::Unexpected error
> Traceback (most recent call last):
>   File "/usr/share/vdsm/storage/task.py", line 873, in _run
>     return fn(*args, **kargs)
>   File "/usr/share/vdsm/storage/task.py", line 332, in run
>     return self.cmd(*self.argslist, **self.argsdict)
>   File "/usr/share/vdsm/storage/securable.py", line 77, in wrapper
>     return method(self, *args, **kwargs)
>   File "/usr/share/vdsm/storage/sp.py", line 1304, in extendVolumeSize
>     .produceVolume(imgUUID, volUUID).extendSize(int(newSize))
>   File "/usr/share/vdsm/storage/volume.py", line 552, in extendSize
>     raise se.VolumeNonWritable(self.volUUID)
> VolumeNonWritable: Volume cannot be access to writes:
> ["u'b7ee712c-d029-452e-bbb4-1ee3b451f97b'"]
> 
> 
> I opened a new bug for it:
> https://bugzilla.redhat.com/show_bug.cgi?id=1253974
> And also one for the fact that the task is not cleared on VDSM:
> https://bugzilla.redhat.com/show_bug.cgi?id=1253975
> 
> 
> Greg, should I move the bug back to ASSIGNED?

Thanks for testing Elad.  After looking at this, I can see what happened and it's just due to an engine vs vdsm discrepancy.  There's a vdsm patch that this bz relies upon, but this bz was unmarked as a dependency of that one.  The vdsm patch took a while to get merged and consequently wasn't in the version you tested.

See bug 1232481, and gerrit change id 43407 which addresses this very issue.  You'll need a vdsm containing commit 745f03e2e0115de4ec59e45c36f7af7a2e4f3e27 which was merged Aug 5.

Please re-test when you have a new enough vdsm, and in the meantime I'll fix the bz references.

Comment 7 Greg Padgett 2015-08-17 18:27:56 UTC
(In reply to Greg Padgett from comment #6)
> Thanks for testing Elad.  After looking at this, I can see what happened and
> it's just due to an engine vs vdsm discrepancy.  There's a vdsm patch that
> this bz relies upon, but this bz was unmarked as a dependency of that one. 
> The vdsm patch took a while to get merged and consequently wasn't in the
> version you tested.
> 
> See bug 1232481, and gerrit change id 43407 which addresses this very issue.
> You'll need a vdsm containing commit
> 745f03e2e0115de4ec59e45c36f7af7a2e4f3e27 which was merged Aug 5.
> 
> Please re-test when you have a new enough vdsm, and in the meantime I'll fix
> the bz references.

On second thought after revisiting the vdsm bz--there are two scenarios here, and testing without that patch is valid:

Scenario A) new engine, older (pre-patched) vdsm.  After attempting Live Merge, you should see the error you got above (instead of the libvirt error).  The data in the disk(s) will be valid and un-touched.

Scenario B) new engine, vdsm with the above fix.  This would allow the merge to take place, extending the disk as necessary.

Comment 8 Kevin Alon Goldblatt 2015-08-20 13:35:25 UTC
Tested using the following version:
---------------------------------------
rhevm-3.6.0-0.11.master.el6.noarch - running on Rhel6.7
vdsm-4.17.2-1.el7ev.noarch - running on Rhel7.2

Tested using the following steps:
Scenario 1:
----------------
1. Create a VM with 1gb preallocated disk (probably thin as well, didn't test that) in a block-based domain.
2. Create a snapshot.
3. Extend the disk to 2gb.
4. Start the VM.
5. Delete the snapshot - Live merge works well!

Scenario 2:
--------------------
1) Create a VM with 1gb preallocated disk in a block-based domain.
2) Create a snapshot.
3) Create a second snapshot.
4) Extend the disk to 2gb.
5) Start the VM.
6) Delete the second snapshot - Live merge works well!

Moving to VERIFIED!

Comment 9 Allon Mureinik 2016-03-10 10:39:57 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE

Comment 10 Allon Mureinik 2016-03-10 10:40:03 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE

Comment 11 Allon Mureinik 2016-03-10 10:45:38 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE

Comment 12 Allon Mureinik 2016-03-10 12:02:15 UTC
RHEV 3.6.0 has been released, setting status to CLOSED CURRENTRELEASE


Note You need to log in before you can comment on or make changes to this bug.