Bug 1070863

Created attachment 868599 [details]
Logs from 17:00-17:30 with first failed live storage migration from 26/02/14

Description of problem:
Targeted storage should not deny live storage migration to it because of existing LV from previous failed storage migration.

Live storage migration fails on disk being under dd and then roll-back doesn't delete LV from targeted storage, so when user tries to do live storage migration again, process fails because of on targeted storage already exists LV from previous failed migration.

Version-Release number of selected component (if applicable):

ovirt-engine-3.4.0-0.7.beta2.el6.noarch
qemu-kvm-rhev-0.12.1.2-2.415.el6_5.4.x86_64
libvirt-0.10.2-29.el6_5.3.x86_64
sanlock-2.8-1.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
1.First part:
1.1.Create 3 different storage domains, VNX1, VNX2 and XIO, while VNX1&2 should be placed on same physical storage, whereas XIO should be placed on another physical storage.
1.2.Assemble cluster of two hosts, host-a and host-b, while host-b should be SPM.
1.3.Create VM on host-a with OS RHEL on 15Gig preallocated disk called "OS", while disk should be created on VNX2.
1.4.Create ISCSI 2Gig disk as thinprovision called "test", while disk should be created on VNX1.
1.5.Hot plug disk "test" to VM.
1.6.Start writing to disk "test" using "dd if=/dev/zero of=/dev/vda bs=1M count=1889" continuously, i.e. start writing again to it when finished continuously.
1.7.While dd is running continuously to disk "test", start live storage migration for both "test" and "OS" disks (choose both disks virtual machines->VM->disks->ctrl+select both disks of VM->right click on them->move to the same storage domain XIO).
1.8.Continue writing to disk "test" using dd, during whole migration process.
1.9.Expect for disk "test" migration failure and check that disk "OS" should succeed migration.

2.Second part:
2.1.Shut down VM properly, not power-off.
2.2.Delete auto-created live snapshot.
2.3.Start VM.
2.4.Try live migration for disk "test" again from VNX1 to XIO, while disk "OS" remains on XIO from previous live storage migration.
2.5.Start writing to disk "test" using "dd if=/dev/zero of=/dev/vda bs=1M count=1889" continuously, i.e. start writing again to it when finished continuously.
2.6.While dd is running continuously to disk "test", start live storage migration only for disk "test" to storage domain XIO.
2.7.Expect this operation to fail with errors in log, as described bellow and attached as "Logs from 14:24 with second failed live storage migration from 27/02/14.".

Actual results:
Live storage migration fails for disk "test" under dd for the first time because of unclear cause, it rolled-back to source storage, while LV on targeted storage not deleted.

After first failed storage migration user tries to move disk "test" once again and fails, now because of not deleted LV on target storage.

da2cd858-9cc6-452a-a160-ef4113512b08::ERROR::2014-02-27 14:24:47,565::volume::466::Storage.Volume::(create) Failed to create volume: /rhev/data-center/4036
785a-511a-4c85-9df7-4dde5092b8a8/1d5a8dcf-6fd2-4365-a3f8-aa1ed5800354/images/5b3c3d25-b8f6-401d-983d-cb3c28bc8913/178ad568-1e64-4985-af36-7786d32cabf6, vol
ume already exists
da2cd858-9cc6-452a-a160-ef4113512b08::ERROR::2014-02-27 14:24:47,565::volume::502::Storage.Volume::(create) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/volume.py", line 468, in create
    raise e
CannotCreateLogicalVolume: Cannot create Logical Volume: ('1d5a8dcf-6fd2-4365-a3f8-aa1ed5800354', '178ad568-1e64-4985-af36-7786d32cabf6')
da2cd858-9cc6-452a-a160-ef4113512b08::ERROR::2014-02-27 14:24:47,588::image::390::Storage.Image::(_createTargetImage) Unexpected error 

Expected results:
For the first time disk being migrated should the migration fail for some reason, LV on targeted storage should be marked as temporary and be removed in case of migration failure.

For the second attempt of live storage migration, process should not fail because of already existing LV on targeted storage, migration should write to another LV.

Targeted storage should create temporary LVs until migration is complete and remain temporary during migration process, if migration process successful, only then LV on targeted storage should become active and LV at the source storage be deleted. 


Additional info:

Logs from 17:00-17:30 with first failed live storage migration from 26/02/14, these logs contain vdsm logs from 2 hosts vdsm.log and vdsm.log.1.xz (HSM-nsednev_host_a-camel-vdsb.qa.lab.tlv.redhat.com-10.35.116.2 and SPM-nsednev_host_b-camel-vdsc.qa.lab.tlv.redhat.com-10.35.116.3) and engine logs engine.log and server.log.
Logs from 14:24 with second failed live storage migration from 27/02/14.