Created attachment 742719 [details] full logs from engine and vds Description of problem: I restarted ovirt-engine right when DeleteImage was sent to the vmds for the LSM flow. the DeleteImage task for the source image was sent to the vdsm although engine thinks it was not and when engine starts it rolls back on the LSM command and sends DeleteImage on the destination which leaves a vm with no disks. Version-Release number of selected component (if applicable): sf15 vdsm-4.10.2-17.0.el6ev.x86_64 How reproducible: 100% when restart is sent at the correct time (which is right when DeleteImage is sent to the vdsm). Steps to Reproduce: 1. on iscsi storage, start a vm with one disk on two hosts (my vm was running on the hsm) 2. live migrate the vms disk 3. right when the DeleteImage for the source disk is sent, restart ovirt-engine Actual results: the task was sent to the spm and the source image is removed. when engine restarts we think that the move failed and engine sends DeleteImage to the destination image as part of the rollback -> leaving a vm with no disk. Expected results: engine should see if there was a deleteImage task in vdsm before sending the second deleteImage. Additional info: full engine and vdsm logs. this is the first deleteImage task: b483e310-3648-4722-9300-19cebc4d7766 : verb = deleteImage id = b483e310-3648-4722-9300-19cebc4d7766 ae5688bb-a669-4559-aa3b-bb30e11ec442 : verb = syncImageData id = ae5688bb-a669-4559-aa3b-bb30e11ec442 second deleteImage task: b483e310-3648-4722-9300-19cebc4d7766 : verb = deleteImage id = b483e310-3648-4722-9300-19cebc4d7766 a6f9de58-3722-4d0d-a8cd-31dac2dcc9fa : verb = deleteImage id = a6f9de58-3722-4d0d-a8cd-31dac2dcc9fa ae5688bb-a669-4559-aa3b-bb30e11ec442 : verb = syncImageData id = ae5688bb-a669-4559-aa3b-bb30e11ec442 lvs before the live storage migration (we have two templates and one disk): [root@gold-vdsd ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert 9d48ee3a-6c5c-4b74-a59a-3784b50fdf10 38755249-4bb3-4841-bf5b-05f4a521514d -wi------ 1.00g e886fb6e-f91f-46ae-a488-0f24e47f5731 38755249-4bb3-4841-bf5b-05f4a521514d -wi-a---- 2.00g ids 38755249-4bb3-4841-bf5b-05f4a521514d -wi-ao--- 128.00m inbox 38755249-4bb3-4841-bf5b-05f4a521514d -wi-a---- 128.00m leases 38755249-4bb3-4841-bf5b-05f4a521514d -wi-a---- 2.00g master 38755249-4bb3-4841-bf5b-05f4a521514d -wi-a---- 1.00g metadata 38755249-4bb3-4841-bf5b-05f4a521514d -wi-a---- 512.00m outbox 38755249-4bb3-4841-bf5b-05f4a521514d -wi-a---- 128.00m 9d48ee3a-6c5c-4b74-a59a-3784b50fdf10 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 1.00g e886fb6e-f91f-46ae-a488-0f24e47f5731 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 2.00g ids 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-ao--- 128.00m inbox 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 128.00m leases 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 2.00g master 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 1.00g metadata 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 512.00m outbox 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 128.00m 3d7adfe5-499d-401e-8a11-89d5d31b4f33 d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-ao--- 5.00g 9d48ee3a-6c5c-4b74-a59a-3784b50fdf10 d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 1.00g e886fb6e-f91f-46ae-a488-0f24e47f5731 d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 2.00g ids d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-ao--- 128.00m inbox d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 128.00m leases d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 2.00g master d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 1.00g metadata d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 512.00m outbox d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 128.00m lv_root vg0 -wi-ao--- 120.79g lv_swap vg0 -wi-ao--- 15.74g [root@gold-vdsd ~]# here is the lvs after the live storage migration (we have only two templates): [root@gold-vdsd ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert 9d48ee3a-6c5c-4b74-a59a-3784b50fdf10 38755249-4bb3-4841-bf5b-05f4a521514d -wi------ 1.00g e886fb6e-f91f-46ae-a488-0f24e47f5731 38755249-4bb3-4841-bf5b-05f4a521514d -wi-a---- 2.00g ids 38755249-4bb3-4841-bf5b-05f4a521514d -wi-ao--- 128.00m inbox 38755249-4bb3-4841-bf5b-05f4a521514d -wi-a---- 128.00m leases 38755249-4bb3-4841-bf5b-05f4a521514d -wi-a---- 2.00g master 38755249-4bb3-4841-bf5b-05f4a521514d -wi-a---- 1.00g metadata 38755249-4bb3-4841-bf5b-05f4a521514d -wi-a---- 512.00m outbox 38755249-4bb3-4841-bf5b-05f4a521514d -wi-a---- 128.00m 9d48ee3a-6c5c-4b74-a59a-3784b50fdf10 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 1.00g e886fb6e-f91f-46ae-a488-0f24e47f5731 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 2.00g ids 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-ao--- 128.00m inbox 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 128.00m leases 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 2.00g master 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 1.00g metadata 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 512.00m outbox 81ef11d0-4c0c-47b4-8953-d61a6af442d8 -wi-a---- 128.00m 9d48ee3a-6c5c-4b74-a59a-3784b50fdf10 d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 1.00g e886fb6e-f91f-46ae-a488-0f24e47f5731 d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 2.00g ids d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-ao--- 128.00m inbox d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 128.00m leases d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 2.00g master d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 1.00g metadata d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 512.00m outbox d326916a-89b6-41c8-9c14-4b9cf8a1c979 -wi-a---- 128.00m lv_root vg0 -wi-ao--- 120.79g lv_swap vg0 -wi-ao--- 15.74g
the disk is no longer removed but we have other issues due to the fix which I will open new bugs for. verified on sf16
*** Bug 963719 has been marked as a duplicate of this bug. ***
3.2 has been released