Description of problem: Doing a live storage migration of a vm's disks (multiple) increases the size to what seems wrong values (~80 Gb). This is part of our automation efforts so the scenario here may be not the usual but is the one running. Version-Release number of selected component (if applicable): 3.5.0 vt13.7 How reproducible: 100% Steps to Reproduce: 1. Create a vm with a 10Gb Thin and OS install on it (rhel7 in this case). 3. start the vm 3. Create and attach multiple disk of 1 GB of different type to the vm. In our run we have 1 of each of [interface - format - sparse]: virtio - cow - sparse virtio_scsi - cow sparse, irito - raw - no sparse, virtio_scsi - raw - no sparse. 3. Start a live storage migration of all the disks, even the bootable disk, the order in the migration: ['virtio_cow_True_iscsi_disk', 'live_storage_migration_iscsi_Disk1', 'virtio_scsi_raw_False_iscsi_disk', 'virtio_scsi_cow_True_iscsi_disk', 'virtio_raw_False_iscsi_disk'] 4. After the migration the ACTUAL DISK of the disks: virtio_scsi - cow - True: 22548578304 os' boot disk (cow sparse) : 67645734912 virtio_scsi - raw - no sparse(changed to cow sparse): 41875931136 virtio - cow - sparse : 88046829568 virtio - raw - no sparse (changed to cow sparse): 13958643712 Actual results: Seems unreasonable disk sizes after a storage migration. Expected results: Not sure what size the actual snapshots before the migration is but should be fairly less. Additional info: I'll add the relevant logs.
Created attachment 984687 [details] engine.log from the run
Can you please provide the exact versions of vdsm, libvirt, qemu and sanlock? Thanks!
Tentatively targeting to 3.5.1 until we clear it up.
Values for the run: Build (vt13.7) VDS(vdsm-4.16.8.1-5.el7ev.x86_64) VDS(libvirt-1.2.8-10.el7.x86_64) VDS(libvirt-1.1.1-29.el7_0.4.x86_64 I made another run but didn't see the issue (on NFS) with vt13.8: VDS(vdsm-4.16.8.1-6.el7ev.x86_64) VDS(libvirt-1.2.8-13.el7.x86_64) VDS(libvirt-1.1.1-29.el7_0.4.x86_64) I'll try to reproduce once more/see if I missed something.
(In reply to Carlos Mestre González from comment #5) > Values for the run: > Build (vt13.7) > VDS(vdsm-4.16.8.1-5.el7ev.x86_64) > VDS(libvirt-1.2.8-10.el7.x86_64) > VDS(libvirt-1.1.1-29.el7_0.4.x86_64 Carlos, I don't understand this input - how can you have two versions of libvirt? Or are these separate runs on separate hosts?
The described behavior is expected for block domains. Since each live migration generates a new snapshot of all VM disks, all disks sizes increase on each migration. The migrated disk size increases by additional blocks as part of the block mirroring step, which is required to acquire extra blocks for data synchronization - see bug 1101515 (each block is defined as 1GB). A workaround for testing is either using a file domain or invoke multiple disks live migration concurrently (currently available through the UI, should be added to the rest-api as part of bug 1132478). *** This bug has been marked as a duplicate of bug 1101515 ***
Daniel thanks for you reply, can I ask you for clarification. In this tested case there are 5 disk and all of them are migrating, according to your comment then each disk will increse to 1Gb for the mirroring step, so 5 Disk * 5 live migration = 25 Gb. As I showed in my first comment I ended up with a disk of 80Gb (virtio - cow - sparse)
(In reply to Carlos Mestre González from comment #11) > Daniel thanks for you reply, can I ask you for clarification. > > In this tested case there are 5 disk and all of them are migrating, > according to your comment then each disk will increse to 1Gb for the > mirroring step, so 5 Disk * 5 live migration = 25 Gb. As I showed in my > first comment I ended up with a disk of 80Gb (virtio - cow - sparse) That behavior seems to be due to another bug (bz1176673).
*** This bug has been marked as a duplicate of bug 1176673 ***
Opening this issue again because I just test it with vt14.1 and is still happening after bz1196049 was fixed (but could still be that is just the normal behaviour after each LSM increases the disk size). I just want to make sure this makes sense. Same scenario: 1. 10Gb Thin bootable disk with OS installed and UP. (live_storage_migration_iscsi_Disk1 is the one with the OS - 3GB) 2. Adding 4 different disk types 1 GB 3. Start migrating the disks LSM (last migration does not happen because the domain is out of space) So I'm adding the migration order and how the actual size of the disks change after each migration finishes (only including disks that change size): 1. Migrating virtio_scsi_raw_False_iscsi_disk: virtio_scsi_raw_False_iscsi_disk -> 7516192768 7 GB 2. Migrating virtio_cow_True_iscsi_disk virtio_scsi_raw_False_iscsi_disk -> 35433480192 33 GB virtio_cow_True_iscsi_disk -> 8589934592 8 GB 3. Migrating live_storage_migration_iscsi virtio_scsi_raw_False_iscsi_disk -> 52613349376 49 GB virtio_cow_True_iscsi_disk -> 27917287424 26 GB live_storage_migration_iscsi_Disk1 -> 3221225472 3GB 4. Migrating virtio_scsi_cow_True_iscsi_disk virtio_scsi_raw_False_iscsi_disk -> 75161927680 70 GB virtio_cow_True_iscsi_disk -> 52613349376 49 GB live_storage_migration_iscsi_Disk1 -> 26843545600 25 GB virtio_scsi_cow_True_iscsi_disk -> 7516192768 7 GB 5. Migrating virtio_raw_False_iscsi_disk virtio_scsi_raw_False_iscsi_disk 84825604096 79Gb virtio_cow_True_iscsi_disk 62277025792 58Gb live_storage_migration_iscsi_Disk1 50465865728 47Gb virtio_scsi_cow_True_iscsi_disk 30064771072 28Gb virtio_raw_False_iscsi_disk 1073741824 1Gb (last migration does not happen because the domain is out of space) Is this the expected behaviour?
please list what vdsm, libvirt and qemu RPMs you're using, and attach logs from the new run.
Created attachment 1009218 [details] engine.log engine.log for the run, starts at 18:13:29
Created attachment 1009219 [details] collection of vdsm.log for all hosts
Packages versions: qemu-img-rhev-2.1.2-23.el7_1.1.x86_64 libvirt-1.2.8-16.el7.x86_64 vdsm-4.16.12.1-3.el7ev.x86_64
(In reply to Carlos Mestre González from comment #18) > Packages versions: > qemu-img-rhev-2.1.2-23.el7_1.1.x86_64 > libvirt-1.2.8-16.el7.x86_64 > vdsm-4.16.12.1-3.el7ev.x86_64 @Nir - is the fix for bug 1196049 should be included in the aforementioned build?
(In reply to Daniel Erez from comment #20) > (In reply to Carlos Mestre González from comment #18) > > Packages versions: > > qemu-img-rhev-2.1.2-23.el7_1.1.x86_64 > > libvirt-1.2.8-16.el7.x86_64 > > vdsm-4.16.12.1-3.el7ev.x86_64 > > @Nir - is the fix for bug 1196049 should be included in the aforementioned > build? No, you need libvirt-1.2.8-16.el7_1.1 and libvirt-python-1.2.8-6.el7_1.1 or later, see https://gerrit.ovirt.org/#/c/38201/3//COMMIT_MSG
(In reply to Nir Soffer from comment #21) > (In reply to Daniel Erez from comment #20) > > (In reply to Carlos Mestre González from comment #18) > > > Packages versions: > > > qemu-img-rhev-2.1.2-23.el7_1.1.x86_64 > > > libvirt-1.2.8-16.el7.x86_64 > > > vdsm-4.16.12.1-3.el7ev.x86_64 > > > > @Nir - is the fix for bug 1196049 should be included in the aforementioned > > build? > > No, you need libvirt-1.2.8-16.el7_1.1 and libvirt-python-1.2.8-6.el7_1.1 > or later, see https://gerrit.ovirt.org/#/c/38201/3//COMMIT_MSG @Carlos - can you please retest the described scenario with the libvirt and libvirt-python versions mentioned by Nir?
Hi Daniel, tested again in vt14.2 (same libvirt mentioned here) and works, each of the disks end up being the same size as the OS disk. So closing it again as duplicated of bz1196049 . *** This bug has been marked as a duplicate of bug 1196049 ***