Paolo: do you think it's just a matter of adjusting the timeouts (in libvirt), or should we change the behavior of the commands involved?

I tried with my rhevm, but cannot reproduce.

Scenario 1:
1. create VMs on iscsi storage A
2. start them
3. move disk to iscsi storage B through clicking 'Move' in 'Disks' tab
Result:
rhevm told me the disk have finished to move to another domain

Scenario 2:
Repeat above steps on NFS storage
Result:
rhevm told me have failed to move the disk to another domain

Will try more VMs to load stress and see if reproducible next week, also append some questions here.

Hi Dafna,
 Can you please tell:
1. if template is necessary to reproduce your issue? 
2. all of VMs became non respsonsive in your case, or just some of them?
3. in my case, I don't see any "__com.redhat_drive-reopen" event in libvirtd.log but "blockdev_snapshot-sync " as well as " __com.redhat_drive-mirror" when disk was moved to new domain. It seems failed to open the disk path. Is there any chance we hit same issue?

Any steps I am missing?

(In reply to comment #5)
> Any steps I am missing?

yes. 

1. you need 3 domains not two (look at the bug's headline)
2. run about 20-30 vm's linked to template
3. install rhel and Win OS on the vms
4. use 2 hosts

Dafna,

Are you running qemu-kvm-0.12.1.2-2.295.el6_3.10, qemu-kvm-0.12.1.2-2.344.el6 or later? The kvm-autotest guys were seeing a somewhat similar issue while testing the fix for bug 881732, so I think it's worth it to eliminate the possibility of a regression.

this was tested on: 

Version-Release number of selected component (if applicable):

qemu-img-rhev-0.12.1.2-2.295.el6_3.5.x86_64
libvirt-0.9.10-21.el6_3.5.x86_64
vdsm-4.9.6-42.0.el6_3.x86_64

No qemu-kvm version listed. I could guess it's 0.12.1.2-2.295.el6_3.5 because of qemu-img, but it would be good to confirm it.

(In reply to comment #9)
> No qemu-kvm version listed. I could guess it's 0.12.1.2-2.295.el6_3.5
> because of qemu-img, but it would be good to confirm it.

qemu-kvm version is 0.12.1.2-2.295.el6_3.5

Ok, so this has nothing to do with bug 881732. Thanks.

The query-blockstats command is very cheap to execute inside QEMU, it does not acquire resources or perform blocking operations.  There are two issues that could prevent QEMU from replying:
1. The host is under heavy load and the QEMU process is not being scheduled.
2. A QEMU thread is not releasing the global mutex, causing the monitor (iothread) to become unresponsive.

Due to the test environment requirements (oVirt, 20 Windows XP VMs, multiple hosts, etc) I cannot easily reproduce the problem to dig deeper.  It would be easiest if you can provide access to a machine.

From the bug report it's not clear whether only libvirt is hung or whether both libvirt and QEMU are hung.  What does "the vm's become non-responsive" mean?  Does the guest stop responding to network ping?

What happens at the libvirt when you "move the disk to the 3ed domain"?  Is this invoking the libvirt blockpull command?

Please post the thread backtraces for both libvirtd and the qemu process of a hung VM.  You can collect this by installing the debuginfo packages for libvirtd and qemu-kvm.  Then use "ps aux" to find the libvirtd and relevant qemu processes and the pstack(1) command to collect backtraces.

Thanks,
Stefan

I have been working on this issue these days. But have no luck to reproduce this issue. I have 18VMs in total running on 1st storage domain and 2 on 2nd. Then move all os disks to 3rd storage domain. After waiting for a while, most of them migrated successfully, some failed on error below[failure-1] but still in UP state. 2 VMs PAUSED on error bellow[failure-2].

***[failure-1]***
" �<11>vdsm vm.Vm ERROR vmId=`5333c042-6140-437a-9b5d-94f4fce776ba`::Unable to prepare the volume path for the disk: vdb#012Traceback (most recent call last):#012  File "/usr/share/vdsm/libvirtvm.py", line 1860, in snapshot#012    self.cif.prepareVolumePath(newDrives[vmDevName])#012  File "/usr/share/vdsm/clientIF.py", line 278, in prepareVolumePath#012    raise vm.VolumeError(drive)#012VolumeError: Bad volume specification {'domainID': 'a6fb82ba-8ed1-4665-aa18-6c683e6e5344', 'name': u'vdb', 'format': 'cow', 'volumeID': '8abf7f6b-df39-4e78-895c-95978fc83313', 'imageID': 'e3b3c87a-9a52-4220-9755-83e312045ad8', 'poolID': '2a58c701-8bab-4613-8b33-2d8c862f7355', 'device': 'disk'}"

***[failure-2]***
Failed to complete Snapshot Auto-generated for Live Storage Migration of xxx creation for VM xxxx
User admin@internal have failed to move the disk xxx to domain xxx

Related packages:
qemu-kvm-rhev-0.12.1.2-2.348.el6.x86_64
libvirt-0.10.2-11.el6.x86_64
vdsm-4.10.2-1.0.el6.x86_64

Steps:
1. start 18 VMs on 1st storage domain on host1
2. start 2 VMs on 2nd storage domain on host2, VMs are created from the same template
3. move OS disk to 3nd storage domain one by one

I will retry again this scenario: all VMs created from template.

(In reply to comment #13)
> I have been working on this issue these days. But have no luck to reproduce
> this issue. I have 18VMs in total running on 1st storage domain and 2 on
> 2nd. Then move all os disks to 3rd storage domain. After waiting for a
> while, most of them migrated successfully, some failed on error
> below[failure-1] but still in UP state. 2 VMs PAUSED on error
> bellow[failure-2].
> 
> ***[failure-1]***
> " �<11>vdsm vm.Vm ERROR vmId=`5333c042-6140-437a-9b5d-94f4fce776ba`::Unable
> to prepare the volume path for the disk: vdb#012Traceback (most recent call
> last):#012  File "/usr/share/vdsm/libvirtvm.py", line 1860, in snapshot#012 
> self.cif.prepareVolumePath(newDrives[vmDevName])#012  File
> "/usr/share/vdsm/clientIF.py", line 278, in prepareVolumePath#012    raise
> vm.VolumeError(drive)#012VolumeError: Bad volume specification {'domainID':
> 'a6fb82ba-8ed1-4665-aa18-6c683e6e5344', 'name': u'vdb', 'format': 'cow',
> 'volumeID': '8abf7f6b-df39-4e78-895c-95978fc83313', 'imageID':
> 'e3b3c87a-9a52-4220-9755-83e312045ad8', 'poolID':
> '2a58c701-8bab-4613-8b33-2d8c862f7355', 'device': 'disk'}"
> 
> ***[failure-2]***
> Failed to complete Snapshot Auto-generated for Live Storage Migration of xxx
> creation for VM xxxx
> User admin@internal have failed to move the disk xxx to domain xxx
> 
> Related packages:
> qemu-kvm-rhev-0.12.1.2-2.348.el6.x86_64
> libvirt-0.10.2-11.el6.x86_64
> vdsm-4.10.2-1.0.el6.x86_64
> 
> Steps:
> 1. start 18 VMs on 1st storage domain on host1
> 2. start 2 VMs on 2nd storage domain on host2, VMs are created from the same
> template
> 3. move OS disk to 3nd storage domain one by one
> 
> I will retry again this scenario: all VMs created from template.


it sounds like you are creating an independent vms (not linked to the template) 
also, run the vms sporadically - I did not assign a specific vm to a specific host

From earlier discussion (on IRC) with Federico and Dafna, the reason is that there is extremely high I/O load.

Flush operations time out in the guest, and as a result qemu_aio_flush() is called from the IDE layer.  These take up to 60-90 seconds, during which the monitor is blocked.

I asked Dafna at the time about logs from the targets, or network usage statistics, or anything that could help understanding the root cause.

Retried with Dafna's instructions:
1. copy templates to 3 storage domains
2. create pool and define the number of VMs linked to template, target 1st storage domain
3. repeat step 1 with different template, target 2nd storage domain
4. move all of disks to 3rd storage domain one by one

Actual Result:
ALL failed to move to 3rd storage domain. Failure log like below[failure].

I will retry again with less VMs but high i/o in them then update here.

***[failure]***
2013-Jan-10, 10:31
	
User admin@internal have failed to move the disk a-rhel6u4-64bit_Disk1 to domain storage-on-amd.
	
	
2013-Jan-10, 10:30
	
User admin@internal failed to move the disk a-rhel6u4-64bit_Disk1 to domain storage-on-amd.
	
	
2013-Jan-10, 10:30
	
User admin@internal failed to move the disk a-rhel6u4-64bit_Disk1 to domain storage-on-amd.
	
	
2013-Jan-10, 10:27
	
User admin@internal moving the disk a-rhel6u4-64bit_Disk1 to domain storage-on-amd.
	
	
2013-Jan-10, 10:27
	
User admin@internal moving the disk a-rhel6u4-64bit_Disk1 to domain storage-on-amd.
	
	
2013-Jan-10, 10:26
	
User admin@internal moving the disk a-rhel6u4-64bit_Disk1 to domain storage-on-amd.
	
	
2013-Jan-10, 10:26
	
User admin@internal moving the disk a-rhel6u4-64bit_Disk1 to domain storage-on-amd.
	
	
2013-Jan-10, 10:26
	
Snapshot Auto-generated for Live Storage Migration of a-rhel6u4-64bit_Disk1 creation for VM pool-B-rhel6u4-3 has been completed.
	
	
2013-Jan-10, 10:25
	
User admin@internal moving the disk a-rhel6u4-64bit_Disk1 to domain storage-on-amd.
	
	
2013-Jan-10, 10:25
	
Snapshot Auto-generated for Live Storage Migration of a-rhel6u4-64bit_Disk1 creation for VM pool-B-rhel6u4-3 was initiated by admin@internal.
	
64f0d3f5
	
2013-Jan-09, 17:53
	
VM pool-B-rhel6u4-3 started on Host hp-dl385g7-11
	
7175b86a
	
2013-Jan-09, 17:52
	
VM pool-B-rhel6u4-3 was started by admin@internal (Host: hp-dl385g7-11).
	
7175b86a
	
2013-Jan-09, 17:45
	
VM pool-B-rhel6u4-3 was removed from VM Pool pool-B-rhel6u4 by admin@internal.
	
7f899c12
	
2013-Jan-09, 17:44
	
VM pool-B-rhel6u4-3 creation has been completed.
	
7b3488f4
	
2013-Jan-09, 17:44
	
VM pool-B-rhel6u4-3 creation was initiated by admin@internal.
	
7b3488f4

(In reply to comment #16)
> Retried with Dafna's instructions:
> 1. copy templates to 3 storage domains
> 2. create pool and define the number of VMs linked to template, target 1st
> storage domain
> 3. repeat step 1 with different template, target 2nd storage domain

Correction: repeat step 2 with different template, target 2nd storage domain
20 VMs on 1st storage domain, 10 on 2nd one
> 4. move all of disks to 3rd storage domain one by one

What storage did you use?  If iSCSI, was the target Linux too?  Can you gather any logs or even just network usage statistics?

Is it possible to get a setup that I can use from my home office to reproduce (and ssh into all RHEV-H machines + hopefully iSCSI targets too)?

Hi, Paolo

Would you please have a look comment21? Any suggestion to KVM QE for reproducing this bug? Thanks.

Hi juzhang,

at this point I'm not even sure this is a qemu-kvm bug.  It may be simply that migration is exhausting the I/O or network bandwidth.  After I manage to reproduce it on chayang's infrastructure (probably tomorrow), I'll summarize what's going on.

(In reply to comment #23)
> Hi juzhang,
> 
> at this point I'm not even sure this is a qemu-kvm bug.  It may be simply
> that migration is exhausting the I/O or network bandwidth.  After I manage
> to reproduce it on chayang's infrastructure (probably tomorrow), I'll
> summarize what's going on.

Ok, Please update in this bz if you need KVM QE further testing.

Dafna, chayang, can you please confirm that the source and destination LUNs are mapped to _different_ disks, i.e. that you are not doing a "fake" migration?

(In reply to comment #31)
> Dafna, chayang, can you please confirm that the source and destination LUNs
> are mapped to _different_ disks, i.e. that you are not doing a "fake"
> migration?

I'm not doing fake migration - the luns in dst and src are different.

Reassigning it back to Paolo.

Yes, it's certainly reproducible.  QEMU is happily sending a lot of flush commands, which are very expensive.  I might have a fix.

Managed to reproduce 

From engine.log
===============
2015-12-28 10:05:07,767 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [2a9f7c7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM VM-19 is not responding.
2015-12-28 10:05:07,827 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [2a9f7c7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM VM-20 is not responding.
2015-12-28 10:05:07,927 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [2a9f7c7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM VM-14 is not responding.
2015-12-28 10:05:08,027 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [2a9f7c7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM VM-15 is not responding.
2015-12-28 10:05:08,059 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [2a9f7c7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM VM-6 is not responding.
2015-12-28 10:05:08,164 WARN  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [2a9f7c7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM VM-4 is not responding.



Reproduction steps 
==================
create the following setup: 
create 3 iscsi storage domains 100GB each
create 20 pool vm's from template on 1 domain
create 2 vm's from the same template, as thin provision and one as clone of template on the second domain

1. Run all vms on 2 hosts
2. Move the disks to the 3ed domain


Versions
========
vdsm-4.16.31-1.el6ev.x86_64
libvirt-0.10.2-54.el6_7.2.x86_64
qemu-kvm-rhev-0.12.1.2-2.479.el6_7.3.x86_64
qemu-img-rhev-0.12.1.2-2.479.el6_7.3.x86_64
RHEL 6.7 - kernel-2.6.32-573.el6.x86_64

The patches for this are ready, and I will submit them upstream in a week or two, for inclusion in QEMU 2.8.

Cause: completing a disk migration tried to quiesce all disks

Consequence: when many migrations were in progress at the same time, quiescence took a long time and could cause I/O aborts in the guest

Fix: completing a particular disk migration only quiesce the source of that migration job

Result: quiescence doesn't take a long time to achieve

Fix included in qemu-kvm-0.12.1.2-2.499.el6

Hi Aharo,

Would you please help verify this bz? Since kvm qe was not able to reproduce this bz, and also we have no proper environment for now.
Many thanks.

Qianqian

I replaced Aharon in managing the storage QE. 
This bug is verified, non of the VMs bocome non-responsive during migration

(In reply to Raz Tamir from comment #67)
> I replaced Aharon in managing the storage QE. 
> This bug is verified, non of the VMs bocome non-responsive during migration

Thanks, Raz

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0621.html