Created attachment 594993 [details]
/var/log/libvirt/qmeu/mig-0.log

Description of problem:
Can not migrate several guests concurrently

Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.295.el6.x86_64
libvirt-0.9.10-21.el6.x86_64

How reproducible:
60%

Steup:
1. Prepare 2 hosts and prepare a nfs which is mounted on both hosts, and setting the virt_use_nfs boolean on both sides

  # setsebool -P virt_use_nfs 1

   and close the iptable on both sides

  # iptables -F

2. Prepare 10 guests on each side (20 total) with shared image on nfs named mig-0 ~ mig-9 on source and mig-10 ~ mig-19 on target

3. On both side, setting tcp connection environment

a. Edit /etc/sysconfig/libvirtd
       LIBVIRTD_ARGS="--listen"

b. Edit /etc/libvirt/libvirtd.conf
       listen_tls = 0
       listen_tcp=1
       auth_tcp="none"

c. # service libvirtd restart

4. Prepare p2p migration environment on source and target hosts.
	On source:
	  # ssh-copy-id -i ~/.ssh/id_rsa.pub root@souce-ip
	On target:
	  # ssh-copy-id -i ~/.ssh/id_rsa.pub root@target-ip

Actions:
1.Start all the guest on both sides

   on source:

   # for i in {0..9}; do virsh start mig$i; done

   on target:

   # for i in {10..19}; do virsh start mig$i; done

2. Do migration on source with commands:
   
   on source:
   # for i in {0..9}; do virsh migrate --live mig-$i qemu+tcp://10.66.6.38/system --verbose & echo mig-$i; done
   
   And to the best of your abilities to run following command at the same time,
   On target:
   # for i in {0..9}; do virsh migrate --live mig-1$i qemu+tcp://10.66.5.143/system --verbose & echo mig-1$i; done

Actual results:
	There are some guests could not migrate to target with errors like:
	error: internal error Invalid file descriptor while waiting for monitor

	error: internal error Unable to set monitor close-on-exec flag

	error: Unable to write to monitor: Bad file descriptor

	error: An error occurred, but the cause is unknown

	error: internal error Error while processing monitor IO

	error: error: internal error Unknown JSON reply '{"execute":"qmp_capabilities","id":"libvirt-1"}'
	Unable to read from monitor: Bad file descriptor

	After migration:
	On target:
	# virsh list 
	 Id    Name                           State
	----------------------------------------------------
	 1     mig-10                         running
	 2     mig-11                         running
	 4     mig-13                         running
	 5     mig-14                         running
	 6     mig-15                         running
	 8     mig-17                         running
	 9     mig-18                         running
	 10    mig-19                         running
	 11    mig-0                          running
	 12    mig-8                          running
	 18    mig-9                          running
	 19    mig-1                          running
	 20    mig-4                          running

	On source:
	# virsh list 
	 Id    Name                           State
	----------------------------------------------------
	 3     mig-2                          running
	 4     mig-3                          running
	 6     mig-5                          running
	 7     mig-6                          running
	 8     mig-7                          running
	 12    mig-12                         running
	 20    mig-16                         running

Expected results:
	Migration succeed without error, and all 20 guests migrate to destination respectively.

Additional info:
	I will attach the source and target libvirtd.log and also for a /var/log/libvirt/qemu/$guest.log(almost all failed guest.log are same).
	
	I can only reproduce the bug in some special environment, but not sure what is the ultimate reason for this bug.
	
	And get same result using qemu+ssh to migrate the guests.