Description of problem:
Start a guest with hugepage option '-mem-path', then do migration within the host, the migration could not finish even after 'total time: 95118867 milliseconds'
Version-Release number of selected component (if applicable):
host kernel: 3.10.0-302.el7.ppc64le
Guest kernel: 3.10.0-295.el7.ppc64le
Qemu-kvm-rhev: qemu-kvm-rhev-2.3.0-13.el7.ppc64le
How reproducible:
100%
Steps to Reproduce:
1. Mount hugepage in the host:
# mount -t hugetlbfs none /mnt/kvm_hugepage/ -o pagesize=16M
2. Echo 1024 nr_hugepages:
# echo 1024 > /proc/sys/vm/nr_hugepages
3. Start a guest with hugepage option '-mem-path':
# /usr/libexec/qemu-kvm \
> -name 'virt-tests-vm1' \
> -sandbox off \
> -machine pseries \
> -nodefaults \
> -vga std \
> -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20150803-144757-zdzQgUOr,server,nowait \
> -mon chardev=qmp_id_qmpmonitor1,mode=control \
> -chardev socket,id=qmp_id_catch_monitor,path=/tmp/monitor-catch_monitor-20150803-144757-zdzQgUOr,server,nowait \
> -mon chardev=qmp_id_catch_monitor,mode=control \
> -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20150803-144757-zdzQgUOr,server,nowait \
> -device spapr-vty,chardev=serial_id_serial0 \
> -device pci-ohci,id=usb1,bus=pci.0,addr=03 \
> -device (qemu)
(qemu) virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=04 \
> -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=/media/ngu1/autotest-devel/client/tests/virt/shared/data/images/RHEL-Server-7.2-ppc64le-virtio-scsi.qcow2 \
> -device scsi-hd,id=image1,drive=drive_image1 \
> -device virtio-net-pci,mac=9a:e4:e5:e6:e7:e8,id=idpXVEnf,vectors=4,netdev=idOOWXSB,bus=pci.0,addr=05 \
> -netdev tap,id=idOOWXSB,vhost=on \
> -m 8192 \
> -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
> -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
> -vnc :0 \
> -rtc base=utc,clock=host,driftfix=slew \
> -boot order=cdn,once=c,menu=off,strict=off \
> -mem-path /mnt/kvm_hugepage \
> -enable-kvm \
> -monitor stdio \
> -device usb-kbd,id=input0 -device usb-mouse,id=input1
QEMU 2.3.0 monitor - type 'help' for more information
(qemu) migrate -d tcp:0:5200
(qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off
Migration status: active
total time: 2521 milliseconds
expected downtime: 300 milliseconds
setup: 13 milliseconds
transferred ram: 82200 kbytes
throughput: 268.44 mbps
remaining ram: 7881020 kbytes
total ram: 8405312 kbytes
duplicate: 110805 pages
skipped: 0 pages
normal: 20267 pages
normal bytes: 81068 kbytes
dirty sync count: 0
(qemu)
......
(qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off
Migration status: active
total time: 95118867 milliseconds
expected downtime: 2723 milliseconds
setup: 13 milliseconds
transferred ram: 3118411303 kbytes
throughput: 268.64 mbps
remaining ram: 379452 kbytes
total ram: 8405312 kbytes
duplicate: 536319527 pages
skipped: 0 pages
normal: 776905124 pages
normal bytes: 3107620496 kbytes
dirty sync count: 12743
dirty pages rate: 22328 pages
(qemu)
(qemu) q
4. Start a destination guest in the same host with the same qemu cmd lines as above ones except changing '-vnc :1' and appending '-incoming tcp:0:5200' in the end
5. Begin to do migration and check migration status from the source guest from time to time
Actual results:
The migration could not finish even after 'total time: 95118867 milliseconds'
Expected results:
The migration could finish
Additional info:
Found the problem on LE guest, will check it on BE guest. And for x86_64, have done test for 2M hugepage, there is no the problem
Even if the system is idle, it can touch one or two 16MB pages easily, and it takes more time to migrate a 16 MB pages than to the guest to modify it.
I've checked with dgilbert and quintela, this is not a bug:
- on x86, this doesn't happen because hugepages are smaller (less time to transfer them),
- if we stop the VM or increase the migration speed (for instance, migrate_set_speed 10G), it works well,
So, if the migration can't be done at the given speed, the admin can either stop the VM or increase bandwidth.
In my case, a speed of 130M (~ 1040 mbps) is enough to migrate in 90 cycles.