Bug 1250031 - Migration could not finish for guest started with hugepage option '-mem-path'
Migration could not finish for guest started with hugepage option '-mem-path'
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.2
ppc64le Unspecified
unspecified Severity low
: rc
: ---
Assigned To: Laurent Vivier
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-04 07:49 EDT by Gu Nini
Modified: 2015-08-11 09:58 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-08-11 09:58:46 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Gu Nini 2015-08-04 07:49:41 EDT
Description of problem:
Start a guest with hugepage option '-mem-path', then do migration within the host, the migration could not finish even after 'total time: 95118867 milliseconds'

Version-Release number of selected component (if applicable):
host kernel: 3.10.0-302.el7.ppc64le
Guest kernel: 3.10.0-295.el7.ppc64le
Qemu-kvm-rhev: qemu-kvm-rhev-2.3.0-13.el7.ppc64le

How reproducible:
100%


Steps to Reproduce:
1. Mount hugepage in the host:
# mount -t hugetlbfs none /mnt/kvm_hugepage/ -o pagesize=16M

2. Echo 1024 nr_hugepages:
# echo 1024 > /proc/sys/vm/nr_hugepages

3. Start a guest with hugepage option '-mem-path':

# /usr/libexec/qemu-kvm \
>     -name 'virt-tests-vm1'  \
>     -sandbox off  \
>     -machine pseries  \
>     -nodefaults  \
>     -vga std  \
>     -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20150803-144757-zdzQgUOr,server,nowait \
>     -mon chardev=qmp_id_qmpmonitor1,mode=control  \
>     -chardev socket,id=qmp_id_catch_monitor,path=/tmp/monitor-catch_monitor-20150803-144757-zdzQgUOr,server,nowait \
>     -mon chardev=qmp_id_catch_monitor,mode=control  \
>     -chardev socket,id=serial_id_serial0,path=/tmp/serial-serial0-20150803-144757-zdzQgUOr,server,nowait \
>     -device spapr-vty,chardev=serial_id_serial0 \
>     -device pci-ohci,id=usb1,bus=pci.0,addr=03 \
>     -device (qemu) 
(qemu) virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=04 \
>     -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=/media/ngu1/autotest-devel/client/tests/virt/shared/data/images/RHEL-Server-7.2-ppc64le-virtio-scsi.qcow2 \
>     -device scsi-hd,id=image1,drive=drive_image1 \
>     -device virtio-net-pci,mac=9a:e4:e5:e6:e7:e8,id=idpXVEnf,vectors=4,netdev=idOOWXSB,bus=pci.0,addr=05  \
>     -netdev tap,id=idOOWXSB,vhost=on  \
>     -m 8192  \
>     -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \
>     -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
>     -vnc :0  \
>     -rtc base=utc,clock=host,driftfix=slew  \
>     -boot order=cdn,once=c,menu=off,strict=off  \
>     -mem-path /mnt/kvm_hugepage \
>     -enable-kvm \
>     -monitor stdio \
>     -device usb-kbd,id=input0 -device usb-mouse,id=input1
QEMU 2.3.0 monitor - type 'help' for more information
(qemu) migrate -d tcp:0:5200
(qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off 
Migration status: active
total time: 2521 milliseconds
expected downtime: 300 milliseconds
setup: 13 milliseconds
transferred ram: 82200 kbytes
throughput: 268.44 mbps
remaining ram: 7881020 kbytes
total ram: 8405312 kbytes
duplicate: 110805 pages
skipped: 0 pages
normal: 20267 pages
normal bytes: 81068 kbytes
dirty sync count: 0
(qemu) 
......
(qemu) info migrate
capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off 
Migration status: active
total time: 95118867 milliseconds
expected downtime: 2723 milliseconds
setup: 13 milliseconds
transferred ram: 3118411303 kbytes
throughput: 268.64 mbps
remaining ram: 379452 kbytes
total ram: 8405312 kbytes
duplicate: 536319527 pages
skipped: 0 pages
normal: 776905124 pages
normal bytes: 3107620496 kbytes
dirty sync count: 12743
dirty pages rate: 22328 pages
(qemu) 
(qemu) q

4. Start a destination guest in the same host with the same qemu cmd lines as above ones except changing '-vnc :1' and appending '-incoming tcp:0:5200' in the end
5. Begin to do migration and check migration status from the source guest from time to time

Actual results:
The migration could not finish even after 'total time: 95118867 milliseconds'

Expected results:
The migration could finish

Additional info:
Found the problem on LE guest, will check it on BE guest. And for x86_64, have done test for 2M hugepage, there is no the problem
Comment 2 Laurent Vivier 2015-08-11 09:58:46 EDT
Even if the system is idle, it can touch one or two 16MB pages easily, and it takes more time to migrate a 16 MB pages than to the guest to modify it.

I've checked with dgilbert and quintela, this is not a bug:
- on x86, this doesn't happen because hugepages are smaller (less time to transfer them),
- if we stop the VM or increase the migration speed (for instance, migrate_set_speed 10G), it works well,

So, if the migration can't be done at the given speed, the admin can either stop the VM or increase bandwidth.

In my case, a speed of 130M (~ 1040 mbps) is enough to migrate in 90 cycles.

Note You need to log in before you can comment on or make changes to this bug.