Bug 695394

Is this bug same as Bug 601619  ? Dor said libvirt could handle it in that bug

(In reply to comment #1)
> Is this bug same as Bug 601619  ? Dor said libvirt could handle it in that bug

It's similar but this one is more limited in scope.

this bug can be reproduce

host info:
# uname -r
2.6.32-158.el6.x86_64

# rpm -qa|grep kvm
qemu-kvm-0.12.1.2-2.165.el6.x86_64

guest info

rhel6.1

# uname -r
2.6.32-131.0.10.el6.x86_64

steps to reproduce(local migrate):

1. src 
/usr/libexec/qemu-kvm -M rhel6.1.0 -enable-kvm -name rhel6.1-64 -smp 2 -m 2G -uuid 9e6f04cf-2ad7-45aa-9333-2d2ee26570c6 -boot c -drive file=/dev/vg0/data2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:19:3b:80,bus=pci.0 -balloon none -monitor stdio  -vga cirrus -vnc :11

2.des
/usr/libexec/qemu-kvm -M rhel6.1.0 -enable-kvm -name rhel6.1-64 -smp 2 -m 2G -uuid 9e6f04cf-2ad7-45aa-9333-2d2ee26570c6 -boot c -drive file=/dev/vg0/data2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:19:3b:80,bus=pci.0 -balloon none -monitor stdio  -vga cirrus -vnc :12 -incoming tcp:0:5555

3. stress --vm 2  in guest

4. migrate -d tcp:localhost:5555

5. after 30 mins, migration still can not successfully

6. migrate_set_speed 1G in src

7. check migrate status
(qemu) info migrate
Migration status: completed

And stalls happen or not?
Test something like ping from outside to guest, and see if there are too much packet lost during migration.



Later, Juan.

(In reply to comment #5)
> And stalls happen or not?
> Test something like ping from outside to guest, and see if there are too much
> packet lost during migration.
> 
> 
> 
> Later, Juan.

when do local migration, migration default transfer speed is about 37M/sec
after changed migrate_set_speed to 1G, migration transfer speed is about 170M.
if test ping from outside to guest packet don't lost during migration.

host info:
# uname -r
2.6.32-192.el6.x86_64
# rpm -qa|grep kvm
qemu-kvm-0.12.1.2-2.184.el6.x86_64

guest info:
# uname -r
2.6.32-191.el6.i686

# cat /etc/issue
Red Hat Enterprise Linux Server release 6.2 Beta (Santiago)

default speed info:

(qemu) info migrate
Migration status: active
transferred ram: 90881 kbytes
remaining ram: 3993092 kbytes
total ram: 4214796 kbytes
QEMU 0.9.1 monitor - type 'help' for more information

(qemu) info migrate
Migration status: active
transferred ram: 127135 kbytes
remaining ram: 3956908 kbytes
total ram: 4214796 kbytes
QEMU 0.9.1 monitor - type 'help' for more information

(qemu) info migrate
Migration status: active
transferred ram: 179874 kbytes
remaining ram: 3904272 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 212834 kbytes
remaining ram: 3871376 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 245791 kbytes
remaining ram: 3838484 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 291936 kbytes
remaining ram: 3792428 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 324897 kbytes
remaining ram: 3759532 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 361151 kbytes
remaining ram: 3723348 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 397149 kbytes
remaining ram: 3634812 kbytes
total ram: 4214796 kbytes


after setting migrate speed 1G, migration info:

(qemu) info migrate
Migration status: active
transferred ram: 782433 kbytes
remaining ram: 3237796 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 944833 kbytes
remaining ram: 3074260 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 1165022 kbytes
remaining ram: 2854524 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 1301184 kbytes
remaining ram: 2718660 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 1456158 kbytes
remaining ram: 2564552 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 1596031 kbytes
remaining ram: 2424972 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 1749662 kbytes
remaining ram: 2271892 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 1898016 kbytes
remaining ram: 2124208 kbytes
total ram: 4214796 kbytes

Should the default be changed in libvirt, or perhaps RHEV-M?

We tried to increase the limit in the past but faced a bug in qemu, it wasn't responding to monitor commands until the migration finished. Juan, can you confirm this was fixed and we can safely increase the limit?

OK, I found bug 752138, which is already fixed. I suppose that's the bug in qemu we faced when increasing the limit.

I can reproduce this with:
libvirt-0.9.13-3.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.297.el6_3.x86_64


I run # stress --vm 2 in guest

After 40 min migration not finished like:

[root@zhpeng ~]# virsh migrate --live aaa qemu+ssh://10.66.7.230/system --verbose
root.7.230's password: 
Migration: [ 88 %]

I use Gigabit Ethernet and migrate-getspeed is 1G, averange speed is about 110M/s

# ifconfig em1|grep 'TX bytes';sleep 10; ifconfig em1|grep 'TX bytes'
          RX bytes:7201162804 (6.7 GiB)  TX bytes:305255032191 (284.2 GiB)
          RX bytes:7227112468 (6.7 GiB)  TX bytes:306448336749 (285.4 GiB)

This collect in migration:
virsh # domjobinfo aaa
Job type:         Unbounded   
Time elapsed:     1000025      ms
Data processed:   107.860 GiB
Data remaining:   455.246 MiB
Data total:       4.016 GiB
Memory processed: 107.860 GiB
Memory remaining: 455.246 MiB
Memory total:     4.016 GiB

SPEED = Data processed / Time elapsed     ~= 110M/s

Patch sent upstream: https://www.redhat.com/archives/libvir-list/2012-August/msg00268.html

Fixed upstream by v0.10.0-rc0-62-g6cfdeaa:

commit 6cfdeaac552e1ef744f269ed4cef3d74e663d677
Author: Jiri Denemark <jdenemar>
Date:   Fri Aug 3 18:34:06 2012 +0200

    qemu: Migrate at unlimited speed by default

This "unexpectedly failed" migration errors come from qemu-kvm. You may find some details about it in /var/log/libvirt/qemu/VM.log (on both source and destination host) if you are lucky.

kvm pkg:qemu-kvm-rhev-0.12.1.2-2.307.el6.x86_64

Reproduced steps:
For example:
libvirt-0.9.10-21.el6_3.3.x86_64.rpm

1, i prepare a guest with 2 vcpus and 4G mem, 
   in guest i run # stress --vm 2

2, migrate guest to dst

3, for i in {1..10000}; do virsh domjobinfo aaa >> aaa.jobinfo;done
...
Job type:         Unbounded   
Time elapsed:     104163       ms
Data processed:   3.167 GB
Data remaining:   423.777 MB
Data total:       3.922 GB
Memory processed: 3.167 GB
Memory remaining: 423.777 MB
Memory total:     3.922 GB
...

default ~= 30M/s

Verify steps:
pks: libvirt-0.10.1-1.el6.x86_64

1, i prepare a guest with 2 vcpus and 4G mem, 
   in guest i run # stress --vm 2

2, migrate guest to dst

3, for i in {1..10000}; do virsh domjobinfo aaa >> aaa.jobinfo.new;done
...
Job type:         Unbounded
Time elapsed:     80217        ms
Data processed:   7.573 GiB
Data remaining:   409.293 MiB
Data total:       3.922 GiB
Memory processed: 7.573 GiB
Memory remaining: 409.293 MiB
Memory total:     3.922 GiB
...

default ~=100M/s   (Gigabyte network, depends on network status.)

About comment 16 error, i can't reproduce it this time, i tried above 10 times.
And no qemu log in src/dst log. So, i will file it if i can reproduce it again.

Therefor, above the steps, i think this is verified, default migration speed is changed.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0276.html