RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 695394 - default migration speed is too low for guests with heavy IO
Summary: default migration speed is too low for guests with heavy IO
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.1
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: ---
Assignee: Jiri Denemark
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 752138
Blocks: 798682
TreeView+ depends on / blocked
 
Reported: 2011-04-11 15:32 UTC by Jes Sorensen
Modified: 2013-02-21 07:06 UTC (History)
18 users (show)

Fixed In Version: libvirt-0.10.0-0rc1.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-02-21 07:06:31 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2013:0276 0 normal SHIPPED_LIVE Moderate: libvirt security, bug fix, and enhancement update 2013-02-20 21:18:26 UTC

Description Jes Sorensen 2011-04-11 15:32:36 UTC 
Description of problem:
Running a 4GB guest, 2vCPUs, storage on LVM volumes or NFS, and heavy
buffered IO within the guest. Migration will stall and never complete
due to the buffered IO thrashing the memory of the guest.

Increasing the migration speed with migration_set_speed to a higher
value than the default makes the issue go away.

Version-Release number of selected component (if applicable):


How reproducible:
Every time

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Mike Cao 2011-05-29 08:51:58 UTC 
Is this bug same as Bug 601619  ? Dor said libvirt could handle it in that bug
Comment 2 Dor Laor 2011-05-29 10:21:14 UTC 
(In reply to comment #1)
> Is this bug same as Bug 601619  ? Dor said libvirt could handle it in that bug

It's similar but this one is more limited in scope.
Comment 4 FuXiangChun 2011-06-20 04:51:55 UTC 
this bug can be reproduce

host info:
# uname -r
2.6.32-158.el6.x86_64

# rpm -qa|grep kvm
qemu-kvm-0.12.1.2-2.165.el6.x86_64

guest info

rhel6.1

# uname -r
2.6.32-131.0.10.el6.x86_64

steps to reproduce(local migrate):

1. src 
/usr/libexec/qemu-kvm -M rhel6.1.0 -enable-kvm -name rhel6.1-64 -smp 2 -m 2G -uuid 9e6f04cf-2ad7-45aa-9333-2d2ee26570c6 -boot c -drive file=/dev/vg0/data2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:19:3b:80,bus=pci.0 -balloon none -monitor stdio  -vga cirrus -vnc :11

2.des
/usr/libexec/qemu-kvm -M rhel6.1.0 -enable-kvm -name rhel6.1-64 -smp 2 -m 2G -uuid 9e6f04cf-2ad7-45aa-9333-2d2ee26570c6 -boot c -drive file=/dev/vg0/data2,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:19:3b:80,bus=pci.0 -balloon none -monitor stdio  -vga cirrus -vnc :12 -incoming tcp:0:5555

3. stress --vm 2  in guest

4. migrate -d tcp:localhost:5555

5. after 30 mins, migration still can not successfully

6. migrate_set_speed 1G in src

7. check migrate status
(qemu) info migrate
Migration status: completed
Comment 5 Juan Quintela 2011-08-05 15:16:27 UTC 
And stalls happen or not?
Test something like ping from outside to guest, and see if there are too much packet lost during migration.



Later, Juan.
Comment 6 FuXiangChun 2011-08-30 10:53:11 UTC 
(In reply to comment #5)
> And stalls happen or not?
> Test something like ping from outside to guest, and see if there are too much
> packet lost during migration.
> 
> 
> 
> Later, Juan.

when do local migration, migration default transfer speed is about 37M/sec
after changed migrate_set_speed to 1G, migration transfer speed is about 170M.
if test ping from outside to guest packet don't lost during migration.

host info:
# uname -r
2.6.32-192.el6.x86_64
# rpm -qa|grep kvm
qemu-kvm-0.12.1.2-2.184.el6.x86_64

guest info:
# uname -r
2.6.32-191.el6.i686

# cat /etc/issue
Red Hat Enterprise Linux Server release 6.2 Beta (Santiago)

default speed info:

(qemu) info migrate
Migration status: active
transferred ram: 90881 kbytes
remaining ram: 3993092 kbytes
total ram: 4214796 kbytes
QEMU 0.9.1 monitor - type 'help' for more information

(qemu) info migrate
Migration status: active
transferred ram: 127135 kbytes
remaining ram: 3956908 kbytes
total ram: 4214796 kbytes
QEMU 0.9.1 monitor - type 'help' for more information

(qemu) info migrate
Migration status: active
transferred ram: 179874 kbytes
remaining ram: 3904272 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 212834 kbytes
remaining ram: 3871376 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 245791 kbytes
remaining ram: 3838484 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 291936 kbytes
remaining ram: 3792428 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 324897 kbytes
remaining ram: 3759532 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 361151 kbytes
remaining ram: 3723348 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 397149 kbytes
remaining ram: 3634812 kbytes
total ram: 4214796 kbytes


after setting migrate speed 1G, migration info:

(qemu) info migrate
Migration status: active
transferred ram: 782433 kbytes
remaining ram: 3237796 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 944833 kbytes
remaining ram: 3074260 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 1165022 kbytes
remaining ram: 2854524 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 1301184 kbytes
remaining ram: 2718660 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 1456158 kbytes
remaining ram: 2564552 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 1596031 kbytes
remaining ram: 2424972 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 1749662 kbytes
remaining ram: 2271892 kbytes
total ram: 4214796 kbytes

QEMU 0.9.1 monitor - type 'help' for more information
(qemu) info migrate
Migration status: active
transferred ram: 1898016 kbytes
remaining ram: 2124208 kbytes
total ram: 4214796 kbytes
Comment 9 Karen Noel 2012-07-11 12:24:23 UTC 
Should the default be changed in libvirt, or perhaps RHEV-M?
Comment 10 Jiri Denemark 2012-07-12 09:09:29 UTC 
We tried to increase the limit in the past but faced a bug in qemu, it wasn't responding to monitor commands until the migration finished. Juan, can you confirm this was fixed and we can safely increase the limit?
Comment 11 Jiri Denemark 2012-07-19 09:26:37 UTC 
OK, I found bug 752138, which is already fixed. I suppose that's the bug in qemu we faced when increasing the limit.
Comment 12 zhpeng 2012-07-25 08:53:32 UTC 
I can reproduce this with:
libvirt-0.9.13-3.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.297.el6_3.x86_64


I run # stress --vm 2 in guest

After 40 min migration not finished like:

[root@zhpeng ~]# virsh migrate --live aaa qemu+ssh://10.66.7.230/system --verbose
root.7.230's password: 
Migration: [ 88 %]

I use Gigabit Ethernet and migrate-getspeed is 1G, averange speed is about 110M/s

# ifconfig em1|grep 'TX bytes';sleep 10; ifconfig em1|grep 'TX bytes'
          RX bytes:7201162804 (6.7 GiB)  TX bytes:305255032191 (284.2 GiB)
          RX bytes:7227112468 (6.7 GiB)  TX bytes:306448336749 (285.4 GiB)

This collect in migration:
virsh # domjobinfo aaa
Job type:         Unbounded   
Time elapsed:     1000025      ms
Data processed:   107.860 GiB
Data remaining:   455.246 MiB
Data total:       4.016 GiB
Memory processed: 107.860 GiB
Memory remaining: 455.246 MiB
Memory total:     4.016 GiB

SPEED = Data processed / Time elapsed     ~= 110M/s
Comment 13 Jiri Denemark 2012-08-03 18:18:01 UTC 
Patch sent upstream: https://www.redhat.com/archives/libvir-list/2012-August/msg00268.html
Comment 14 Jiri Denemark 2012-08-09 14:45:01 UTC 
Fixed upstream by v0.10.0-rc0-62-g6cfdeaa:

commit 6cfdeaac552e1ef744f269ed4cef3d74e663d677
Author: Jiri Denemark <jdenemar>
Date:   Fri Aug 3 18:34:06 2012 +0200

    qemu: Migrate at unlimited speed by default
Comment 17 Jiri Denemark 2012-09-06 09:57:44 UTC 
This "unexpectedly failed" migration errors come from qemu-kvm. You may find some details about it in /var/log/libvirt/qemu/VM.log (on both source and destination host) if you are lucky.
Comment 18 zhpeng 2012-09-11 03:22:38 UTC 
kvm pkg:qemu-kvm-rhev-0.12.1.2-2.307.el6.x86_64

Reproduced steps:
For example:
libvirt-0.9.10-21.el6_3.3.x86_64.rpm

1, i prepare a guest with 2 vcpus and 4G mem, 
   in guest i run # stress --vm 2

2, migrate guest to dst

3, for i in {1..10000}; do virsh domjobinfo aaa >> aaa.jobinfo;done
...
Job type:         Unbounded   
Time elapsed:     104163       ms
Data processed:   3.167 GB
Data remaining:   423.777 MB
Data total:       3.922 GB
Memory processed: 3.167 GB
Memory remaining: 423.777 MB
Memory total:     3.922 GB
...

default ~= 30M/s

Verify steps:
pks: libvirt-0.10.1-1.el6.x86_64

1, i prepare a guest with 2 vcpus and 4G mem, 
   in guest i run # stress --vm 2

2, migrate guest to dst

3, for i in {1..10000}; do virsh domjobinfo aaa >> aaa.jobinfo.new;done
...
Job type:         Unbounded
Time elapsed:     80217        ms
Data processed:   7.573 GiB
Data remaining:   409.293 MiB
Data total:       3.922 GiB
Memory processed: 7.573 GiB
Memory remaining: 409.293 MiB
Memory total:     3.922 GiB
...

default ~=100M/s   (Gigabyte network, depends on network status.)

About comment 16 error, i can't reproduce it this time, i tried above 10 times.
And no qemu log in src/dst log. So, i will file it if i can reproduce it again.

Therefor, above the steps, i think this is verified, default migration speed is changed.
Comment 19 errata-xmlrpc 2013-02-21 07:06:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0276.html
Note You need to log in before you can comment on or make changes to this bug.