Bug 985958 (MigrationThrottle)

Summary: Throttle-down guest to help with live migration convergence (backport to RHEL7.0)
Product: Red Hat Enterprise Linux 7 Reporter: Hai Huang <hhuang>
Component: qemu-kvmAssignee: Nigel Croxon <ncroxon>
Status: CLOSED CURRENTRELEASE QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 7.0CC: acathrow, adaora.onyia, hhuang, jshortt, juzhang, knoel, lagarcia, mrezanin, mtessun, nigel.croxon, peterm, qzhang, virt-maint
Target Milestone: rc   
Target Release: 7.0   
Hardware: Unspecified   
OS: Linux   
Whiteboard: FutureFeature
Fixed In Version: qemu-kvm-1.5.2-3.el7 Doc Type: Release Note
Doc Text:
[Note: This feature is not complete in RHEL 7.0 Beta. It is missing libvirt support and thus should be described in a release note.] Live Migration Convergence Busy enterprise workloads hosted on large-sized virtual machines tend to dirty memory faster than the transfer rate achieved using live guest migration. If the user chooses to force convergence of their migration by using a new migration capability,"auto-converge", then this change auto-detects the lack of convergence scenario and triggers a slowdown of the workload by explicitly disallowing the virtual CPUs from spending much time in the virtual machine context.
 Story Points: ---
Clone Of:
: 990319 (view as bug list) Environment:
Last Closed: 2014-06-13 12:12:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 975927, 723986, 861643, 990319, 1004480    

Description Hai Huang 2013-07-18 15:14:55 UTC 
Description of problem:

Backport "throttle down in live migration" feature in QEMU 1.6 to RHEL7.0:

http://thread.gmane.org/gmane.comp.emulators.qemu/218577
-------------------------------------------
From: Chegu Vinod <chegu_vinod <at> hp.com>
Subject: [PATCH v8 0/3] Throttle-down guest to help with live migration convergence
Newsgroups: gmane.comp.emulators.qemu
Date: 2013-06-24 09:49:40 GMT (3 weeks, 3 days, 5 hours and 19 minutes ago)

Busy enterprise workloads hosted on large sized VM's tend to dirty
memory faster than the transfer rate achieved via live guest migration.
Despite some good recent improvements (& using dedicated 10Gig NICs
between hosts) the live migration does NOT converge.

If a user chooses to force convergence of their migration via a new
migration capability "auto-converge" then this change will auto-detect
lack of convergence scenario and trigger a slow down of the workload
by explicitly disallowing the VCPUs from spending much time in the VM
context.

The migration thread tries to catchup and this eventually leads
to convergence in some "deterministic" amount of time. Yes it does
impact the performance of all the VCPUs but in my observation that
lasts only for a short duration of time. i.e. end up entering
stage 3 (downtime phase) soon after that. No external trigger is
required.

Thanks to Juan and Paolo for their useful suggestions.


Version-Release number of selected component (if applicable):
RHEL7.0


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Miroslav Rezanina 2013-08-09 11:44:09 UTC 
Fix included in qemu-kvm-1.5.2-3.el7
Comment 5 Qunfang Zhang 2013-12-02 05:29:05 UTC 
Tested this bug on qemu-kvm-1.5.3-19.el7.x86_64, with and without auto-converge options. 

Two test cases are suggested by Orit:

1. Migration doesn't finish without auto-converge(off), you need to confirm that
it finishes with auto-converge on. This is missing.

2. Migration finishes without auto-converge, in this case auto-converge should
not have any effect.

And currently I can only cover the second test cases, have not hit the point that migration doesn't finish without auto-converge and could finish with auto-converge on.

Steps:
1. Boot up a guest with -m 2G and running stress tools inside guest.
 #stress --vm-bytes *** --vm 1 --vm-keep

Result:

+------------------------------------------------------------------------------+
|    |downtime|migration speed | auto-converge | --vm-bytes  |migration finish?|
+------------------------------------------------------------------------------+
| 1  |default | default        |  on/off       | 256000000   |  No             |
+------------------------------------------------------------------------------+
| 2  |default | default        |  on/off       | 64000000    |  No             |
+------------------------------------------------------------------------------+
| 3  |   1s   | 62MB/s         |  on/off       | 64000000    |  Yes            |
+------------------------------------------------------------------------------+
| 4  |   1s   | 61MB/s         |  on/off       | 64000000    |   No            |
+------------------------------------------------------------------------------+
| 5  |   0.5s | 100MB/s        |  on/off       | >=50000000  |   No            |
+------------------------------------------------------------------------------+
| 6  |   0.5S | 100MB/s        |  on/off       | <=48000000  |  Yes            |
+------------------------------------------------------------------------------+

Compare the item 3& item 4, and also the item 5& item 6, the effect of the auto-converge is not that obvious in my experiment. Please fix me and give some suggestion on that. 

Thanks,
Qunfang
Comment 6 Orit Wasserman 2013-12-05 07:30:58 UTC 
(In reply to Qunfang Zhang from comment #5)
> Tested this bug on qemu-kvm-1.5.3-19.el7.x86_64, with and without
> auto-converge options. 
> 
> Two test cases are suggested by Orit:
> 
> 1. Migration doesn't finish without auto-converge(off), you need to confirm
> that
> it finishes with auto-converge on. This is missing.
> 
> 2. Migration finishes without auto-converge, in this case auto-converge
> should
> not have any effect.
> 
> And currently I can only cover the second test cases, have not hit the point
> that migration doesn't finish without auto-converge and could finish with
> auto-converge on.
> 
> Steps:
> 1. Boot up a guest with -m 2G and running stress tools inside guest.
>  #stress --vm-bytes *** --vm 1 --vm-keep
> 
> Result:
> 
> +----------------------------------------------------------------------------
> --+
> |    |downtime|migration speed | auto-converge | --vm-bytes  |migration
> finish?|
> +----------------------------------------------------------------------------
> --+
> | 1  |default | default        |  on/off       | 256000000   |  No          
> |
> +----------------------------------------------------------------------------
> --+
> | 2  |default | default        |  on/off       | 64000000    |  No          
> |
> +----------------------------------------------------------------------------
> --+
> | 3  |   1s   | 62MB/s         |  on/off       | 64000000    |  Yes         
> |
> +----------------------------------------------------------------------------
> --+
> | 4  |   1s   | 61MB/s         |  on/off       | 64000000    |   No         
> |
> +----------------------------------------------------------------------------
> --+
> | 5  |   0.5s | 100MB/s        |  on/off       | >=50000000  |   No         
> |
> +----------------------------------------------------------------------------
> --+
> | 6  |   0.5S | 100MB/s        |  on/off       | <=48000000  |  Yes         
> |
> +----------------------------------------------------------------------------
> --+
> 
> Compare the item 3& item 4, and also the item 5& item 6, the effect of the
> auto-converge is not that obvious in my experiment. Please fix me and give
> some suggestion on that. 
> 
> Thanks,
> Qunfang

Hi Qunfang,
This table for test case 2 looks good.
I would suggest also comparing CPU and memory usage of QEMU during migration (using top for example).
Comment 8 Qunfang Zhang 2014-01-07 07:41:58 UTC 
I test this bug on qemu-kvm-1.5.3-31.el7.x86_64 with Orit and Vinod(hp)'s suggestion, running specjbb inside guest, we could see big difference between auto-converge on and off. 

Steps:
1. Boot up a guest on host, (I use -m 2G, not big configuration but we could see the auto-converge magic performance enhancement).

 /usr/libexec/qemu-kvm -cpu SandyBridge -enable-kvm -m 2048 -smp 2,sockets=2,cores=1,threads=1 -enable-kvm -name t2-rhel6.4-64 -uuid 61b6c504-5a8b-4fe1-8347-6c929b750dde -k en-us -rtc base=localtime,clock=host,driftfix=slew -no-kvm-pit-reinjection -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=input0 -drive file=/mnt/rhel6.5-32-bak.qcow2,if=none,id=disk0,format=qcow2,werror=stop,rerror=stop,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x3,drive=disk0,id=disk0  -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,drive=drive-ide0-1-0,bus=ide.1,unit=0,id=cdrom -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=44:37:E6:5E:91:85,bus=pci.0,addr=0x5 -monitor stdio -qmp tcp:0:6666,server,nowait -chardev socket,path=/tmp/isa-serial,server,nowait,id=isa1 -device isa-serial,chardev=isa1,id=isa-serial1 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x8 -chardev socket,id=charchannel0,path=/tmp/serial-socket,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,path=/tmp/foo,server,nowait,id=foo -device virtconsole,chardev=foo,id=console0  -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -vnc :10 -k en-us -boot c -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 -device virtserialport,bus=virtio-serial0.0,chardev=qga0,name=org.qemu.guest_agent.0  -global  PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0

2. Running specjbb2005 inside guest. 

#sh  run_qzhang.sh

Note:

Some configuration:
In the SPECjbb.props file: 

input.jvm_instances=1
input.starting_number_warehouses=2
input.increment_number_warehouses=4
input.ending_number_warehouses=2

...

input.measurement_seconds=2400 

My run_qzhang.sh:

# cat run_qzhang.sh 
##
## This is an example of what a run sh script might look like
##

date
echo $CLASSPATH
CLASSPATH=./jbb.jar:./check.jar:$CLASSPATH
echo $CLASSPATH
export CLASSPATH

java -fullversion

java  -Xms8192k -Xmx512m -Xss256k -XX:+UseParallelOldGC -XX:+AggressiveOpts -XX:+UseBiasedLocking  -XX:SurvivorRatio=24 spec.jbb.JBBmain -propfile SPECjbb.props

date 

3. Set migration speed as 100M (my physical network is 1GB nic), and set migration downtime as 2s. (I used the default downtime 30ms, but migration never finish for both auto-converge on and off, so I use 2s here)

4. Migrate guest to destination host. 

Note:
The "top" info inside guest:
 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 2984 root      20   0  761m 275m  10m S 190.6 14.5  19:50.03 java              
    1 root      20   0  2900 1436 1216 S  0.0  0.1   0:00.95 init      

=====================

Result:

(1) For auto-converge=on: Migration always finished around 25s.

(qemu) info migrate
capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: on zero-blocks: off 
Migration status: completed
total time: 25295 milliseconds
downtime: 2044 milliseconds
setup: 5 milliseconds
transferred ram: 2415439 kbytes
throughput: 775.54 mbps
remaining ram: 0 kbytes
total ram: 2114008 kbytes
duplicate: 424825 pages
skipped: 0 pages
normal: 5252393 pages
normal bytes: 21009572 kbytes
(qemu) 

(2) While for auto-converge=off:

Migration has not finished after 6 mins (I used same configuration as when auto-converge=on).

(qemu) info migrate
capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: off zero-blocks: off 
Migration status: active
total time: 367866 milliseconds
expected downtime: 976 milliseconds
setup: 8 milliseconds
transferred ram: 33821285 kbytes
throughput: 765.27 mbps
remaining ram: 204824 kbytes
total ram: 2114008 kbytes
duplicate: 151319 pages
skipped: 0 pages
normal: 8439549 pages
normal bytes: 33758196 kbytes
dirty pages rate: 22809 pages
(qemu) 

(3) Cancel the migration process (auto-converge=off experiment), and then turn on auto-converge and re-migrate again. 

Result: Migration will finish after 20s. 

So, I think this bug could be verified. 

Hi, Orit

Could you help double confirm my steps and results?

Thanks
Qunfang
Comment 10 Orit Wasserman 2014-01-07 15:19:07 UTC 
(In reply to Qunfang Zhang from comment #8)
> I test this bug on qemu-kvm-1.5.3-31.el7.x86_64 with Orit and Vinod(hp)'s
> suggestion, running specjbb inside guest, we could see big difference
> between auto-converge on and off. 
> 
> Steps:
> 1. Boot up a guest on host, (I use -m 2G, not big configuration but we could
> see the auto-converge magic performance enhancement).
> 
>  /usr/libexec/qemu-kvm -cpu SandyBridge -enable-kvm -m 2048 -smp
> 2,sockets=2,cores=1,threads=1 -enable-kvm -name t2-rhel6.4-64 -uuid
> 61b6c504-5a8b-4fe1-8347-6c929b750dde -k en-us -rtc
> base=localtime,clock=host,driftfix=slew -no-kvm-pit-reinjection -device
> piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=input0
> -drive
> file=/mnt/rhel6.5-32-bak.qcow2,if=none,id=disk0,format=qcow2,werror=stop,
> rerror=stop,aio=native -device
> virtio-blk-pci,bus=pci.0,addr=0x3,drive=disk0,id=disk0  -drive
> if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device
> ide-drive,drive=drive-ide0-1-0,bus=ide.1,unit=0,id=cdrom -netdev
> tap,id=hostnet0,vhost=on -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=44:37:E6:5E:91:85,bus=pci.0,
> addr=0x5 -monitor stdio -qmp tcp:0:6666,server,nowait -chardev
> socket,path=/tmp/isa-serial,server,nowait,id=isa1 -device
> isa-serial,chardev=isa1,id=isa-serial1 -device
> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x8 -chardev
> socket,id=charchannel0,path=/tmp/serial-socket,server,nowait -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,
> name=com.redhat.rhevm.vdsm -chardev
> socket,path=/tmp/foo,server,nowait,id=foo -device
> virtconsole,chardev=foo,id=console0  -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x9 -vnc :10 -k en-us -boot c
> -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 -device
> virtserialport,bus=virtio-serial0.0,chardev=qga0,name=org.qemu.guest_agent.0
> -global  PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0
> 
> 2. Running specjbb2005 inside guest. 
> 
> #sh  run_qzhang.sh
> 
> Note:
> 
> Some configuration:
> In the SPECjbb.props file: 
> 
> input.jvm_instances=1
> input.starting_number_warehouses=2
> input.increment_number_warehouses=4
> input.ending_number_warehouses=2
> 
> ...
> 
> input.measurement_seconds=2400 
> 
> My run_qzhang.sh:
> 
> # cat run_qzhang.sh 
> ##
> ## This is an example of what a run sh script might look like
> ##
> 
> date
> echo $CLASSPATH
> CLASSPATH=./jbb.jar:./check.jar:$CLASSPATH
> echo $CLASSPATH
> export CLASSPATH
> 
> java -fullversion
> 
> java  -Xms8192k -Xmx512m -Xss256k -XX:+UseParallelOldGC -XX:+AggressiveOpts
> -XX:+UseBiasedLocking  -XX:SurvivorRatio=24 spec.jbb.JBBmain -propfile
> SPECjbb.props
> 
> date 
> 
> 3. Set migration speed as 100M (my physical network is 1GB nic), and set
> migration downtime as 2s. (I used the default downtime 30ms, but migration
> never finish for both auto-converge on and off, so I use 2s here)
> 
> 4. Migrate guest to destination host. 
> 
> Note:
> The "top" info inside guest:
>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
> 
>  2984 root      20   0  761m 275m  10m S 190.6 14.5  19:50.03 java          
> 
>     1 root      20   0  2900 1436 1216 S  0.0  0.1   0:00.95 init      
> 
> =====================
> 
> Result:
> 
> (1) For auto-converge=on: Migration always finished around 25s.
> 
> (qemu) info migrate
> capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: on zero-blocks:
> off 
> Migration status: completed
> total time: 25295 milliseconds
> downtime: 2044 milliseconds
> setup: 5 milliseconds
> transferred ram: 2415439 kbytes
> throughput: 775.54 mbps
> remaining ram: 0 kbytes
> total ram: 2114008 kbytes
> duplicate: 424825 pages
> skipped: 0 pages
> normal: 5252393 pages
> normal bytes: 21009572 kbytes
> (qemu) 
> 
> (2) While for auto-converge=off:
> 
> Migration has not finished after 6 mins (I used same configuration as when
> auto-converge=on).
> 
> (qemu) info migrate
> capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: off
> zero-blocks: off 
> Migration status: active
> total time: 367866 milliseconds
> expected downtime: 976 milliseconds
> setup: 8 milliseconds
> transferred ram: 33821285 kbytes
> throughput: 765.27 mbps
> remaining ram: 204824 kbytes
> total ram: 2114008 kbytes
> duplicate: 151319 pages
> skipped: 0 pages
> normal: 8439549 pages
> normal bytes: 33758196 kbytes
> dirty pages rate: 22809 pages
> (qemu) 
> 
> (3) Cancel the migration process (auto-converge=off experiment), and then
> turn on auto-converge and re-migrate again. 
> 
> Result: Migration will finish after 20s. 
> 
> So, I think this bug could be verified. 
> 
> Hi, Orit
> 
> Could you help double confirm my steps and results?
> 
> Thanks
> Qunfang

Your steps and results are great.

Can you try it with migration speed set to 1G?
Comment 11 juzhang 2014-01-08 02:24:24 UTC 
Hi Qunfang,

Could you have a try and update the result in the bz?

Best Regards,
Junyi
Comment 12 Qunfang Zhang 2014-01-08 03:33:40 UTC 
I will test it after I find the hosts and setup the environment.
Comment 13 Qunfang Zhang 2014-01-17 09:15:13 UTC 
Verified this bug with 1GB migration speed on qemu-kvm-1.5.3-37.el7.x86_64, the auto-converge works. 

Steps:

0. Host configuration: 
(1) Plug two Mellanox 40GB cards on two large hosts (32GB memory and 32 cpus)
# lspci | grep Ether
07:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

(2) Connect the two NIC cards directly and configure the IP address, I used the private static IP. 

1. Boot up a guest on source host with "-m 32G -smp 10".

# /usr/libexec/qemu-kvm  -cpu SandyBridge -M pc -enable-kvm -m 32G -smp 10,sockets=10,cores=1,threads=1 -name rhel6.4-64 -uuid 1519507c-ada7-4a20-b8c4-723198532cc0 -nodefconfig -nodefaults -monitor stdio -rtc base=utc,clock=host,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/qzhang/RHEL-Server-6.5-64-virtio.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d5:51:8a,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc :10 -vga std  -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6  -drive if=none,id=drive-fdc0-0-0,format=raw,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -qmp tcp:0:5555,server,nowait -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0

2. Boot up the guest on destination host with listening mode "-incoming tcp:0:5800".

3. Run specjbb inside guest, see comment 8. But this time I used the following configuration:

In the SPECjbb.props file: 

input.jvm_instances=1
input.starting_number_warehouses=10
input.increment_number_warehouses=4
input.ending_number_warehouses=10

In the run_qzhang.sh:

 java  -Xms8192m -Xmx8192m -Xss256k -XX:+UseParallelOldGC -XX:+AggressiveOpts
 -XX:+UseBiasedLocking  -XX:SurvivorRatio=24 spec.jbb.JBBmain -propfile
 SPECjbb.props

4. Set migration speed as 1GB, migration downtime as 1(s). 

5. Migrate guest without auto-converge first:

(qemu) info migrate
capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: off zero-blocks: off 
Migration status: active
total time: 121515 milliseconds
expected downtime: 983 milliseconds
setup: 26 milliseconds
transferred ram: 122767277 kbytes
throughput: 8590.17 mbps
remaining ram: 2295840 kbytes
total ram: 33571288 kbytes
duplicate: 7275156 pages
skipped: 0 pages
normal: 30631583 pages
normal bytes: 122526332 kbytes
dirty pages rate: 257867 pages


**Result**:
Migration does not finish after 2 minutes. 

6. Cancel the migration process and re-migrate with auto-onverge *on*.

(qemu) info migrate
capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: on zero-blocks: off 
Migration status: completed
total time: 24446 milliseconds
downtime: 1075 milliseconds
setup: 20 milliseconds
transferred ram: 22111874 kbytes
throughput: 8590.17 mbps
remaining ram: 0 kbytes
total ram: 33571288 kbytes
duplicate: 14485885 pages
skipped: 0 pages
normal: 27199722 pages
normal bytes: 108798888 kbytes

**Result**:

Migration could finish after 24 seconds.  I tried 5 times with and without auto-converge, all get the similar results. 

Hi, Orit

So based on above, I think we could verify this bug? 

Thanks,
Qunfang
Comment 14 Qunfang Zhang 2014-01-17 09:18:33 UTC 
Host info:

[root@hp-dl380pg8-02 ~]# cat /proc/cpuinfo 
......

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 45
model name	: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
stepping	: 7
microcode	: 0x710
cpu MHz		: 1995.086
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 7
cpu cores	: 8
apicid		: 47
initial apicid	: 47
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 3994.47
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:


[root@hp-dl380pg8-02 ~]# free -m
             total       used       free     shared    buffers     cached
Mem:         31880       3265      28614          0         43       2502
-/+ buffers/cache:        719      31160
Swap:        16135          0      16135


[root@hp-dl380pg8-02 ~]# lspci | grep Ether
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
03:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
03:00.2 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
03:00.3 Ethernet controller: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01)
07:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

[root@hp-dl380pg8-02 ~]# ethtool -i eth2
driver: mlx4_en
version: 2.0 (Dec 2011)
firmware-version: 2.11.500
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Comment 15 Qunfang Zhang 2014-01-21 07:04:34 UTC 
Hi, Orit

I would like to change the status to VERIFIED according to comment 8 and comment 13. Please point out if there's something wrong or missing. 

Thanks,
Qunfang
Comment 16 Orit Wasserman 2014-01-21 10:33:59 UTC 
(In reply to Qunfang Zhang from comment #13)
> Verified this bug with 1GB migration speed on qemu-kvm-1.5.3-37.el7.x86_64,
> the auto-converge works. 
> 
> Steps:
> 
> 0. Host configuration: 
> (1) Plug two Mellanox 40GB cards on two large hosts (32GB memory and 32 cpus)
> # lspci | grep Ether
> 07:00.0 Ethernet controller: Mellanox Technologies MT27500 Family
> [ConnectX-3]
> 
> (2) Connect the two NIC cards directly and configure the IP address, I used
> the private static IP. 
> 
> 1. Boot up a guest on source host with "-m 32G -smp 10".
> 
> # /usr/libexec/qemu-kvm  -cpu SandyBridge -M pc -enable-kvm -m 32G -smp
> 10,sockets=10,cores=1,threads=1 -name rhel6.4-64 -uuid
> 1519507c-ada7-4a20-b8c4-723198532cc0 -nodefconfig -nodefaults -monitor stdio
> -rtc base=utc,clock=host,driftfix=slew -no-kvm-pit-reinjection -no-shutdown
> -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
> file=/home/qzhang/RHEL-Server-6.5-64-virtio.qcow2,if=none,id=drive-virtio-
> disk0,format=qcow2,cache=none -device
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,
> id=virtio-disk0,bootindex=1 -drive
> if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device
> ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev
> tap,id=hostnet0,vhost=on -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d5:51:8a,bus=pci.0,
> addr=0x3 -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc
> :10 -vga std  -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 
> -drive if=none,id=drive-fdc0-0-0,format=raw,cache=none -global
> isa-fdc.driveA=drive-fdc0-0-0 -qmp tcp:0:5555,server,nowait -global
> PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0
> 
> 2. Boot up the guest on destination host with listening mode "-incoming
> tcp:0:5800".
> 
> 3. Run specjbb inside guest, see comment 8. But this time I used the
> following configuration:
> 
> In the SPECjbb.props file: 
> 
> input.jvm_instances=1
> input.starting_number_warehouses=10
> input.increment_number_warehouses=4
> input.ending_number_warehouses=10
> 
> In the run_qzhang.sh:
> 
>  java  -Xms8192m -Xmx8192m -Xss256k -XX:+UseParallelOldGC -XX:+AggressiveOpts
>  -XX:+UseBiasedLocking  -XX:SurvivorRatio=24 spec.jbb.JBBmain -propfile
>  SPECjbb.props
> 
> 4. Set migration speed as 1GB, migration downtime as 1(s). 
> 
> 5. Migrate guest without auto-converge first:
> 
> (qemu) info migrate
> capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: off
> zero-blocks: off 
> Migration status: active
> total time: 121515 milliseconds
> expected downtime: 983 milliseconds
> setup: 26 milliseconds
> transferred ram: 122767277 kbytes
> throughput: 8590.17 mbps
> remaining ram: 2295840 kbytes
> total ram: 33571288 kbytes
> duplicate: 7275156 pages
> skipped: 0 pages
> normal: 30631583 pages
> normal bytes: 122526332 kbytes
> dirty pages rate: 257867 pages
> 
> 
> **Result**:
> Migration does not finish after 2 minutes. 
> 
> 6. Cancel the migration process and re-migrate with auto-onverge *on*.
> 
> (qemu) info migrate
> capabilities: xbzrle: off x-rdma-pin-all: off auto-converge: on zero-blocks:
> off 
> Migration status: completed
> total time: 24446 milliseconds
> downtime: 1075 milliseconds
> setup: 20 milliseconds
> transferred ram: 22111874 kbytes
> throughput: 8590.17 mbps
> remaining ram: 0 kbytes
> total ram: 33571288 kbytes
> duplicate: 14485885 pages
> skipped: 0 pages
> normal: 27199722 pages
> normal bytes: 108798888 kbytes
> 
> **Result**:
> 
> Migration could finish after 24 seconds.  I tried 5 times with and without
> auto-converge, all get the similar results. 
> 
> Hi, Orit
> 
> So based on above, I think we could verify this bug? 
> 
> Thanks,
> Qunfang

Yes, the results look good.

Orit
Comment 17 Ludek Smid 2014-06-13 12:12:18 UTC 
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.