Bug 857578

Summary:	Improved live migration statistics - support in libvirt
Product:	Red Hat Enterprise Linux 7	Reporter:	Karen Noel <knoel>
Component:	libvirt	Assignee:	Jiri Denemark <jdenemar>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.0	CC:	adaora.onyia, chegu_vinod, cwei, dyuan, mzhan, sherold, virt-maint, weizhan, zpeng
Target Milestone:	rc	Keywords:	FutureFeature
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-1.0.3-1.el7	Doc Type:	Enhancement
Doc Text:		Story Points:	---
Clone Of:	857577
Clones:	RHEV_MigrationStats 1061570 (view as bug list)		Environment:
Last Closed:	2014-06-13 10:11:28 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	857577
Bug Blocks:	801536, 918410, 1061570

Description Karen Noel 2012-09-14 22:41:22 UTC

+++ This bug was initially created as a clone of Bug #857577 +++

Description of problem:

Information about live migration is not available to analyze performance and to tune. Additional statistics should be added.

- Total migration time
- Expected downtime
- Dirty page rate
- Additional stats requested by performance team, partners, libvirt team, etc.

Comment 1 Jiri Denemark 2013-02-25 10:08:47 UTC

Implemented upstream by commits v1.0.2-239-g4dd00f4 through v1.0.2-245-g8def329:

commit 8def32916d65ace1e9b0d35f2d71ac2938882c8a
Author: Jiri Denemark <jdenemar>
Date:   Tue Feb 12 20:29:17 2013 +0100

    qemu: Implement virDomainGetJobStats

commit 4121a77c1a9dc477bbbfcac7c44715f78cad4b61
Author: Jiri Denemark <jdenemar>
Date:   Fri Feb 8 09:58:03 2013 +0100

    qemu: Parse more fields from query-migrate QMP command
    
    As a side effect, this also fixes reporting disk migration process.
    It was added to memory migration progress, which was wrong. Disk
    progress has dedicated fields in virDomainJobInfo structure.

commit 38ab12251ecb5ed7d6f12b13047cb15480a3a55b
Author: Jiri Denemark <jdenemar>
Date:   Fri Feb 8 09:55:17 2013 +0100

    virsh: Use virDomainGetJobStats in domjobinfo if available

commit 34fd94278a9f478f59a080640bd8c0317dc13e8d
Author: Jiri Denemark <jdenemar>
Date:   Wed Jan 30 10:49:28 2013 +0100

    remote: Implement virDomainGetJobStats

commit 98b9acf5aa02551dd37d0209339aba2e22e4004a
Author: Jiri Denemark <jdenemar>
Date:   Wed Jan 30 10:28:54 2013 +0100

    remote: Auto-allocate params in remoteDeserializeTypedParameters
    
    remoteDeserializeTypedParameters can now be called with either
    preallocated params array (size of which is announced by nparams) or it
    can allocate params array according to the number of parameters received
    from the server.

commit 8eb7ab95c2d459968bf097dcd2d2bd068a76e628
Author: Jiri Denemark <jdenemar>
Date:   Mon Feb 18 23:20:08 2013 +0100

    python: Implement virDomainGetJobStats wrapper

commit 4dd00f4238df262092550d533496bbf0bc7522a2
Author: Jiri Denemark <jdenemar>
Date:   Sat Jan 26 00:30:49 2013 +0100

    Introduce virDomainGetJobStats API
    
    This is an extensible version of virDomainGetJobInfo.

Comment 2 zhe peng 2013-03-08 02:46:11 UTC

verify with build:
libvirt-1.0.3-1.el7.x86_64
qemu-kvm-1.3.0-8.el7.x86_64

step:
1: prepare two host for migration
2: start live migration without setting cache size
# virsh migrate-compcache rhel6
Compression cache: 64.000 MiB
# virsh migrate --live rhel6 qemu+ssh://ibm-x3650m4-06.qe.lab.eng.nay.redhat.com/system --unsafe --compressed

check domjobinfo
# virsh domjobinfo rhel6
Job type:         Unbounded   
Time elapsed:     1603         ms
Data processed:   47.004 MiB
Data remaining:   658.633 MiB
Data total:       1.125 GiB
Memory processed: 47.004 MiB
Memory remaining: 658.633 MiB
Memory total:     1.125 GiB
Constant pages:   114382      
Normal pages:     12005       
Normal data:      46.895 MiB
Expected downtime: 0            ms
Compression cache: 64.000 MiB
Compressed data:  0.000 B
Compressed pages: 0            
Compression cache misses: 12005        
Compression overflows: 0        

3:live migration with setting cache size
# virsh migrate-compcache rhel6 --size=640000000
Compression cache: 512.000 MiB
# virsh migrate --live rhel6 qemu+ssh://ibm-x3650m4-06.qe.lab.eng.nay.redhat.com/system --unsafe --compressed

check domjobinfo
# virsh domjobinfo rhel6
Job type:         Unbounded   
Time elapsed:     1358         ms
Data processed:   42.033 MiB
Data remaining:   1.017 GiB
Data total:       1.125 GiB
Memory processed: 42.033 MiB
Memory remaining: 1.017 GiB
Memory total:     1.125 GiB
Constant pages:   17747       
Normal pages:     10756       
Normal data:      42.016 MiB
Expected downtime: 0            ms
Compression cache: 512.000 MiB
Compressed data:  0.000 B
Compressed pages: 0            
Compression cache misses: 10756        
Compression overflows: 0            

move to verify.

Comment 3 Vinod Chegu 2014-02-09 17:18:05 UTC

A comment/question:
Currently there doesn't seem to be a way for libvirt to track and extract the statistics for a *completed* live guest migration (e.g. Total Migration time, Actual Downtime, Throughput, # of pages (or kB) migrated etc.). Understand from (an offine chat with Jiri Denemark ) that this is not a trivial thing to do in libvirt.  Curious to know if any of the KVM/RHEV customers today have really asked for this type of information ?  Would this become important some day in the future when there are benchmarks that include migration as part of their measured metrics?

Comment 4 Scott Herold 2014-02-10 20:46:24 UTC

Vinod, 1063486 created as a requirement for RHEV for statistics of completed live migration.

Comment 5 Vinod Chegu 2014-02-10 21:17:42 UTC

Thanks Scott. At the moment I don't seem to have the right credentials to get to see that specific BZ...but would be curious to know about its progress.

Comment 6 Ludek Smid 2014-06-13 10:11:28 UTC

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.