Bug 1278727

Summary: "virsh domjobinfo" hangs on destination host during migration.
Product: Red Hat Enterprise Linux 7 Reporter: Fangge Jin <fjin>
Component: libvirtAssignee: Jiri Denemark <jdenemar>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.2CC: dyuan, mzhan, rbalakri, zpeng
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-1.3.3-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-03 18:30:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
libvirtd log on target host none

Description Fangge Jin 2015-11-06 09:57:12 UTC
Created attachment 1090546 [details]
libvirtd log on target host

Description of problem:
"virsh domjobinfo" hangs on destination host during migration.

Version-Release number of selected component (if applicable):
libvirt-1.2.17-13.el7.x86_64
qemu-kvm-rhev-2.3.0-31.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Prepare a running guest on source host.
# virsh list
 Id    Name                           State
----------------------------------------------------
 35    ceph-admin                     running

2.Set the migration speed to a small value: 1
# virsh migrate-setspeed ceph-admin 1

3.Migrate the guest to another host:
# virsh migrate ceph-admin qemu+ssh://10.66.85.92/system --live --verbose
root.85.92's password:
Migration:[1%]

Before migration completed, get the domjobinfo on target host, the command hangs for a while, then returns with error:
# virsh domjobinfo ceph-admin
error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePrepare3Params)

After migration completed, get the domjobinfo on target host again:
# virsh domjobinfo ceph-admin --completed
Job type:         Completed  
Time elapsed:     408155       ms
Time elapsed w/o network: 408144       ms
Data processed:   417.593 MiB
Data remaining:   0.000 B
Data total:       1.102 GiB
Memory processed: 417.593 MiB
Memory remaining: 0.000 B
Memory total:     1.102 GiB
Memory bandwidth: 1.026 MiB/s
Constant pages:   220266      
Normal pages:     106204      
Normal data:      414.859 MiB
Total downtime:   73           ms
Downtime w/o network: 62           ms
Setup time:       22           ms


Actual results:
Can't get domjobinfo on target host before migration completed.

Expected results:
"Get domjobinfo" successfully on target host before migration completed.


Additional info:
"virsh domjobinfo <domain>" can get domjobinfo successfully on source host during migration.

Comment 1 Jiri Denemark 2016-03-01 15:44:12 UTC
This is fixed upstream by commit v1.3.2-47-g3ecd73c:

commit 3ecd73c4d8962acfbc4bac05bcd47663f2bd791f
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Feb 29 12:34:08 2016 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Tue Mar 1 15:59:00 2016 +0100

    qemu: Don't try to fetch migration stats on destination
    
    Migration statistics are not available on the destination host and
    starting a query job during incoming migration is not allowed. Trying to
    do that would result in
    
        Timed out during operation: cannot acquire state change lock (held
        by remoteDispatchDomainMigratePrepare3Params)
    
    error. We should not even try to start the job.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1278727
    
    Signed-off-by: Jiri Denemark <jdenemar>

Comment 2 Mike McCune 2016-03-28 22:45:31 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 4 zhe peng 2016-04-11 08:37:59 UTC
I can reproduce this.
verified with build:
libvirt-1.3.3-1.el7.x86_64
qemu-kvm-rhev-2.5.0-4.el7.x86_64

step same as bug's description

during the migration, check domjobinfo in target machine.
#virsh domjobinfo test
error: Operation not supported: migration statistics are available only on the source host
worked as expect. move to verified.

Comment 6 errata-xmlrpc 2016-11-03 18:30:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html