Bug 1278727 - "virsh domjobinfo" hangs on destination host during migration.
"virsh domjobinfo" hangs on destination host during migration.
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.2
x86_64 Linux
unspecified Severity medium
: rc
: ---
Assigned To: Jiri Denemark
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-06 04:57 EST by Fangge Jin
Modified: 2016-11-03 14:30 EDT (History)
4 users (show)

See Also:
Fixed In Version: libvirt-1.3.3-1.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-11-03 14:30:27 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
libvirtd log on target host (203.53 KB, application/x-gzip)
2015-11-06 04:57 EST, Fangge Jin
no flags Details

  None (edit)
Description Fangge Jin 2015-11-06 04:57:12 EST
Created attachment 1090546 [details]
libvirtd log on target host

Description of problem:
"virsh domjobinfo" hangs on destination host during migration.

Version-Release number of selected component (if applicable):
libvirt-1.2.17-13.el7.x86_64
qemu-kvm-rhev-2.3.0-31.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Prepare a running guest on source host.
# virsh list
 Id    Name                           State
----------------------------------------------------
 35    ceph-admin                     running

2.Set the migration speed to a small value: 1
# virsh migrate-setspeed ceph-admin 1

3.Migrate the guest to another host:
# virsh migrate ceph-admin qemu+ssh://10.66.85.92/system --live --verbose
root@10.66.85.92's password:
Migration:[1%]

Before migration completed, get the domjobinfo on target host, the command hangs for a while, then returns with error:
# virsh domjobinfo ceph-admin
error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainMigratePrepare3Params)

After migration completed, get the domjobinfo on target host again:
# virsh domjobinfo ceph-admin --completed
Job type:         Completed  
Time elapsed:     408155       ms
Time elapsed w/o network: 408144       ms
Data processed:   417.593 MiB
Data remaining:   0.000 B
Data total:       1.102 GiB
Memory processed: 417.593 MiB
Memory remaining: 0.000 B
Memory total:     1.102 GiB
Memory bandwidth: 1.026 MiB/s
Constant pages:   220266      
Normal pages:     106204      
Normal data:      414.859 MiB
Total downtime:   73           ms
Downtime w/o network: 62           ms
Setup time:       22           ms


Actual results:
Can't get domjobinfo on target host before migration completed.

Expected results:
"Get domjobinfo" successfully on target host before migration completed.


Additional info:
"virsh domjobinfo <domain>" can get domjobinfo successfully on source host during migration.
Comment 1 Jiri Denemark 2016-03-01 10:44:12 EST
This is fixed upstream by commit v1.3.2-47-g3ecd73c:

commit 3ecd73c4d8962acfbc4bac05bcd47663f2bd791f
Author:     Jiri Denemark <jdenemar@redhat.com>
AuthorDate: Mon Feb 29 12:34:08 2016 +0100
Commit:     Jiri Denemark <jdenemar@redhat.com>
CommitDate: Tue Mar 1 15:59:00 2016 +0100

    qemu: Don't try to fetch migration stats on destination
    
    Migration statistics are not available on the destination host and
    starting a query job during incoming migration is not allowed. Trying to
    do that would result in
    
        Timed out during operation: cannot acquire state change lock (held
        by remoteDispatchDomainMigratePrepare3Params)
    
    error. We should not even try to start the job.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1278727
    
    Signed-off-by: Jiri Denemark <jdenemar@redhat.com>
Comment 2 Mike McCune 2016-03-28 18:45:31 EDT
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Comment 4 zhe peng 2016-04-11 04:37:59 EDT
I can reproduce this.
verified with build:
libvirt-1.3.3-1.el7.x86_64
qemu-kvm-rhev-2.5.0-4.el7.x86_64

step same as bug's description

during the migration, check domjobinfo in target machine.
#virsh domjobinfo test
error: Operation not supported: migration statistics are available only on the source host
worked as expect. move to verified.
Comment 6 errata-xmlrpc 2016-11-03 14:30:27 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html

Note You need to log in before you can comment on or make changes to this bug.