Bug 1598015 - libvirtd crashed on target host when do migration with '--tls'
Summary: libvirtd crashed on target host when do migration with '--tls'
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: Fangge Jin
URL:
Whiteboard:
: 1598412 (view as bug list)
Depends On:
Blocks: 1600342
TreeView+ depends on / blocked
 
Reported: 2018-07-04 07:07 UTC by yafu
Modified: 2018-10-30 09:58 UTC (History)
7 users (show)

Fixed In Version: libvirt-4.5.0-3.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1600342 (view as bug list)
Environment:
Last Closed: 2018-10-30 09:57:31 UTC
Target Upstream Version:


Attachments (Terms of Use)
libvirtd log - error message (5.25 MB, application/x-tar)
2018-07-25 11:29 UTC, Fangge Jin
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:3113 None None None 2018-10-30 09:58:30 UTC
Red Hat Bugzilla 1598412 None CLOSED libvirtd crash on target Power9 host when migration from RHEL-7.5 to RHEL-ALT-7.6 with --tls 2019-10-28 11:58:38 UTC

Internal Links: 1598412

Description yafu 2018-07-04 07:07:46 UTC
Description of problem:
libvirtd crashed on target host when do migration with '--tls'

Version-Release number of selected component (if applicable):
libvirt-4.5.0-1.el7.x86_64
qemu-kvm-rhev-2.12.0-6.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Do migration with '--tls':
# virsh migrate iommu1 qemu+ssh://10.66.69.35/system --live --verbose --tls --unsafe
error: End of file while reading data: Ncat: Broken pipe.: Input/output error

Actual results:
libvirtd crashed on target host when do migration with '--tls'

Expected results:
Migration should be completed successfully.

Additional info:
1.backtrace of libvirtd process:
Core was generated by `/usr/sbin/libvirtd'.
Program terminated with signal 11, Segmentation fault.
#0  virJSONValueObjectGet (object=0x0, key=key@entry=0x7f392ec0ffe3 "qom-type") at util/virjson.c:818
818	    if (object->type != VIR_JSON_TYPE_OBJECT)

(gdb) bt
#0  virJSONValueObjectGet (object=0x0, key=key@entry=0x7f392ec0ffe3 "qom-type") at util/virjson.c:818
#1  0x00007f3979bab251 in virJSONValueObjectGetString (object=<optimized out>, key=key@entry=0x7f392ec0ffe3 "qom-type")
    at util/virjson.c:1258
#2  0x00007f392eb791bd in qemuMonitorAddObject (mon=0x7f394c015f50, props=props@entry=0x7f3967ecd4c0, 
    alias=alias@entry=0x7f3967ecd430) at qemu/qemu_monitor.c:3090
#3  0x00007f392eb2bb23 in qemuDomainAddTLSObjects (driver=driver@entry=0x7f39081151a0, vm=vm@entry=0x7f3908292b30, 
    asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_MIGRATION_IN, secProps=secProps@entry=0x7f3967ecd4c0, 
    tlsProps=tlsProps@entry=0x7f3967ecd4b8) at qemu/qemu_hotplug.c:1342
#4  0x00007f392eb6cad2 in qemuMigrationParamsEnableTLS (driver=driver@entry=0x7f39081151a0, vm=0x7f3908292b30, 
    tlsListen=tlsListen@entry=true, asyncJob=asyncJob@entry=2, tlsAlias=tlsAlias@entry=0x7f3967ecd608, 
    hostname=hostname@entry=0x0, migParams=migParams@entry=0x7f394c0008c0) at qemu/qemu_migration_params.c:878
#5  0x00007f392eb5aaa2 in qemuMigrationDstPrepareAny (driver=driver@entry=0x7f39081151a0, dconn=dconn@entry=0x7f395801f3a0, 
    cookiein=cookiein@entry=0x7f394c0032a0 "<qemu-migration>\n  <name>iommu1</name>\n  <uuid>1b3268d6-b59c-406b-a14c-33b000b15b6c</uuid>\n  <hostname>wlan-69-35.nay.redhat.com</hostname>\n  <hostuuid>5cd9f881-5529-11cb-b989-b4c8b0f5dd17</hostuuid>\n"..., cookieinlen=cookieinlen@entry=597, cookieout=cookieout@entry=0x7f3967ecdaa8, cookieoutlen=cookieoutlen@entry=0x7f3967ecda9c, 
    def=def@entry=0x7f3967ecd900, origname=origname@entry=0x0, st=st@entry=0x0, protocol=<optimized out>, port=49153, 
    autoPort=true, listenAddress=listenAddress@entry=0x0, nmigrate_disks=nmigrate_disks@entry=0, 
    migrate_disks=migrate_disks@entry=0x0, nbdPort=nbdPort@entry=0, migParams=migParams@entry=0x7f394c0008c0, 
    flags=flags@entry=65537) at qemu/qemu_migration.c:2505
#6  0x00007f392eb5d31b in qemuMigrationDstPrepareDirect (driver=driver@entry=0x7f39081151a0, 
    dconn=dconn@entry=0x7f395801f3a0, 
    cookiein=cookiein@entry=0x7f394c0032a0 "<qemu-migration>\n  <name>iommu1</name>\n  <uuid>1b3268d6-b59c-406b-a14c-33b000b15b6c</uuid>\n  <hostname>wlan-69-35.nay.redhat.com</hostname>\n  <hostuuid>5cd9f881-5529-11cb-b989-b4c8b0f5dd17</hostuuid>\n"..., cookieinlen=cookieinlen@entry=597, cookieout=cookieout@entry=0x7f3967ecdaa8, cookieoutlen=cookieoutlen@entry=0x7f3967ecda9c, 
    uri_in=0x0, uri_out=uri_out@entry=0x7f394c000ad0, def=def@entry=0x7f3967ecd900, origname=0x0, listenAddress=0x0, 
    nmigrate_disks=nmigrate_disks@entry=0, migrate_disks=0x0, nbdPort=0, migParams=migParams@entry=0x7f394c0008c0, 
    flags=flags@entry=65537) at qemu/qemu_migration.c:2833
#7  0x00007f392eba73b7 in qemuDomainMigratePrepare3Params (dconn=0x7f395801f3a0, params=<optimized out>, 
    nparams=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    cookiein=0x7f394c0032a0 "<qemu-migration>\n  <name>iommu1</name>\n  <uuid>1b3268d6-b59c-406b-a14c-33b000b15b6c</uuid>\n  <hostname>wlan-69-35.nay.redhat.com</hostname>\n  <hostuuid>5cd9f881-5529-11cb-b989-b4c8b0f5dd17</hostuuid>\n"..., 
    cookieinlen=597, cookieout=0x7f3967ecdaa8, cookieoutlen=0x7f3967ecda9c, uri_out=0x7f394c000ad0, flags=65537)
    at qemu/qemu_driver.c:12668
#8  0x00007f3979e38d6d in virDomainMigratePrepare3Params (dconn=0x7f395801f3a0, params=0x7f394c000990, nparams=1, 
    cookiein=0x7f394c0032a0 "<qemu-migration>\n  <name>iommu1</name>\n  <uuid>1b3268d6-b59c-406b-a14c-33b000b15b6c</uuid>\n  <hostname>wlan-69-35.nay.redhat.com</hostname>\n  <hostuuid>5cd9f881-5529-11cb-b989-b4c8b0f5dd17</hostuuid>\n"..., 
    cookieinlen=597, cookieout=cookieout@entry=0x7f3967ecdaa8, cookieoutlen=cookieoutlen@entry=0x7f3967ecda9c, 
    uri_out=0x7f394c000ad0, flags=65537) at libvirt-domain.c:4877
#9  0x0000562aecd80349 in remoteDispatchDomainMigratePrepare3Params (server=<optimized out>, msg=0x562aed10b300, 
    ret=0x7f394c000b40, args=0x7f394c000b10, rerr=0x7f3967ecdbc0, client=<optimized out>)
    at remote/remote_daemon_dispatch.c:5395
#10 remoteDispatchDomainMigratePrepare3ParamsHelper (server=<optimized out>, client=<optimized out>, msg=0x562aed10b300, 
    rerr=0x7f3967ecdbc0, args=0x7f394c000b10, ret=0x7f394c000b40) at remote/remote_daemon_dispatch_stubs.h:8347
#11 0x00007f3979d235e5 in virNetServerProgramDispatchCall (msg=0x562aed10b300, client=0x562aed10a480, server=0x562aed0b8fa0, 
    prog=0x562aed107650) at rpc/virnetserverprogram.c:437
#12 virNetServerProgramDispatch (prog=0x562aed107650, server=server@entry=0x562aed0b8fa0, client=client@entry=0x562aed10a480, 
    msg=msg@entry=0x562aed10b300) at rpc/virnetserverprogram.c:304
#13 0x00007f3979d2c34a in virNetServerProcessMsg (srv=srv@entry=0x562aed0b8fa0, client=0x562aed10a480, prog=<optimized out>, 
    msg=0x562aed10b300) at rpc/virnetserver.c:143
#14 0x00007f3979d2c798 in virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x562aed0b8fa0) at rpc/virnetserver.c:164
#15 0x00007f3979c0aa81 in virThreadPoolWorker (opaque=opaque@entry=0x562aed0acb00) at util/virthreadpool.c:167
#16 0x00007f3979c09850 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#17 0x00007f3976f21dd5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f3976c4baed in clone () from /lib64/libc.so.6





Additional info:

Comment 3 Peter Krempa 2018-07-04 07:31:10 UTC
This was most probably caused by:

commit c2f71bb295162a81c9085a184972cb93e5964451
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Tue May 22 07:38:22 2018 +0200

    qemu: hotplug: Refactor 'secret' props formatting to qemuMonitorCreateObjectProps

Comment 4 Dan Zheng 2018-07-06 05:59:15 UTC
From the automated job log, I can see libvirt-4.4.0-2.el7.ppc64le has not this issue.

Comment 5 Peter Krempa 2018-07-09 07:40:06 UTC
*** Bug 1598412 has been marked as a duplicate of this bug. ***

Comment 6 Peter Krempa 2018-07-10 15:29:45 UTC
Fixed upstream:

commit fac0dacd54c02b842c995d0999d9450d09d1e7cd
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Wed Jul 4 16:36:37 2018 +0200

    qemu: monitor: Make qemuMonitorAddObject more robust against programming errors
    
    Document and check that @props contains a pointer to a json object and
    check that both necessary fields are present. Also mark @props as
    NONNULL.
    
    Signed-off-by: Peter Krempa <pkrempa@redhat.com>
    Reviewed-by: Ján Tomko <jtomko@redhat.com>

commit 62ef8227e2717618c96fa17f2d4f5b7570bbe980
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Wed Jul 4 16:28:58 2018 +0200

    qemu: hotplug: Do not try to add secret object for TLS if it does not exist
    
    The check whether the object holding secret for decryption of the TLS
    environment was wrong and would always attempt to add the object. This
    lead to a crash due to recent refactors.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1598015
    
    Signed-off-by: Peter Krempa <pkrempa@redhat.com>
    Reviewed-by: Ján Tomko <jtomko@redhat.com>

Comment 9 Fangge Jin 2018-07-25 11:28:37 UTC
Reproduced with libvirt-4.5.0-1.el7.x86_64

Verified with libvirt-4.5.0-4.el7.x86_64, migration succeeds, but there is error in libvirtd.log:

Source:
2018-07-25 11:12:25.884+0000: 8375: debug : qemuMonitorJSONCheckError:385 : unable to execute QEMU command {"execute":"object-del","arguments":{"id":"objlibvirt_migrate_tls0"},"id":"libvirt-34"}: {"id":"libvirt-34","error":{"class":"GenericError","desc":"object 'objlibvirt_migrate_tls0' not found"}}


Target:
2018-07-25 11:12:25.618+0000: 10709: debug : qemuMonitorJSONCheckError:385 : unable to execute QEMU command {"execute":"object-del","arguments":{"id":"objlibvirt_migrate_tls0"},"id":"libvirt-13"}: {"id":"libvirt-13","error":{"class":"GenericError","desc":"object 'objlibvirt_migrate_tls0' not found"}}

2018-07-25 11:12:52.821+0000: 10710: debug : qemuMonitorJSONCheckError:385 : unable to execute QEMU command {"execute":"object-del","arguments":{"id":"libvirt_migrate-secret0"},"id":"libvirt-30"}: {"id":"libvirt-30","error":{"class":"GenericError","desc":"object 'libvirt_migrate-secret0' not found"}}

Comment 10 Fangge Jin 2018-07-25 11:29:35 UTC
Created attachment 1470503 [details]
libvirtd log -  error message

Comment 11 Peter Krempa 2018-07-30 07:40:13 UTC
That is expected. We always try to delete the object even when it is not present in qemu. Note that this was happening already for some time.

Comment 12 Fangge Jin 2018-07-30 07:43:11 UTC
(In reply to Peter Krempa from comment #11)
> That is expected. We always try to delete the object even when it is not
> present in qemu. Note that this was happening already for some time.

Thanks. I didn't check the libvirtd.log before during testing.

Comment 14 errata-xmlrpc 2018-10-30 09:57:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113


Note You need to log in before you can comment on or make changes to this bug.