RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1598015 - libvirtd crashed on target host when do migration with '--tls'
Summary: libvirtd crashed on target host when do migration with '--tls'
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: Fangge Jin
URL:
Whiteboard:
: 1598412 (view as bug list)
Depends On:
Blocks: 1600342
TreeView+ depends on / blocked
 
Reported: 2018-07-04 07:07 UTC by yafu
Modified: 2018-10-30 09:58 UTC (History)
7 users (show)

Fixed In Version: libvirt-4.5.0-3.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1600342 (view as bug list)
Environment:
Last Closed: 2018-10-30 09:57:31 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
libvirtd log - error message (5.25 MB, application/x-tar)
2018-07-25 11:29 UTC, Fangge Jin 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1598412 0 unspecified CLOSED libvirtd crash on target Power9 host when migration from RHEL-7.5 to RHEL-ALT-7.6 with --tls 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHSA-2018:3113 0 None None None 2018-10-30 09:58:30 UTC

Internal Links: 1598412

Description yafu 2018-07-04 07:07:46 UTC 
Description of problem:
libvirtd crashed on target host when do migration with '--tls'

Version-Release number of selected component (if applicable):
libvirt-4.5.0-1.el7.x86_64
qemu-kvm-rhev-2.12.0-6.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Do migration with '--tls':
# virsh migrate iommu1 qemu+ssh://10.66.69.35/system --live --verbose --tls --unsafe
error: End of file while reading data: Ncat: Broken pipe.: Input/output error

Actual results:
libvirtd crashed on target host when do migration with '--tls'

Expected results:
Migration should be completed successfully.

Additional info:
1.backtrace of libvirtd process:
Core was generated by `/usr/sbin/libvirtd'.
Program terminated with signal 11, Segmentation fault.
#0  virJSONValueObjectGet (object=0x0, key=key@entry=0x7f392ec0ffe3 "qom-type") at util/virjson.c:818
818	    if (object->type != VIR_JSON_TYPE_OBJECT)

(gdb) bt
#0  virJSONValueObjectGet (object=0x0, key=key@entry=0x7f392ec0ffe3 "qom-type") at util/virjson.c:818
#1  0x00007f3979bab251 in virJSONValueObjectGetString (object=<optimized out>, key=key@entry=0x7f392ec0ffe3 "qom-type")
    at util/virjson.c:1258
#2  0x00007f392eb791bd in qemuMonitorAddObject (mon=0x7f394c015f50, props=props@entry=0x7f3967ecd4c0, 
    alias=alias@entry=0x7f3967ecd430) at qemu/qemu_monitor.c:3090
#3  0x00007f392eb2bb23 in qemuDomainAddTLSObjects (driver=driver@entry=0x7f39081151a0, vm=vm@entry=0x7f3908292b30, 
    asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_MIGRATION_IN, secProps=secProps@entry=0x7f3967ecd4c0, 
    tlsProps=tlsProps@entry=0x7f3967ecd4b8) at qemu/qemu_hotplug.c:1342
#4  0x00007f392eb6cad2 in qemuMigrationParamsEnableTLS (driver=driver@entry=0x7f39081151a0, vm=0x7f3908292b30, 
    tlsListen=tlsListen@entry=true, asyncJob=asyncJob@entry=2, tlsAlias=tlsAlias@entry=0x7f3967ecd608, 
    hostname=hostname@entry=0x0, migParams=migParams@entry=0x7f394c0008c0) at qemu/qemu_migration_params.c:878
#5  0x00007f392eb5aaa2 in qemuMigrationDstPrepareAny (driver=driver@entry=0x7f39081151a0, dconn=dconn@entry=0x7f395801f3a0, 
    cookiein=cookiein@entry=0x7f394c0032a0 "<qemu-migration>\n  <name>iommu1</name>\n  <uuid>1b3268d6-b59c-406b-a14c-33b000b15b6c</uuid>\n  <hostname>wlan-69-35.nay.redhat.com</hostname>\n  <hostuuid>5cd9f881-5529-11cb-b989-b4c8b0f5dd17</hostuuid>\n"..., cookieinlen=cookieinlen@entry=597, cookieout=cookieout@entry=0x7f3967ecdaa8, cookieoutlen=cookieoutlen@entry=0x7f3967ecda9c, 
    def=def@entry=0x7f3967ecd900, origname=origname@entry=0x0, st=st@entry=0x0, protocol=<optimized out>, port=49153, 
    autoPort=true, listenAddress=listenAddress@entry=0x0, nmigrate_disks=nmigrate_disks@entry=0, 
    migrate_disks=migrate_disks@entry=0x0, nbdPort=nbdPort@entry=0, migParams=migParams@entry=0x7f394c0008c0, 
    flags=flags@entry=65537) at qemu/qemu_migration.c:2505
#6  0x00007f392eb5d31b in qemuMigrationDstPrepareDirect (driver=driver@entry=0x7f39081151a0, 
    dconn=dconn@entry=0x7f395801f3a0, 
    cookiein=cookiein@entry=0x7f394c0032a0 "<qemu-migration>\n  <name>iommu1</name>\n  <uuid>1b3268d6-b59c-406b-a14c-33b000b15b6c</uuid>\n  <hostname>wlan-69-35.nay.redhat.com</hostname>\n  <hostuuid>5cd9f881-5529-11cb-b989-b4c8b0f5dd17</hostuuid>\n"..., cookieinlen=cookieinlen@entry=597, cookieout=cookieout@entry=0x7f3967ecdaa8, cookieoutlen=cookieoutlen@entry=0x7f3967ecda9c, 
    uri_in=0x0, uri_out=uri_out@entry=0x7f394c000ad0, def=def@entry=0x7f3967ecd900, origname=0x0, listenAddress=0x0, 
    nmigrate_disks=nmigrate_disks@entry=0, migrate_disks=0x0, nbdPort=0, migParams=migParams@entry=0x7f394c0008c0, 
    flags=flags@entry=65537) at qemu/qemu_migration.c:2833
#7  0x00007f392eba73b7 in qemuDomainMigratePrepare3Params (dconn=0x7f395801f3a0, params=<optimized out>, 
    nparams=<optimized out>, 
---Type <return> to continue, or q <return> to quit---
    cookiein=0x7f394c0032a0 "<qemu-migration>\n  <name>iommu1</name>\n  <uuid>1b3268d6-b59c-406b-a14c-33b000b15b6c</uuid>\n  <hostname>wlan-69-35.nay.redhat.com</hostname>\n  <hostuuid>5cd9f881-5529-11cb-b989-b4c8b0f5dd17</hostuuid>\n"..., 
    cookieinlen=597, cookieout=0x7f3967ecdaa8, cookieoutlen=0x7f3967ecda9c, uri_out=0x7f394c000ad0, flags=65537)
    at qemu/qemu_driver.c:12668
#8  0x00007f3979e38d6d in virDomainMigratePrepare3Params (dconn=0x7f395801f3a0, params=0x7f394c000990, nparams=1, 
    cookiein=0x7f394c0032a0 "<qemu-migration>\n  <name>iommu1</name>\n  <uuid>1b3268d6-b59c-406b-a14c-33b000b15b6c</uuid>\n  <hostname>wlan-69-35.nay.redhat.com</hostname>\n  <hostuuid>5cd9f881-5529-11cb-b989-b4c8b0f5dd17</hostuuid>\n"..., 
    cookieinlen=597, cookieout=cookieout@entry=0x7f3967ecdaa8, cookieoutlen=cookieoutlen@entry=0x7f3967ecda9c, 
    uri_out=0x7f394c000ad0, flags=65537) at libvirt-domain.c:4877
#9  0x0000562aecd80349 in remoteDispatchDomainMigratePrepare3Params (server=<optimized out>, msg=0x562aed10b300, 
    ret=0x7f394c000b40, args=0x7f394c000b10, rerr=0x7f3967ecdbc0, client=<optimized out>)
    at remote/remote_daemon_dispatch.c:5395
#10 remoteDispatchDomainMigratePrepare3ParamsHelper (server=<optimized out>, client=<optimized out>, msg=0x562aed10b300, 
    rerr=0x7f3967ecdbc0, args=0x7f394c000b10, ret=0x7f394c000b40) at remote/remote_daemon_dispatch_stubs.h:8347
#11 0x00007f3979d235e5 in virNetServerProgramDispatchCall (msg=0x562aed10b300, client=0x562aed10a480, server=0x562aed0b8fa0, 
    prog=0x562aed107650) at rpc/virnetserverprogram.c:437
#12 virNetServerProgramDispatch (prog=0x562aed107650, server=server@entry=0x562aed0b8fa0, client=client@entry=0x562aed10a480, 
    msg=msg@entry=0x562aed10b300) at rpc/virnetserverprogram.c:304
#13 0x00007f3979d2c34a in virNetServerProcessMsg (srv=srv@entry=0x562aed0b8fa0, client=0x562aed10a480, prog=<optimized out>, 
    msg=0x562aed10b300) at rpc/virnetserver.c:143
#14 0x00007f3979d2c798 in virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x562aed0b8fa0) at rpc/virnetserver.c:164
#15 0x00007f3979c0aa81 in virThreadPoolWorker (opaque=opaque@entry=0x562aed0acb00) at util/virthreadpool.c:167
#16 0x00007f3979c09850 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#17 0x00007f3976f21dd5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f3976c4baed in clone () from /lib64/libc.so.6





Additional info:
Comment 3 Peter Krempa 2018-07-04 07:31:10 UTC 
This was most probably caused by:

commit c2f71bb295162a81c9085a184972cb93e5964451
Author: Peter Krempa <pkrempa>
Date:   Tue May 22 07:38:22 2018 +0200

    qemu: hotplug: Refactor 'secret' props formatting to qemuMonitorCreateObjectProps
Comment 4 Dan Zheng 2018-07-06 05:59:15 UTC 
From the automated job log, I can see libvirt-4.4.0-2.el7.ppc64le has not this issue.
Comment 5 Peter Krempa 2018-07-09 07:40:06 UTC 
*** Bug 1598412 has been marked as a duplicate of this bug. ***
Comment 6 Peter Krempa 2018-07-10 15:29:45 UTC 
Fixed upstream:

commit fac0dacd54c02b842c995d0999d9450d09d1e7cd
Author: Peter Krempa <pkrempa>
Date:   Wed Jul 4 16:36:37 2018 +0200

    qemu: monitor: Make qemuMonitorAddObject more robust against programming errors
    
    Document and check that @props contains a pointer to a json object and
    check that both necessary fields are present. Also mark @props as
    NONNULL.
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>

commit 62ef8227e2717618c96fa17f2d4f5b7570bbe980
Author: Peter Krempa <pkrempa>
Date:   Wed Jul 4 16:28:58 2018 +0200

    qemu: hotplug: Do not try to add secret object for TLS if it does not exist
    
    The check whether the object holding secret for decryption of the TLS
    environment was wrong and would always attempt to add the object. This
    lead to a crash due to recent refactors.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1598015
    
    Signed-off-by: Peter Krempa <pkrempa>
    Reviewed-by: Ján Tomko <jtomko>
Comment 9 Fangge Jin 2018-07-25 11:28:37 UTC 
Reproduced with libvirt-4.5.0-1.el7.x86_64

Verified with libvirt-4.5.0-4.el7.x86_64, migration succeeds, but there is error in libvirtd.log:

Source:
2018-07-25 11:12:25.884+0000: 8375: debug : qemuMonitorJSONCheckError:385 : unable to execute QEMU command {"execute":"object-del","arguments":{"id":"objlibvirt_migrate_tls0"},"id":"libvirt-34"}: {"id":"libvirt-34","error":{"class":"GenericError","desc":"object 'objlibvirt_migrate_tls0' not found"}}


Target:
2018-07-25 11:12:25.618+0000: 10709: debug : qemuMonitorJSONCheckError:385 : unable to execute QEMU command {"execute":"object-del","arguments":{"id":"objlibvirt_migrate_tls0"},"id":"libvirt-13"}: {"id":"libvirt-13","error":{"class":"GenericError","desc":"object 'objlibvirt_migrate_tls0' not found"}}

2018-07-25 11:12:52.821+0000: 10710: debug : qemuMonitorJSONCheckError:385 : unable to execute QEMU command {"execute":"object-del","arguments":{"id":"libvirt_migrate-secret0"},"id":"libvirt-30"}: {"id":"libvirt-30","error":{"class":"GenericError","desc":"object 'libvirt_migrate-secret0' not found"}}
Comment 10 Fangge Jin 2018-07-25 11:29:35 UTC 
Created attachment 1470503 [details]
libvirtd log -  error message
Comment 11 Peter Krempa 2018-07-30 07:40:13 UTC 
That is expected. We always try to delete the object even when it is not present in qemu. Note that this was happening already for some time.
Comment 12 Fangge Jin 2018-07-30 07:43:11 UTC 
(In reply to Peter Krempa from comment #11)
> That is expected. We always try to delete the object even when it is not
> present in qemu. Note that this was happening already for some time.

Thanks. I didn't check the libvirtd.log before during testing.
Comment 14 errata-xmlrpc 2018-10-30 09:57:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113
Note You need to log in before you can comment on or make changes to this bug.