Bug 1598015
| Summary: | libvirtd crashed on target host when do migration with '--tls' | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | yafu <yafu> | ||||
| Component: | libvirt | Assignee: | Peter Krempa <pkrempa> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Fangge Jin <fjin> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.6 | CC: | dyuan, dzheng, fjin, lmen, pkrempa, xuzhang, yanqzhan | ||||
| Target Milestone: | rc | Keywords: | Regression | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | libvirt-4.5.0-3.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1600342 (view as bug list) | Environment: | |||||
| Last Closed: | 2018-10-30 09:57:31 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1600342 | ||||||
| Attachments: |
|
||||||
This was most probably caused by:
commit c2f71bb295162a81c9085a184972cb93e5964451
Author: Peter Krempa <pkrempa>
Date: Tue May 22 07:38:22 2018 +0200
qemu: hotplug: Refactor 'secret' props formatting to qemuMonitorCreateObjectProps
From the automated job log, I can see libvirt-4.4.0-2.el7.ppc64le has not this issue. *** Bug 1598412 has been marked as a duplicate of this bug. *** Fixed upstream:
commit fac0dacd54c02b842c995d0999d9450d09d1e7cd
Author: Peter Krempa <pkrempa>
Date: Wed Jul 4 16:36:37 2018 +0200
qemu: monitor: Make qemuMonitorAddObject more robust against programming errors
Document and check that @props contains a pointer to a json object and
check that both necessary fields are present. Also mark @props as
NONNULL.
Signed-off-by: Peter Krempa <pkrempa>
Reviewed-by: Ján Tomko <jtomko>
commit 62ef8227e2717618c96fa17f2d4f5b7570bbe980
Author: Peter Krempa <pkrempa>
Date: Wed Jul 4 16:28:58 2018 +0200
qemu: hotplug: Do not try to add secret object for TLS if it does not exist
The check whether the object holding secret for decryption of the TLS
environment was wrong and would always attempt to add the object. This
lead to a crash due to recent refactors.
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1598015
Signed-off-by: Peter Krempa <pkrempa>
Reviewed-by: Ján Tomko <jtomko>
Reproduced with libvirt-4.5.0-1.el7.x86_64
Verified with libvirt-4.5.0-4.el7.x86_64, migration succeeds, but there is error in libvirtd.log:
Source:
2018-07-25 11:12:25.884+0000: 8375: debug : qemuMonitorJSONCheckError:385 : unable to execute QEMU command {"execute":"object-del","arguments":{"id":"objlibvirt_migrate_tls0"},"id":"libvirt-34"}: {"id":"libvirt-34","error":{"class":"GenericError","desc":"object 'objlibvirt_migrate_tls0' not found"}}
Target:
2018-07-25 11:12:25.618+0000: 10709: debug : qemuMonitorJSONCheckError:385 : unable to execute QEMU command {"execute":"object-del","arguments":{"id":"objlibvirt_migrate_tls0"},"id":"libvirt-13"}: {"id":"libvirt-13","error":{"class":"GenericError","desc":"object 'objlibvirt_migrate_tls0' not found"}}
2018-07-25 11:12:52.821+0000: 10710: debug : qemuMonitorJSONCheckError:385 : unable to execute QEMU command {"execute":"object-del","arguments":{"id":"libvirt_migrate-secret0"},"id":"libvirt-30"}: {"id":"libvirt-30","error":{"class":"GenericError","desc":"object 'libvirt_migrate-secret0' not found"}}
Created attachment 1470503 [details]
libvirtd log - error message
That is expected. We always try to delete the object even when it is not present in qemu. Note that this was happening already for some time. (In reply to Peter Krempa from comment #11) > That is expected. We always try to delete the object even when it is not > present in qemu. Note that this was happening already for some time. Thanks. I didn't check the libvirtd.log before during testing. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:3113 |
Description of problem: libvirtd crashed on target host when do migration with '--tls' Version-Release number of selected component (if applicable): libvirt-4.5.0-1.el7.x86_64 qemu-kvm-rhev-2.12.0-6.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.Do migration with '--tls': # virsh migrate iommu1 qemu+ssh://10.66.69.35/system --live --verbose --tls --unsafe error: End of file while reading data: Ncat: Broken pipe.: Input/output error Actual results: libvirtd crashed on target host when do migration with '--tls' Expected results: Migration should be completed successfully. Additional info: 1.backtrace of libvirtd process: Core was generated by `/usr/sbin/libvirtd'. Program terminated with signal 11, Segmentation fault. #0 virJSONValueObjectGet (object=0x0, key=key@entry=0x7f392ec0ffe3 "qom-type") at util/virjson.c:818 818 if (object->type != VIR_JSON_TYPE_OBJECT) (gdb) bt #0 virJSONValueObjectGet (object=0x0, key=key@entry=0x7f392ec0ffe3 "qom-type") at util/virjson.c:818 #1 0x00007f3979bab251 in virJSONValueObjectGetString (object=<optimized out>, key=key@entry=0x7f392ec0ffe3 "qom-type") at util/virjson.c:1258 #2 0x00007f392eb791bd in qemuMonitorAddObject (mon=0x7f394c015f50, props=props@entry=0x7f3967ecd4c0, alias=alias@entry=0x7f3967ecd430) at qemu/qemu_monitor.c:3090 #3 0x00007f392eb2bb23 in qemuDomainAddTLSObjects (driver=driver@entry=0x7f39081151a0, vm=vm@entry=0x7f3908292b30, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_MIGRATION_IN, secProps=secProps@entry=0x7f3967ecd4c0, tlsProps=tlsProps@entry=0x7f3967ecd4b8) at qemu/qemu_hotplug.c:1342 #4 0x00007f392eb6cad2 in qemuMigrationParamsEnableTLS (driver=driver@entry=0x7f39081151a0, vm=0x7f3908292b30, tlsListen=tlsListen@entry=true, asyncJob=asyncJob@entry=2, tlsAlias=tlsAlias@entry=0x7f3967ecd608, hostname=hostname@entry=0x0, migParams=migParams@entry=0x7f394c0008c0) at qemu/qemu_migration_params.c:878 #5 0x00007f392eb5aaa2 in qemuMigrationDstPrepareAny (driver=driver@entry=0x7f39081151a0, dconn=dconn@entry=0x7f395801f3a0, cookiein=cookiein@entry=0x7f394c0032a0 "<qemu-migration>\n <name>iommu1</name>\n <uuid>1b3268d6-b59c-406b-a14c-33b000b15b6c</uuid>\n <hostname>wlan-69-35.nay.redhat.com</hostname>\n <hostuuid>5cd9f881-5529-11cb-b989-b4c8b0f5dd17</hostuuid>\n"..., cookieinlen=cookieinlen@entry=597, cookieout=cookieout@entry=0x7f3967ecdaa8, cookieoutlen=cookieoutlen@entry=0x7f3967ecda9c, def=def@entry=0x7f3967ecd900, origname=origname@entry=0x0, st=st@entry=0x0, protocol=<optimized out>, port=49153, autoPort=true, listenAddress=listenAddress@entry=0x0, nmigrate_disks=nmigrate_disks@entry=0, migrate_disks=migrate_disks@entry=0x0, nbdPort=nbdPort@entry=0, migParams=migParams@entry=0x7f394c0008c0, flags=flags@entry=65537) at qemu/qemu_migration.c:2505 #6 0x00007f392eb5d31b in qemuMigrationDstPrepareDirect (driver=driver@entry=0x7f39081151a0, dconn=dconn@entry=0x7f395801f3a0, cookiein=cookiein@entry=0x7f394c0032a0 "<qemu-migration>\n <name>iommu1</name>\n <uuid>1b3268d6-b59c-406b-a14c-33b000b15b6c</uuid>\n <hostname>wlan-69-35.nay.redhat.com</hostname>\n <hostuuid>5cd9f881-5529-11cb-b989-b4c8b0f5dd17</hostuuid>\n"..., cookieinlen=cookieinlen@entry=597, cookieout=cookieout@entry=0x7f3967ecdaa8, cookieoutlen=cookieoutlen@entry=0x7f3967ecda9c, uri_in=0x0, uri_out=uri_out@entry=0x7f394c000ad0, def=def@entry=0x7f3967ecd900, origname=0x0, listenAddress=0x0, nmigrate_disks=nmigrate_disks@entry=0, migrate_disks=0x0, nbdPort=0, migParams=migParams@entry=0x7f394c0008c0, flags=flags@entry=65537) at qemu/qemu_migration.c:2833 #7 0x00007f392eba73b7 in qemuDomainMigratePrepare3Params (dconn=0x7f395801f3a0, params=<optimized out>, nparams=<optimized out>, ---Type <return> to continue, or q <return> to quit--- cookiein=0x7f394c0032a0 "<qemu-migration>\n <name>iommu1</name>\n <uuid>1b3268d6-b59c-406b-a14c-33b000b15b6c</uuid>\n <hostname>wlan-69-35.nay.redhat.com</hostname>\n <hostuuid>5cd9f881-5529-11cb-b989-b4c8b0f5dd17</hostuuid>\n"..., cookieinlen=597, cookieout=0x7f3967ecdaa8, cookieoutlen=0x7f3967ecda9c, uri_out=0x7f394c000ad0, flags=65537) at qemu/qemu_driver.c:12668 #8 0x00007f3979e38d6d in virDomainMigratePrepare3Params (dconn=0x7f395801f3a0, params=0x7f394c000990, nparams=1, cookiein=0x7f394c0032a0 "<qemu-migration>\n <name>iommu1</name>\n <uuid>1b3268d6-b59c-406b-a14c-33b000b15b6c</uuid>\n <hostname>wlan-69-35.nay.redhat.com</hostname>\n <hostuuid>5cd9f881-5529-11cb-b989-b4c8b0f5dd17</hostuuid>\n"..., cookieinlen=597, cookieout=cookieout@entry=0x7f3967ecdaa8, cookieoutlen=cookieoutlen@entry=0x7f3967ecda9c, uri_out=0x7f394c000ad0, flags=65537) at libvirt-domain.c:4877 #9 0x0000562aecd80349 in remoteDispatchDomainMigratePrepare3Params (server=<optimized out>, msg=0x562aed10b300, ret=0x7f394c000b40, args=0x7f394c000b10, rerr=0x7f3967ecdbc0, client=<optimized out>) at remote/remote_daemon_dispatch.c:5395 #10 remoteDispatchDomainMigratePrepare3ParamsHelper (server=<optimized out>, client=<optimized out>, msg=0x562aed10b300, rerr=0x7f3967ecdbc0, args=0x7f394c000b10, ret=0x7f394c000b40) at remote/remote_daemon_dispatch_stubs.h:8347 #11 0x00007f3979d235e5 in virNetServerProgramDispatchCall (msg=0x562aed10b300, client=0x562aed10a480, server=0x562aed0b8fa0, prog=0x562aed107650) at rpc/virnetserverprogram.c:437 #12 virNetServerProgramDispatch (prog=0x562aed107650, server=server@entry=0x562aed0b8fa0, client=client@entry=0x562aed10a480, msg=msg@entry=0x562aed10b300) at rpc/virnetserverprogram.c:304 #13 0x00007f3979d2c34a in virNetServerProcessMsg (srv=srv@entry=0x562aed0b8fa0, client=0x562aed10a480, prog=<optimized out>, msg=0x562aed10b300) at rpc/virnetserver.c:143 #14 0x00007f3979d2c798 in virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x562aed0b8fa0) at rpc/virnetserver.c:164 #15 0x00007f3979c0aa81 in virThreadPoolWorker (opaque=opaque@entry=0x562aed0acb00) at util/virthreadpool.c:167 #16 0x00007f3979c09850 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #17 0x00007f3976f21dd5 in start_thread () from /lib64/libpthread.so.0 #18 0x00007f3976c4baed in clone () from /lib64/libc.so.6 Additional info: