Bug 1878450
Summary: | libvirtd memory leak when reconnecting to guests | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | yafu <yafu> | ||||
Component: | libvirt | Assignee: | Ján Tomko <jtomko> | ||||
Status: | CLOSED ERRATA | QA Contact: | yafu <yafu> | ||||
Severity: | low | Docs Contact: | |||||
Priority: | low | ||||||
Version: | 8.3 | CC: | ddepaula, jdenemar, jtomko, virt-maint, yalzhang | ||||
Target Milestone: | rc | Keywords: | Upstream | ||||
Target Release: | 8.3 | ||||||
Hardware: | All | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | libvirt-6.10.0-1.el8 | Doc Type: | If docs needed, set a value | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2021-05-25 06:43:34 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | 6.8.0 | ||||
Embargoed: | |||||||
Attachments: |
|
Description
yafu
2020-09-13 03:01:18 UTC
The interesting backtrace: ==83983== 16 bytes in 1 blocks are definitely lost in loss record 604 of 1,892 ==83983== at 0x4C3321A: calloc (vg_replace_malloc.c:760) ==83983== by 0x8CD11FD: g_malloc0 (gmem.c:129) ==83983== by 0x24C5C6E0: qemuJobAllocPrivate (qemu_domain.c:95) ==83983== by 0x24C7A7A5: qemuDomainObjInitJob (qemu_domainjob.c:170) ==83983== by 0x24C5B337: qemuDomainObjPrivateAlloc (qemu_domain.c:1656) ==83983== by 0x54FA6E8: virDomainObjNew (domain_conf.c:3620) ==83983== by 0x552FFB0: virDomainObjParseXML (domain_conf.c:22337) ==83983== by 0x552FFB0: virDomainObjParseNode (domain_conf.c:22523) ==83983== by 0x55309C1: virDomainObjParseFile (domain_conf.c:22537) ==83983== by 0x55401CC: virDomainObjListLoadStatus (virdomainobjlist.c:549) ==83983== by 0x55401CC: virDomainObjListLoadAllConfigs (virdomainobjlist.c:613) ==83983== by 0x24D6D509: qemuStateInitialize (qemu_driver.c:948) ==83983== by 0x57162CE: virStateInitialize (libvirt.c:654) ==83983== by 0x140AC3: daemonRunStateInit (remote_daemon.c:598) This leak happens regardless of TLS setup and leaks ~16 bytes per running guest at libvirtd startup. Note that you can use --suppressions to ignore the possible "g_type_register_static" leaks we do not care about: https://gitlab.com/libvirt/libvirt/-/blob/master/tests/.valgrind.supp#L151 Patches posted upstream: https://www.redhat.com/archives/libvir-list/2020-September/msg00797.html Pushed upstream as: commit af16e754cd4efc3ca1df2b2e725945be743e87f9 Author: Ján Tomko <jtomko> CommitDate: 2020-09-14 18:10:56 +0200 qemuProcessReconnect: clear 'oldjob' After we started copying the privateData pointer in qemuDomainObjRestoreJob, we should also free them once we're done with them. Register the clear function and use g_auto. Also add a check for job->cb to qemuDomainObjClearJob, to prevent freeing an uninitialized job. https://bugzilla.redhat.com/show_bug.cgi?id=1878450 Signed-off-by: Ján Tomko <jtomko> Fixes: aca37c3fb2e8d733c2788ca4b796c153ea7ce391 git describe: v6.7.0-200-gaf16e754cd Reproduced with libvirt-6.6.0-4.el8.x86_64. Steps: 1.Start guest: #virsh start vm1 2.Start libvirtd with valgrind: # systemctl stop libvirtd # systemctl stop virtlogd # virtlogd -d # valgrind --leak-check=full --suppressions=.valgrind.supp --trace-children=no --child-silent-after-fork=yes libvirtd 3.Enter 'Ctrl+C' to interrupt the valgrind after libvirtd starting: ^C==48839== ==48839== HEAP SUMMARY: ==48839== in use at exit: 1,205,195 bytes in 13,246 blocks ==48839== total heap usage: 284,762 allocs, 271,516 frees, 1,293,122,529 bytes allocated ==48839== ==48839== 16 bytes in 1 blocks are definitely lost in loss record 583 of 1,789 ==48839== at 0x4C3321A: calloc (vg_replace_malloc.c:760) ==48839== by 0x8CD32FD: g_malloc0 (in /usr/lib64/libglib-2.0.so.0.5600.4) ==48839== by 0x277096E0: ??? (in /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so) ==48839== by 0x277277A5: qemuDomainObjInitJob (in /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so) ==48839== by 0x27708337: ??? (in /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so) ==48839== by 0x54FA6E8: virDomainObjNew (in /usr/lib64/libvirt.so.0.6006.0) ==48839== by 0x552FFB0: virDomainObjParseNode (in /usr/lib64/libvirt.so.0.6006.0) ==48839== by 0x55309C1: virDomainObjParseFile (in /usr/lib64/libvirt.so.0.6006.0) ==48839== by 0x55401CC: virDomainObjListLoadAllConfigs (in /usr/lib64/libvirt.so.0.6006.0) ==48839== by 0x2781A509: ??? (in /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so) ==48839== by 0x57162CE: virStateInitialize (in /usr/lib64/libvirt.so.0.6006.0) ==48839== by 0x140AC3: ??? (in /usr/sbin/libvirtd) ==48839== ==48839== 32 bytes in 1 blocks are possibly lost in loss record 795 of 1,789 ==48839== at 0x4C3321A: calloc (vg_replace_malloc.c:760) ==48839== by 0x8CD32FD: g_malloc0 (in /usr/lib64/libglib-2.0.so.0.5600.4) ==48839== by 0x8A5EAFA: g_type_class_ref (in /usr/lib64/libgobject-2.0.so.0.5600.4) ==48839== by 0x8A5EB1F: g_type_class_ref (in /usr/lib64/libgobject-2.0.so.0.5600.4) ==48839== by 0x8A4D357: g_param_spec_enum (in /usr/lib64/libgobject-2.0.so.0.5600.4) ==48839== by 0x8704DF0: ??? (in /usr/lib64/libgio-2.0.so.0.5600.4) ==48839== by 0x8A5E97A: g_type_class_ref (in /usr/lib64/libgobject-2.0.so.0.5600.4) ==48839== by 0x8A45201: g_object_new_valist (in /usr/lib64/libgobject-2.0.so.0.5600.4) ==48839== by 0x86EA6BD: g_initable_new_valist (in /usr/lib64/libgio-2.0.so.0.5600.4) ==48839== by 0x86EA77C: g_initable_new (in /usr/lib64/libgio-2.0.so.0.5600.4) ==48839== by 0x277ABA2A: qemuMonitorOpen (in /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so) ==48839== by 0x27764425: ??? (in /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so) ==48839== ==48839== 80 bytes in 1 blocks are possibly lost in loss record 1,292 of 1,789 ==48839== at 0x4C3321A: calloc (vg_replace_malloc.c:760) ==48839== by 0x8CD32FD: g_malloc0 (in /usr/lib64/libglib-2.0.so.0.5600.4) ==48839== by 0x8A5EAFA: g_type_class_ref (in /usr/lib64/libgobject-2.0.so.0.5600.4) ==48839== by 0x8A5EB1F: g_type_class_ref (in /usr/lib64/libgobject-2.0.so.0.5600.4) ==48839== by 0x8A60707: g_type_create_instance (in /usr/lib64/libgobject-2.0.so.0.5600.4) ==48839== by 0x8A4928A: g_param_spec_internal (in /usr/lib64/libgobject-2.0.so.0.5600.4) ==48839== by 0x8A4D38A: g_param_spec_enum (in /usr/lib64/libgobject-2.0.so.0.5600.4) ==48839== by 0x8704DF0: ??? (in /usr/lib64/libgio-2.0.so.0.5600.4) ==48839== by 0x8A5E97A: g_type_class_ref (in /usr/lib64/libgobject-2.0.so.0.5600.4) ==48839== by 0x8A45201: g_object_new_valist (in /usr/lib64/libgobject-2.0.so.0.5600.4) ==48839== by 0x86EA6BD: g_initable_new_valist (in /usr/lib64/libgio-2.0.so.0.5600.4) ==48839== by 0x86EA77C: g_initable_new (in /usr/lib64/libgio-2.0.so.0.5600.4) ==48839== ==48839== 608 bytes in 1 blocks are possibly lost in loss record 1,736 of 1,789 ==48839== at 0x4C3321A: calloc (vg_replace_malloc.c:760) ==48839== by 0x4012421: allocate_dtv (in /usr/lib64/ld-2.28.so) ==48839== by 0x4012DB1: _dl_allocate_tls (in /usr/lib64/ld-2.28.so) ==48839== by 0x91A5DA2: pthread_create@@GLIBC_2.2.5 (in /usr/lib64/libpthread-2.28.so) ==48839== by 0x8D14723: ??? (in /usr/lib64/libglib-2.0.so.0.5600.4) ==48839== by 0x8CF6156: ??? (in /usr/lib64/libglib-2.0.so.0.5600.4) ==48839== by 0x53E514C: virEventThreadNew (in /usr/lib64/libvirt.so.0.6006.0) ==48839== by 0x2770D609: qemuDomainObjStartWorker (in /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so) ==48839== by 0x2777B2DF: ??? (in /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so) ==48839== by 0x54753D2: ??? (in /usr/lib64/libvirt.so.0.6006.0) ==48839== by 0x91A5149: start_thread (in /usr/lib64/libpthread-2.28.so) ==48839== by 0x98D5762: clone (in /usr/lib64/libc-2.28.so) ==48839== ==48839== LEAK SUMMARY: ==48839== definitely lost: 16 bytes in 1 blocks ==48839== indirectly lost: 0 bytes in 0 blocks ==48839== possibly lost: 720 bytes in 3 blocks ==48839== still reachable: 997,112 bytes in 12,263 blocks ==48839== of which reachable via heuristic: ==48839== length64 : 144 bytes in 3 blocks ==48839== newarray : 1,584 bytes in 19 blocks ==48839== suppressed: 196,147 bytes in 915 blocks ==48839== Reachable blocks (those to which a pointer was found) are not shown. ==48839== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==48839== ==48839== For lists of detected and suppressed errors, rerun with: -s ==48839== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 20 from 20) Verified with libvirt-daemon-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64. Test steps: 1.Start guest: #virsh start vm1 2.Start libvirtd with valgrind: # systemctl stop libvirtd # systemctl stop virtlogd # virtlogd -d # valgrind --leak-check=full --suppressions=.valgrind.supp --trace-children=no --child-silent-after-fork=yes libvirtd 3.Enter 'Ctrl+C' to interrupt the valgrind after libvirtd starting: ... ==51860== LEAK SUMMARY: ==51860== definitely lost: 0 bytes in 0 blocks ==51860== indirectly lost: 0 bytes in 0 blocks ==51860== possibly lost: 1,288 bytes in 5 blocks ==51860== still reachable: 1,050,704 bytes in 13,033 blocks ==51860== of which reachable via heuristic: ==51860== length64 : 728 bytes in 14 blocks ==51860== newarray : 1,728 bytes in 28 blocks ==51860== suppressed: 196,219 bytes in 912 blocks ==51860== Reachable blocks (those to which a pointer was found) are not shown. ==51860== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==51860== ==51860== For lists of detected and suppressed errors, rerun with: -s ==51860== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 21 from 21) Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098 |