Red Hat Bugzilla – Bug 1158715
A memory error report when use domstats
Last modified: 2015-03-05 02:46:39 EST
description of problem: A memory error report when use domstats Version-Release number of selected component (if applicable): libvirt-1.2.8-5.el7.x86_64 qemu-kvm-rhev-2.1.2-4.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.# valgrind -v --leak-check=full virsh domstats Domain: 'test3' state.state=1 state.reason=1 cpu.time=1096447270349 cpu.user=328510000000 cpu.system=695060000000 balloon.current=1048576 balloon.maximum=1048576 vcpu.current=1 vcpu.maximum=2 vcpu.0.state=1 vcpu.0.time=19110000000 net.count=1 net.0.name=vnet0 net.0.rx.bytes=196827 net.0.rx.pkts=3746 net.0.rx.errs=0 net.0.rx.drop=0 net.0.tx.bytes=3795 net.0.tx.pkts=39 net.0.tx.errs=0 net.0.tx.drop=0 block.count=1 block.0.name=hda block.0.rd.reqs=45898 block.0.rd.bytes=88999424 block.0.rd.times=8177028688 block.0.wr.reqs=779 block.0.wr.bytes=4816896 block.0.wr.times=1541997742 block.0.fl.reqs=272 block.0.fl.times=3521448382 block.0.allocation=3221089792 block.0.capacity=4294967296 block.0.physical=4294967296 Domain: 'r6' state.state=5 state.reason=2 balloon.maximum=1048576 vcpu.current=2 vcpu.maximum=2 vcpu.0.state=0 vcpu.1.state=0 ==30515== ==30515== HEAP SUMMARY: ==30515== in use at exit: 109,021 bytes in 1,093 blocks ==30515== total heap usage: 5,680 allocs, 4,587 frees, 1,484,320 bytes allocated ==30515== ==30515== Searching for pointers to 1,093 not-freed blocks ==30515== Checked 1,487,152 bytes ==30515== ==30515== 40 bytes in 1 blocks are possibly lost in loss record 118 of 189 ==30515== at 0x4C29BBD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==30515== by 0xBA21FDB: _PR_Getfd (prfdcach.c:112) ==30515== by 0xBA3BCB6: pt_SetMethods.isra.11 (ptio.c:3303) ==30515== by 0xBA3C2A4: PR_OpenFile (ptio.c:3581) ==30515== by 0xD951989: blapi_SHVerifyFile (shvfy.c:355) ==30515== by 0xD951CB0: blapi_SHVerify (shvfy.c:289) ==30515== by 0xD927AE9: freebl_fipsSoftwareIntegrityTest (fipsfreebl.c:1541) ==30515== by 0xD927AE9: bl_startup_tests (fipsfreebl.c:1732) ==30515== by 0x400F502: call_init (dl-init.c:82) ==30515== by 0x400F502: _dl_init (dl-init.c:131) ==30515== by 0x4001459: ??? (in /usr/lib64/ld-2.17.so) ==30515== by 0x1: ??? ==30515== by 0xFFF0002EE: ??? ==30515== by 0xFFF0002F4: ??? ==30515== ==30515== 48 bytes in 1 blocks are possibly lost in loss record 123 of 189 ==30515== at 0x4C29BBD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==30515== by 0xBA21FC9: _PR_Getfd (prfdcach.c:109) ==30515== by 0xBA3BCB6: pt_SetMethods.isra.11 (ptio.c:3303) ==30515== by 0xBA3C2A4: PR_OpenFile (ptio.c:3581) ==30515== by 0xD951989: blapi_SHVerifyFile (shvfy.c:355) ==30515== by 0xD951CB0: blapi_SHVerify (shvfy.c:289) ==30515== by 0xD927AE9: freebl_fipsSoftwareIntegrityTest (fipsfreebl.c:1541) ==30515== by 0xD927AE9: bl_startup_tests (fipsfreebl.c:1732) ==30515== by 0x400F502: call_init (dl-init.c:82) ==30515== by 0x400F502: _dl_init (dl-init.c:131) ==30515== by 0x4001459: ??? (in /usr/lib64/ld-2.17.so) ==30515== by 0x1: ??? ==30515== by 0xFFF0002EE: ??? ==30515== by 0xFFF0002F4: ??? ==30515== ==30515== LEAK SUMMARY: ==30515== definitely lost: 0 bytes in 0 blocks ==30515== indirectly lost: 0 bytes in 0 blocks ==30515== possibly lost: 88 bytes in 2 blocks ==30515== still reachable: 108,933 bytes in 1,091 blocks ==30515== suppressed: 0 bytes in 0 blocks ==30515== Reachable blocks (those to which a pointer was found) are not shown. ==30515== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==30515== ==30515== Use --track-origins=yes to see where uninitialised values come from ==30515== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 2 from 2) ==30515== ==30515== 1 errors in context 1 of 3: ==30515== Conditional jump or move depends on uninitialised value(s) ==30515== at 0x85E9402: xdr_array (xdr_array.c:88) ==30515== by 0x4FD8FC9: xdr_remote_connect_get_all_domain_stats_args (remote_protocol.c:6473) ==30515== by 0x4FE72F2: virNetMessageEncodePayload (virnetmessage.c:350) ==30515== by 0x4FDD21C: virNetClientProgramCall (virnetclientprogram.c:326) ==30515== by 0x4FB4D01: callFull.isra.2 (remote_driver.c:6667) ==30515== by 0x4FCBD45: call (remote_driver.c:6689) ==30515== by 0x4FCBD45: remoteConnectGetAllDomainStats (remote_driver.c:7793) ==30515== by 0x4FA0E75: virConnectGetAllDomainStats (libvirt.c:21678) ==30515== by 0x147FD1: cmdDomstats (virsh-domain-monitor.c:2148) ==30515== by 0x13006B: vshCommandRun (virsh.c:1915) ==30515== by 0x12A9E1: main (virsh.c:3699) ==30515== --30515-- --30515-- used_suppression: 2 glibc-2.5.x-on-SUSE-10.2-(PPC)-2a /usr/lib64/valgrind/default.supp:1296 ==30515== ==30515== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 2 from 2) Actual results: there is a memory error when use domstats,seems there is some problem Expected results: no memory error Additional info: There is a upstream patch write by Jincheng Miao: commit 28b7601dc7acf99d06277267afb63fff4167b755 Author: Jincheng Miao <jmiao@redhat.com> Date: Fri Oct 24 12:03:13 2014 +0800 remote: fix jump depends on uninitialised value Currently remote driver only initializes partial fields of remote_connect_get_all_domain_stats_args. But xdr_array() will check the uninitialised field 'doms_val'. For safty reason, memset all fields of args is better. Fix the following error from valgrind, like: ==30515== 1 errors in context 1 of 3: ==30515== Conditional jump or move depends on uninitialised value(s) ==30515== at 0x85E9402: xdr_array (xdr_array.c:88) ==30515== by 0x4FD8FC9: xdr_remote_connect_get_all_domain_stats_args (remote_protocol.c:6473) ==30515== by 0x4FE72F2: virNetMessageEncodePayload (virnetmessage.c:350) ==30515== by 0x4FDD21C: virNetClientProgramCall (virnetclientprogram.c:326) ==30515== by 0x4FB4D01: callFull.isra.2 (remote_driver.c:6667) ==30515== by 0x4FCBD45: call (remote_driver.c:6689) ==30515== by 0x4FCBD45: remoteConnectGetAllDomainStats (remote_driver.c:7793) ==30515== by 0x4FA0E75: virConnectGetAllDomainStats (libvirt.c:21678) ==30515== by 0x147FD1: cmdDomstats (virsh-domain-monitor.c:2148) ==30515== by 0x13006B: vshCommandRun (virsh.c:1915) ==30515== by 0x12A9E1: main (virsh.c:3699) Signed-off-by: Jincheng Miao <jmiao@redhat.com>
A memory leak happen when with --domain and maybe forget free args: # valgrind -v --leak-check=full virsh domstats test3 test4 ==8372== ==8372== HEAP SUMMARY: ==8372== in use at exit: 109,145 bytes in 1,098 blocks ==8372== total heap usage: 5,100 allocs, 4,002 frees, 1,575,606 bytes allocated ==8372== ==8372== Searching for pointers to 1,098 not-freed blocks ==8372== Checked 1,487,240 bytes ==8372== ==8372== 40 bytes in 1 blocks are possibly lost in loss record 119 of 192 ==8372== at 0x4C29BBD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==8372== by 0xBA23FDB: _PR_Getfd (prfdcach.c:112) ==8372== by 0xBA3DCB6: pt_SetMethods.isra.11 (ptio.c:3303) ==8372== by 0xBA3E2A4: PR_OpenFile (ptio.c:3581) ==8372== by 0xD953989: blapi_SHVerifyFile (shvfy.c:355) ==8372== by 0xD953CB0: blapi_SHVerify (shvfy.c:289) ==8372== by 0xD929AE9: freebl_fipsSoftwareIntegrityTest (fipsfreebl.c:1541) ==8372== by 0xD929AE9: bl_startup_tests (fipsfreebl.c:1732) ==8372== by 0x400F502: call_init (dl-init.c:82) ==8372== by 0x400F502: _dl_init (dl-init.c:131) ==8372== by 0x4001459: ??? (in /usr/lib64/ld-2.17.so) ==8372== by 0x3: ??? ==8372== by 0xFFF0002DE: ??? ==8372== by 0xFFF0002E4: ??? ==8372== ==8372== 48 bytes in 1 blocks are possibly lost in loss record 125 of 192 ==8372== at 0x4C29BBD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==8372== by 0xBA23FC9: _PR_Getfd (prfdcach.c:109) ==8372== by 0xBA3DCB6: pt_SetMethods.isra.11 (ptio.c:3303) ==8372== by 0xBA3E2A4: PR_OpenFile (ptio.c:3581) ==8372== by 0xD953989: blapi_SHVerifyFile (shvfy.c:355) ==8372== by 0xD953CB0: blapi_SHVerify (shvfy.c:289) ==8372== by 0xD929AE9: freebl_fipsSoftwareIntegrityTest (fipsfreebl.c:1541) ==8372== by 0xD929AE9: bl_startup_tests (fipsfreebl.c:1732) ==8372== by 0x400F502: call_init (dl-init.c:82) ==8372== by 0x400F502: _dl_init (dl-init.c:131) ==8372== by 0x4001459: ??? (in /usr/lib64/ld-2.17.so) ==8372== by 0x3: ??? ==8372== by 0xFFF0002DE: ??? ==8372== by 0xFFF0002E4: ??? ==8372== ==8372== 64 bytes in 1 blocks are definitely lost in loss record 134 of 192 ==8372== at 0x4C2B934: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so) ==8372== by 0x4E922EC: virAllocN (viralloc.c:191) ==8372== by 0x4FCD2BB: remoteConnectGetAllDomainStats (remote_driver.c:7781) ==8372== by 0x4FA26B5: virDomainListGetStats (libvirt.c:21776) ==8372== by 0x147ED7: cmdDomstats (virsh-domain-monitor.c:2142) ==8372== by 0x13006B: vshCommandRun (virsh.c:1915) ==8372== by 0x12A9E1: main (virsh.c:3699) ==8372== ==8372== LEAK SUMMARY: ==8372== definitely lost: 64 bytes in 1 blocks ==8372== indirectly lost: 0 bytes in 0 blocks ==8372== possibly lost: 88 bytes in 2 blocks ==8372== still reachable: 108,993 bytes in 1,095 blocks ==8372== suppressed: 0 bytes in 0 blocks ==8372== Reachable blocks (those to which a pointer was found) are not shown. ==8372== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==8372== ==8372== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 2 from 2) --8372-- --8372-- used_suppression: 2 glibc-2.5.x-on-SUSE-10.2-(PPC)-2a /usr/lib64/valgrind/default.supp:1296 ==8372== ==8372== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 2 from 2)
The issue is now fixed upstream: commit bf1f8e280c330f51833189d40a99dc894205a129 Author: Peter Krempa <pkrempa@redhat.com> Date: Wed Nov 5 12:34:51 2014 +0100 remote: Fix memory leak in remoteConnectGetAllDomainStats The remote call actually doesn't free the arguments array so we leak memory in case a domain list is specified. As the remote domain list array consists only of stolen pointers from the actual domain objects it's sufficient just to free the array. Valgrind message: ==1081452== 64 bytes in 1 blocks are definitely lost in loss record 632 of 726 ==1081452== at 0x4C296D0: calloc (vg_replace_malloc.c:618) ==1081452== by 0x4EA5CB4: virAllocN (viralloc.c:191) ==1081452== by 0x505D21E: remoteConnectGetAllDomainStats (remote_driver.c:7785) ==1081452== by 0x50081AA: virDomainListGetStats (libvirt-domain.c:11080) ==1081452== by 0x155249: cmdDomstats (virsh-domain-monitor.c:2147) ==1081452== by 0x12FB73: vshCommandRun (virsh.c:1935) ==1081452== by 0x133FEB: main (virsh.c:3719)
Verify this bug with libvirt-1.2.8-7.el7: # valgrind -v --leak-check=full virsh domstats test3 test4 ==3839== ==3839== LEAK SUMMARY: ==3839== definitely lost: 0 bytes in 0 blocks ==3839== indirectly lost: 0 bytes in 0 blocks ==3839== possibly lost: 88 bytes in 2 blocks ==3839== still reachable: 108,993 bytes in 1,095 blocks ==3839== suppressed: 0 bytes in 0 blocks ==3839== Reachable blocks (those to which a pointer was found) are not shown. ==3839== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==3839== # valgrind -v --leak-check=full virsh domstats --list-active --block ==4124== ==4124== LEAK SUMMARY: ==4124== definitely lost: 0 bytes in 0 blocks ==4124== indirectly lost: 0 bytes in 0 blocks ==4124== possibly lost: 88 bytes in 2 blocks ==4124== still reachable: 108,981 bytes in 1,093 blocks ==4124== suppressed: 0 bytes in 0 blocks ==4124== Reachable blocks (those to which a pointer was found) are not shown. ==4124== To see them, rerun with: --leak-check=full --show-leak-kinds=all And do some extend test, no memory leak found when use domstats
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0323.html