Bug 1316433

Summary: cmd domstats cause libvirtd memleak
Product: Red Hat Enterprise Linux 7 Reporter: Luyao Huang <lhuang>
Component: libvirtAssignee: Peter Krempa <pkrempa>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.3CC: jishao, pkrempa, rbalakri
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-1.3.3-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-03 18:39:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Luyao Huang 2016-03-10 08:47:14 UTC
Description of problem:
cmd domstats/schedinfo cause libvirtd memleak

Version-Release number of selected component (if applicable):
libvirt-1.3.2-1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.
prepare a guest
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 3     r7                             running

2. start valgrind on another terminal:
# service libvirtd stop
# valgrind --leak-check=full libvirtd

3.
# virsh domstats r7
Domain: 'r7'
  state.state=1
  state.reason=1
  cpu.time=94396896375
  cpu.user=21880000000
  cpu.system=48060000000
  balloon.current=1048576
  balloon.maximum=1048576
  vcpu.current=1
  vcpu.maximum=4
  vcpu.0.state=1
  vcpu.0.time=19060000000
  vcpu.0.wait=0
  net.count=1
  net.0.name=vnet0
  net.0.rx.bytes=23454
  net.0.rx.pkts=442
  net.0.rx.errs=0
  net.0.rx.drop=0
  net.0.tx.bytes=0
  net.0.tx.pkts=0
  net.0.tx.errs=0
  net.0.tx.drop=0
  block.count=1
  block.0.name=vda
  block.0.path=/var/lib/libvirt/images/r7_ext4.raw
  block.0.rd.reqs=8450
  block.0.rd.bytes=142745600
  block.0.rd.times=48333131974
  block.0.wr.reqs=360
  block.0.wr.bytes=3290112
  block.0.wr.times=22059101371
  block.0.fl.reqs=89
  block.0.fl.times=3088680104
  block.0.allocation=9116335616
  block.0.capacity=10737418240
  block.0.physical=10737426432

3. on the terminal which run valgrind

^C

==3408== 2 bytes in 2 blocks are definitely lost in loss record 3 of 866
==3408==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==3408==    by 0x82F4FE9: strdup (in /usr/lib64/libc-2.17.so)
==3408==    by 0x5509544: virStrdup (virstring.c:712)
==3408==    by 0x5509993: virStringSplitCount (virstring.c:107)
==3408==    by 0x1E89F5D4: qemuGetSchedInfo (qemu_driver.c:1402)
==3408==    by 0x1E89F5D4: qemuDomainHelperGetVcpus (qemu_driver.c:1564)
==3408==    by 0x1E89FA3D: qemuDomainGetStatsVcpu (qemu_driver.c:18974)
==3408==    by 0x1E884CA0: qemuDomainGetStats (qemu_driver.c:19409)
==3408==    by 0x1E884CA0: qemuConnectGetAllDomainStats (qemu_driver.c:19499)
==3408==    by 0x55BFF55: virDomainListGetStats (libvirt-domain.c:11371)
==3408==    by 0x121B44: remoteDispatchConnectGetAllDomainStats (remote.c:6190)
==3408==    by 0x121B44: remoteDispatchConnectGetAllDomainStatsHelper (remote_dispatch.h:615)
==3408==    by 0x561B1F1: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==3408==    by 0x561B1F1: virNetServerProgramDispatch (virnetserverprogram.c:307)
==3408==    by 0x561641C: virNetServerProcessMsg (virnetserver.c:135)
==3408==    by 0x561641C: virNetServerHandleJob (virnetserver.c:156)
==3408==    by 0x550EDF4: virThreadPoolWorker (virthreadpool.c:145)

...

==3408== 1,868 bytes in 32 blocks are definitely lost in loss record 840 of 866
==3408==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==3408==    by 0x82F5039: strndup (in /usr/lib64/libc-2.17.so)
==3408==    by 0x55096D6: virStrndup (virstring.c:758)
==3408==    by 0x550981E: virStringSplitCount (virstring.c:96)
==3408==    by 0x1E89F5D4: qemuGetSchedInfo (qemu_driver.c:1402)
==3408==    by 0x1E89F5D4: qemuDomainHelperGetVcpus (qemu_driver.c:1564)
==3408==    by 0x1E89FA3D: qemuDomainGetStatsVcpu (qemu_driver.c:18974)
==3408==    by 0x1E884CA0: qemuDomainGetStats (qemu_driver.c:19409)
==3408==    by 0x1E884CA0: qemuConnectGetAllDomainStats (qemu_driver.c:19499)
==3408==    by 0x55BFF55: virDomainListGetStats (libvirt-domain.c:11371)
==3408==    by 0x121B44: remoteDispatchConnectGetAllDomainStats (remote.c:6190)
==3408==    by 0x121B44: remoteDispatchConnectGetAllDomainStatsHelper (remote_dispatch.h:615)
==3408==    by 0x561B1F1: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==3408==    by 0x561B1F1: virNetServerProgramDispatch (virnetserverprogram.c:307)
==3408==    by 0x561641C: virNetServerProcessMsg (virnetserver.c:135)
==3408==    by 0x561641C: virNetServerHandleJob (virnetserver.c:156)
==3408==    by 0x550EDF4: virThreadPoolWorker (virthreadpool.c:145)

4. also i can easy see libvirtd memleak via ps

# ps aux|grep libvirtd
root      3544  0.7  0.3 899324 23164 ?        Ssl  16:39   0:00 /usr/sbin/libvirtd
root      4010  0.0  0.0 112644   952 pts/1    S+   16:40   0:00 grep 
--color=auto libvirtd

# virsh schedinfo r7
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : -1
emulator_period: 100000
emulator_quota : -1

# ps aux|grep libvirtd
root      3544  0.7  0.3 964860 23244 ?        Ssl  16:39   0:00 /usr/sbin/libvirtd
root      4014  0.0  0.0 112644   956 pts/1    S+   16:40   0:00 grep --color=auto libvirtd


Actual results:
libvirtd memleak when use command getstats/schedinfo

Expected results:
no memleak

Additional info:

i afraid readonly command cause libvirtd memleak will be a security problem (DDos), so i set it private first

Comment 1 Peter Krempa 2016-03-10 13:02:17 UTC
Fixed upstream:

commit a0143d5242e7425883a670ad0ff01b6630e83fae
Author: Peter Krempa <pkrempa>
Date:   Thu Mar 10 09:57:13 2016 +0100

    qemu: Fix memory leak in qemuGetSchedInfo
    
    Memory returned from virStringSplit shall be freed with
    virStringFreeList rather than VIR_FREE. Introduced in commit 511e7c5b.

Comment 2 Peter Krempa 2016-03-10 13:02:33 UTC
v1.3.2-102-ga0143d5

Comment 3 Mike McCune 2016-03-28 23:19:36 UTC
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 5 Jingjing Shao 2016-08-09 08:41:15 UTC
Hi Peter,

I try to verify this bugļ¼Œbut I get the result as below. I think the issue still exists

# rpm -q libvirt
libvirt-2.0.0-3.el7.x86_64

# virsh list 
 Id    Name                           State
----------------------------------------------------
 7     r7.2                           running

start valgrind on another terminal:
# service libvirtd stop
# valgrind --leak-check=full libvirtd

# virsh domstats r7.2  
 Domain: 'r7.2'
  state.state=1
  state.reason=1
  cpu.time=34367393971
  cpu.user=2120000000
  cpu.system=8550000000
  balloon.current=615424
  balloon.maximum=615424
  vcpu.current=3
  vcpu.maximum=4
  vcpu.0.state=1
  vcpu.0.time=13950000000
  vcpu.0.wait=0
  vcpu.1.state=1
  vcpu.1.time=6930000000
  vcpu.1.wait=0
  vcpu.2.state=1
  vcpu.2.time=7810000000
  vcpu.2.wait=0
  net.count=2
  net.0.name=vnet0
  net.0.rx.bytes=166688
  net.0.rx.pkts=3197
  net.0.rx.errs=0
  net.0.rx.drop=0
  net.0.tx.bytes=732
  net.0.tx.pkts=10
  net.0.tx.errs=0
  net.0.tx.drop=0
  net.1.name=vnet1
  net.1.rx.bytes=166688
  net.1.rx.pkts=3197
  net.1.rx.errs=0
  net.1.rx.drop=0
  net.1.tx.bytes=732
  net.1.tx.pkts=10
  net.1.tx.errs=0
  net.1.tx.drop=0
  block.count=2
  block.0.name=vda
  block.0.path=/var/lib/libvirt/images/r7.2.img
  block.0.rd.reqs=6268
  block.0.rd.bytes=147808256
  block.0.rd.times=2594141293
  block.0.wr.reqs=976
  block.0.wr.bytes=10103296
  block.0.wr.times=1503970051
  block.0.fl.reqs=223
  block.0.fl.times=2063729523
  block.0.allocation=1648623616
  block.0.capacity=107374182400
  block.0.physical=1648627712
  block.1.name=hda
  block.1.rd.reqs=4
  block.1.rd.bytes=152
  block.1.rd.times=113308
  block.1.wr.reqs=0
  block.1.wr.bytes=0
  block.1.wr.times=0
  block.1.fl.reqs=0
  block.1.fl.times=0
  block.1.allocation=0


on the terminal which run valgrind
....
==395== 8 bytes in 1 blocks are definitely lost in loss record 222 of 2,191
==395==    at 0x4C29BFD: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==395==    by 0x4C2BACB: realloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==395==    by 0x54B294F: virReallocN (in /usr/lib64/libvirt.so.0.2000.0)
==395==    by 0x54B2A19: virExpandN (in /usr/lib64/libvirt.so.0.2000.0)
==395==    by 0x54B2C6C: virInsertElementsN (in /usr/lib64/libvirt.so.0.2000.0)
==395==    by 0x556F29D: virDomainObjListConvert (in /usr/lib64/libvirt.so.0.2000.0)
==395==    by 0x220035EC: ??? (in /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so)
==395==    by 0x55D1AD5: virDomainListGetStats (in /usr/lib64/libvirt.so.0.2000.0)
==395==    by 0x128F94: ??? (in /usr/sbin/libvirtd)
==395==    by 0x562F3D1: virNetServerProgramDispatch (in /usr/lib64/libvirt.so.0.2000.0)
==395==    by 0x15BBCC: ??? (in /usr/sbin/libvirtd)
==395==    by 0x551D610: ??? (in /usr/lib64/libvirt.so.0.2000.0)
.....

==395== 80 bytes in 1 blocks are definitely lost in loss record 1,287 of 2,191
==395==    at 0x4C2B974: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==395==    by 0x54B2853: virAlloc (in /usr/lib64/libvirt.so.0.2000.0)
==395==    by 0x54CE1D3: ??? (in /usr/lib64/libvirt.so.0.2000.0)
==395==    by 0x54CFDB8: virResetLastError (in /usr/lib64/libvirt.so.0.2000.0)
==395==    by 0x55D19AD: virDomainListGetStats (in /usr/lib64/libvirt.so.0.2000.0)
==395==    by 0x128F94: ??? (in /usr/sbin/libvirtd)
==395==    by 0x562F3D1: virNetServerProgramDispatch (in /usr/lib64/libvirt.so.0.2000.0)
==395==    by 0x15BBCC: ??? (in /usr/sbin/libvirtd)
==395==    by 0x551D610: ??? (in /usr/lib64/libvirt.so.0.2000.0)
==395==    by 0x551C997: ??? (in /usr/lib64/libvirt.so.0.2000.0)
==395==    by 0x8079DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==395==    by 0x8384CEC: clone (in /usr/lib64/libc-2.17.so)

Comment 6 Peter Krempa 2016-09-06 14:51:23 UTC
The backtrace you've posted is different from the original issue. I'm also not able to reporduce it. Could you please re-try it with the most recent package and preferably file a new bug if it prevails?

Comment 7 Jingjing Shao 2016-09-07 07:24:16 UTC
I try this issue with the newest version of libvirt and can not reproduce it, the result is as below, so I change the status to verified


# rpm -q libvirt
libvirt-2.0.0-6.el7.x86_64


# virsh list
 Id    Name                           State
----------------------------------------------------
 4     r7.2                           running


start valgrind on another terminal:
# service libvirtd stop
# valgrind --leak-check=full libvirtd

# virsh domstats r7.2
Domain: 'r7.2'
  state.state=1
  state.reason=1
  cpu.time=31869063404
  cpu.user=1400000000
  cpu.system=5900000000
  balloon.current=1048576
  balloon.maximum=1048576
  vcpu.current=4
  vcpu.maximum=4
  vcpu.0.state=1
  vcpu.0.time=10400000000
  vcpu.0.wait=0
  vcpu.1.state=1
  vcpu.1.time=3030000000
  vcpu.1.wait=0
  vcpu.2.state=1
  vcpu.2.time=1950000000
  vcpu.2.wait=0
  vcpu.3.state=1
  vcpu.3.time=2400000000
  vcpu.3.wait=0
  net.count=0
  block.count=2
  block.0.name=vda
  block.0.path=/var/lib/libvirt/images/yy.qcow2
  block.0.rd.reqs=7083
  block.0.rd.bytes=141354496
  block.0.rd.times=37454694049
  block.0.wr.reqs=2439
  block.0.wr.bytes=8241664
  block.0.wr.times=82438856660
  block.0.fl.reqs=57
  block.0.fl.times=1002698963
  block.0.allocation=2472279552
  block.0.capacity=10737418240
  block.0.physical=2472280064
  block.1.name=hda
  block.1.rd.reqs=3
  block.1.rd.bytes=82
  block.1.rd.times=39907
  block.1.wr.reqs=0
  block.1.wr.bytes=0
  block.1.wr.times=0
  block.1.fl.reqs=0
  block.1.fl.times=0
  block.1.allocation=0



# valgrind --leak-check=full libvirtd
==4345== Memcheck, a memory error detector
==4345== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==4345== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==4345== Command: libvirtd
==4345== 
mmap: Invalid argument
==4419== 
==4419== HEAP SUMMARY:
==4419==     in use at exit: 2,960,151 bytes in 21,792 blocks
==4419==   total heap usage: 115,781 allocs, 93,989 frees, 853,914,189 bytes allocated
==4419== 
==4419== 5 bytes in 1 blocks are possibly lost in loss record 106 of 1,979
==4419==    at 0x4C27BE3: malloc (vg_replace_malloc.c:299)
==4419==    by 0x8349889: strdup (strdup.c:42)
==4419==    by 0x60463E4: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD810: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 5 bytes in 1 blocks are possibly lost in loss record 107 of 1,979
==4419==    at 0x4C27BE3: malloc (vg_replace_malloc.c:299)
==4419==    by 0x8349889: strdup (strdup.c:42)
==4419==    by 0x60463E4: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD890: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 7 bytes in 1 blocks are possibly lost in loss record 156 of 1,979
==4419==    at 0x4C27BE3: malloc (vg_replace_malloc.c:299)
==4419==    by 0x8349889: strdup (strdup.c:42)
==4419==    by 0x60463E4: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD7C8: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 7 bytes in 1 blocks are possibly lost in loss record 157 of 1,979
==4419==    at 0x4C27BE3: malloc (vg_replace_malloc.c:299)
==4419==    by 0x8349889: strdup (strdup.c:42)
==4419==    by 0x60463E4: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD7E0: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 7 bytes in 1 blocks are possibly lost in loss record 158 of 1,979
==4419==    at 0x4C27BE3: malloc (vg_replace_malloc.c:299)
==4419==    by 0x8349889: strdup (strdup.c:42)
==4419==    by 0x60463E4: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD848: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 7 bytes in 1 blocks are possibly lost in loss record 159 of 1,979
==4419==    at 0x4C27BE3: malloc (vg_replace_malloc.c:299)
==4419==    by 0x8349889: strdup (strdup.c:42)
==4419==    by 0x60463E4: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD878: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 8 bytes in 1 blocks are possibly lost in loss record 216 of 1,979
==4419==    at 0x4C27BE3: malloc (vg_replace_malloc.c:299)
==4419==    by 0x8349889: strdup (strdup.c:42)
==4419==    by 0x60463E4: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD7F8: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 9 bytes in 1 blocks are possibly lost in loss record 248 of 1,979
==4419==    at 0x4C27BE3: malloc (vg_replace_malloc.c:299)
==4419==    by 0x8349889: strdup (strdup.c:42)
==4419==    by 0x60463E4: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD860: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 13 bytes in 2 blocks are possibly lost in loss record 327 of 1,979
==4419==    at 0x4C27BE3: malloc (vg_replace_malloc.c:299)
==4419==    by 0x8349889: strdup (strdup.c:42)
==4419==    by 0x60463E4: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 32 bytes in 1 blocks are possibly lost in loss record 776 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x60463D1: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD7C8: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 32 bytes in 1 blocks are possibly lost in loss record 777 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x60463D1: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD7E0: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 32 bytes in 1 blocks are possibly lost in loss record 778 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x60463D1: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD7F8: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 32 bytes in 1 blocks are possibly lost in loss record 779 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x60463D1: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD810: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 32 bytes in 1 blocks are possibly lost in loss record 780 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x60463D1: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD848: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 32 bytes in 1 blocks are possibly lost in loss record 781 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x60463D1: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD860: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 32 bytes in 1 blocks are possibly lost in loss record 782 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x60463D1: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD878: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 32 bytes in 1 blocks are possibly lost in loss record 783 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x60463D1: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDD890: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 48 bytes in 2 blocks are possibly lost in loss record 1,038 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x6039EF1: nl_addr_alloc (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x6039F23: nl_addr_build (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDEBA6: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x603CCA8: nl_cache_parse (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CCFA: ??? (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x604300B: nl_recvmsgs_report (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x6043478: nl_recvmsgs (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603BFF1: ??? (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CC1C: nl_cache_pickup (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CE62: nl_cache_refill (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CF6A: nl_cache_alloc_and_fill (in /usr/lib64/libnl-3.so.200.23.0)
==4419== 
==4419== 64 bytes in 2 blocks are possibly lost in loss record 1,134 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x60463D1: __trans_list_add (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x400F1E2: call_init (dl-init.c:82)
==4419==    by 0x400F1E2: _dl_init (dl-init.c:131)
==4419==    by 0x4001219: ??? (in /usr/lib64/ld-2.17.so)
==4419== 
==4419== 72 bytes in 2 blocks are possibly lost in loss record 1,182 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x6039EF1: nl_addr_alloc (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x6039F23: nl_addr_build (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDEB50: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x603CCA8: nl_cache_parse (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CCFA: ??? (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x604300B: nl_recvmsgs_report (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x6043478: nl_recvmsgs (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603BFF1: ??? (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CC1C: nl_cache_pickup (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CE62: nl_cache_refill (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CF6A: nl_cache_alloc_and_fill (in /usr/lib64/libnl-3.so.200.23.0)
==4419== 
==4419== 72 bytes in 3 blocks are possibly lost in loss record 1,183 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x6039EF1: nl_addr_alloc (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x6039F23: nl_addr_build (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDECAF: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x603CCA8: nl_cache_parse (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CCFA: ??? (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x604300B: nl_recvmsgs_report (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x6043478: nl_recvmsgs (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603BFF1: ??? (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CC1C: nl_cache_pickup (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CE62: nl_cache_refill (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CF6A: nl_cache_alloc_and_fill (in /usr/lib64/libnl-3.so.200.23.0)
==4419== 
==4419== 720 bytes in 1 blocks are possibly lost in loss record 1,751 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x4011C24: allocate_dtv (dl-tls.c:317)
==4419==    by 0x4011C24: _dl_allocate_tls (dl-tls.c:533)
==4419==    by 0x80AF960: allocate_stack (allocatestack.c:570)
==4419==    by 0x80AF960: pthread_create@@GLIBC_2.2.5 (pthread_create.c:451)
==4419==    by 0x551D0EF: virThreadCreateFull (virthread.c:241)
==4419==    by 0x22430563: qemuProcessReconnectHelper (qemu_process.c:3434)
==4419==    by 0x556E2F9: virDomainObjListHelper (virdomainobjlist.c:781)
==4419==    by 0x54D9A4C: virHashForEach (virhash.c:606)
==4419==    by 0x556F820: virDomainObjListForEach (virdomainobjlist.c:796)
==4419==    by 0x2242CAD8: qemuProcessReconnectAll (qemu_process.c:3466)
==4419==    by 0x22469163: qemuStateInitialize (qemu_driver.c:923)
==4419==    by 0x55AF3BE: virStateInitialize (libvirt.c:770)
==4419==    by 0x12254A: daemonRunStateInit (libvirtd.c:959)
==4419== 
==4419== 800 bytes in 5 blocks are possibly lost in loss record 1,757 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x6043682: nl_object_alloc (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x5DDEA34: ??? (in /usr/lib64/libnl-route-3.so.200.23.0)
==4419==    by 0x603CCA8: nl_cache_parse (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CCFA: ??? (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x604300B: nl_recvmsgs_report (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x6043478: nl_recvmsgs (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603BFF1: ??? (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CC1C: nl_cache_pickup (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CE62: nl_cache_refill (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x603CF6A: nl_cache_alloc_and_fill (in /usr/lib64/libnl-3.so.200.23.0)
==4419==    by 0x1A574A6D: ??? (in /usr/lib64/libnetcf.so.1.4.0)
==4419== 
==4419== 1,312 (184 direct, 1,128 indirect) bytes in 1 blocks are definitely lost in loss record 1,822 of 1,979
==4419==    at 0x4C29975: calloc (vg_replace_malloc.c:711)
==4419==    by 0x708AAE5: ??? (in /usr/lib64/libdbus-1.so.3.7.4)
==4419==    by 0x708F11E: ??? (in /usr/lib64/libdbus-1.so.3.7.4)
==4419==    by 0x709772F: ??? (in /usr/lib64/libdbus-1.so.3.7.4)
==4419==    by 0x7097862: ??? (in /usr/lib64/libdbus-1.so.3.7.4)
==4419==    by 0x70981D5: ??? (in /usr/lib64/libdbus-1.so.3.7.4)
==4419==    by 0x70988E5: ??? (in /usr/lib64/libdbus-1.so.3.7.4)
==4419==    by 0x70975EE: ??? (in /usr/lib64/libdbus-1.so.3.7.4)
==4419==    by 0x7080D7B: ??? (in /usr/lib64/libdbus-1.so.3.7.4)
==4419==    by 0x708175C: ??? (in /usr/lib64/libdbus-1.so.3.7.4)
==4419==    by 0x7081D39: dbus_connection_send_with_reply_and_block (in /usr/lib64/libdbus-1.so.3.7.4)
==4419==    by 0x707DD53: dbus_bus_register (in /usr/lib64/libdbus-1.so.3.7.4)
==4419== 
==4419== LEAK SUMMARY:
==4419==    definitely lost: 184 bytes in 1 blocks
==4419==    indirectly lost: 1,128 bytes in 2 blocks
==4419==      possibly lost: 2,100 bytes in 33 blocks
==4419==    still reachable: 2,956,739 bytes in 21,756 blocks
==4419==                       of which reachable via heuristic:
==4419==                         stdstring          : 3,876 bytes in 131 blocks
==4419==         suppressed: 0 bytes in 0 blocks
==4419== Reachable blocks (those to which a pointer was found) are not shown.
==4419== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==4419== 
==4419== For counts of detected and suppressed errors, rerun with: -v
==4419== ERROR SUMMARY: 24 errors from 24 contexts (suppressed: 0 from 0)
==4345== Thread 3:
==4345== Syscall param open(filename) points to unaddressable byte(s)
==4345==    at 0x80B5C3D: ??? (syscall-template.S:81)
==4345==    by 0x55162D4: UnknownInlinedFun (fcntl2.h:53)
==4345==    by 0x55162D4: virStorageSourceUpdateBlockPhysicalSize (virstoragefile.c:3166)
==4345==    by 0x22486B21: qemuDomainGetStatsOneBlock.isra.13 (qemu_driver.c:18883)
==4345==    by 0x22486EBB: qemuDomainGetStatsBlock (qemu_driver.c:18940)
==4345==    by 0x2246B8F0: qemuDomainGetStats (qemu_driver.c:19107)
==4345==    by 0x2246B8F0: qemuConnectGetAllDomainStats (qemu_driver.c:19197)
==4345==    by 0x55D2405: virDomainListGetStats (libvirt-domain.c:11615)
==4345==    by 0x128FD4: remoteDispatchConnectGetAllDomainStats (remote.c:6032)
==4345==    by 0x128FD4: remoteDispatchConnectGetAllDomainStatsHelper (remote_dispatch.h:665)
==4345==    by 0x562FC11: virNetServerProgramDispatchCall (virnetserverprogram.c:437)
==4345==    by 0x562FC11: virNetServerProgramDispatch (virnetserverprogram.c:307)
==4345==    by 0x15BC6C: virNetServerProcessMsg (virnetserver.c:148)
==4345==    by 0x15BC6C: virNetServerHandleJob (virnetserver.c:169)
==4345==    by 0x551D990: virThreadPoolWorker (virthreadpool.c:167)
==4345==    by 0x551CD17: virThreadHelper (virthread.c:206)
==4345==    by 0x80AEDC4: start_thread (pthread_create.c:308)
==4345==  Address 0x0 is not stack'd, malloc'd or (recently) free'd

Comment 9 errata-xmlrpc 2016-11-03 18:39:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html