Bug 977430
Summary: | libvirt leaks cpuset mask | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 6 | Reporter: | Ján Tomko <jtomko> |
Component: | libvirt | Assignee: | Ján Tomko <jtomko> |
Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 6.4 | CC: | acathrow, ajia, cwei, dyuan, gsun, jtomko |
Target Milestone: | rc | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | libvirt-0.10.2-19.el6 | Doc Type: | Bug Fix |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2013-11-21 09:03:46 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Ján Tomko
2013-06-24 14:15:52 UTC
Downstream patch posted: http://post-office.corp.redhat.com/archives/rhvirt-patches/2013-June/msg00565.html (In reply to Jan Tomko from comment #0) > Steps to Reproduce: > 1. Edit a domain with <vcpu cpuset='...'/> present > > Actual results: > Valgrind shows a leak: > ==2281== Thread 1: > ==2281== 128 bytes in 1 blocks are definitely lost in loss record 505 of 655 > ==2281== at 0x4A0577B: calloc (vg_replace_malloc.c:593) > ==2281== by 0x31A4C5853B: virAllocN (memory.c:128) > ==2281== by 0x31A4C453D3: virBitmapNew (bitmap.c:76) > ==2281== by 0x31A4C4556D: virBitmapParse (bitmap.c:300) > ==2281== by 0x31A4C9CC0C: virDomainDefParseXML (domain_conf.c:8875) > ==2281== by 0x31A4CA3813: virDomainDefParseNode (domain_conf.c:10344) > ==2281== by 0x31A4CA395E: virDomainDefParse (domain_conf.c:10294) > ==2281== by 0x460F0C: qemudDomainDefine (qemu_driver.c:5700) > ==2281== by 0x31A4CE7C83: virDomainDefineXML (libvirt.c:7957) > ==2281== by 0x43F93D: remoteDispatchDomainDefineXMLHelper > (remote_dispatch.h:1224) > ==2281== by 0x31A4D3F161: virNetServerProgramDispatch > (virnetserverprogram.c:431) > ==2281== by 0x31A4D3FDFD: virNetServerProcessMsg (virnetserver.c:170) > Hi Jan, Is it a libvirt daemon leak? or a libvirt client leak? but I haven't seen the leaks on libvirt daemon or client with <vcpu cpuset='...'/> edition in domain XML configuration, however, I saw other leaks on client and daemon. # to use valgrind to start libvirt daemon and then # use virsh edit with <vcpu cpuset='...'/> and type 'n' to exit virsh edit, # virsh edit foo error: XML error: topology cpuset syntax error Failed. Try again? [y,n,f,?]: n Leaks on libvirt daemon: ==2395== 373 (232 direct, 141 indirect) bytes in 1 blocks are definitely lost in loss record 50 of 63 ==2395== at 0x4A04A28: calloc (vg_replace_malloc.c:467) ==2395== by 0x3EC4C5855D: virAlloc (memory.c:100) ==2395== by 0x4258C6: daemonConfigNew (libvirtd-config.c:231) ==2395== by 0x422DC9: main (libvirtd.c:976) ==2395== ==2395== LEAK SUMMARY: ==2395== definitely lost: 232 bytes in 1 blocks To use valgrind to directly run virsh edit with <vcpu cpuset='...'/> and type 'y' first time then type 'n' to exit virsh edit. ==2343== 4,443 bytes in 3 blocks are definitely lost in loss record 54 of 59 ==2343== at 0x4A05FDE: malloc (vg_replace_malloc.c:236) ==2343== by 0x33F691497D: xdr_string (xdr.c:709) ==2343== by 0x3EC4D3088D: xdr_remote_nonnull_string (remote_protocol.c:31) ==2343== by 0x3EC4D31658: xdr_remote_domain_get_xml_desc_ret (remote_protocol.c:1543) ==2343== by 0x3EC4D40F0B: virNetMessageDecodePayload (virnetmessage.c:389) ==2343== by 0x3EC4D3B15C: virNetClientProgramCall (virnetclientprogram.c:382) ==2343== by 0x3EC4D1503D: callWithFD (remote_driver.c:5536) ==2343== by 0x3EC4D150BB: call (remote_driver.c:5557) ==2343== by 0x3EC4D1F48F: remoteDomainGetXMLDesc (remote_client_bodies.h:976) ==2343== by 0x3EC4CF5C2D: virDomainGetXMLDesc (libvirt.c:4371) ==2343== by 0x418618: cmdEdit (virsh-edit.c:105) ==2343== by 0x40D150: vshCommandRun (virsh.c:1579) ==2343== ==2343== 8,193 bytes in 1 blocks are definitely lost in loss record 57 of 59 ==2343== at 0x4A05FDE: malloc (vg_replace_malloc.c:236) ==2343== by 0x4A06167: realloc (vg_replace_malloc.c:525) ==2343== by 0x3EC4C585AB: virReallocN (memory.c:160) ==2343== by 0x3EC4C67359: virFileReadLimFD (util.c:400) ==2343== by 0x3EC4C674C7: virFileReadAll (util.c:461) ==2343== by 0x40DDF0: vshEditReadBackFile (virsh.c:680) ==2343== by 0x4185EC: cmdEdit (virsh-edit.c:89) ==2343== by 0x40D150: vshCommandRun (virsh.c:1579) ==2343== by 0x410734: main (virsh.c:2994) ==2343== ==2343== LEAK SUMMARY: ==2343== definitely lost: 12,636 bytes in 4 blocks # rpm -q libvirt libvirt-0.10.2-18.el6.x86_64 Sorry, my steps to reproduce it weren't accurate. The correct steps are: 1. Have a domain with any cpuset: <vcpu placement='static' cpuset='1'>1</vcpu> 2. Successfully edit that domain. This leak happens in libvirtd when we dispose of the old domain definition. From your valgrind logs, the first libvirtd leak doesn't seem significant, since we need to have the libvirtd config allocated the whole time the daemon is running and the next two are not as important since they are in virsh, but I'll take a look at them. (In reply to Jan Tomko from comment #4) > Sorry, my steps to reproduce it weren't accurate. > The correct steps are: > 1. Have a domain with any cpuset: > <vcpu placement='static' cpuset='1'>1</vcpu> > 2. Successfully edit that domain. > > This leak happens in libvirtd when we dispose of the old domain definition. > > > From your valgrind logs, the first libvirtd leak doesn't seem significant, > since we need to have the libvirtd config allocated the whole time the > daemon is running and the next two are not as important since they are in > virsh, but I'll take a look at them. I still can't see libvirtd leaks with above operations, but I saw new memory errors except previous 'daemonConfigNew' leak. ==13922== Invalid read of size 8 ==13922== at 0x33F693016D: __strspn_sse42 (emmintrin.h:679) ==13922== by 0x3EC4C64DA2: virDiskNameToIndex (util.c:2169) ==13922== by 0x3EC4C858CE: virDomainDiskInsertPreAlloced (domain_conf.c:7950) ==13922== by 0x3EC4C9F540: virDomainDefParseXML (domain_conf.c:9493) ==13922== by 0x3EC4CA3813: virDomainDefParseNode (domain_conf.c:10344) ==13922== by 0x3EC4CA395E: virDomainDefParse (domain_conf.c:10294) ==13922== by 0x3EC4CA4359: virDomainLoadAllConfigs (domain_conf.c:14418) ==13922== by 0x44BE6A: qemudStartup (qemu_driver.c:878) ==13922== by 0x3EC4CEF2FA: virStateInitialize (libvirt.c:796) ==13922== by 0x422640: daemonRunStateInit (libvirtd.c:738) ==13922== by 0x3EC4C62538: virThreadHelper (threads-pthread.c:161) ==13922== by 0x38BAA077F0: start_thread (pthread_create.c:301) ==13922== Address 0x137c7240 is 0 bytes inside a block of size 4 alloc'd ==13922== at 0x4A05FDE: malloc (vg_replace_malloc.c:236) ==13922== by 0x3400CA67DD: xmlStrndup (xmlstring.c:45) ==13922== by 0x3EC4C90F66: virDomainDiskDefParseXML (domain_conf.c:3679) ==13922== by 0x3EC4C9F523: virDomainDefParseXML (domain_conf.c:9483) ==13922== by 0x3EC4CA3813: virDomainDefParseNode (domain_conf.c:10344) ==13922== by 0x3EC4CA395E: virDomainDefParse (domain_conf.c:10294) ==13922== by 0x3EC4CA4359: virDomainLoadAllConfigs (domain_conf.c:14418) ==13922== by 0x44BE6A: qemudStartup (qemu_driver.c:878) ==13922== by 0x3EC4CEF2FA: virStateInitialize (libvirt.c:796) ==13922== by 0x422640: daemonRunStateInit (libvirtd.c:738) ==13922== by 0x3EC4C62538: virThreadHelper (threads-pthread.c:161) ==13922== by 0x38BAA077F0: start_thread (pthread_create.c:301) ==13922== ==13922== Invalid read of size 8 ==13922== at 0x33F693016D: __strspn_sse42 (emmintrin.h:679) ==13922== by 0x3EC4C64DA2: virDiskNameToIndex (util.c:2169) ==13922== by 0x3EC4C858DA: virDomainDiskInsertPreAlloced (domain_conf.c:7950) ==13922== by 0x3EC4C9F540: virDomainDefParseXML (domain_conf.c:9493) ==13922== by 0x3EC4CA3813: virDomainDefParseNode (domain_conf.c:10344) ==13922== by 0x3EC4CA395E: virDomainDefParse (domain_conf.c:10294) ==13922== by 0x3EC4CA4359: virDomainLoadAllConfigs (domain_conf.c:14418) ==13922== by 0x44BE6A: qemudStartup (qemu_driver.c:878) ==13922== by 0x3EC4CEF2FA: virStateInitialize (libvirt.c:796) ==13922== by 0x422640: daemonRunStateInit (libvirtd.c:738) ==13922== by 0x3EC4C62538: virThreadHelper (threads-pthread.c:161) ==13922== by 0x38BAA077F0: start_thread (pthread_create.c:301) ==13922== Address 0x137c8440 is 0 bytes inside a block of size 4 alloc'd ==13922== at 0x4A05FDE: malloc (vg_replace_malloc.c:236) ==13922== by 0x3400CA67DD: xmlStrndup (xmlstring.c:45) ==13922== by 0x3EC4C90F66: virDomainDiskDefParseXML (domain_conf.c:3679) ==13922== by 0x3EC4C9F523: virDomainDefParseXML (domain_conf.c:9483) ==13922== by 0x3EC4CA3813: virDomainDefParseNode (domain_conf.c:10344) ==13922== by 0x3EC4CA395E: virDomainDefParse (domain_conf.c:10294) ==13922== by 0x3EC4CA4359: virDomainLoadAllConfigs (domain_conf.c:14418) ==13922== by 0x44BE6A: qemudStartup (qemu_driver.c:878) ==13922== by 0x3EC4CEF2FA: virStateInitialize (libvirt.c:796) ==13922== by 0x422640: daemonRunStateInit (libvirtd.c:738) ==13922== by 0x3EC4C62538: virThreadHelper (threads-pthread.c:161) ==13922== by 0x38BAA077F0: start_thread (pthread_create.c:301) I just checked again, it only shows up with valgrind --leak-check=full. A different way to reproduce it is undefining that domain: virsh define /dev/stdin <<EOF <domain type='qemu'> <name>duck</name> <memory unit='MiB'>32</memory> <vcpu placement='static' cpuset='1'>1</vcpu> <os> <type arch='x86_64' machine='pc'>hvm</type> </os> </domain> EOF virsh undefine duck The leak in virsh edit from comment 3 is now fixed upstream: commit 1e3a252974c8e5c650f1d84dc2b167f0ae8cee3c Author: Ján Tomko <jtomko> AuthorDate: 2013-06-25 15:10:27 +0200 Commit: Ján Tomko <jtomko> CommitDate: 2013-06-25 15:37:20 +0200 virsh: edit: don't leak XML string on reedit or redefine Free the old XML strings before overwriting them if the user has chosen to reedit the file or force the redefinition. Found by Alex Jia trying to reproduce another bug: https://bugzilla.redhat.com/show_bug.cgi?id=977430#c3 git describe: v1.1.0-rc1-2-g1e3a252 For the invalid reads in comment 5, could you please share the domain XML? (In reply to Jan Tomko from comment #6) > I just checked again, it only shows up with valgrind --leak-check=full. > A different way to reproduce it is undefining that domain: > > virsh define /dev/stdin <<EOF > <domain type='qemu'> > <name>duck</name> > <memory unit='MiB'>32</memory> > <vcpu placement='static' cpuset='1'>1</vcpu> > <os> > <type arch='x86_64' machine='pc'>hvm</type> > </os> > </domain> > EOF > > virsh undefine duck It's very weird for me, I still can't reproduce it based on following steps, maybe, I'm missing some important stuff. [root@202 Workspace]# service libvirtd status libvirtd is stopped [root@202 Workspace]# valgrind -v --leak-check=full --log-file=/tmp/virtd.leaks /usr/sbin/libvirtd -d [root@202 Workspace]# ps -ef|grep valgrind|grep -v grep root 18226 1 68 17:57 ? 00:00:16 valgrind -v --leak-check=full --log-file=/tmp/virtd.leaks /usr/sbin/libvirtd -d [root@202 virt-test]# virsh define /dev/stdin <<EOF <domain type='qemu'> <name>duck</name> <memory unit='MiB'>32</memory> <vcpu placement='static' cpuset='1'>1</vcpu> <os> <type arch='x86_64' machine='pc'>hvm</type> </os> </domain> EOF Domain duck defined from /dev/stdin [root@202 virt-test]# virsh undefine duck Domain duck has been undefined And then check 'definitely lost' in /tmp/virtd.leaks, I only saw the following leak: ==18224== 373 (232 direct, 141 indirect) bytes in 1 blocks are definitely lost in loss record 50 of 63 ==18224== at 0x4A04A28: calloc (vg_replace_malloc.c:467) ==18224== by 0x3EC4C5855D: virAlloc (memory.c:100) ==18224== by 0x4258C6: daemonConfigNew (libvirtd-config.c:231) ==18224== by 0x422DC9: main (libvirtd.c:976) > For the invalid reads in comment 5, could you please share the domain XML? <domain type='qemu'> <name>hello</name> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='i686' machine='rhel6.2.0'>hvm</type> <boot dev='hd'/> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/var/lib/libvirt/images/hello'/> <target dev='hda' bus='ide'/> </disk> <controller type='usb' index='0'> </controller> <controller type='ide' index='0'> </controller> <interface type='network'> <source network='default'/> <model type='virtio'/> </interface> <memballoon model='virtio'> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'/> </domain> In you define a guest with above XML, you also should see new libvirtd memory error as follows: ==18226== Invalid read of size 8 ==18226== at 0x33F693016D: __strspn_sse42 (emmintrin.h:679) ==18226== by 0x3EC4C64DA2: virDiskNameToIndex (util.c:2169) ==18226== by 0x3EC4C84584: virDomainDiskDefAssignAddress (domain_conf.c:3071) ==18226== by 0x3EC4C9275F: virDomainDiskDefParseXML (domain_conf.c:4241) ==18226== by 0x3EC4C9F523: virDomainDefParseXML (domain_conf.c:9483) ==18226== by 0x3EC4CA3813: virDomainDefParseNode (domain_conf.c:10344) ==18226== by 0x3EC4CA395E: virDomainDefParse (domain_conf.c:10294) ==18226== by 0x460F0C: qemudDomainDefine (qemu_driver.c:5700) ==18226== by 0x3EC4CE7C83: virDomainDefineXML (libvirt.c:7957) ==18226== by 0x43F93D: remoteDispatchDomainDefineXMLHelper (remote_dispatch.h:1224) ==18226== by 0x3EC4D3F161: virNetServerProgramDispatch (virnetserverprogram.c:431) ==18226== by 0x3EC4D3FDFD: virNetServerProcessMsg (virnetserver.c:170) ==18226== Address 0x149f22b0 is 0 bytes inside a block of size 4 alloc'd ==18226== at 0x4A05FDE: malloc (vg_replace_malloc.c:236) ==18226== by 0x3400CA67DD: xmlStrndup (xmlstring.c:45) ==18226== by 0x3EC4C90F66: virDomainDiskDefParseXML (domain_conf.c:3679) ==18226== by 0x3EC4C9F523: virDomainDefParseXML (domain_conf.c:9483) ==18226== by 0x3EC4CA3813: virDomainDefParseNode (domain_conf.c:10344) ==18226== by 0x3EC4CA395E: virDomainDefParse (domain_conf.c:10294) ==18226== by 0x460F0C: qemudDomainDefine (qemu_driver.c:5700) ==18226== by 0x3EC4CE7C83: virDomainDefineXML (libvirt.c:7957) ==18226== by 0x43F93D: remoteDispatchDomainDefineXMLHelper (remote_dispatch.h:1224) ==18226== by 0x3EC4D3F161: virNetServerProgramDispatch (virnetserverprogram.c:431) ==18226== by 0x3EC4D3FDFD: virNetServerProcessMsg (virnetserver.c:170) ==18226== by 0x3EC4D4049B: virNetServerHandleJob (virnetserver.c:191) (In reply to Alex Jia from comment #7) > [root@202 Workspace]# ps -ef|grep valgrind|grep -v grep > root 18226 1 68 17:57 ? 00:00:16 valgrind -v > --leak-check=full --log-file=/tmp/virtd.leaks /usr/sbin/libvirtd -d > If you run libvirtd with -d, it will fork the daemon in the background and exit and valgrind won't catch any leaks after the exit. It should be reproducible without the '-d'. However I can't reproduce that invalid read, or see how it could possibly happen. (In reply to Jan Tomko from comment #8) > (In reply to Alex Jia from comment #7) > > [root@202 Workspace]# ps -ef|grep valgrind|grep -v grep > > root 18226 1 68 17:57 ? 00:00:16 valgrind -v > > --leak-check=full --log-file=/tmp/virtd.leaks /usr/sbin/libvirtd -d > > > > If you run libvirtd with -d, it will fork the daemon in the background and > exit and valgrind won't catch any leaks after the exit. > > It should be reproducible without the '-d'. > > Retest it without '-d', I saw the same leak like you, but also got extra leaks. ==17479== Thread 1: ==17479== 74 bytes in 2 blocks are definitely lost in loss record 530 of 835 ==17479== at 0x4A05FDE: malloc (vg_replace_malloc.c:236) ==17479== by 0x3400CA67DD: xmlStrndup (xmlstring.c:45) ==17479== by 0x3EC4C7F172: virNetDevVPortProfileParse (netdev_vport_profile_conf.c:78) ==17479== by 0x3EC4C8D760: virDomainNetDefParseXML (domain_conf.c:4885) ==17479== by 0x3EC4C9F8C4: virDomainDefParseXML (domain_conf.c:9581) ==17479== by 0x3EC4CA3813: virDomainDefParseNode (domain_conf.c:10344) ==17479== by 0x3EC4CA395E: virDomainDefParse (domain_conf.c:10294) ==17479== by 0x3EC4CA4359: virDomainLoadAllConfigs (domain_conf.c:14418) ==17479== by 0x44BE6A: qemudStartup (qemu_driver.c:878) ==17479== by 0x3EC4CEF2FA: virStateInitialize (libvirt.c:796) ==17479== by 0x422640: daemonRunStateInit (libvirtd.c:738) ==17479== by 0x3EC4C62538: virThreadHelper (threads-pthread.c:161) ==17479== ==17479== 128 bytes in 1 blocks are definitely lost in loss record 581 of 835 ==17479== at 0x4A04A28: calloc (vg_replace_malloc.c:467) ==17479== by 0x3EC4C5853B: virAllocN (memory.c:128) ==17479== by 0x3EC4C453D3: virBitmapNew (bitmap.c:76) ==17479== by 0x3EC4C4556D: virBitmapParse (bitmap.c:300) ==17479== by 0x3EC4C9CC0C: virDomainDefParseXML (domain_conf.c:8875) ==17479== by 0x3EC4CA3813: virDomainDefParseNode (domain_conf.c:10344) ==17479== by 0x3EC4CA395E: virDomainDefParse (domain_conf.c:10294) ==17479== by 0x460F0C: qemudDomainDefine (qemu_driver.c:5700) ==17479== by 0x3EC4CE7C83: virDomainDefineXML (libvirt.c:7957) ==17479== by 0x43F93D: remoteDispatchDomainDefineXMLHelper (remote_dispatch.h:1224) ==17479== by 0x3EC4D3F161: virNetServerProgramDispatch (virnetserverprogram.c:431) ==17479== by 0x3EC4D3FDFD: virNetServerProcessMsg (virnetserver.c:170) Notes, to use ctrl+c to exit libvirtd and then check /tmp/virtd.leaks. > However I can't reproduce that invalid read, or see how it could possibly > happen. ==17479== Thread 12: ==17479== Invalid read of size 8 ==17479== at 0x33F693016D: __strspn_sse42 (emmintrin.h:679) ==17479== by 0x3EC4C64DA2: virDiskNameToIndex (util.c:2169) ==17479== by 0x3EC4C858DA: virDomainDiskInsertPreAlloced (domain_conf.c:7950) ==17479== by 0x3EC4C9F540: virDomainDefParseXML (domain_conf.c:9493) ==17479== by 0x3EC4CA3813: virDomainDefParseNode (domain_conf.c:10344) ==17479== by 0x3EC4CA395E: virDomainDefParse (domain_conf.c:10294) ==17479== by 0x3EC4CA4359: virDomainLoadAllConfigs (domain_conf.c:14418) ==17479== by 0x44BE6A: qemudStartup (qemu_driver.c:878) ==17479== by 0x3EC4CEF2FA: virStateInitialize (libvirt.c:796) ==17479== by 0x422640: daemonRunStateInit (libvirtd.c:738) ==17479== by 0x3EC4C62538: virThreadHelper (threads-pthread.c:161) ==17479== by 0x38BAA077F0: start_thread (pthread_create.c:301) ==17479== Address 0x133117f0 is 0 bytes inside a block of size 4 alloc'd ==17479== at 0x4A05FDE: malloc (vg_replace_malloc.c:236) ==17479== by 0x3400CA67DD: xmlStrndup (xmlstring.c:45) ==17479== by 0x3EC4C90F66: virDomainDiskDefParseXML (domain_conf.c:3679) ==17479== by 0x3EC4C9F523: virDomainDefParseXML (domain_conf.c:9483) ==17479== by 0x3EC4CA3813: virDomainDefParseNode (domain_conf.c:10344) ==17479== by 0x3EC4CA395E: virDomainDefParse (domain_conf.c:10294) ==17479== by 0x3EC4CA4359: virDomainLoadAllConfigs (domain_conf.c:14418) ==17479== by 0x44BE6A: qemudStartup (qemu_driver.c:878) ==17479== by 0x3EC4CEF2FA: virStateInitialize (libvirt.c:796) ==17479== by 0x422640: daemonRunStateInit (libvirtd.c:738) ==17479== by 0x3EC4C62538: virThreadHelper (threads-pthread.c:161) ==17479== by 0x38BAA077F0: start_thread (pthread_create.c:301) Test it with steps just like comment 9 and no leak happens. libvirt pkg: libvirt-0.10.2-19.el6.x86_64 So, it's verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1581.html |