Bug 1357358

Summary: Memory leaks when virt-admin client is disconnected forcely.
Product: Red Hat Enterprise Linux 7 Reporter: Fangge Jin <fjin>
Component: libvirtAssignee: Erik Skultety <eskultet>
Status: CLOSED ERRATA QA Contact: Lili Zhu <lizhu>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.3CC: dyuan, jdenemar, mzhan, rbalakri, xuzhang, yafu, zpeng
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-2.5.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 17:11:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Fangge Jin 2016-07-18 04:34:06 UTC
Description of problem:
Memory leaks when virt-admin client is disconnected forcely.

Version-Release number of selected component:
libvirt-2.0.0-2.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Connect to admin server:
# virt-admin -c libvirtd:///system
Welcome to virt-admin, the administrating virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit


virt-admin #


2. Check client id and disconnect the client:
# virt-admin srv-clients-list admin
 Id    Transport       Connected since
--------------------------------------------------
 44    unix            2016-07-18 09:40:31+0800  ===> This client is the one opened in step1
 45    unix            2016-07-18 09:40:36+0800

# virt-admin client-disconnect admin 44
Client '44' disconnected


3. Quit the virt-admin client opened in step1:
virt-admin # error: Disconnected from libvirtd:///system due to I/O error

virt-admin # quit

error: One or more references were leaked after disconnect from the hypervisor


Actual results:
There is memory leak when quit a virt-admin client which is disconnected forcely.

Expected results:
No memory leak.


Additional info:
The valgrind output is as below:

# valgrind --leak-check=full virt-admin -c libvirtd:///system
==15330== Memcheck, a memory error detector
==15330== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==15330== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info
==15330== Command: virt-admin -c libvirtd:///system
==15330==
Welcome to virt-admin, the administrating virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virt-admin # error: Disconnected from libvirtd:///system due to I/O error

==15330== Syscall param sendmsg(msg.msg_name) points to uninitialised byte(s)
==15330==    at 0x7EB978D: ??? (in /usr/lib64/libpthread-2.17.so)
==15330==    by 0x505830E: readline (in /usr/lib64/libreadline.so.6.2)
==15330==    by 0x10D5A0: main (virt-admin.c:1405)
==15330==  Address 0xffeffff92 is on thread 1's stack
==15330==  in frame #1, created by readline (???)
==15330==
virt-admin # quit

error: One or more references were leaked after disconnect from the hypervisor
==15330==
==15330== HEAP SUMMARY:
==15330==     in use at exit: 126,683 bytes in 547 blocks
==15330==   total heap usage: 2,218 allocs, 1,671 frees, 407,686 bytes allocated
==15330==
==15330== 717 (88 direct, 629 indirect) bytes in 1 blocks are definitely lost in loss record 112 of 139
==15330==    at 0x4C2B974: calloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==15330==    by 0x52F0A54: virAllocVar (viralloc.c:560)
==15330==    by 0x533F695: virObjectNew (virobject.c:193)
==15330==    by 0x533FAB3: virObjectLockableNew (virobject.c:219)
==15330==    by 0x4E3D288: virAdmConnectNew (datatypes.c:900)
==15330==    by 0x4E37E62: virAdmConnectOpen (libvirt-admin.c:216)
==15330==    by 0x10EF32: vshAdmConnect (virt-admin.c:161)
==15330==    by 0x10EF32: vshAdmReconnect (virt-admin.c:215)
==15330==    by 0x10D4FB: vshAdmInit (virt-admin.c:1017)
==15330==    by 0x10D4FB: main (virt-admin.c:1385)
==15330==
==15330== LEAK SUMMARY:
==15330==    definitely lost: 88 bytes in 1 blocks
==15330==    indirectly lost: 629 bytes in 9 blocks
==15330==      possibly lost: 0 bytes in 0 blocks
==15330==    still reachable: 125,966 bytes in 537 blocks
==15330==         suppressed: 0 bytes in 0 blocks
==15330== Reachable blocks (those to which a pointer was found) are not shown.
==15330== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==15330==
==15330== For counts of detected and suppressed errors, rerun with: -v
==15330== Use --track-origins=yes to see where uninitialised values come from
==15330== ERROR SUMMARY: 3 errors from 2 contexts (suppressed: 1 from 1)

Comment 2 Erik Skultety 2016-11-14 11:27:02 UTC
Fixed upstream by:

commit e555ed6f7b06c8dce6198b6f822f6b4e5fd478d5
Author:     Erik Skultety <eskultet>
AuthorDate: Thu Nov 10 13:21:29 2016 +0100
Commit:     Erik Skultety <eskultet>
CommitDate: Mon Nov 14 10:18:56 2016 +0100

    admin: Use the newly introduced close callback handling helpers
    
    Use the newly introduced close callback helpers to make the code look just a
    bit cleaner and more importantly, to fix the following memleak regarding a
    dangling virAdmConnect object reference caused by assigning NULL to the
    close callback data once the catch-disconnect routine used the callback 
    followed by a comparison of NULL to the originally defined close callback
    (which at that moment had already been NULL'd by 
    remoteAdminClientCloseFunc) in virAdmConnectCloseCallbackUnregister.
    
    717 (88 direct, 629 indirect) bytes in 1 blocks are definitely lost record
     110 of 141
        at 0x4C2A988: calloc (vg_replace_malloc.c:711)
        by 0x530696F: virAllocVar (viralloc.c:560)
        by 0x53689E6: virObjectNew (virobject.c:193)
        by 0x5368B5E: virObjectLockableNew (virobject.c:219)
        by 0x4E3E7EE: virAdmConnectNew (datatypes.c:900)
        by 0x4E398BB: virAdmConnectOpen (libvirt-admin.c:220)
        by 0x10D3E3: vshAdmConnect (virt-admin.c:161)
        by 0x10D624: vshAdmReconnect (virt-admin.c:215)
        by 0x10DB0A: cmdConnect (virt-admin.c:353)
        by 0x11288F: vshCommandRun (vsh.c:1313)
        by 0x10FDB6: main (virt-admin.c:1439)
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1357358
    
    Signed-off-by: Erik Skultety <eskultet>

v2.4.0-85-ge555ed6

Comment 4 Lili Zhu 2017-06-14 05:04:49 UTC
Reproduce the bug with build libvirt-2.0.0-10.el7.x86_64
Steps to reproduce
1. In terminal 1,
# virt-admin -c libvirtd:///system
Welcome to virt-admin, the administrating virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virt-admin #

2. In terminal 2,
# virt-admin srv-clients-list admin
 Id    Transport       Connected since
--------------------------------------------------
 6     unix            2017-06-14 12:38:28+0800
 7     unix            2017-06-14 12:38:34+0800

# virt-admin client-disconnect admin 6
Client '6' disconnected

3. In terminal 1,
virt-admin # error: Disconnected from libvirtd:///system due to I/O error
quit

error: One or more references were leaked after disconnect from the hypervisor

Verify this bug with build libvirt-3.2.0-9.el7.x86_64
1. in terminal 1

# virt-admin -c libvirtd:///system
Welcome to virt-admin, the administrating virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virt-admin # 

2. In terminal 2
# virt-admin client-list admin
 Id    Transport       Connected since
--------------------------------------------------
 1     unix            2017-06-14 12:53:09+0800
 2     unix            2017-06-14 12:53:39+0800

# virt-admin client-disconnect admin 1
Client '1' disconnected

3. In terminal 1
virt-admin # error: Disconnected from libvirtd:///system due to I/O error
quit

Additional info:
The valgrind output is as below:
#  valgrind --leak-check=full virt-admin -c libvirtd:///system
==4773== Memcheck, a memory error detector
==4773== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==4773== Using Valgrind-3.12.0 and LibVEX; rerun with -h for copyright info
==4773== Command: virt-admin -c libvirtd:///system
==4773== 
Welcome to virt-admin, the administrating virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virt-admin # error: Disconnected from libvirtd:///system due to I/O error
quit
==4773== Syscall param sendmsg(msg.msg_name) points to uninitialised byte(s)
==4773==    at 0x811CC6D: ??? (in /usr/lib64/libpthread-2.17.so)
==4773==    by 0x505A30E: readline (in /usr/lib64/libreadline.so.6.2)
==4773==    by 0x10DA50: main (virt-admin.c:1551)
==4773==  Address 0xffeffff72 is on thread 1's stack
==4773==  in frame #1, created by readline (???:)
==4773== 

==4773== 
==4773== HEAP SUMMARY:
==4773==     in use at exit: 125,935 bytes in 538 blocks
==4773==   total heap usage: 2,257 allocs, 1,719 frees, 490,281 bytes allocated
==4773== 
==4773== LEAK SUMMARY:
==4773==    definitely lost: 0 bytes in 0 blocks
==4773==    indirectly lost: 0 bytes in 0 blocks
==4773==      possibly lost: 0 bytes in 0 blocks
==4773==    still reachable: 125,935 bytes in 538 blocks
==4773==         suppressed: 0 bytes in 0 blocks
==4773== Reachable blocks (those to which a pointer was found) are not shown.
==4773== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==4773== 
==4773== For counts of detected and suppressed errors, rerun with: -v
==4773== Use --track-origins=yes to see where uninitialised values come from
==4773== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

As the result match the expectation, mark the bug as verified.

Comment 5 errata-xmlrpc 2017-08-01 17:11:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 6 errata-xmlrpc 2017-08-01 23:53:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846