Bug 1599545

Summary: libvirtd crashed randomly during start
Product: Red Hat Enterprise Linux 7 Reporter: Luyao Huang <lhuang>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: Luyao Huang <lhuang>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.6CC: dyuan, eskultet, fjin, lmen, xuzhang, yafu, yalzhang
Target Milestone: rcKeywords: Upstream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-4.5.0-3.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-30 09:57:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Luyao Huang 2018-07-10 03:57:59 UTC
Description of problem:
libvirtd crashed randomly during start

Version-Release number of selected component (if applicable):
libvirt-4.5.0-2.el7.x86_64

How reproducible:
40%

Steps to Reproduce:
1. restart libvirtd on a host which have heavy workload
2.
3.

Actual results:

libvirtd crashed during start

Expected results:

No invalid memory access

Additional info:

valgrind:

==11847== For counts of detected and suppressed errors, rerun with: -v
==11847== ERROR SUMMARY: 23 errors from 23 contexts (suppressed: 0 from 0)
==11781== Thread 19:
==11781== Conditional jump or move depends on uninitialised value(s)
==11781==    at 0x4C2AC70: free (vg_replace_malloc.c:530)
==11781==    by 0x536F159: virFree (viralloc.c:582)
==11781==    by 0x54A1C33: virDomainConfNWFilterInstantiate (domain_nwfilter.c:122)
==11781==    by 0x321D4C05: qemuProcessFiltersInstantiate (qemu_process.c:3028)
==11781==    by 0x321D4C05: qemuProcessReconnect (qemu_process.c:7658)
==11781==    by 0x54208A4: virThreadHelper (virthread.c:206)
==11781==    by 0x8429DD4: start_thread (in /usr/lib64/libpthread-2.17.so)
==11781==    by 0x873BEAC: clone (in /usr/lib64/libc-2.17.so)
==11781== 
==11781== Invalid free() / delete / delete[] / realloc()
==11781==    at 0x4C2ACBD: free (vg_replace_malloc.c:530)
==11781==    by 0x536F159: virFree (viralloc.c:582)
==11781==    by 0x54A1C33: virDomainConfNWFilterInstantiate (domain_nwfilter.c:122)
==11781==    by 0x321D4C05: qemuProcessFiltersInstantiate (qemu_process.c:3028)
==11781==    by 0x321D4C05: qemuProcessReconnect (qemu_process.c:7658)
==11781==    by 0x54208A4: virThreadHelper (virthread.c:206)
==11781==    by 0x8429DD4: start_thread (in /usr/lib64/libpthread-2.17.so)
==11781==    by 0x873BEAC: clone (in /usr/lib64/libc-2.17.so)
==11781==  Address 0x2bc7 is not stack'd, malloc'd or (recently) free'd
==11781== 

GDB:

#0  0x00007ffff428d59c in free () from /lib64/libc.so.6
#1  0x00007ffff721314a in virFree (ptrptr=ptrptr@entry=0x7fffc67f1b00) at util/viralloc.c:582
#2  0x00007ffff7345ac4 in virDomainConfNWFilterInstantiate (vmname=<optimized out>, vmuuid=vmuuid@entry=0x7fffc017e898 "߉\237\\۔H\262\206z\340\302f\265\233z", net=<optimized out>, 
    ignoreExists=ignoreExists@entry=true) at conf/domain_nwfilter.c:122
#3  0x00007fffca5a77f6 in qemuProcessFiltersInstantiate (ignoreExists=true, def=0x7fffc017e890) at qemu/qemu_process.c:3028
#4  qemuProcessReconnect (opaque=<optimized out>) at qemu/qemu_process.c:7653
#5  0x00007ffff72c4895 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#6  0x00007ffff45dcdd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007ffff4305ead in clone () from /lib64/libc.so.6

Comment 1 Luyao Huang 2018-07-10 04:03:50 UTC
This upstream commit should fix this issue:

commit d7557f5f6f51264c1b5623e97f7b4d5259d4d3b6
Author: Luyao Huang <lhuang>
Date:   Thu Jul 5 12:34:11 2018 +0800

    virDomainConfNWFilterInstantiate: initialize @xml to avoid random crash
    
    If the code jump to the cleanup before assigning value to @xml
    libvirtd may crash when it tries to free an uninitialized pointer.
    
    backtrace:
    
    0  0x00007ffff428d59c in free () from /lib64/libc.so.6
    1  0x00007ffff721314a in virFree (ptrptr=ptrptr@entry=0x7fffc67f1b00) at util/viralloc.c:582
    2  0x00007ffff7345ac4 in virDomainConfNWFilterInstantiate (vmname=<optimized out>,
       vmuuid=vmuuid@entry=0x7fffc0181ca8 "߉\237\\۔H\262\206z\340\302f\265\233z", net=<optimized out>,
       ignoreExists=ignoreExists@entry=true) at conf/domain_nwfilter.c:122
    3  0x00007fffca5a77f6 in qemuProcessFiltersInstantiate (ignoreExists=true, def=0x7fffc0181ca0) at qemu/qemu_process.c:3028
    4  qemuProcessReconnect (opaque=<optimized out>) at qemu/qemu_process.c:7653
    5  0x00007ffff72c4895 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
    6  0x00007ffff45dcdd5 in start_thread () from /lib64/libpthread.so.0
    7  0x00007ffff4305ead in clone () from /lib64/libc.so.6
    
    Signed-off-by: Luyao Huang <lhuang>

v4.5.0-44-gd7557f5

Comment 5 Luyao Huang 2018-08-14 06:53:51 UTC
Verify this bug with libvirt-4.5.0-6.el7.x86_64:

1. use valgrind to start libvirtd cannot find invalid memory free error

2. restart libvirtd in a heavy workload host several times, no libvirtd crash happened

Comment 7 errata-xmlrpc 2018-10-30 09:57:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113