Bug 1612009

Summary: libvirtd crash when using perl-Sys-Virt API: get_node_sev_info()
Product: Red Hat Enterprise Linux 7 Reporter: Dan Zheng <dzheng>
Component: libvirtAssignee: Erik Skultety <eskultet>
Status: CLOSED ERRATA QA Contact: Dan Zheng <dzheng>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.6CC: eskultet, tburke, xuzhang
Target Milestone: rcKeywords: Automation
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-4.5.0-7.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1619150 (view as bug list) Environment:
Last Closed: 2018-10-30 09:58:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1619150    

Description Dan Zheng 2018-08-03 08:53:51 UTC
Description of problem:
libvirtd crash when using perl-Sys-Virt API: get_node_sev_info(0)

Version-Release number of selected component (if applicable):
libvirt-4.5.0-6.el7.x86_64
qemu-kvm-rhev-2.12.0-9.el7.x86_64
kernel-3.10.0-931.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Using perl-Sys-Virt API

use warnings;

use Sys::Virt;
use Sys::Virt::Domain;

my $uri = shift @ARGV;
my $domname = shift @ARGV;

my $c = Sys::Virt->new(uri => $uri);

my $dom = $c->get_domain_by_name($domname);

my $info = $c->get_node_sev_info(0);        <===libvirtd carsh


2.
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fca9ae36700 (LWP 20767)]
0x00007fca88f1dfd1 in qemuGetSEVInfoToParams (flags=<optimized out>, nparams=0x7fca9ae35afc, params=0x7fca9ae35b00, qemuCaps=0x7fca8018db60) at qemu/qemu_driver.c:21500
21500	    if (virTypedParamsAddString(&sevParams, &n, &maxpar,
(gdb) thread apply all bt

Thread 17 (Thread 0x7fca9be38700 (LWP 20765)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952a71518, m=m@entry=0x55b952a714f0) at util/virthread.c:154
#2  0x00007fcaabca2323 in virThreadPoolWorker (opaque=opaque@entry=0x55b952a65f80) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca9be38700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 16 (Thread 0x7fca9b637700 (LWP 20766)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952a71518, m=m@entry=0x55b952a714f0) at util/virthread.c:154
#2  0x00007fcaabca2323 in virThreadPoolWorker (opaque=opaque@entry=0x55b952a65ec0) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca9b637700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 15 (Thread 0x7fca9ae36700 (LWP 20767)):
#0  0x00007fca88f1dfd1 in qemuGetSEVInfoToParams (flags=<optimized out>, nparams=0x7fca9ae35afc, params=0x7fca9ae35b00, qemuCaps=0x7fca8018db60) at qemu/qemu_driver.c:21500
#1  qemuNodeGetSEVInfo (conn=<optimized out>, params=0x7fca9ae35b00, nparams=0x7fca9ae35afc, flags=<optimized out>) at qemu/qemu_driver.c:21551
#2  0x00007fcaabe49837 in virNodeGetSEVInfo (conn=0x7fca5c0009a0, params=params@entry=0x7fca9ae35b00, nparams=nparams@entry=0x7fca9ae35afc, flags=4) at libvirt-host.c:1679
#3  0x000055b9511977f9 in remoteDispatchNodeGetSevInfo (server=0x55b952a713c0, msg=0x55b952ac3c90, ret=0x7fca64000910, args=0x7fca640008f0, rerr=0x7fca9ae35c10, client=0x55b952ac3700)
    at remote/remote_daemon_dispatch.c:5071
#4  remoteDispatchNodeGetSevInfoHelper (server=0x55b952a713c0, client=0x55b952ac3700, msg=0x55b952ac3c90, rerr=0x7fca9ae35c10, args=0x7fca640008f0, ret=0x7fca64000910)
    at remote/remote_daemon_dispatch_stubs.h:15145
#5  0x00007fcaabd64015 in virNetServerProgramDispatchCall (msg=0x55b952ac3c90, client=0x55b952ac3700, server=0x55b952a713c0, prog=0x55b952ac0c50) at rpc/virnetserverprogram.c:437
#6  virNetServerProgramDispatch (prog=0x55b952ac0c50, server=server@entry=0x55b952a713c0, client=0x55b952ac3700, msg=0x55b952ac3c90) at rpc/virnetserverprogram.c:304
#7  0x00007fcaabd6a7ed in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x55b952a713c0) at rpc/virnetserver.c:143
#8  virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x55b952a713c0) at rpc/virnetserver.c:164
#9  0x00007fcaabca2271 in virThreadPoolWorker (opaque=opaque@entry=0x55b952a65e00) at util/virthreadpool.c:167
#10 0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#11 0x00007fcaa908edd5 in start_thread (arg=0x7fca9ae36700) at pthread_create.c:307
#12 0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 14 (Thread 0x7fca9a635700 (LWP 20768)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952a71518, m=m@entry=0x55b952a714f0) at util/virthread.c:154
#2  0x00007fcaabca2323 in virThreadPoolWorker (opaque=opaque@entry=0x55b952a65d40) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca9a635700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 13 (Thread 0x7fca99e34700 (LWP 20769)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952a71518, m=m@entry=0x55b952a714f0) at util/virthread.c:154
#2  0x00007fcaabca2323 in virThreadPoolWorker (opaque=opaque@entry=0x55b952a65ec0) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca99e34700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
---Type <return> to continue, or q <return> to quit--- 

Thread 12 (Thread 0x7fca99633700 (LWP 20770)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952a715b8, m=m@entry=0x55b952a714f0) at util/virthread.c:154
#2  0x00007fcaabca22bb in virThreadPoolWorker (opaque=opaque@entry=0x55b952a65d40) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca99633700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 11 (Thread 0x7fca98e32700 (LWP 20771)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952a715b8, m=m@entry=0x55b952a714f0) at util/virthread.c:154
#2  0x00007fcaabca22bb in virThreadPoolWorker (opaque=opaque@entry=0x55b952a65ec0) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca98e32700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 10 (Thread 0x7fca98631700 (LWP 20772)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952a715b8, m=m@entry=0x55b952a714f0) at util/virthread.c:154
#2  0x00007fcaabca22bb in virThreadPoolWorker (opaque=opaque@entry=0x55b952a65d40) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca98631700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 9 (Thread 0x7fca97e30700 (LWP 20773)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952a715b8, m=m@entry=0x55b952a714f0) at util/virthread.c:154
#2  0x00007fcaabca22bb in virThreadPoolWorker (opaque=opaque@entry=0x55b952a65e00) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca97e30700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 8 (Thread 0x7fca9762f700 (LWP 20774)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952a715b8, m=m@entry=0x55b952a714f0) at util/virthread.c:154
#2  0x00007fcaabca22bb in virThreadPoolWorker (opaque=opaque@entry=0x55b952a65f80) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca9762f700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 7 (Thread 0x7fca889b3700 (LWP 20775)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952ac0e58, m=m@entry=0x55b952ac0e30) at util/virthread.c:154
#2  0x00007fcaabca2323 in virThreadPoolWorker (opaque=opaque@entry=0x55b952ac0f60) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca889b3700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

---Type <return> to continue, or q <return> to quit---
Thread 6 (Thread 0x7fca881b2700 (LWP 20776)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952ac0e58, m=m@entry=0x55b952ac0e30) at util/virthread.c:154
#2  0x00007fcaabca2323 in virThreadPoolWorker (opaque=opaque@entry=0x55b952ac12e0) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca881b2700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7fca879b1700 (LWP 20777)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952ac0e58, m=m@entry=0x55b952ac0e30) at util/virthread.c:154
#2  0x00007fcaabca2323 in virThreadPoolWorker (opaque=opaque@entry=0x55b952ac1660) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca879b1700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7fca871b0700 (LWP 20778)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952ac0e58, m=m@entry=0x55b952ac0e30) at util/virthread.c:154
#2  0x00007fcaabca2323 in virThreadPoolWorker (opaque=opaque@entry=0x55b952ac19e0) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca871b0700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7fca869af700 (LWP 20779)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x55b952ac0e58, m=m@entry=0x55b952ac0e30) at util/virthread.c:154
#2  0x00007fcaabca2323 in virThreadPoolWorker (opaque=opaque@entry=0x55b952ac12e0) at util/virthreadpool.c:124
#3  0x00007fcaabca15f8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca869af700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7fca859ad700 (LWP 20819)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007fcaabca1866 in virCondWait (c=c@entry=0x7fca800f3e40, m=m@entry=0x7fca800f3e00) at util/virthread.c:154
#2  0x00007fca89a3f168 in udevEventHandleThread (opaque=<optimized out>) at node_device/node_device_udev.c:1604
#3  0x00007fcaabca1622 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007fcaa908edd5 in start_thread (arg=0x7fca859ad700) at pthread_create.c:307
#5  0x00007fcaa8db7ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7fcaac7588c0 (LWP 20764)):
#0  0x00007fcaa8dad20d in poll () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007fcaabc423e6 in poll (__timeout=-1, __nfds=12, __fds=<optimized out>) at /usr/include/bits/poll2.h:46
#2  virEventPollRunOnce () at util/vireventpoll.c:641
#3  0x00007fcaabc40ec2 in virEventRunDefaultImpl () at util/virevent.c:327
#4  0x00007fcaabd6a06d in virNetDaemonRun (dmn=dmn@entry=0x55b952a71140) at rpc/virnetdaemon.c:850
#5  0x000055b9511938a9 in main (argc=<optimized out>, argv=<optimized out>) at remote/remote_daemon.c:1461


Actual results:


Expected results:


Additional info:

Comment 2 Erik Skultety 2018-08-15 15:32:39 UTC
A more simple reproducer for python3 would be:
1) start libvirtd (need to cache the qemu capabilities)
2) restart libvirtd (this reloads the qemu capabilities from a cache)
3) use the following python3 snippet:

import libvirt
conn = libvirt.open()
conn.getSEVInfo()

4) libvirtd encounters SIGSEGV

Comment 3 Erik Skultety 2018-08-15 15:33:07 UTC
patches proposed upstream:
https://www.redhat.com/archives/libvir-list/2018-August/msg00940.html

Comment 4 Erik Skultety 2018-08-20 05:22:19 UTC
Fixed upstream by commit:

commit 77f51ab52049734d80a8ccb79b80189c7fb95c41
Refs: v4.6.0-211-g77f51ab520
Author:     Erik Skultety <eskultet>
AuthorDate: Thu Aug 9 15:27:26 2018 +0200
Commit:     Erik Skultety <eskultet>
CommitDate: Mon Aug 20 07:18:21 2018 +0200

    qemu: caps: Format SEV platform data into qemuCaps cache

    Since we're not saving the platform-specific data into a cache, we're
    not going to populate the structure, which in turn will cause a crash
    upon calling virNodeGetSEVInfo because of a NULL pointer dereference.
    Ultimately, we should start caching this data along with host-specific
    capabilities like NUMA and SELinux stuff into a separate cache, but for
    the time being, this is a semi-proper fix for a potential crash.

    Backtrace (requires libvirtd restart to load qemu caps from cache):
        #0 qemuGetSEVInfoToParams
        #1 qemuNodeGetSEVInfo
        #2 virNodeGetSEVInfo
        #3 remoteDispatchNodeGetSevInfo
        #4 remoteDispatchNodeGetSevInfoHelper
        #5 virNetServerProgramDispatchCall
        #6 virNetServerProgramDispatch
        #7 virNetServerProcessMsg
        #8 virNetServerHandleJob
  	#9 virThreadPoolWorker
        #10 virThreadHelper

    https: //bugzilla.redhat.com/show_bug.cgi?id=1612009
    Signed-off-by: Erik Skultety <eskultet>
    Acked-by: Peter Krempa <pkrempa>
    Tested-by: Brijesh Singh <brijesh.singh>

Comment 7 Dan Zheng 2018-09-25 03:51:36 UTC
# rpm -q libvirt perl-Sys-Virt
libvirt-4.5.0-9.el7.x86_64
perl-Sys-Virt-4.5.0-2.el7.x86_64

Check libvirtd 
# systemctl status libvirtd
 Main PID: 9658 (libvirtd)

 ./test_1612009.pl qemu:///system cc
libvirt error code: 84, message: Operation not supported: QEMU does not support SEV guest

Note:  ./test_1612009.pl is same with the scripts in description.

Check libvirtd again
# systemctl status libvirtd
 Main PID: 9658 (libvirtd)


NO libvirtd crash any more. Verify.

Comment 9 errata-xmlrpc 2018-10-30 09:58:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113