1464300 – libvirt crashed with SIGSEGV after applying update

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1464300 - libvirt crashed with SIGSEGV after applying update

Summary: libvirt crashed with SIGSEGV after applying update

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	7.4
Hardware:	x86_64
OS:	Linux
Priority:	low
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Jiri Denemark
QA Contact:	yafu
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1420851
TreeView+	depends on / blocked

Reported:	2017-06-23 02:57 UTC by Germano Veit Michel
Modified:	2021-03-11 15:21 UTC (History)
CC List:	7 users (show)
Fixed In Version:	libvirt-3.9.0-1.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-04-10 10:50:46 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Knowledge Base (Article)	4000221	0	None	None	None	2019-03-21 02:24:19 UTC
Red Hat Product Errata	RHEA-2018:0704	0	None	None	None	2018-04-10 10:52:02 UTC

Description Germano Veit Michel 2017-06-23 02:57:00 UTC

Description of problem:

libvirtd failed to restart after yum update:

abrt-hook-ccpp[12809]: Process 12736 (libvirtd) of user 0 killed by SIGSEGV - dumping core

Version-Release number of selected component (if applicable):
libvirt-3.2.0-9.el7.x86_64

How reproducible:
0%

Actual results:
libvirt crashed

Expected results:
libvirt run

Additional info:

Program terminated with signal 11, Segmentation fault.
#0  virDomainXMLOptionNew (config=0x7f3c4f2060e0 <virLXCDriverDomainDefParserConfig>, priv=0x7f3c4f206120 <virLXCDriverPrivateDataCallbacks>, 
    xmlns=0x7f3c4f206160 <virLXCDriverDomainXMLNamespace>, abi=0x128, abi@entry=0x0, saveCookie=saveCookie@entry=0x0) at conf/domain_conf.c:1081
1081            xmlopt->abi = *abi;

Thread 1 (Thread 0x7f3c4c57d700 (LWP 12752)):
#0  virDomainXMLOptionNew (config=0x7f3c4f2060e0 <virLXCDriverDomainDefParserConfig>, priv=0x7f3c4f206120 <virLXCDriverPrivateDataCallbacks>, 
    xmlns=0x7f3c4f206160 <virLXCDriverDomainXMLNamespace>, abi=0x128, abi@entry=0x0, saveCookie=saveCookie@entry=0x0) at conf/domain_conf.c:1081
#1  0x00007f3c4efd32af in lxcDomainXMLConfInit () at lxc/lxc_conf.c:213
#2  0x00007f3c4efe9029 in lxcStateInitialize (privileged=<optimized out>, callback=<optimized out>, opaque=<optimized out>) at lxc/lxc_driver.c:1678
#3  0x00007f3c7b15150f in virStateInitialize (privileged=true, callback=callback@entry=0x557832935970 <daemonInhibitCallback>, opaque=opaque@entry=0x557834037c70) at libvirt.c:770
#4  0x00005578329359cb in daemonRunStateInit (opaque=0x557834037c70) at libvirtd.c:881
#5  0x00007f3c7b0b8352 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#6  0x00007f3c784cfe25 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f3c781fd34d in clone () from /lib64/libc.so.6

Comment 2 Peter Krempa 2017-08-03 13:37:17 UTC

Logs from the sosreport hint that the daemon was restarted while being updated:
Jun 16 18:12:33 hostname yum[12141]: Updated: libvirt-daemon-3.2.0-9.el7.x86_64
Jun 16 18:12:34 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-core-3.2.0-9.el7.x86_64
Jun 16 18:12:34 hostname yum[12141]: Updated: libvirt-daemon-driver-network-3.2.0-9.el7.x86_64
Jun 16 18:12:34 hostname yum[12141]: Updated: libvirt-daemon-driver-nwfilter-3.2.0-9.el7.x86_64
Jun 16 18:12:35 hostname yum[12141]: Updated: libvirt-daemon-driver-qemu-3.2.0-9.el7.x86_64
Jun 16 18:12:35 hostname yum[12141]: Updated: libvirt-daemon-driver-interface-3.2.0-9.el7.x86_64
Jun 16 18:12:35 hostname yum[12141]: Updated: libvirt-daemon-driver-nodedev-3.2.0-9.el7.x86_64
Jun 16 18:12:36 hostname yum[12141]: Updated: libvirt-daemon-driver-secret-3.2.0-9.el7.x86_64
Jun 16 18:12:36 hostname systemd[1]: Stopping Virtualization daemon...
Jun 16 18:12:36 hostname systemd[1]: Starting Virtualization daemon...
Jun 16 18:12:36 hostname systemd[1]: Started Virtualization daemon.
Jun 16 18:12:36 hostname yum[12141]: Updated: libvirt-daemon-config-nwfilter-3.2.0-9.el7.x86_64
Jun 16 18:12:36 hostname dnsmasq[3678]: read /etc/hosts - 2 addresses
Jun 16 18:12:36 hostname dnsmasq[3678]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jun 16 18:12:36 hostname dnsmasq-dhcp[3678]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Jun 16 18:12:37 hostname yum[12141]: Updated: libvirt-daemon-driver-lxc-3.2.0-9.el7.x86_64
Jun 16 18:12:37 hostname yum[12141]: Updated: libvirt-daemon-config-network-3.2.0-9.el7.x86_64
Jun 16 18:12:38 hostname kvm[12798]: 1 guest now active
Jun 16 18:12:38 hostname kvm[12800]: 0 guests now active
Jun 16 18:12:38 hostname kernel: libvirtd[12752]: segfault at 128 ip 00007f3c7b0d9f41 sp 00007f3c4c57cc60 error 4 in libvirt.so.0.3002.0[7f3c7afd3000+37a000]
Jun 16 18:12:38 hostname abrt-hook-ccpp[12809]: Process 12736 (libvirtd) of user 0 killed by SIGSEGV - dumping core
Jun 16 18:12:38 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-rbd-3.2.0-9.el7.x86_64
Jun 16 18:12:38 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-mpath-3.2.0-9.el7.x86_64
Jun 16 18:12:38 hostname abrt-hook-ccpp[12809]: Failed to create core_backtrace: waitpid failed: No child processes
Jun 16 18:12:38 hostname systemd[1]: libvirtd.service: main process exited, code=killed, status=11/SEGV
Jun 16 18:12:38 hostname systemd[1]: Unit libvirtd.service entered failed state.
Jun 16 18:12:38 hostname systemd[1]: libvirtd.service failed.
Jun 16 18:12:38 hostname systemd[1]: libvirtd.service holdoff time over, scheduling restart.
Jun 16 18:12:38 hostname systemd[1]: Starting Virtualization daemon...
Jun 16 18:12:38 hostname systemd[1]: Started Virtualization daemon.
Jun 16 18:12:39 hostname dnsmasq[3678]: read /etc/hosts - 2 addresses
Jun 16 18:12:39 hostname dnsmasq[3678]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jun 16 18:12:39 hostname dnsmasq-dhcp[3678]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Jun 16 18:12:39 hostname kvm[12874]: 1 guest now active
Jun 16 18:12:39 hostname kvm[12876]: 0 guests now active
Jun 16 18:12:39 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-disk-3.2.0-9.el7.x86_64
Jun 16 18:12:39 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-gluster-3.2.0-9.el7.x86_64
Jun 16 18:12:41 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-logical-3.2.0-9.el7.x86_64
Jun 16 18:12:41 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-iscsi-3.2.0-9.el7.x86_64
Jun 16 18:12:42 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-scsi-3.2.0-9.el7.x86_64
Jun 16 18:12:42 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-3.2.0-9.el7.x86_64

According to the ccpp generated after the crash, the binary belongs to libvirt-daemon-3.2.0-6.el7

I'm suspecting that the libvirt daemon was restarted during upgrade and ten tried to load the lxc driver module from the new version, which crashed, since we changed the internal API between the modules. I suspect that it's not supposed to restart until the update finishes, but I'm not a spec file master, so I don't know where the problem is.

Comment 3 Jiri Denemark 2017-09-07 12:30:54 UTC

Well, the daemon gets restarted once the libvirt subpackage which contains it (libvirt-daemon) is updated. But since individual hypervisor drivers are optional, libvirt-daemon does not depend on them and it can be updated (and restarted) before drivers are updated.

I'll try to investigate if there's a way we could postpone restarting the daemon until all libvirt packages are updated.

Comment 5 Jiri Denemark 2017-11-01 12:27:19 UTC

Fixed upstream by

commit 1bf893406637e852daeaafec6617d3ee3716de25
Refs: v3.9.0-rc2-1-g1bf893406
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Oct 23 13:06:44 2017 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Tue Oct 31 16:00:50 2017 +0100

    spec: Restart libvirtd in posttrans

    When upgrading libvirt packages, there's no strict ordering for the
    installation or removal of the individual libvirt sub packages. Thus
    libvirt-daemon may be upgraded (and its %postun scriptlet) started
    before all sub packages with driver libraries are upgraded. When
    libvirt-daemon's %postun scriptlet restarts the daemon old drivers may
    still be laying around and the daemon may crash when it tries to use
    them.

    Let's restart the daemon in %posttrans to make sure libvirtd is
    restarted only after all sub packages are at the same version.

    https://bugzilla.redhat.com/show_bug.cgi?id=1464300

    Signed-off-by: Jiri Denemark <jdenemar>

Comment 7 yafu 2018-01-10 10:15:03 UTC

Hi,Jiri,

Could you help to give some advise about how to verify this bug please?
Thanks in advance.

Comment 8 Jiri Denemark 2018-01-19 16:19:29 UTC

Whatch the logs while updating libvirt. You should see when individual sub packages were updated and when the daemon was restarted. Without the fix, you should see the daemon to be restarted several times and even before all sub packages are updated. The fixed package should only restart libvirtd once all packages are updated.

Beware, you need to update twice to see the fix. You should still see the old behavior the first time you update from an unfixed version of libvirt, because it's the libvirt which is being uninstalled which used to restart the daemon. Everything should be OK once you upgrade from a fixed version of libvirt, e.g., from libvirt-3.9.0-1.el7 to libvirt-3.9.0-8.el7.

Comment 9 yafu 2018-01-22 02:52:34 UTC

(In reply to Jiri Denemark from comment #8)
> Whatch the logs while updating libvirt. You should see when individual sub
> packages were updated and when the daemon was restarted. Without the fix,
> you should see the daemon to be restarted several times and even before all
> sub packages are updated. The fixed package should only restart libvirtd
> once all packages are updated.
> 
> Beware, you need to update twice to see the fix. You should still see the
> old behavior the first time you update from an unfixed version of libvirt,
> because it's the libvirt which is being uninstalled which used to restart
> the daemon. Everything should be OK once you upgrade from a fixed version of
> libvirt, e.g., from libvirt-3.9.0-1.el7 to libvirt-3.9.0-8.el7.

Got it. Thanks for your professional and detailed explanation. It helps me a lot.

Comment 10 yafu 2018-01-30 10:13:20 UTC

Reproduced with update libvirt-3.2.0-14.el7.x86_64 to libvirt-3.2.0-14.el7_4.9.x86_64.

Test steps:
1.#yum -y update libvirt
2.Check the yum log after update completed, could see libvirtd service try to start during updating sub pkgs:
Jan 30 01:12:58 dell-per730-44 yum[28931]: Updated: libvirt-libs-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:58 dell-per730-44 yum[28931]: Updated: libvirt-daemon-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:58 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-core-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:58 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-network-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:58 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-nwfilter-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:58 dell-per730-44 systemd: Stopping Virtualization daemon...
Jan 30 01:12:58 dell-per730-44 systemd: Starting Virtualization daemon...
Jan 30 01:12:58 dell-per730-44 systemd: Started Virtualization daemon.
Jan 30 01:12:58 dell-per730-44 yum[28931]: Updated: libvirt-daemon-config-nwfilter-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-lxc-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-config-network-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 dnsmasq[4458]: read /etc/hosts - 7 addresses
Jan 30 01:12:59 dell-per730-44 dnsmasq[4458]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Jan 30 01:12:59 dell-per730-44 dnsmasq-dhcp[4458]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-qemu-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-disk-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-gluster-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-scsi-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-rbd-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-logical-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-mpath-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-nodedev-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-interface-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-secret-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-client-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-3.2.0-14.el7_4.9.x86_64
Jan 30 01:12:59 dell-per730-44 systemd: Reloading.

Verified pass with update libvirt-3.9.0-8.el7.x86_64 to libvirt-3.9.0-9.el7.x86_64.
Test steps:
1.#yum update libvirt
2.Check yum log after yum completed, could see libvirtd service only restart after all pkgs are updated:
Jan 30 16:43:05 dhcp-66-71-72 yum[15160]: Updated: libvirt-libs-3.9.0-9.el7.x86_64
Jan 30 16:43:06 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-3.9.0-9.el7.x86_64
Jan 30 16:43:06 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-core-3.9.0-9.el7.x86_64
Jan 30 16:43:06 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-network-3.9.0-9.el7.x86_64
Jan 30 16:43:07 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-nwfilter-3.9.0-9.el7.x86_64
Jan 30 16:43:07 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-config-nwfilter-3.9.0-9.el7.x86_64
Jan 30 16:43:08 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-qemu-3.9.0-9.el7.x86_64
Jan 30 16:43:08 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-lxc-3.9.0-9.el7.x86_64
Jan 30 16:43:09 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-config-network-3.9.0-9.el7.x86_64
Jan 30 16:43:09 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-iscsi-3.9.0-9.el7.x86_64
Jan 30 16:43:09 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-rbd-3.9.0-9.el7.x86_64
Jan 30 16:43:10 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-logical-3.9.0-9.el7.x86_64
Jan 30 16:43:10 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-disk-3.9.0-9.el7.x86_64
Jan 30 16:43:10 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-gluster-3.9.0-9.el7.x86_64
Jan 30 16:43:11 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-mpath-3.9.0-9.el7.x86_64
Jan 30 16:43:11 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-scsi-3.9.0-9.el7.x86_64
Jan 30 16:43:11 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-3.9.0-9.el7.x86_64
Jan 30 16:43:12 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-secret-3.9.0-9.el7.x86_64
Jan 30 16:43:12 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-interface-3.9.0-9.el7.x86_64
Jan 30 16:43:12 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-nodedev-3.9.0-9.el7.x86_64
Jan 30 16:43:13 dhcp-66-71-72 yum[15160]: Updated: libvirt-client-3.9.0-9.el7.x86_64
Jan 30 16:43:13 dhcp-66-71-72 yum[15160]: Updated: libvirt-3.9.0-9.el7.x86_64
Jan 30 16:43:20 dhcp-66-71-72 systemd: Reloading.
Jan 30 16:43:21 dhcp-66-71-72 systemd: Unit virtlockd.service cannot be reloaded because it is inactive.
Jan 30 16:43:21 dhcp-66-71-72 systemd: Unit virtlogd.service cannot be reloaded because it is inactive.
Jan 30 16:43:21 dhcp-66-71-72 systemd: Stopping Virtualization daemon...
Jan 30 16:43:21 dhcp-66-71-72 systemd: Starting Virtualization daemon...
Jan 30 16:43:21 dhcp-66-71-72 systemd: Started Virtualization daemon.

Comment 14 errata-xmlrpc 2018-04-10 10:50:46 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0704

Note You need to log in before you can comment on or make changes to this bug.