Bug 1464300
| Summary: | libvirt crashed with SIGSEGV after applying update | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Germano Veit Michel <gveitmic> |
| Component: | libvirt | Assignee: | Jiri Denemark <jdenemar> |
| Status: | CLOSED ERRATA | QA Contact: | yafu <yafu> |
| Severity: | medium | Docs Contact: | |
| Priority: | low | ||
| Version: | 7.4 | CC: | cww, dyuan, gsun, jdenemar, rbalakri, xuzhang, zpeng |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | libvirt-3.9.0-1.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-04-10 10:50:46 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1420851 | ||
Logs from the sosreport hint that the daemon was restarted while being updated: Jun 16 18:12:33 hostname yum[12141]: Updated: libvirt-daemon-3.2.0-9.el7.x86_64 Jun 16 18:12:34 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-core-3.2.0-9.el7.x86_64 Jun 16 18:12:34 hostname yum[12141]: Updated: libvirt-daemon-driver-network-3.2.0-9.el7.x86_64 Jun 16 18:12:34 hostname yum[12141]: Updated: libvirt-daemon-driver-nwfilter-3.2.0-9.el7.x86_64 Jun 16 18:12:35 hostname yum[12141]: Updated: libvirt-daemon-driver-qemu-3.2.0-9.el7.x86_64 Jun 16 18:12:35 hostname yum[12141]: Updated: libvirt-daemon-driver-interface-3.2.0-9.el7.x86_64 Jun 16 18:12:35 hostname yum[12141]: Updated: libvirt-daemon-driver-nodedev-3.2.0-9.el7.x86_64 Jun 16 18:12:36 hostname yum[12141]: Updated: libvirt-daemon-driver-secret-3.2.0-9.el7.x86_64 Jun 16 18:12:36 hostname systemd[1]: Stopping Virtualization daemon... Jun 16 18:12:36 hostname systemd[1]: Starting Virtualization daemon... Jun 16 18:12:36 hostname systemd[1]: Started Virtualization daemon. Jun 16 18:12:36 hostname yum[12141]: Updated: libvirt-daemon-config-nwfilter-3.2.0-9.el7.x86_64 Jun 16 18:12:36 hostname dnsmasq[3678]: read /etc/hosts - 2 addresses Jun 16 18:12:36 hostname dnsmasq[3678]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses Jun 16 18:12:36 hostname dnsmasq-dhcp[3678]: read /var/lib/libvirt/dnsmasq/default.hostsfile Jun 16 18:12:37 hostname yum[12141]: Updated: libvirt-daemon-driver-lxc-3.2.0-9.el7.x86_64 Jun 16 18:12:37 hostname yum[12141]: Updated: libvirt-daemon-config-network-3.2.0-9.el7.x86_64 Jun 16 18:12:38 hostname kvm[12798]: 1 guest now active Jun 16 18:12:38 hostname kvm[12800]: 0 guests now active Jun 16 18:12:38 hostname kernel: libvirtd[12752]: segfault at 128 ip 00007f3c7b0d9f41 sp 00007f3c4c57cc60 error 4 in libvirt.so.0.3002.0[7f3c7afd3000+37a000] Jun 16 18:12:38 hostname abrt-hook-ccpp[12809]: Process 12736 (libvirtd) of user 0 killed by SIGSEGV - dumping core Jun 16 18:12:38 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-rbd-3.2.0-9.el7.x86_64 Jun 16 18:12:38 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-mpath-3.2.0-9.el7.x86_64 Jun 16 18:12:38 hostname abrt-hook-ccpp[12809]: Failed to create core_backtrace: waitpid failed: No child processes Jun 16 18:12:38 hostname systemd[1]: libvirtd.service: main process exited, code=killed, status=11/SEGV Jun 16 18:12:38 hostname systemd[1]: Unit libvirtd.service entered failed state. Jun 16 18:12:38 hostname systemd[1]: libvirtd.service failed. Jun 16 18:12:38 hostname systemd[1]: libvirtd.service holdoff time over, scheduling restart. Jun 16 18:12:38 hostname systemd[1]: Starting Virtualization daemon... Jun 16 18:12:38 hostname systemd[1]: Started Virtualization daemon. Jun 16 18:12:39 hostname dnsmasq[3678]: read /etc/hosts - 2 addresses Jun 16 18:12:39 hostname dnsmasq[3678]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses Jun 16 18:12:39 hostname dnsmasq-dhcp[3678]: read /var/lib/libvirt/dnsmasq/default.hostsfile Jun 16 18:12:39 hostname kvm[12874]: 1 guest now active Jun 16 18:12:39 hostname kvm[12876]: 0 guests now active Jun 16 18:12:39 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-disk-3.2.0-9.el7.x86_64 Jun 16 18:12:39 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-gluster-3.2.0-9.el7.x86_64 Jun 16 18:12:41 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-logical-3.2.0-9.el7.x86_64 Jun 16 18:12:41 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-iscsi-3.2.0-9.el7.x86_64 Jun 16 18:12:42 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-scsi-3.2.0-9.el7.x86_64 Jun 16 18:12:42 hostname yum[12141]: Updated: libvirt-daemon-driver-storage-3.2.0-9.el7.x86_64 According to the ccpp generated after the crash, the binary belongs to libvirt-daemon-3.2.0-6.el7 I'm suspecting that the libvirt daemon was restarted during upgrade and ten tried to load the lxc driver module from the new version, which crashed, since we changed the internal API between the modules. I suspect that it's not supposed to restart until the update finishes, but I'm not a spec file master, so I don't know where the problem is. Well, the daemon gets restarted once the libvirt subpackage which contains it (libvirt-daemon) is updated. But since individual hypervisor drivers are optional, libvirt-daemon does not depend on them and it can be updated (and restarted) before drivers are updated. I'll try to investigate if there's a way we could postpone restarting the daemon until all libvirt packages are updated. Fixed upstream by
commit 1bf893406637e852daeaafec6617d3ee3716de25
Refs: v3.9.0-rc2-1-g1bf893406
Author: Jiri Denemark <jdenemar>
AuthorDate: Mon Oct 23 13:06:44 2017 +0200
Commit: Jiri Denemark <jdenemar>
CommitDate: Tue Oct 31 16:00:50 2017 +0100
spec: Restart libvirtd in posttrans
When upgrading libvirt packages, there's no strict ordering for the
installation or removal of the individual libvirt sub packages. Thus
libvirt-daemon may be upgraded (and its %postun scriptlet) started
before all sub packages with driver libraries are upgraded. When
libvirt-daemon's %postun scriptlet restarts the daemon old drivers may
still be laying around and the daemon may crash when it tries to use
them.
Let's restart the daemon in %posttrans to make sure libvirtd is
restarted only after all sub packages are at the same version.
https://bugzilla.redhat.com/show_bug.cgi?id=1464300
Signed-off-by: Jiri Denemark <jdenemar>
Hi,Jiri, Could you help to give some advise about how to verify this bug please? Thanks in advance. Whatch the logs while updating libvirt. You should see when individual sub packages were updated and when the daemon was restarted. Without the fix, you should see the daemon to be restarted several times and even before all sub packages are updated. The fixed package should only restart libvirtd once all packages are updated. Beware, you need to update twice to see the fix. You should still see the old behavior the first time you update from an unfixed version of libvirt, because it's the libvirt which is being uninstalled which used to restart the daemon. Everything should be OK once you upgrade from a fixed version of libvirt, e.g., from libvirt-3.9.0-1.el7 to libvirt-3.9.0-8.el7. (In reply to Jiri Denemark from comment #8) > Whatch the logs while updating libvirt. You should see when individual sub > packages were updated and when the daemon was restarted. Without the fix, > you should see the daemon to be restarted several times and even before all > sub packages are updated. The fixed package should only restart libvirtd > once all packages are updated. > > Beware, you need to update twice to see the fix. You should still see the > old behavior the first time you update from an unfixed version of libvirt, > because it's the libvirt which is being uninstalled which used to restart > the daemon. Everything should be OK once you upgrade from a fixed version of > libvirt, e.g., from libvirt-3.9.0-1.el7 to libvirt-3.9.0-8.el7. Got it. Thanks for your professional and detailed explanation. It helps me a lot. Reproduced with update libvirt-3.2.0-14.el7.x86_64 to libvirt-3.2.0-14.el7_4.9.x86_64. Test steps: 1.#yum -y update libvirt 2.Check the yum log after update completed, could see libvirtd service try to start during updating sub pkgs: Jan 30 01:12:58 dell-per730-44 yum[28931]: Updated: libvirt-libs-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:58 dell-per730-44 yum[28931]: Updated: libvirt-daemon-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:58 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-core-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:58 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-network-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:58 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-nwfilter-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:58 dell-per730-44 systemd: Stopping Virtualization daemon... Jan 30 01:12:58 dell-per730-44 systemd: Starting Virtualization daemon... Jan 30 01:12:58 dell-per730-44 systemd: Started Virtualization daemon. Jan 30 01:12:58 dell-per730-44 yum[28931]: Updated: libvirt-daemon-config-nwfilter-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-lxc-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-config-network-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 dnsmasq[4458]: read /etc/hosts - 7 addresses Jan 30 01:12:59 dell-per730-44 dnsmasq[4458]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses Jan 30 01:12:59 dell-per730-44 dnsmasq-dhcp[4458]: read /var/lib/libvirt/dnsmasq/default.hostsfile Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-qemu-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-disk-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-gluster-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-scsi-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-rbd-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-iscsi-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-logical-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-mpath-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-storage-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-nodedev-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-interface-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-daemon-driver-secret-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-client-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 yum[28931]: Updated: libvirt-3.2.0-14.el7_4.9.x86_64 Jan 30 01:12:59 dell-per730-44 systemd: Reloading. Verified pass with update libvirt-3.9.0-8.el7.x86_64 to libvirt-3.9.0-9.el7.x86_64. Test steps: 1.#yum update libvirt 2.Check yum log after yum completed, could see libvirtd service only restart after all pkgs are updated: Jan 30 16:43:05 dhcp-66-71-72 yum[15160]: Updated: libvirt-libs-3.9.0-9.el7.x86_64 Jan 30 16:43:06 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-3.9.0-9.el7.x86_64 Jan 30 16:43:06 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-core-3.9.0-9.el7.x86_64 Jan 30 16:43:06 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-network-3.9.0-9.el7.x86_64 Jan 30 16:43:07 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-nwfilter-3.9.0-9.el7.x86_64 Jan 30 16:43:07 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-config-nwfilter-3.9.0-9.el7.x86_64 Jan 30 16:43:08 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-qemu-3.9.0-9.el7.x86_64 Jan 30 16:43:08 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-lxc-3.9.0-9.el7.x86_64 Jan 30 16:43:09 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-config-network-3.9.0-9.el7.x86_64 Jan 30 16:43:09 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-iscsi-3.9.0-9.el7.x86_64 Jan 30 16:43:09 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-rbd-3.9.0-9.el7.x86_64 Jan 30 16:43:10 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-logical-3.9.0-9.el7.x86_64 Jan 30 16:43:10 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-disk-3.9.0-9.el7.x86_64 Jan 30 16:43:10 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-gluster-3.9.0-9.el7.x86_64 Jan 30 16:43:11 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-mpath-3.9.0-9.el7.x86_64 Jan 30 16:43:11 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-scsi-3.9.0-9.el7.x86_64 Jan 30 16:43:11 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-storage-3.9.0-9.el7.x86_64 Jan 30 16:43:12 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-secret-3.9.0-9.el7.x86_64 Jan 30 16:43:12 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-interface-3.9.0-9.el7.x86_64 Jan 30 16:43:12 dhcp-66-71-72 yum[15160]: Updated: libvirt-daemon-driver-nodedev-3.9.0-9.el7.x86_64 Jan 30 16:43:13 dhcp-66-71-72 yum[15160]: Updated: libvirt-client-3.9.0-9.el7.x86_64 Jan 30 16:43:13 dhcp-66-71-72 yum[15160]: Updated: libvirt-3.9.0-9.el7.x86_64 Jan 30 16:43:20 dhcp-66-71-72 systemd: Reloading. Jan 30 16:43:21 dhcp-66-71-72 systemd: Unit virtlockd.service cannot be reloaded because it is inactive. Jan 30 16:43:21 dhcp-66-71-72 systemd: Unit virtlogd.service cannot be reloaded because it is inactive. Jan 30 16:43:21 dhcp-66-71-72 systemd: Stopping Virtualization daemon... Jan 30 16:43:21 dhcp-66-71-72 systemd: Starting Virtualization daemon... Jan 30 16:43:21 dhcp-66-71-72 systemd: Started Virtualization daemon. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:0704 |
Description of problem: libvirtd failed to restart after yum update: abrt-hook-ccpp[12809]: Process 12736 (libvirtd) of user 0 killed by SIGSEGV - dumping core Version-Release number of selected component (if applicable): libvirt-3.2.0-9.el7.x86_64 How reproducible: 0% Actual results: libvirt crashed Expected results: libvirt run Additional info: Program terminated with signal 11, Segmentation fault. #0 virDomainXMLOptionNew (config=0x7f3c4f2060e0 <virLXCDriverDomainDefParserConfig>, priv=0x7f3c4f206120 <virLXCDriverPrivateDataCallbacks>, xmlns=0x7f3c4f206160 <virLXCDriverDomainXMLNamespace>, abi=0x128, abi@entry=0x0, saveCookie=saveCookie@entry=0x0) at conf/domain_conf.c:1081 1081 xmlopt->abi = *abi; Thread 1 (Thread 0x7f3c4c57d700 (LWP 12752)): #0 virDomainXMLOptionNew (config=0x7f3c4f2060e0 <virLXCDriverDomainDefParserConfig>, priv=0x7f3c4f206120 <virLXCDriverPrivateDataCallbacks>, xmlns=0x7f3c4f206160 <virLXCDriverDomainXMLNamespace>, abi=0x128, abi@entry=0x0, saveCookie=saveCookie@entry=0x0) at conf/domain_conf.c:1081 #1 0x00007f3c4efd32af in lxcDomainXMLConfInit () at lxc/lxc_conf.c:213 #2 0x00007f3c4efe9029 in lxcStateInitialize (privileged=<optimized out>, callback=<optimized out>, opaque=<optimized out>) at lxc/lxc_driver.c:1678 #3 0x00007f3c7b15150f in virStateInitialize (privileged=true, callback=callback@entry=0x557832935970 <daemonInhibitCallback>, opaque=opaque@entry=0x557834037c70) at libvirt.c:770 #4 0x00005578329359cb in daemonRunStateInit (opaque=0x557834037c70) at libvirtd.c:881 #5 0x00007f3c7b0b8352 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #6 0x00007f3c784cfe25 in start_thread () from /lib64/libpthread.so.0 #7 0x00007f3c781fd34d in clone () from /lib64/libc.so.6