Bug 1164773
Summary: | libvirt-1.2.10-2.fc22 fails to connect to hypervisor; error: undefined symbol: virStorageFileCreate | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Kashyap Chamarthy <kchamart> | ||||||
Component: | libvirt | Assignee: | Libvirt Maintainers <libvirt-maint> | ||||||
Status: | CLOSED NOTABUG | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
Severity: | unspecified | Docs Contact: | |||||||
Priority: | unspecified | ||||||||
Version: | rawhide | CC: | ads.kuknus, agedosier, berrange, clalancette, crobinso, fradisel, itamar, jforbes, kchamart, laine, libvirt-maint, mprivozn, rjones, veillard, virt-maint | ||||||
Target Milestone: | --- | ||||||||
Target Release: | --- | ||||||||
Hardware: | Unspecified | ||||||||
OS: | Unspecified | ||||||||
Whiteboard: | |||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||
Doc Text: | Story Points: | --- | |||||||
Clone Of: | Environment: | ||||||||
Last Closed: | 2014-11-20 15:27:01 UTC | Type: | Bug | ||||||
Regression: | --- | Mount Type: | --- | ||||||
Documentation: | --- | CRM: | |||||||
Verified Versions: | Category: | --- | |||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||
Embargoed: | |||||||||
Attachments: |
|
Description
Kashyap Chamarthy
2014-11-17 12:32:15 UTC
Created attachment 958228 [details]
Contextual libvirtd debug logs
This log was captured was after enabling debug
logs in /etc/libvirt/libvirtd.conf:
log_level = 1
log_outputs="1:file:/var/tmp/libvirtd.log"
And restarting libvirtd:
systemctl restart libvirtd
Strangely, explicitly installing the libvirt RPM (which was missing earlier) and a few deps it brings along, resovled it: $ yum install libvirt --enablerepo=rawhide -y Installing: libvirt x86_64 1.2.10-2.fc22 rawhide 42 k Installing for dependencies: libvirt-daemon-driver-libxl x86_64 1.2.10-2.fc22 rawhide 147 k libvirt-daemon-driver-lxc x86_64 1.2.10-2.fc22 rawhide 637 k libvirt-daemon-driver-uml x86_64 1.2.10-2.fc22 rawhide 94 k libvirt-daemon-driver-vbox x86_64 1.2.10-2.fc22 rawhide 317 k libvirt-daemon-driver-xen $ systemctl restart libvirtd $ virsh nodeinfo CPU model: x86_64 CPU(s): 4 CPU frequency: 1726 MHz CPU socket(s): 1 Core(s) per socket: 4 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 16154688 KiB Does removing those packages again bring back the broken state? From a broke state, can you show: rpm -qa | grep libvirt On Fedora 21 I just updated libvirt\* from rawhide, restarted libvirtd, but virsh works fine FWIW Updated: libvirt-client.x86_64 0:1.2.10-2.fc22 libvirt-daemon.x86_64 0:1.2.10-2.fc22 libvirt-daemon-config-network.x86_64 0:1.2.10-2.fc22 libvirt-daemon-config-nwfilter.x86_64 0:1.2.10-2.fc22 libvirt-daemon-driver-interface.x86_64 0:1.2.10-2.fc22 libvirt-daemon-driver-lxc.x86_64 0:1.2.10-2.fc22 libvirt-daemon-driver-network.x86_64 0:1.2.10-2.fc22 libvirt-daemon-driver-nodedev.x86_64 0:1.2.10-2.fc22 libvirt-daemon-driver-nwfilter.x86_64 0:1.2.10-2.fc22 libvirt-daemon-driver-qemu.x86_64 0:1.2.10-2.fc22 libvirt-daemon-driver-secret.x86_64 0:1.2.10-2.fc22 libvirt-daemon-driver-storage.x86_64 0:1.2.10-2.fc22 libvirt-daemon-kvm.x86_64 0:1.2.10-2.fc22 libvirt-debuginfo.x86_64 0:1.2.10-2.fc22 libvirt-designer-libs.x86_64 0:0.0.1-10.fc22 libvirt-devel.x86_64 0:1.2.10-2.fc22 libvirt-docs.x86_64 0:1.2.10-2.fc22 libvirt-gconfig.x86_64 0:0.1.9-1.fc22 libvirt-glib.x86_64 0:0.1.9-1.fc22 libvirt-gobject.x86_64 0:0.1.9-1.fc22 libvirt-python.x86_64 0:1.2.10-1.fc22 I've seen 'error: failed to connect to the hypervisor' as well. It's infuriatingly hard to reproduce however, which is why I didn't file a bug about it. Also I suspect it has several underlying causes, and the undefined symbol in this bug is just one of them. Rich, are you running rawhide host? Also, does it reproduce with 1.2.10-1 ? The patches added in the -2 update are pretty minor and don't use the StorageCreateFile API. Maybe this is rawhide brokenness (In reply to Cole Robinson from comment #6) > Rich, are you running rawhide host? > > Also, does it reproduce with 1.2.10-1 ? The patches added in the -2 update > are pretty minor and don't use the StorageCreateFile API. Maybe this is > rawhide brokenness As I recall it has happened with recent upgrades to F21 and Rawhide. In neither case was it easy to reproduce, and both went away after rebooting. (In reply to Cole Robinson from comment #3) > Does removing those packages again bring back the broken state? Yes, I can. (Sorry, I forgot to note this in my initial description.) $ yum remove libvirt -y Dependencies resolved. [. . .] Removing: libvirt x86_64 1.2.10-2.fc22 @System 0 libvirt-daemon-driver-lxc x86_64 1.2.10-2.fc22 @System 1.6 M libvirt-daemon-driver-uml x86_64 1.2.10-2.fc22 @System 130 k libvirt-daemon-driver-vbox x86_64 1.2.10-2.fc22 @System 1.7 M libvirt-daemon-driver-xen x86_64 1.2.10-2.fc22 @System 234 k Transaction Summary ====================================================================================================================================================== [. . .] Removed: libvirt.x86_64 1.2.10-2.fc22 libvirt-daemon-driver-lxc.x86_64 1.2.10-2.fc22 libvirt-daemon-driver-uml.x86_64 1.2.10-2.fc22 libvirt-daemon-driver-vbox.x86_64 1.2.10-2.fc22 libvirt-daemon-driver-xen.x86_64 1.2.10-2.fc22 $ systemctl restart libvirtd $ systemctl status libvirtd | grep Active Active: active (running) since Tue 2014-11-18 02:58:05 EST; 2min 1s ago $ virsh nodeinfo error: failed to connect to the hypervisor error: no valid connection error: no connection driver available for <null> error: Failed to reconnect to the hypervisor > From a broke state, can you show: rpm -qa | grep libvirt There we go: ----- $ rpm -qa | grep libvirt libvirt-daemon-driver-storage-1.2.10-2.fc22.x86_64 libvirt-docs-1.2.10-2.fc22.x86_64 libvirt-daemon-qemu-1.2.10-2.fc22.x86_64 libvirt-devel-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-interface-1.2.10-2.fc22.x86_64 libvirt-daemon-config-nwfilter-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-libxl-1.2.10-2.fc22.x86_64 libvirt-client-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-nodedev-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-nwfilter-1.2.10-2.fc22.x86_64 libvirt-daemon-kvm-1.2.10-2.fc22.x86_64 libvirt-python-1.2.10-1.fc22.x86_64 libvirt-daemon-config-network-1.2.10-2.fc22.x86_64 libvirt-daemon-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-qemu-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-secret-1.2.10-2.fc22.x86_64 libvirt-daemon-driver-network-1.2.10-2.fc22.x86_64 ----- Created attachment 958464 [details]
gdb session when libvirt fails to connect to hypervisor
That's exactly how I tried:
$ ps -C libvirtd
PID TTY TIME CMD
1360 ? 00:00:00 libvirtd
$ gdb libvirtd 1360
(gdb)
[. . .]
(gdb) set logging on
Copying output to gdb.txt.
(gdb) cont
Continuing.
Perform the test:
$ virsh nodeinfo
error: failed to connect to the hypervisor
error: no valid connection
error: no connection driver available for <null>
error: Failed to reconnect to the hypervisor
Then on the `gdb` instance:
(gdb) ^C
(gdb) thread apply all bt
(gdb) quit
Output in gdb.txt.
Another point I should note here is, one oddity with this machine's default libvirt URI somehow ended up being LXC: $ virsh uri lxc:/// It wasn't set in any of the libvirt config files either: $ cat /etc/libvirt/libvirtd.conf /etc/libvirt/libvirt.conf \ | grep -i uri | grep -v ^# | grep -v ^$ $ echo $? 1 And, as noted in this bug[*], epxlicitly setting export LIBVIRT_DEFAULT_URI=qemu:///system didn't help either: $ virsh uri error: failed to connect to the hypervisor error: no valid connection error: no connection driver available for qemu:///system error: Failed to reconnect to the hypervisor (However, an you can notice from comment#8, libvirt-daemon-driver-qemu, libvirt-daemon-qemu RPMs are installed on the host.) [*] https://bugzilla.redhat.com/show_bug.cgi?id=1165068 -- libvirt-1.2.10-2.fc22: fails to define KVM guest XML - internal error: unexpected domain type kvm, expecting lxc Okay, we found (thanks to DanPB for the hint to take a look at`journalctl` libvirt logs ) the root cause : device-mapper RPM version should be this: device-mapper-1.02.90-1.fc21.x86_64 (instead of: device-mapper-1.02.88-2.fc21.x86_64) From `journalctl`: $ journalctl -u libvirtd --since=yesterday -p err [. . .] failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so: symbol dm_task_get_info_with_deferred_remove, version Base not defined in file libdevmapper.so.1.02 with link time reference . . [. . .] failed to load module /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so: undefined symbol: virStorageFileCreate Updating to device-mapper-1.02.90-1.fc21.x86_64 solved the issue: $ virsh uri qemu:///system $ virsh nodeinfo CPU model: x86_64 CPU(s): 4 CPU frequency: 1568 MHz CPU socket(s): 1 Core(s) per socket: 4 Thread(s) per core: 1 NUMA cell(s): 1 Memory size: 16154688 KiB (In reply to Kashyap Chamarthy from comment #11) > Okay, we found (thanks to DanPB for the hint to take a look at`journalctl` > libvirt logs ) the root cause : device-mapper RPM version should be this: > device-mapper-1.02.90-1.fc21.x86_64 (instead of: > device-mapper-1.02.88-2.fc21.x86_64) > > From `journalctl`: > > $ journalctl -u libvirtd --since=yesterday -p err > [. . .] failed to load module > /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so > /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so: symbol > dm_task_get_info_with_deferred_remove, version Base not defined in file > libdevmapper.so.1.02 with link time reference So libdevmapper.so (from device-mapper-libs) hasn't provided the symbol. Hence, storage driver has failed to load. > . > . > [. . .] failed to load module > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so: undefined > symbol: virStorageFileCreate > And subsequently did qemu driver, which needs some symbols (e.g. virStorageFileCreate) from storage driver. Therefore, the default URI is not qemu:///system but lxc:/// > > Updating to device-mapper-1.02.90-1.fc21.x86_64 solved the issue: Exactly! this is a device-mapper-libs bug where they just didn't export some symbol(s) for a several versions. I think this can be closed as NOTABUG, can't it? (In reply to Michal Privoznik from comment #12) > (In reply to Kashyap Chamarthy from comment #11) > > Okay, we found (thanks to DanPB for the hint to take a look at`journalctl` > > libvirt logs ) the root cause : device-mapper RPM version should be this: > > device-mapper-1.02.90-1.fc21.x86_64 (instead of: > > device-mapper-1.02.88-2.fc21.x86_64) > > > > From `journalctl`: > > > > $ journalctl -u libvirtd --since=yesterday -p err > > [. . .] failed to load module > > /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so > > /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so: symbol > > dm_task_get_info_with_deferred_remove, version Base not defined in file > > libdevmapper.so.1.02 with link time reference > > So libdevmapper.so (from device-mapper-libs) hasn't provided the symbol. > Hence, storage driver has failed to load. > > > . > > . > > [. . .] failed to load module > > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so > > /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so: undefined > > symbol: virStorageFileCreate > > > > And subsequently did qemu driver, which needs some symbols (e.g. > virStorageFileCreate) from storage driver. Therefore, the default URI is not > qemu:///system but lxc:/// > > > > > Updating to device-mapper-1.02.90-1.fc21.x86_64 solved the issue: > > Exactly! this is a device-mapper-libs bug where they just didn't export some > symbol(s) for a several versions. Yes, and the Fixed-In-Version device-mapper-libs-1.02.90-1.fc21.x86_64.rpm (provided as part of lvm2-2.02.111-1.fc21) is in Fedora-21 stable: $ bodhi -L lvm2 | grep f21 [. . .] f21-updates-testing lvm2-2.02.111-1.fc21 f21 lvm2-2.02.111-1.fc21 f21-updates-candidate lvm2-2.02.111-1.fc21 > I think this can be closed as NOTABUG, > can't it? Yes, this is not a libvirt bug, so, closing the bug per rationale noted above. FWIW I just hit this error on a fresh F22 @Core install. The cause was because I didn't have 'qemu' & 'libvirt-daemon-kvm' installed. The error was in no way obvious. |