Bug 1314881

Summary:	libvirtd.service in a continuous restart loop if systemd compat libraries are not present (systemd>v229 and archlinux)
Product:	[Community] Virtualization Tools	Reporter:	Richard Tollerton <rich.tollerton>
Component:	libvirt	Assignee:	Libvirt Maintainers <libvirt-maint>
Status:	CLOSED NEXTRELEASE	QA Contact:
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	unspecified	CC:	agx, berrange, crobinso, dr4ke, dyuan, jtomko, pupykin.s, rbalakri, rich.tollerton, yafu
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-06-07 08:55:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Richard Tollerton 2016-03-04 18:09:58 UTC

libvirtd.service is Type=notify, using sd_notify to report service startup. Presently libvirt links against libsystemd-daemon to call sd_notify, but this library has been renamed to libsystemd in newer versions of systemd, with the older library present in systemd's compat library support.

This support has been removed in systemd git: https://github.com/systemd/systemd/commit/4de282cf9324ab13d17ac334244d0d7cae2df37d . When this change is present on the installed systemd,

- old builds of libvirt.so will fail to load because of a lack of libsystemd-daemon.so
- new builds of libvirt will appear to work, but will have silently disabled sd_notify support. As a result, libvirtd.service will go into a continuous restart loop.

systemd 229 is presently the latest release, but Arch jumped the gun and disabled compat library support in its 229 packages, cf https://bugs.archlinux.org/task/48440 .

Debian fixed this last August but failed to upstream the fix. Boooooo. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779775

Comment 1 Daniel Berrangé 2016-03-04 18:18:16 UTC

We should just ditch use of sd_notify and not link to any systemd library. It is trivial to implementation the notification in libvirt directly - its just a matter of connecting to the UNIX socket path from $NOTIFY_SOCKET env var and sending the string "READY=1".

Comment 2 Richard Tollerton 2016-03-04 18:34:41 UTC

Sure... because systemd has never broken compatibility in the past, right? ;)

Annoying though it is, libvirt probably gains some abstraction benefit out of delegating this work to sd_notify, and if/when they deprecate that API in the future, the issue would manifest itself as a build failure, when otherwise (by hand-stuffing the socket) it would be altogether harder to diagnose.

Comment 3 Cole Robinson 2016-04-10 22:10:10 UTC

CCing the debian maintainer

guido, comment #0 mentions https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779775 as a fix, sounds like arch wants this too so maybe we find a way to get it upstream? (but I might be missing the details)

Comment 4 Guido Günther 2016-04-11 07:32:33 UTC

We're linking against libsystemdaemon since some time in Debian:

    https://anonscm.debian.org/cgit/pkg-libvirt/libvirt.git/tree/debian/patches/debian/libsystemd.patch

I did not get around to clean this up to link against either libsystemd-daemon yet.

Comment 5 Ján Tomko 2016-06-07 08:55:27 UTC

Now pushed upstream:
commit c0bc172383c2c955394589e5808457935ae06f1d
Author:     Daniel P. Berrange <berrange>
AuthorDate: 2016-06-06 15:03:27 +0100
Commit:     Daniel P. Berrange <berrange>
CommitDate: 2016-06-07 09:37:57 +0100

    systemd: directly notify systemd instead of using sd_notify
    
    The sd_notify method is used to tell systemd when libvirtd
    has finished starting up. All it does is send a datagram
    containing the string parameter to systemd on a UNIX socket
    named in the NOTIFY_SOCKET environment variable. Rather than
    pulling in the systemd libraries for this, just code the
    notification directly in libvirt as this is a stable ABI
    from systemd's POV which explicitly allows independant
    implementations:
    
    See "Reimplementable Independently" column in the
    "$NOTIFY_SOCKET Daemon Notifications" row:
    
    https://www.freedesktop.org/wiki/Software/systemd/InterfacePortabilityAndStabilityChart/
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1314881
    
    Signed-off-by: Daniel P. Berrange <berrange>

git describe: v1.3.5-11-gc0bc172