Bug 1830380

Summary: pipewire jack/pulseaudio replacements have issues with library file layout and automatic library provides
Product: [Fedora] Fedora Reporter: Adam Williamson <awilliam>
Component: pipewireAssignee: Wim Taymans <wtaymans>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 33CC: brunovern.a, goeran, jmaibaum, mcatanza, robatino, wtaymans
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard: openqa
Fixed In Version: pipewire-0.3.19-2.fc33 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-01-12 01:32:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Adam Williamson 2020-05-01 19:25:37 UTC
pipewire 0.34 seems to break GNOME, both on Rawhide and in Fedora 32 updates-testing (where it has been submitted). Starting a GNOME session shows the "oops! Something has gone wrong" screen. The system logs show multiple errors like this:

gnome-shell[1607]: Failed to load shared library 'libgvc.so' referenced by the typelib: libpulse-mainloop-glib.so.0: cannot open shared object file: No such file or directory"

affecting gnome-shell, and errors like this:

gsd-media-keys[2199]: /usr/libexec/gsd-media-keys: error while loading shared libraries: libpulse-mainloop-glib.so.0: cannot open shared object file: No such file or directory

affecting gsd-media-keys. This seems to be the root of the problem.

/usr/lib64/pipewire-0.3/pulse/libpulse-mainloop-glib.so.0 is included in pipewire-libpulse in 0.3.4 builds, and pipewire-pulseaudio now conflicts with and provides (but does not obsolete) pulseaudio-libs and pulseaudio-libs-glib2. It seems the live image winds up with the following pipewire and pulseaudio packages installed:

pipewire0.2-libs-0.2.7-2.fc32
pipewire-0.3.4-1.fc33
pipewire-libpulse-0.3.4-1.fc33
pipewire-libs-0.3.4-1.fc33
pulseaudio-13.99.1-3.fc33
pulseaudio-libs-13.99.1-3.fc33
pulseaudio-module-bluetooth-13.99.1-3.fc33
pulseaudio-module-x11-13.99.1-3.fc33
pulseaudio-utils-13.99.1-3.fc33

Notes:

1. We don't have pulseaudio-libs-glib2. This is where the "original" of libpulse-mainloop-glib.so.0 is from, and in that package it's in /usr/lib64, not a subdirectory of it
2. We don't have pipewire-pulseaudio, but we do have pipewire-libpulse
3. pipewire-libpulse claims to "provide" libpulse-mainloop-glib.so.0()(64bit), even though it only ships the library in a subdirectory of /usr/lib64 which is not on the linker path AFAICT (it doesn't include a /etc/ld.so.conf.d file). I guess it is "beating" pulseaudio-libs-glib2 to this Provide, but then not actually making the library available
4. It seems odd that gnome-shell thinks pipewire-libpulse's version of the library references libgvc.so, because neither 'ldd' nor 'strings' thinks so, there is no trace of 'gvc' in the pipewire build logs - https://kojipkgs.fedoraproject.org//packages/pipewire/0.3.4/1.fc33/data/logs/x86_64/build.log - and the only libgvc.so I can find is in graphviz-devel, which doesn't seem like something a sound library would link against, so I'm not sure what's going on there

So there are various little weird alleyways here, but the broad picture is "pipewire is currently about half replacing pulseaudio and claiming to provide a library it is not actually shipping in the linker path, and this is breaking stuff badly".

Comment 1 Adam Williamson 2020-05-01 19:27:14 UTC
Oh, forgot to note, I think this may be breaking KDE too as the KDE live isn't reaching a desktop either, but I haven't yet booted that locally to confirm it's the same problem.

Proposing as an F33 Beta blocker, obviously failing to load GNOME is a blocker, criterion "A system installed with a release-blocking desktop must boot to a log in screen where it is possible to log in to a working desktop using a user account created during installation or a 'first boot' utility." - https://fedoraproject.org/wiki/Basic_Release_Criteria#Expected_installed_system_boot_behavior

Comment 2 Adam Williamson 2020-05-01 19:40:01 UTC
OK, so looking at the package build, it seems like the idea is that the underlying packages pipewire-libpulse and pipewire-libjack ship the library files in the subdirectory, and then there are 'pipewire-jack-audio-connection-kit' and 'pipewire-pulseaudio' packages that are intended as replacements for jack-audio-connection-kit and pulseaudio-libs{-glib2} respectively, and which ship *symlinks* to those files in %{_libdir}. But because the underlying library packages wind up with Provides: for the libraries (even though they aren't shipping them in the library path), this system is breaking down a bit, dependencies are pulling in the underlying packages but not the ones with the symlinks in %{_libdir}.

I'm not sure it's actually allowed to ship libraries as symlinks like this, or if it works, but if it is, the package at least needs to munge the provides so the pipewire-jack-audio-connection-kit and pipewire-pulseaudio subpackages have the library provides rather than pipewire-libjack and pipewire-libpulse.

Comment 3 Adam Williamson 2020-05-01 20:15:38 UTC
I've done a -2 build which suppresses the automatic library provides/requires for everything under %{_libdir}/pipewire-%{apiversion} . That should get Rawhide working again at least as it'll mean no pipewire package provides the library dependencies any more, and Rawhide installs should get the original pulseaudio-libs-glib2 package instead and things should work.

I was not able to get the automatic library provides to appear in the pipewire-jack-audio-connection-kit and pipewire-pulseaudio packages instead, yet. Note they do not appear in these packages even *without* the exclusions I added, so I didn't make anything worse. I'm assuming the dependency generator either ignores symlinks, or follows them and applies the Provides to the package containing the real file; either way, it's not adding the Provides to the packages that have the symlinks in %{_libdir}. I'm not sure how to fix this, again, I'm not sure if this layout is actually allowed/workable/a good idea.

Comment 4 Wim Taymans 2020-05-04 08:51:03 UTC
Now it somewhat breaks the original idea (libjackserver.so.0()(64bit) and libjacknet.so.0()(64bit)
are no longer provided and so the JACK examples/devel don't want to install anymore)

What I want to achieve is this:

1) pipewire-libpulse

 * installs compatible replacement libraries in separate directory, not in library path
 * doesn't Provides: any of the libpulse.so 
 * has script to put compat libraries in library path in order to run them with new
   libraries.

The idea is to have a complete regular pulseaudio install and have all apps use regular
pulseaudio. Only with the script can you try out the new stuff. In no way are the PipeWire
libraries supposed to interfere with existing PulseAudio or install.

Your fix to the .spec achieves this.


2) pipewire-pulseaudio

 * brings to compat libraries into the library path
 * provides libpulse.so etc
 * all pulseaudio apps run on PipeWire

This package is in no way ready for general use. It should be completely opt-in, disabled by
default and should in no way ever be added to the default install.

The same setup for libjack, except that the JACK replacement will generally work ok.

With the removal of the Provides: the second option to bring the libraries in the search path
with symlinks doesn't work anymore. 

Maybe what can be done is:

- leave the removal of Provides: in the libpulse/libjack package.
- don't use symlinks but put a copy of the same libs in the library path, this should then add
  them back to Provides for the pipewire-pulseaudio and pipewire-jack-audio-connection-kit
- Make sure pipewire-jack-audio-connection-kit and pipewire-pulseaudio are never added to the
  default install, they should only ever be installed manually.

Not sure how to do the last bit or if it even can be done.. 

Are there other ideas?

Comment 5 Adam Williamson 2020-05-04 14:55:00 UTC
"Now it somewhat breaks the original idea (libjackserver.so.0()(64bit) and libjacknet.so.0()(64bit)
are no longer provided and so the JACK examples/devel don't want to install anymore)"

They should never have been provided by the pipewire-libjack package at all, because it doesn't provide them anywhere anything else can rely on finding them. It literally contains the files, but they're in a directory that is not in the loader path. So any external app that is built against those libraries is not going to find them when it tries to run, if only pipewire-libjack is installed.

They should have been provided by pipewire-jack-audio-connection-kit, since that's the package which included the symlinks in %_libdir. But they never were provided by this package, neither before nor after my change.

Your 'copy' idea should work, at the cost of needlessly having two copies of the files, I guess.

I don't have any great ideas off the top of my head for point #3, really. Well, for Pulseaudio at least, we could have the main pulseaudio package explicitly Recommends: or Suggests: its 'regular' library packages, I think dnf uses soft dependencies to break ties for provides like this, so that would help. But for jack since the daemon and libraries are in the same package, we can't really do that...

Comment 6 Wim Taymans 2020-05-04 15:05:38 UTC
(In reply to Adam Williamson from comment #5)
> They should never have been provided by the pipewire-libjack package at all,
> because it doesn't provide them anywhere anything else can rely on finding
> them. It literally contains the files, but they're in a directory that is
> not in the loader path. So any external app that is built against those
> libraries is not going to find them when it tries to run, if only
> pipewire-libjack is installed.

I agree. I used to have these libraries with a different name but that was also
awkward because the library path trick doesn't work unless I add symlinks with
the proper library names again. Maybe that's not a bad idea after all and then
we can remove the __provides_exclude_from lines.

> 
> Your 'copy' idea should work, at the cost of needlessly having two copies of
> the files, I guess.

In fact, it does not make much sense to have both packages installed at the same
time. It's either installed in the private directory with script or in the
library path.

> 
> I don't have any great ideas off the top of my head for point #3, really.
> Well, for Pulseaudio at least, we could have the main pulseaudio package
> explicitly Recommends: or Suggests: its 'regular' library packages, I think
> dnf uses soft dependencies to break ties for provides like this, so that
> would help. But for jack since the daemon and libraries are in the same
> package, we can't really do that...

Splitting jack-audio-connection-kit into a jack-audio-connection-kit-libs can
also be done to resolve this.

Comment 7 Adam Williamson 2020-05-09 01:16:49 UTC
As we've at least made this stop breaking composes for now, adjusting the summary and dropping the release blocker proposal.

Comment 8 Adam Williamson 2020-05-09 01:17:45 UTC
*** Bug 1830447 has been marked as a duplicate of this bug. ***

Comment 9 Ben Cotton 2020-08-11 15:19:31 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle.
Changing version to 33.

Comment 10 Johannes Maibaum 2021-01-02 11:23:38 UTC
As of pipewire-0.3.18, the libjack*.so provides from pipewire-libjack/pipewire-jack-audio-connection-kit are missing.

Currently (testing on Fedora Silverblue 33, where manually creating symlinks is not possible due to read-only /usr), trying to install any tool depending on JACK fails with output similar to:

$ rpm-ostree install qjackctl
Checking out tree aa6dd6b... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora updates-modular updates-testing fedora-modular updates-testing-modular updates-archive
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-08-25T19:10:34Z
rpm-md repo 'updates' (cached); generated: 2021-01-02T00:56:56Z
rpm-md repo 'fedora' (cached); generated: 2020-10-19T23:27:19Z
rpm-md repo 'updates-modular' (cached); generated: 2020-12-30T01:36:23Z
rpm-md repo 'updates-testing' (cached); generated: 2021-01-02T01:41:41Z
rpm-md repo 'fedora-modular' (cached); generated: 2020-10-19T23:04:43Z
rpm-md repo 'updates-testing-modular' (cached); generated: 2020-12-30T01:59:29Z
rpm-md repo 'updates-archive' (cached); generated: 2021-01-02T01:35:05Z
Importing rpm-md... done
Resolving dependencies... done
error: Could not depsolve transaction; 1 problem detected:
 Problem: conflicting requests
  - package qjackctl-0.6.3-2.fc33.x86_64 requires libjack.so.0()(64bit), but none of the providers can be installed
  - package qjackctl-0.6.2-2.fc33.x86_64 requires libjack.so.0()(64bit), but none of the providers can be installed
  - package pipewire-jack-audio-connection-kit-0.3.18-1.fc33.i686 conflicts with jack-audio-connection-kit provided by jack-audio-connection-kit-1.9.14-5.fc33.x86_64
  - package pipewire-jack-audio-connection-kit-0.3.18-1.fc33.x86_64 conflicts with jack-audio-connection-kit provided by jack-audio-connection-kit-1.9.14-5.fc33.x86_64
  - package pipewire-jack-audio-connection-kit-0.3.13-4.fc33.i686 requires pipewire-libjack(x86-32) = 0.3.13-4.fc33, but none of the providers can be installed
  - package pipewire-libjack-0.3.13-4.fc33.i686 requires pipewire-libs(x86-32) = 0.3.13-4.fc33, but none of the providers can be installed
  - package pipewire-jack-audio-connection-kit-0.3.13-4.fc33.x86_64 requires pipewire-libjack(x86-64) = 0.3.13-4.fc33, but none of the providers can be installed
  - package pipewire-jack-audio-connection-kit-0.3.14-1.fc33.x86_64 requires pipewire-libjack(x86-64) = 0.3.14-1.fc33, but none of the providers can be installed
  - package pipewire-jack-audio-connection-kit-0.3.14-2.fc33.x86_64 requires pipewire-libjack(x86-64) = 0.3.14-2.fc33, but none of the providers can be installed
  - package pipewire-jack-audio-connection-kit-0.3.15-2.fc33.x86_64 requires pipewire-libjack(x86-64) = 0.3.15-2.fc33, but none of the providers can be installed
  - pipewire-libs-0.3.13-4.fc33.i686 has inferior architecture
  - package pipewire-libjack-0.3.13-4.fc33.x86_64 requires pipewire-libs(x86-64) = 0.3.13-4.fc33, but none of the providers can be installed
  - package pipewire-libjack-0.3.14-1.fc33.x86_64 requires pipewire-libs(x86-64) = 0.3.14-1.fc33, but none of the providers can be installed
  - package pipewire-libjack-0.3.14-2.fc33.x86_64 requires pipewire-libs(x86-64) = 0.3.14-2.fc33, but none of the providers can be installed
  - package pipewire-libjack-0.3.15-2.fc33.x86_64 requires pipewire-libs(x86-64) = 0.3.15-2.fc33, but none of the providers can be installed
  - cannot install both pipewire-libs-0.3.13-4.fc33.x86_64 and pipewire-libs-0.3.18-1.fc33.x86_64
  - cannot install both pipewire-libs-0.3.14-1.fc33.x86_64 and pipewire-libs-0.3.18-1.fc33.x86_64
  - cannot install both pipewire-libs-0.3.14-2.fc33.x86_64 and pipewire-libs-0.3.18-1.fc33.x86_64
  - cannot install both pipewire-libs-0.3.15-2.fc33.x86_64 and pipewire-libs-0.3.18-1.fc33.x86_64

$ cat /etc/ld.so.conf.d/pipewire-jack-x86_64.conf 
/usr/lib64/pipewire-0.3/jack/

$ sudo ldconfig -p | grep libjack
	libjackserver.so.0 (libc6,x86-64) => /usr/lib64/pipewire-0.3/jack/libjackserver.so.0
	libjackserver.so (libc6,x86-64) => /usr/lib64/pipewire-0.3/jack/libjackserver.so
	libjacknet.so.0 (libc6,x86-64) => /usr/lib64/pipewire-0.3/jack/libjacknet.so.0
	libjacknet.so (libc6,x86-64) => /usr/lib64/pipewire-0.3/jack/libjacknet.so
	libjack.so.0 (libc6,x86-64) => /usr/lib64/pipewire-0.3/jack/libjack.so.0
	libjack.so (libc6,x86-64) => /usr/lib64/pipewire-0.3/jack/libjack.so

On the bright side, after fighting the long dependency chain of pulseaudio (documented here: https://paste.sr.ht/~jmaibaum/8f9cc331078b14681077308e1b56ad34bb068aeb), I can report that the pulseaudio replacement looks, and works great so far! It's just the JACK replacement libraries which don't yet work as expected.

Comment 11 Fedora Update System 2021-01-10 14:15:23 UTC
FEDORA-2021-219c1b512a has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-219c1b512a

Comment 12 Fedora Update System 2021-01-11 02:02:07 UTC
FEDORA-2021-219c1b512a has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-219c1b512a`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-219c1b512a

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 13 Fedora Update System 2021-01-12 01:32:11 UTC
FEDORA-2021-219c1b512a has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.