Bug 2177912 - Silverblue ostree compose fails since ffmpeg 6.0 added a dep on oneVPL, which installs files to /usr/etc
Summary: Silverblue ostree compose fails since ffmpeg 6.0 added a dep on oneVPL, which...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: oneVPL
Version: rawhide
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Simone Caronni
QA Contact:
URL:
Whiteboard: openqa
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-03-13 21:58 UTC by Adam Williamson
Modified: 2023-03-15 00:18 UTC (History)
14 users (show)

Fixed In Version: oneVPL-2023.1.3-1.fc38
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-03-15 00:18:00 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker FC-783 0 None None None 2023-03-13 21:59:11 UTC

Description Adam Williamson 2023-03-13 21:58:07 UTC
the openQA test that builds a Silverblue ostree then embeds it in an installer image and tests that has been failing for the last few hours. The first update it failed on is https://bodhi.fedoraproject.org/updates/FEDORA-2023-8e38ae984c , gstreamer1-plugins-base-1.22.1-1.fc39 .

On failed attempts, rpm-ostree fails here:

[2023-03-13 13:28:11] Installing packages: 100%
[2023-03-13 13:28:18] [0m[31merror: [0mDirectory not empty (os error 39)

on the last successful attempt, that part looks like this:

[2023-03-13 12:29:19] Installing packages: 100%
[2023-03-13 12:29:22] Updating usr/lib/os-release

so the failure is occurring somewhere between the end of package install - specifically, right after the "Installing packages" line in `install_packages()` in rpmostree-compose-builtin-tree.cxx - and early post-processing (specifically the line `    println!("Updating {}", path);` in `compose_postprocess_mutate_os_release()` in rust/src/composepost.rs).

It seems surprising that the gstreamer update would be the cause here but it (or ffmpeg) may be. The base packages on the test system are the same between the "last good" and "first bad" tests. I can't see every dependency of rpm-ostree that got installed in the "last good" case, but at least rpm-ostree itself is the same version between those two cases, and I can't see anything obviously related that was updated around the same time as gstreamer1-plugins-base. The only packages that differ in the set of packages that get installed into the ostree are these:

[adamw@xps13a tmp]$ diff -u0 /tmp/goodpkgs /tmp/badpkgs
--- /tmp/goodpkgs	2023-03-13 14:39:56.127159388 -0700
+++ /tmp/badpkgs	2023-03-13 14:39:50.817219037 -0700
@@ -382,2 +382,2 @@
-gstreamer1-1.22.1-1.fc39.x86_64 (advisory)
-gstreamer1-plugin-libav-1.22.0-1.fc38.x86_64 (fedora-rawhide)
+gstreamer1-1.22.1-1.fc39.x86_64 (koji-rawhide)
+gstreamer1-plugin-libav-1.22.0-2.fc39.x86_64 (koji-rawhide)
@@ -385 +385 @@
-gstreamer1-plugins-base-1.22.0-1.fc38.x86_64 (fedora-rawhide)
+gstreamer1-plugins-base-1.22.1-1.fc39.x86_64 (advisory)
@@ -565,3 +565,3 @@
-libavcodec-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
-libavfilter-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
-libavformat-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
+libavcodec-free-6.0-1.fc39.x86_64 (koji-rawhide)
+libavfilter-free-6.0-1.fc39.x86_64 (koji-rawhide)
+libavformat-free-6.0-1.fc39.x86_64 (koji-rawhide)
@@ -569 +569 @@
-libavutil-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
+libavutil-free-6.0-1.fc39.x86_64 (koji-rawhide)
@@ -595 +595 @@
-libchromaprint-1.5.1-7.fc38.x86_64 (fedora-rawhide)
+libchromaprint-1.5.1-8.fc39.x86_64 (koji-rawhide)
@@ -727 +727 @@
-libpostproc-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
+libpostproc-free-6.0-1.fc39.x86_64 (koji-rawhide)
@@ -777,2 +777,2 @@
-libswresample-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
-libswscale-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
+libswresample-free-6.0-1.fc39.x86_64 (koji-rawhide)
+libswscale-free-6.0-1.fc39.x86_64 (koji-rawhide)
@@ -933,0 +934,2 @@
+oneVPL-2022.2.2-1.fc38.x86_64 (fedora-rawhide)
+oneVPL-intel-gpu-22.5.3-1.fc38.x86_64 (fedora-rawhide)

those are all the gstreamer package itself or deps of it, or subpackages of ffmpeg - ffmpeg was updated from 5.1.2 to 6.0 around the same time.

Otherwise this could be caused by some dep of rpm-ostree that I can't see in the "last good" test, or by some change to workstation-ostree-config. Two PRs were merged there recently, but they seem to be a few hours too old to be the culprit. The last-known-good test ran "git clone https://pagure.io/workstation-ostree-config.git" at [2023-03-13T16:18:49.674286Z] , and those commits seem to have come an hour or two before that, so it should already have got the latest git state with the changes.

I suspect this will break Silverblue compose with the next official compose, as the openQA test is designed to be as similar to the official compose process as possible.

Comment 1 Adam Williamson 2023-03-13 22:04:31 UTC
It occurs to me we can use F38 to narrow this down a bit. Neither gstreamer1-plugins-base nor ffmpeg has an official update for F38 yet, so I'm doing scratch builds; I'll trigger openQA tests on both and see where this test fails.

Comment 2 Adam Williamson 2023-03-13 22:49:45 UTC
OK, it seems like it's ffmpeg or one of its deps that causes this. I ran the ostree compose test on the ffmpeg update for F38 and on a scratch build of the gstreamer1-plugins-base update too. The ffmpeg test failed the same way:

https://openqa.stg.fedoraproject.org/tests/2659828#step/_ostree_build/41

but the gstreamer1-plugins-base test has passed the failure point and is still going:

https://openqa.stg.fedoraproject.org/tests/2659839

so it looks like the ffmpeg update triggers this, somehow. I'm confirming these results on prod, just to be sure, but that's the way it looks.

Comment 3 Adam Williamson 2023-03-13 23:00:07 UTC
The thing that looks, I guess, most 'suspicious' to me is that oneVPL ships some files in /usr/etc:

/usr/etc/modulefiles
/usr/etc/modulefiles/vpl
/usr/etc/vpl
/usr/etc/vpl/vars.sh

that's rather unusual, I don't think many packages ship stuff there. Also neither oneVPL itself nor anything else owns the directory /usr/etc , though it does own /usr/etc/modulefiles and /usr/etc/vpl .

Comment 4 Colin Walters 2023-03-13 23:50:01 UTC
Thanks for debugging this; indeed packages should not install files in /usr/etc, such a thing just conflicts with ostree's internal use of that directory.

Comment 5 Adam Williamson 2023-03-14 00:12:35 UTC
Maybe we need a CI test that checks for files in /usr/etc or something...

Glad I guessed the right thing, as now I'm down the rabbit hole of "why does oneVPL do this and how do we make it stop?" I've filed an issue - https://github.com/oneapi-src/oneVPL/issues/94 - and I'm testing a hacky 'fix' which may not be good enough for upstream but probably is good enough for our package. I hope.

Comment 6 Adam Williamson 2023-03-14 01:01:13 UTC
OK, I'm running oneVPL builds with the config file path fixed using a slightly hacky patch that should be good enough for us at least. Will edit the F38 one into the ffmpeg update.

Comment 7 Adam Williamson 2023-03-14 01:03:09 UTC
sigh, the F38 update already went stable :( so this is broken till I can get my fix through.

Comment 8 Fedora Update System 2023-03-14 01:12:14 UTC
FEDORA-2023-a433b7cf46 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-a433b7cf46

Comment 9 Fedora Update System 2023-03-15 00:18:00 UTC
FEDORA-2023-a433b7cf46 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.