Bug 2177912

the openQA test that builds a Silverblue ostree then embeds it in an installer image and tests that has been failing for the last few hours. The first update it failed on is https://bodhi.fedoraproject.org/updates/FEDORA-2023-8e38ae984c , gstreamer1-plugins-base-1.22.1-1.fc39 .

On failed attempts, rpm-ostree fails here:

[2023-03-13 13:28:11] Installing packages: 100%
[2023-03-13 13:28:18] [0m[31merror: [0mDirectory not empty (os error 39)

on the last successful attempt, that part looks like this:

[2023-03-13 12:29:19] Installing packages: 100%
[2023-03-13 12:29:22] Updating usr/lib/os-release

so the failure is occurring somewhere between the end of package install - specifically, right after the "Installing packages" line in `install_packages()` in rpmostree-compose-builtin-tree.cxx - and early post-processing (specifically the line `    println!("Updating {}", path);` in `compose_postprocess_mutate_os_release()` in rust/src/composepost.rs).

It seems surprising that the gstreamer update would be the cause here but it (or ffmpeg) may be. The base packages on the test system are the same between the "last good" and "first bad" tests. I can't see every dependency of rpm-ostree that got installed in the "last good" case, but at least rpm-ostree itself is the same version between those two cases, and I can't see anything obviously related that was updated around the same time as gstreamer1-plugins-base. The only packages that differ in the set of packages that get installed into the ostree are these:

[adamw@xps13a tmp]$ diff -u0 /tmp/goodpkgs /tmp/badpkgs
--- /tmp/goodpkgs	2023-03-13 14:39:56.127159388 -0700
+++ /tmp/badpkgs	2023-03-13 14:39:50.817219037 -0700
@@ -382,2 +382,2 @@
-gstreamer1-1.22.1-1.fc39.x86_64 (advisory)
-gstreamer1-plugin-libav-1.22.0-1.fc38.x86_64 (fedora-rawhide)
+gstreamer1-1.22.1-1.fc39.x86_64 (koji-rawhide)
+gstreamer1-plugin-libav-1.22.0-2.fc39.x86_64 (koji-rawhide)
@@ -385 +385 @@
-gstreamer1-plugins-base-1.22.0-1.fc38.x86_64 (fedora-rawhide)
+gstreamer1-plugins-base-1.22.1-1.fc39.x86_64 (advisory)
@@ -565,3 +565,3 @@
-libavcodec-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
-libavfilter-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
-libavformat-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
+libavcodec-free-6.0-1.fc39.x86_64 (koji-rawhide)
+libavfilter-free-6.0-1.fc39.x86_64 (koji-rawhide)
+libavformat-free-6.0-1.fc39.x86_64 (koji-rawhide)
@@ -569 +569 @@
-libavutil-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
+libavutil-free-6.0-1.fc39.x86_64 (koji-rawhide)
@@ -595 +595 @@
-libchromaprint-1.5.1-7.fc38.x86_64 (fedora-rawhide)
+libchromaprint-1.5.1-8.fc39.x86_64 (koji-rawhide)
@@ -727 +727 @@
-libpostproc-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
+libpostproc-free-6.0-1.fc39.x86_64 (koji-rawhide)
@@ -777,2 +777,2 @@
-libswresample-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
-libswscale-free-5.1.2-12.fc39.x86_64 (fedora-rawhide)
+libswresample-free-6.0-1.fc39.x86_64 (koji-rawhide)
+libswscale-free-6.0-1.fc39.x86_64 (koji-rawhide)
@@ -933,0 +934,2 @@
+oneVPL-2022.2.2-1.fc38.x86_64 (fedora-rawhide)
+oneVPL-intel-gpu-22.5.3-1.fc38.x86_64 (fedora-rawhide)

those are all the gstreamer package itself or deps of it, or subpackages of ffmpeg - ffmpeg was updated from 5.1.2 to 6.0 around the same time.

Otherwise this could be caused by some dep of rpm-ostree that I can't see in the "last good" test, or by some change to workstation-ostree-config. Two PRs were merged there recently, but they seem to be a few hours too old to be the culprit. The last-known-good test ran "git clone https://pagure.io/workstation-ostree-config.git" at [2023-03-13T16:18:49.674286Z] , and those commits seem to have come an hour or two before that, so it should already have got the latest git state with the changes.

I suspect this will break Silverblue compose with the next official compose, as the openQA test is designed to be as similar to the official compose process as possible.