the openQA test that builds a Silverblue ostree then embeds it in an installer image and tests that has been failing for the last few hours. The first update it failed on is https://bodhi.fedoraproject.org/updates/FEDORA-2023-8e38ae984c , gstreamer1-plugins-base-1.22.1-1.fc39 . On failed attempts, rpm-ostree fails here: [2023-03-13 13:28:11] Installing packages: 100% [2023-03-13 13:28:18] [0m[31merror: [0mDirectory not empty (os error 39) on the last successful attempt, that part looks like this: [2023-03-13 12:29:19] Installing packages: 100% [2023-03-13 12:29:22] Updating usr/lib/os-release so the failure is occurring somewhere between the end of package install - specifically, right after the "Installing packages" line in `install_packages()` in rpmostree-compose-builtin-tree.cxx - and early post-processing (specifically the line ` println!("Updating {}", path);` in `compose_postprocess_mutate_os_release()` in rust/src/composepost.rs). It seems surprising that the gstreamer update would be the cause here but it (or ffmpeg) may be. The base packages on the test system are the same between the "last good" and "first bad" tests. I can't see every dependency of rpm-ostree that got installed in the "last good" case, but at least rpm-ostree itself is the same version between those two cases, and I can't see anything obviously related that was updated around the same time as gstreamer1-plugins-base. The only packages that differ in the set of packages that get installed into the ostree are these: [adamw@xps13a tmp]$ diff -u0 /tmp/goodpkgs /tmp/badpkgs --- /tmp/goodpkgs 2023-03-13 14:39:56.127159388 -0700 +++ /tmp/badpkgs 2023-03-13 14:39:50.817219037 -0700 @@ -382,2 +382,2 @@ -gstreamer1-1.22.1-1.fc39.x86_64 (advisory) -gstreamer1-plugin-libav-1.22.0-1.fc38.x86_64 (fedora-rawhide) +gstreamer1-1.22.1-1.fc39.x86_64 (koji-rawhide) +gstreamer1-plugin-libav-1.22.0-2.fc39.x86_64 (koji-rawhide) @@ -385 +385 @@ -gstreamer1-plugins-base-1.22.0-1.fc38.x86_64 (fedora-rawhide) +gstreamer1-plugins-base-1.22.1-1.fc39.x86_64 (advisory) @@ -565,3 +565,3 @@ -libavcodec-free-5.1.2-12.fc39.x86_64 (fedora-rawhide) -libavfilter-free-5.1.2-12.fc39.x86_64 (fedora-rawhide) -libavformat-free-5.1.2-12.fc39.x86_64 (fedora-rawhide) +libavcodec-free-6.0-1.fc39.x86_64 (koji-rawhide) +libavfilter-free-6.0-1.fc39.x86_64 (koji-rawhide) +libavformat-free-6.0-1.fc39.x86_64 (koji-rawhide) @@ -569 +569 @@ -libavutil-free-5.1.2-12.fc39.x86_64 (fedora-rawhide) +libavutil-free-6.0-1.fc39.x86_64 (koji-rawhide) @@ -595 +595 @@ -libchromaprint-1.5.1-7.fc38.x86_64 (fedora-rawhide) +libchromaprint-1.5.1-8.fc39.x86_64 (koji-rawhide) @@ -727 +727 @@ -libpostproc-free-5.1.2-12.fc39.x86_64 (fedora-rawhide) +libpostproc-free-6.0-1.fc39.x86_64 (koji-rawhide) @@ -777,2 +777,2 @@ -libswresample-free-5.1.2-12.fc39.x86_64 (fedora-rawhide) -libswscale-free-5.1.2-12.fc39.x86_64 (fedora-rawhide) +libswresample-free-6.0-1.fc39.x86_64 (koji-rawhide) +libswscale-free-6.0-1.fc39.x86_64 (koji-rawhide) @@ -933,0 +934,2 @@ +oneVPL-2022.2.2-1.fc38.x86_64 (fedora-rawhide) +oneVPL-intel-gpu-22.5.3-1.fc38.x86_64 (fedora-rawhide) those are all the gstreamer package itself or deps of it, or subpackages of ffmpeg - ffmpeg was updated from 5.1.2 to 6.0 around the same time. Otherwise this could be caused by some dep of rpm-ostree that I can't see in the "last good" test, or by some change to workstation-ostree-config. Two PRs were merged there recently, but they seem to be a few hours too old to be the culprit. The last-known-good test ran "git clone https://pagure.io/workstation-ostree-config.git" at [2023-03-13T16:18:49.674286Z] , and those commits seem to have come an hour or two before that, so it should already have got the latest git state with the changes. I suspect this will break Silverblue compose with the next official compose, as the openQA test is designed to be as similar to the official compose process as possible.
It occurs to me we can use F38 to narrow this down a bit. Neither gstreamer1-plugins-base nor ffmpeg has an official update for F38 yet, so I'm doing scratch builds; I'll trigger openQA tests on both and see where this test fails.
OK, it seems like it's ffmpeg or one of its deps that causes this. I ran the ostree compose test on the ffmpeg update for F38 and on a scratch build of the gstreamer1-plugins-base update too. The ffmpeg test failed the same way: https://openqa.stg.fedoraproject.org/tests/2659828#step/_ostree_build/41 but the gstreamer1-plugins-base test has passed the failure point and is still going: https://openqa.stg.fedoraproject.org/tests/2659839 so it looks like the ffmpeg update triggers this, somehow. I'm confirming these results on prod, just to be sure, but that's the way it looks.
The thing that looks, I guess, most 'suspicious' to me is that oneVPL ships some files in /usr/etc: /usr/etc/modulefiles /usr/etc/modulefiles/vpl /usr/etc/vpl /usr/etc/vpl/vars.sh that's rather unusual, I don't think many packages ship stuff there. Also neither oneVPL itself nor anything else owns the directory /usr/etc , though it does own /usr/etc/modulefiles and /usr/etc/vpl .
Thanks for debugging this; indeed packages should not install files in /usr/etc, such a thing just conflicts with ostree's internal use of that directory.
Maybe we need a CI test that checks for files in /usr/etc or something... Glad I guessed the right thing, as now I'm down the rabbit hole of "why does oneVPL do this and how do we make it stop?" I've filed an issue - https://github.com/oneapi-src/oneVPL/issues/94 - and I'm testing a hacky 'fix' which may not be good enough for upstream but probably is good enough for our package. I hope.
OK, I'm running oneVPL builds with the config file path fixed using a slightly hacky patch that should be good enough for us at least. Will edit the F38 one into the ffmpeg update.
sigh, the F38 update already went stable :( so this is broken till I can get my fix through.
FEDORA-2023-a433b7cf46 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-a433b7cf46
FEDORA-2023-a433b7cf46 has been pushed to the Fedora 38 stable repository. If problem still persists, please make note of it in this bug report.