Fedora Account System
Red Hat Associate
Red Hat Customer
On current Rawhide openQA tests, `rpm-ostree rebase` and `rpm-ostree install` commands seem to be failing with: error: Error binding to address (GUnixSocketAddress): No such file or directory the system logs don't seem to shed any more light, that's the only obvious error message.
I can reproduce this with the 20230213.n.0 build. Trying an older one to try to see when it started failing.
Latest Fedora CoreOS is affected as well. Reporting upstream.
It has already been reported in https://github.com/coreos/rpm-ostree/issues/4306
I've confirmed that reverting to `glib2-2.74.1-3.fc38.x86_64` fixes this. This is thus likely a `glib2` bug.
This is urgent as this basically affects (breaks) all rpm-ostree variants (CoreOS, IoT, Silverblue, Kinoite, Sericea) and has been pushed to Rawhide and F38 so prevents us from doing work there before the Beta freeze. I'm trying to figure out which change broke it but it's going to take me some time.
"It has already been reported in https://github.com/coreos/rpm-ostree/issues/4306" - actually, this report came first, that one came later. ;) mcatanzaro says a bisect would be helpful here, if anyone has time to do one. I can try but I have a bunch of other things to look after around branching too...
Timothée, when you "confirmed that reverting to `glib2-2.74.1-3.fc38.x86_64` fixes this", how did you *do* that exactly?
I'm still bisecting but I now suspect that this is the culprit: 9151fe94c gio: remove pointless use of g_unix_socket_address_abstract_names_supported with unix:tmpdir= Current range: git log 5b01b47c34a103450183a5eb1a9d67f1f4643461..7b09713963244caf08bc660c848473a6606553c0 To verify / bisect: I booted a fresh Fedora CoreOS rawhide built with glib2-2.74.1-3.fc38.x86_64 and used: - rpm-ostree usroverlay - rpm -Uvh --force glib2-*.rpm - rpm-ostree install foo
That would be a strange place for the bisect to end up. That's basically an internal alternative to dbus-broker. It's only used for (a) its own tests, and (b) running a session bus on Windows.
welp, call it as strange as you like, but it looks like Timothée was right. Following on using his technique (thanks!) I built a glib2 with these two commits reverted: c0a1a3b384a27b4b18628174a5c4d9f3fb0a8c82 gio/gdbusserver: use non-abstract socket for unix:tmpdir= 9151fe94cb21ef78c00b0ce0b7dfebdbc423830c gio: remove pointless use of g_unix_socket_address_abstract_names_supported with unix:tmpdir= and it works fine.
(In reply to Adam Williamson from comment #10) > c0a1a3b384a27b4b18628174a5c4d9f3fb0a8c82 gio/gdbusserver: use non-abstract > socket for unix:tmpdir= This one is much less strange. ;) Adam is going to temporarily revert this for now as an emergency measure, but this needs to be temporary because the change is important. This commit moves the D-Bus server socket from the abstract socket namespace to /tmp to prevent sandboxed applications from accessing it if they have access to the host network namespace. The abstract socket namespace is basically one big sandbox hole, as the only way to block access to it is to deny network access altogether. (I've heard that doesn't necessarily have to be true in theory, but in practice it's true for flatpak today.) So presumably rpm-ostree is running a private D-Bus server using GDBusServer, the D-Bus socket gets created in /tmp, and then the client process does not have permission to access it for some reason. Is the client process itself sandboxed such that it cannot see the host /tmp? Is the client process's view of host /tmp obstructed by an overlay? rpm-ostree developers will probably know.
so since the change in glib is considered intended/correct, let's kick this back to rpm-ostree for now, and see if rpm-ostree devs can see a way to make what they're doing work with the glib change.
*** Bug 2170029 has been marked as a duplicate of this bug. ***
> So presumably rpm-ostree is running a private D-Bus server using GDBusServer, the D-Bus socket gets created in /tmp, and then the client process does not have permission to access it for some reason. Is the client process itself sandboxed such that it cannot see the host /tmp? Ah...good investigation. Indeed that is highly likely and I think we can adapt things to work more properly here. I hadn't realized we were relying on the abstract namespace. That said...still, glib is such a foundational library that I think if we found one thing that broke due to this we probably need to consider making the change to GDBusServer opt-in via an API.
https://gitlab.freedesktop.org/dbus/dbus/-/merge_requests/350 > The merge request with the details.
https://gitlab.gnome.org/GNOME/glib/-/merge_requests/3005 > The one for glib.
(In reply to Colin Walters from comment #14) > That said...still, glib is such a foundational library that I think if we > found one thing that broke due to this we probably need to consider making > the change to GDBusServer opt-in via an API. Problem is GLib should be secure by default, and that means not using the abstract socket namespace. (Use of abstract sockets was actually an implementation detail anyway, not an API guarantee.) Possible workaround: switch to using unix:path addresses instead of unix:tmpdir, so you can place the socket precisely where you want it. (Remember to unlink() the socket when done.)
I agree that this what we want in the end but the issue is that it's unlikely that we'll be able to have something ready before the freeze, and this is blocking us from doing other changes to Fedora Silverblue, etc. I would appreciate if we could temporarily revert this while making sure to land it in F38, just a bit later.
I already reverted it last night. I left the bug open because the revert isn't a correct long-term fix.
> Possible workaround: switch to using unix:path addresses instead of unix:tmpdir, so you can place the socket precisely where you want it. (Remember to unlink() the socket when done.) Right; PR in https://github.com/coreos/rpm-ostree/pull/4308 > Problem is GLib should be secure by default, and that means not using the abstract socket namespace. For sure. But, you also broke an application. I used to do work in glib and personally if it'd been me making this change I probably would have reverted to start. (Sorry, I tried to comment on gnome gitlab but fighting the password reset process right now) ISTM the goal could be achieved by also patching webkit to use `unix:dir` right? The real question is: how many other users of gdbusserver are there and how many of them would be broken by this versus silently fixed? Was that analysis done?
(In reply to Colin Walters from comment #20) > ISTM the goal could be achieved by also patching webkit to use `unix:dir` > right? No. (In fact, this doesn't even matter for WebKit because there the untrusted process does not have network access. It does matter for flatpak.) To achieve this goal, you would have to patch all software that uses 'unix:tmpdir' to use 'unix:dir' or another scheme instead, or implement some custom authentication scheme on top of D-Bus so that each side can ensure it's really talking to the process that it expects. It makes much more sense to fix the problem in GLib. Abstract sockets just have to go away. You surely expected that only trusted processes would be able to connect to your D-Bus server; this change makes that expectation actually true. > The real question is: how many other users of gdbusserver are there and how > many of them would be broken by this versus silently fixed? Was that > analysis done? GLib does try to be as compatible as practical, but it's just not perfect to fully preserve backwards-compatibility in this case because the previous behavior was not secure. Moving a socket from one place to another is an implementation detail, not something that would normally be expected to cause applications to break. rpm-ostree is frankly pretty special -- I'd be surprised if it's a problem for anything else -- and it's likely going to be pretty easy to fix as you only need to change the socket path to someplace else, so I don't think this merits an upstream revert. Also, this change is a follow-up to match the same change made in dbus-daemon as well (comment #15).
(In reply to Michael Catanzaro from comment #21) > but it's just not perfect to > fully preserve backwards-compatibility in this case I meant to write: "just not practical"
(In reply to Adam Williamson from comment #19) > I already reverted it last night. I left the bug open because the revert isn't a correct long-term fix. Thanks. Got fooled by the versions listed at https://src.fedoraproject.org/rpms/glib2
Those would only get updated after a compose, I think. The last branched ran a little before I did the backport, and the last Rawhide failed for unrelated reasons. The most reliable thing to look at if you want to know what's 'tagged stable' (as opposed to what's actually currently on the public mirrors) is Koji tag history: [adamw@xps13a workstation-ostree-config (fix-repo-indent)]$ koji list-history --tag=f39 --package=glib2 Wed Feb 8 13:29:34 2023 package owner mclasen set for glib2 in f39 by humaton [still active] Wed Feb 8 13:29:34 2023 package list entry created: glib2 in f39 by humaton [still active] Wed Feb 8 13:39:39 2023 glib2-2.74.1-3.fc38 tagged into f39 by humaton [still active] Mon Feb 13 12:29:00 2023 glib2-2.75.3-1.fc39 tagged into f39 by bodhi [still active] Tue Feb 14 17:40:03 2023 glib2-2.75.3-3.fc39 tagged into f39 by bodhi [still active] whichever build was most recently tagged and has not since been untagged is the "current stable" (note, this applies even if the most recently tagged build is lower versioned than another build). What Bodhi says is nearly as reliable, but Bodhi doesn't know if we ninja-untag something because it causes a major bug, which we're doing relatively frequently now because openQA testing lets us know quickly when something breaks stuff.
> GLib does try to be as compatible as practical, but it's just not perfect to fully preserve backwards-compatibility in this case because the previous behavior was not secure. That depends on what the app was doing with the socket. rpm-ostree's use case was secure - we serve read-only data from the socket. (Also, SELinux tends to stop unexpected connections, e.g. podman containers can't get to it by default) > Moving a socket from one place to another is an implementation detail, not something that would normally be expected to cause applications to break. I hope we can agree that everything using GDBusServer tends to be a bit special. > To achieve this goal, you would have to patch all software that uses 'unix:tmpdir' to use 'unix:dir' Only the software that we determine actually is vulnerable to something. > and it's likely going to be pretty easy to fix as you only need to change the socket path to someplace else Yes. But now you're introducing the risk of breaking things as this newer glib propagates, but *before* the fixed rpm-ostree propagates. (And the failure mode here is high because it kills OS updates...) I dunno, I don't have the energy or time to debate this endlessly. The last thing I'll say is basically: just consider again how it would feel if glib (or some other software) was on the other end of this and it was a kernel or glibc change and glib had been using some relatively obscure feature and it broke - but what glib was doing wasn't actually insecure to begin with.
Built https://bodhi.fedoraproject.org/updates/FEDORA-2023-ca42354dfa so someone (adam?) can likely follow up soon and revert the revert to glib2.
Thanks, I'll try that later today (I'll do a scratch build first to make sure it works OK).
Pushed to F38 with https://bodhi.fedoraproject.org/updates/FEDORA-2023-811e53475e
Great. Sorry, I didn't get time yet to test if we can put the glib2 change back now, everything's been a bit on fire. I'll try and do it now.
My scratch build test looks good: https://openqa.stg.fedoraproject.org/tests/overview?distri=fedora&version=39&build=Kojitask-97643457-NOREPORT&groupid=2 so I'll revert the revert for real. Thanks, everyone.
I think this may also have broken createrepo_c: https://bodhi.fedoraproject.org/updates/FEDORA-2023-41cd721e93 at least, createrepo_c was recently rebuilt against glib2 with this change put back in, and it's now failing with the same undefined symbol error from https://github.com/coreos/fedora-coreos-tracker/issues/1422 , which was traced back to glib.
Never mind, sorry, this was my bad. It was an issue in the openQA tests (erroneously enabling the koji-rawhide repo on a support test which isn't itself Rawhide).
This is getting bounced around yet still a big problem... Been going on (rpm-ostree transaction failed with: error: Error binding to address (GUnixSocketAddress): No such file or directory) for a month now and I thought it would get ironed out but no and when I search a solution every mention of it gets shut down "closed" saying it's been reported elsewhere and follow the link now and that is pointing back circular >>> It has not been solved. I waited awhile because I am learning and don't have the knowledge to contribute so I feel I should just be gratefully patient. People that make demands and act super entitled for something valuable that they use for free and is maintained by volunteers, infuriates me, so to refrain from becoming a hypocrite I've waited. What should I do?
David: the bug was fixed some time ago. If you're stuck on a bad ostree and don't know how to escape, see https://bugzilla.redhat.com/show_bug.cgi?id=2169622#c8 - you should be able to use that technique to overlay a newer rpm-ostree and update, I believe.
I'm still stuck with the same error. I can't update or upgrade or rollback or rebase. I get the same "Error binding to address (GUnixSocketAddress): No such file or directory". I tried that 3 lines of code above but I couldn't get past the second line - I assume maybe the "*" is a wildcard standin but I don't know what the actual rest is as copy paste didn't work.
Ah, OK. You're meant to literally type * , but you need to download an updated version of glib2 first (and things are a bit complicated because we later reverted the glib2 workaround and fixed the problem in rpm-ostree instead). Try this: wget https://kojipkgs.fedoraproject.org//packages/rpm-ostree/2023.1/4.fc38/x86_64/rpm-ostree-2023.1-4.fc38.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/rpm-ostree/2023.1/4.fc38/x86_64/rpm-ostree-libs-2023.1-4.fc38.x86_64.rpm rpm-ostree usroverlay rpm -Uvh --force rpm-ostree-*.rpm then see if you can `rpm-ostree upgrade`. Sorry for the trouble!
Alright! Looks like it's working! Thank you. I usually always "search and lurk" then repeat, but I couldn't figure it out. What is that "-Uvh" argument? I'll search as you guys seem busy. Thanks.
It's the standard "update a package" recipe for rpm. It's been so long since I learned it that I forget what all the bits do =) U is upgrade (for "install" you use i), v is verbose, and h tells it to print a progress indicator for each package as it goes.