I debugged this a bit with Stephanie yesterday.

The root bridge does a recvfrom() to receive a file descriptor through a Unix socketpair() from the user bridge. This socket is expected to be blocking (as every opened fd is on Unix by default). But on the customer system, recvfrom() fails with EAGAIN, which means that there is no data yet, and instead of waiting for data to arrive, it fails with that error code (shown in the screenshot as "temporarily unavailable"). The bridge is not prepared for EAGAIN, as that should never happen with a normal socket pair. Something on the customer's system causes this, and I have no idea what.

My first hack was to loop recvfrom() until it stops EAGAINing. This is a ridiculous hack, but at least proves what happens. The customer confirmed that this fixed the issue. But of course it's prone to spinning 100% CPU.

The next iteration would try to set the socket back to blocking mode:

--- src/common/cockpitfdpassing.c
+++ src/common/cockpitfdpassing.c
@@ -23,6 +23,8 @@
 
 #include <assert.h>
 #include <errno.h>
+#include <fcntl.h>
+#include <stdio.h>
 #include <string.h>
 #include <unistd.h>
 
@@ -119,6 +121,13 @@ cockpit_socket_receive_fd (int  socket_fd,
                         .msg_controllen = CMSG_LEN (sizeof *out_fd) };
   assert (msg.msg_controllen <= sizeof cmsg);
 
+  int cur_flags = fcntl (socket_fd, F_GETFL);
+  if (cur_flags & O_NONBLOCK)
+    {
+      // fprintf (stderr, "cockpit_socket_receive_fd: socket_fd %i is unexpectedly non-blocking, flags %x, switching to blocking", socket_fd, cur_flags);
+      fcntl (socket_fd, F_SETFL, cur_flags & ~O_NONBLOCK);
+    }
+
   ssize_t s;
   do
     s = recvmsg (socket_fd, &msg, 0);

If this works, that would be a much better fix/workaround, as it avoids looping in user space. I sent an RPM to Stephanie for the customer to test. Once they confirm, I'm happy to send an upstream PR to fix that.

Thanks!

Thanks Stephanie for confirming! I sent https://github.com/cockpit-project/cockpit/pull/18915 to fix this upstream, and eventually in RHEL 9.

I already asked that in the support case, but that was closed by now.

We still don't know about the root cause of that. Dear original reporter, can you please share how you set up this system? There must be something out of the ordinary, stderr shouldn't just randomly become blocking. Custom kernel, logging into a remote system, session recording, non-default PAM libraries, etc.

I can now reproduce this, see https://github.com/cockpit-project/cockpit/pull/18915 . For me this happens when enabling sudo's `log_output` option. We still don't know if it was the same root cause for the original reporter here, but that doesn't matter much now.

on Tumbleweed i get such message in the dbus strace:
1468  recvmsg(27, {msg_namelen=0}, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource temporarily unavailable)

No logging turned on in sudo, or anything special. Normal journal log:

dbus-daemon[1468]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.816' (uid=1000 pid=22597 comm="cockpit-bridge")
systemd[1]: Starting Hostname Service...
dbus-daemon[1468]: [system] Successfully activated service 'org.freedesktop.hostname1'
systemd[1]: Started Hostname Service.
dbus-daemon[1468]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.819' (uid=1000 pid=22597 comm="cockpit-bridge")
systemd[1]: Starting Time & Date Service...
dbus-daemon[1468]: [system] Successfully activated service 'org.freedesktop.timedate1'
systemd[1]: Started Time & Date Service.
dbus-daemon[1468]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.823' (uid=0 pid=22606 comm="cockpit-bridge --privileged")
systemd[1]: Starting PackageKit Daemon...
PackageKit[22755]: daemon start
dbus-daemon[1468]: [system] Successfully activated service 'org.freedesktop.PackageKit'
systemd[1]: Started PackageKit Daemon.

After restart of libvirt-dbus admin can access VMs in cockpit immediately.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (cockpit bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6591