Created attachment 1968416 [details] cockpit-error Description of problem: Local users, or AD users are receiving the bellow error when trying to become administrators thru Cocpkit Problem becoming administrator Cockpit-bridge: recvmsg(stdin) failed: Resource temporarily unavailable Version-Release number of selected component (if applicable): cockpit-bridge-286.1-1.el9.x86_64 How reproducible: I wasn't able to reproduce it in a lab. Steps to Reproduce: 1. Login with a non-root user 2. Try to become an administrator Actual results: Problem becoming administrator Cockpit-bridge: recvmsg(stdin) failed: Resource temporarily unavailable Expected results: Get administrative access on cockpit. Additional info:
I debugged this a bit with Stephanie yesterday. The root bridge does a recvfrom() to receive a file descriptor through a Unix socketpair() from the user bridge. This socket is expected to be blocking (as every opened fd is on Unix by default). But on the customer system, recvfrom() fails with EAGAIN, which means that there is no data yet, and instead of waiting for data to arrive, it fails with that error code (shown in the screenshot as "temporarily unavailable"). The bridge is not prepared for EAGAIN, as that should never happen with a normal socket pair. Something on the customer's system causes this, and I have no idea what. My first hack was to loop recvfrom() until it stops EAGAINing. This is a ridiculous hack, but at least proves what happens. The customer confirmed that this fixed the issue. But of course it's prone to spinning 100% CPU. The next iteration would try to set the socket back to blocking mode: --- src/common/cockpitfdpassing.c +++ src/common/cockpitfdpassing.c @@ -23,6 +23,8 @@ #include <assert.h> #include <errno.h> +#include <fcntl.h> +#include <stdio.h> #include <string.h> #include <unistd.h> @@ -119,6 +121,13 @@ cockpit_socket_receive_fd (int socket_fd, .msg_controllen = CMSG_LEN (sizeof *out_fd) }; assert (msg.msg_controllen <= sizeof cmsg); + int cur_flags = fcntl (socket_fd, F_GETFL); + if (cur_flags & O_NONBLOCK) + { + // fprintf (stderr, "cockpit_socket_receive_fd: socket_fd %i is unexpectedly non-blocking, flags %x, switching to blocking", socket_fd, cur_flags); + fcntl (socket_fd, F_SETFL, cur_flags & ~O_NONBLOCK); + } + ssize_t s; do s = recvmsg (socket_fd, &msg, 0); If this works, that would be a much better fix/workaround, as it avoids looping in user space. I sent an RPM to Stephanie for the customer to test. Once they confirm, I'm happy to send an upstream PR to fix that. Thanks!
Thanks Stephanie for confirming! I sent https://github.com/cockpit-project/cockpit/pull/18915 to fix this upstream, and eventually in RHEL 9.
I already asked that in the support case, but that was closed by now. We still don't know about the root cause of that. Dear original reporter, can you please share how you set up this system? There must be something out of the ordinary, stderr shouldn't just randomly become blocking. Custom kernel, logging into a remote system, session recording, non-default PAM libraries, etc.
I can now reproduce this, see https://github.com/cockpit-project/cockpit/pull/18915 . For me this happens when enabling sudo's `log_output` option. We still don't know if it was the same root cause for the original reporter here, but that doesn't matter much now.