Bug 1925045

I did a bit of experimentation here and I'm not sure this bug is solvable.
It's really a buggy kernel API in Linux.

The socket(2) call succeeds, and connect(2) fails with EAGAIN.

After this, the socket is writable (POLLOUT|POLLHUP), but connecting
fails again immediately.  We just end up in a loop doing:

poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT|POLLHUP}])
connect(3, {sa_family=AF_UNIX, sun_path="/tmp/sock"}, 110) = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT|POLLHUP}])
connect(3, {sa_family=AF_UNIX, sun_path="/tmp/sock"}, 110) = -1 EAGAIN (Resource temporarily unavailable)

which of course uses 100% CPU.

Since the socket on its own does not "remember" which socket we wanted to
connect to, there's no way that poll could do the right thing here.

Here's a simpler reproducer which doesn't require NBD at all
(suggested by Daniel Berrange).

-----------------------------------
#include <sys/socket.h>
#include <sys/un.h>
#include <fcntl.h>
#include <unistd.h>
#include <assert.h>
#include <stdio.h>

int main(int argc, char **argv)
{
  int s = socket(AF_UNIX, SOCK_STREAM|SOCK_NONBLOCK|SOCK_CLOEXEC, 0);
  struct sockaddr_un un = {};
  const char *path = "/tmp/sock";
  int ret;

  assert(s >= 0);

  un.sun_family = AF_UNIX;
  memcpy(un.sun_path, path, strlen(path));

  ret = connect(s, (struct sockaddr *)&un, sizeof(un));
  if (ret < 0) {
    perror("connect");
  }

  sleep(30);

  return 0;
}
-----------------------------------

$ gcc sock.c -o sock

$ socat  UNIX-LISTEN:/tmp/sock,backlog=1 STDOUT
^Z
[1]+  Stopped                 socat UNIX-LISTEN:/tmp/sock,backlog=1 STDOUT

In other windows, run the following command three times:

$ strace ./sock

The first two will connect successfully, and the third will fail with:

connect(3, {sa_family=AF_UNIX, sun_path="/tmp/sock"}, 110) = -1 EAGAIN (Resource temporarily unavailable)

Patch posted:
https://www.redhat.com/archives/libguestfs/2021-February/msg00015.html

BZ 1925345 opened to track the qemu-nbd fix to use SOMAXCONN.

Upstream in:
https://github.com/libguestfs/libnbd/commit/cd231fd94bbfaacdd9b89e7d355ba2bbc83c2aeb
https://github.com/libguestfs/libnbd/commit/85ed74960a658a82d7b61b0be07f43d1b2dcede9

Note the final fix only corrects the error message, referring back
to this bug.  The real fix has to be done on the server side.

Reproduce the bug with builds:
libnbd-1.6.0-2.module+el8.4.0+9341+96cf2672.x86_64

Steps:
1.Create sock.c file as comment 2 
2.Follow steps in comment 2
# gcc sock.c -o sock

#  socat  UNIX-LISTEN:/tmp/sock,backlog=1 STDOUT
^Z
[1]+  Stopped                 socat UNIX-LISTEN:/tmp/sock,backlog=1 STDOUT

3.In other windows, run the following command three times:
# strace ./sock

The command failed in third time and error info like below:
[...]
connect(3, {sa_family=AF_UNIX, sun_path="/tmp/sock"}, 110) = -1 EAGAIN (Resource temporarily unavailable)
dup(2)                                  = 4
fcntl(4, F_GETFL)                       = 0x402 (flags O_RDWR|O_APPEND)
brk(NULL)                               = 0xb39000
brk(0xb5a000)                           = 0xb5a000
brk(NULL)                               = 0xb5a000
fstat(4, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x1), ...}) = 0
write(4, "connect: Resource temporarily un"..., 42connect: Resource temporarily unavailable



Test the bug with builds:
libnbd-1.6.0-3.module+el8.4.0+9856+bbc47853.x86_64

Scenario 1:
Steps as reproduce,but the connection still failed.
# strace ./sock
execve("./sock", ["./sock"], 0x7ffc1c02cd90 /* 37 vars */) = 0
brk(NULL)                               = 0x1dc6000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffee5b8ba90) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=25567, ...}) = 0
mmap(NULL, 25567, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fd870509000
close(3)                                = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\2405\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=3167976, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd870507000
lseek(3, 808, SEEK_SET)                 = 808
read(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32) = 32
mmap(NULL, 3950400, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fd86ff1f000
mprotect(0x7fd8700db000, 2093056, PROT_NONE) = 0
mmap(0x7fd8702da000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bb000) = 0x7fd8702da000
mmap(0x7fd8702e0000, 14144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fd8702e0000
close(3)                                = 0
arch_prctl(ARCH_SET_FS, 0x7fd870508500) = 0
mprotect(0x7fd8702da000, 16384, PROT_READ) = 0
mprotect(0x600000, 4096, PROT_READ)     = 0
mprotect(0x7fd870510000, 4096, PROT_READ) = 0
munmap(0x7fd870509000, 25567)           = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/tmp/sock"}, 110) = -1 EAGAIN (Resource temporarily unavailable)
dup(2)                                  = 4
fcntl(4, F_GETFL)                       = 0x2 (flags O_RDWR)
brk(NULL)                               = 0x1dc6000
brk(0x1de7000)                          = 0x1de7000
brk(NULL)                               = 0x1de7000
fstat(4, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x1), ...}) = 0
write(4, "connect: Resource temporarily un"..., 42connect: Resource temporarily unavailable
) = 42
close(4)                                = 0
nanosleep({tv_sec=30, tv_nsec=0}, 0x7ffee5b8ba00) = 0
exit_group(0)                           = ?
+++ exited with 0 +++


Scenario 2:
Test the bug follow steps in comment 0,still failed.
# rpm -q libnbd
libnbd-1.6.0-3.module+el8.4.0+9856+bbc47853.x86_64
# touch /tmp/disk.img
# truncate -s 10M /tmp/disk.img
# rm /tmp/sock
rm: remove socket '/tmp/sock'? y
#  qemu-nbd -t -k /tmp/sock -f raw /tmp/disk.img 
^Z
[1]+  Stopped                 qemu-nbd -t -k /tmp/sock -f raw /tmp/disk.img
# nbdsh -c 'h.connect_unix("/tmp/sock")' &
[2] 5974
# nbdsh -c 'h.connect_unix("/tmp/sock")' &
[3] 5976
# nbdsh -c 'h.connect_unix("/tmp/sock")' &
[4] 5978
# nbdsh: command line script failed: nbd_connect_unix: connect: server backlog overflowed, see https://bugzilla.redhat.com/1925045: Resource temporarily unavailable

[4]-  Exit 1                  nbdsh -c 'h.connect_unix("/tmp/sock")'

Hi,Rjones
Result as below:
The scenario 2 can refer the problem to bug, but the scenario 1 still give error info like:
connect(3, {sa_family=AF_UNIX, sun_path="/tmp/sock"}, 110) = -1 EAGAIN (Resource temporarily unavailable)

Pls help to check the result, is it a expected result?

Scenario 1 / comment 2 is a reproducer for the server-side problem
that doesn't involve the NBD protocol at all.  It was just there to
demonstrate the problem with hung servers and short backlogs.  So
forget about scenario 1.

Scenario 2 is the real test, and that does show the expected
result (the updated error message), so that verifies the bug.

Thanks for your kindly reply.
As comment 10 and comment 11,now change the bug from on_qa to verified.

Hmm - this patch exposed a latent problem when doing fd passing (systemd style):
https://lists.gnu.org/archive/html/qemu-devel/2021-03/msg03544.html

We'll want to clone this bug to pull in that patch as well.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2098