Does it work any better with an older build of coreutils?  Do we have some steps to reproduce the failure locally?

Well, it worked the day before. ;) We could untag that latest coreutils and run another compose and confirm that works?

As far as reproducers, this is livemedia-creator with the parameters in the koji task. I think /dev might be bind mounted in from the host in the chroot (which would explain why it cannot change perms, but not why it tries and fails).

This is the only change in src/mknod.c upstream between 8.31 and 8.32:

https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=v8.31-61-g04b136e29#patch10

It does not seem to be related...

lchmod() has changed in gnulib quite fundamentally.
The strace diff between 8.31 and 8.32 is:

-mknod("random1", S_IFCHR|0666, makedev(0x1, 0x8)) = 0
-chmod("random1", 0666)                  = 0
+mknod("random2", S_IFCHR|0666, makedev(0x1, 0x8)) = 0
+openat(AT_FDCWD, "random2", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 3
+newfstatat(3, "", {st_mode=S_IFCHR|0644, st_rdev=makedev(0x1, 0x8), ...}, AT_EMPTY_PATH) = 0
+chmod("/proc/self/fd/3", 0666)          = 0
+close(3)                                = 0

I would report this to bug-gnulib and cc Florian Weimer as this looks related:
https://www.sourceware.org/ml/libc-alpha/2020-02/msg00534.html

For lchmod changes see:
https://www.sourceware.org/ml/libc-alpha/2020-02/msg00534.html

Thank you for pointing it out, Pádraig!  The mentioned change looks indeed related.

I'm going to untag coreutils-8.32-1.fc33 for now so we can get a rawhide out. Let me know if you need it tagged back for some reason.

Kevin, could you please check that there is a working /proc mounted in the chroot when it fails?

It would be great to get an strace if possible.

I'll mention this issue now on the upstream patch...

I can reproduce it locally now.  It is sufficient to have a chroot where /proc is not mounted and a build of coreutils where the compile-time check `fchmodat+AT_SYMLINK_NOFOLLOW works on non-symlinks` evaluates to yes.  I will be able to share some details tomorrow hopefully.

Sorry for not getting back with more info sooner. Glad you are able to reproduce it.

Minimal example:

# yum install -y \
    https://kojipkgs.fedoraproject.org/packages/coreutils/8.32/1.fc33/x86_64/coreutils-8.32-1.fc33.x86_64.rpm \
    https://kojipkgs.fedoraproject.org/packages/coreutils/8.32/1.fc33/x86_64/coreutils-common-8.32-1.fc33.x86_64.rpm

# mkdir -p /var/tmp/chroot/{bin,dev,lib64,proc}
# cp -L /lib64/{ld-linux-x86-64.so.2,libselinux.so.1,libc.so.6,libpcre2-8.so.0,libdl.so.2,libpthread.so.0} /var/tmp/chroot/lib64
# cp /bin/mknod /var/tmp/chroot/bin
# chroot /var/tmp/chroot /bin/mknod /dev/random -m0666 c 1 8
/bin/mknod: cannot set permissions of '/dev/random': Operation not supported
# echo $?
1

# rm -fv /var/tmp/chroot/dev/random 
removed '/var/tmp/chroot/dev/random'

# strace chroot /var/tmp/chroot /bin/mknod /dev/random -m0666 c 1 8
[...]
umask(000)                              = 022
umask(022)                              = 000
mknod("/dev/random", S_IFCHR|0666, makedev(0x1, 0x8)) = 0
openat(AT_FDCWD, "/dev/random", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 3
newfstatat(3, "", {st_mode=S_IFCHR|0644, st_rdev=makedev(0x1, 0x8), ...}, AT_EMPTY_PATH) = 0
chmod("/proc/self/fd/3", 0666)          = -1 ENOENT (No such file or directory)
close(3)                                = 0
write(2, "/bin/mknod: ", 12/bin/mknod: )            = 12
write(2, "cannot set permissions of '/dev/"..., 39cannot set permissions of '/dev/random') = 39
write(2, ": Operation not supported", 25: Operation not supported) = 25
write(2, "\n", 1
)                       = 1
close(1)                                = 0
close(2)                                = 0
exit_group(1)                           = ?

# rm -fv /var/tmp/chroot/dev/random 
removed '/var/tmp/chroot/dev/random'
# mount -t proc proc /var/tmp/chroot/proc
# strace chroot /var/tmp/chroot /bin/mknod /dev/random -m0666 c 1 8
umask(000)                              = 022
umask(022)                              = 000
mknod("/dev/random", S_IFCHR|0666, makedev(0x1, 0x8)) = 0
openat(AT_FDCWD, "/dev/random", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 3
newfstatat(3, "", {st_mode=S_IFCHR|0644, st_rdev=makedev(0x1, 0x8), ...}, AT_EMPTY_PATH) = 0
chmod("/proc/self/fd/3", 0666)          = 0
close(3)                                = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?

It turns out that the observed behavior is caused by the implementation of lchmod() in glibc.  A pure rebuild of coreutils-8.31-10.fc32 on a rawhide system triggers the same bug.

You can force the gnulib implementation of lchmod with `configure ac_cv_func_lchmod=no`.
That implementation in 8.32 is similar to the glibc implementation,
but it also falls back to chmod() without /proc,
and so does not have this problem.

Thanks!  I will use it as a workaround until the issue is resolved upstream.

dist-git commit: https://src.fedoraproject.org/rpms/coreutils/c/acfa9e81

I have pushed a downstream change for now to avoid breakage of existing software.  On the other hand, coreutils/gnulib/glibc upstream developers are not going to fix this anytime soon.  So the long term fix needs to go to lorax: it needs to mount a proc file system on /var/tmp/lorax/lorax.7xmi9p_a/installtree/proc before running mknod in the chroot.

coreutils-8.32-3.fc32 has been pushed to the Fedora 32 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-17c2561434

We saw something similar in NixOS, which runs builds in a sandbox.

In our findings, it seems however something in between _binutils_ 2.31.1 and 2.34 caused this bug, not glibc.

We can individually build certain package bumps, and it was certainly one of the binutils bumps introducing the necessity to the `ac_cv_func_lchmod=no` configure hack in coreutils to unbreak it, but after building more, realized other packages needed these fixes as well, like rsync (which caused our linux kernel builds to fail) - see https://github.com/NixOS/nixpkgs/pull/85951#issuecomment-621199816.

Are you sure this is a problem in glibc, and not binutils? Has there been any discussion with upstream about that?

I am not sure about other packages but for coreutils it was really the change in glibc what introduced this bug.  It started to provide its own lchmod() implemetation at some point and the implementation relies on /proc being properly mounted.  If you need to make things work again without rebuilding anything, just make sure that a proc filesystem is mounted on /proc each time you directly or indirectly use glibc's implementation of lchmod().  I cannot see how binutils could ever be related to this bug.

(In reply to Florian Klink from comment #18)
> Has there been any discussion with upstream about that?

Yes.  As I understand it, coreutils/glibc/gnulib upstream developers are not going to fix it until there is an appropriate syscall available in Linux kernel:

https://lists.gnu.org/archive/html/bug-gnulib/2020-03/msg00028.html
https://lists.gnu.org/archive/html/bug-gnulib/2020-03/msg00031.html

At least here, moving from binutils 2.33.1 to 2.34 (https://github.com/NixOS/nixpkgs/pull/85951/commits/efdb29597a76c526dc0d9a55f8adf2ec5c33b0ee) is what's needed to reliably break the coreutils test suite (more context in the comment thread at https://github.com/NixOS/nixpkgs/pull/85951).

We have been on glibc 2.30 before and after this commit, and there's definitely no other glibc available inside the whole build environment.

Thanks for the pointers to the ML - I'll need to dive into how our /proc looks like inside the build sandbox (https://github.com/NixOS/nix/blob/master/src/libstore/build.cc#L3198) - there's also some seccomp magic involved (https://github.com/NixOS/nix/blob/master/src/libstore/build.cc#L2978), and how these could cause the weird behaviour I'm seeing now.

This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle.
Changing version to 33.