Bug 2215027

Description of problem:
A customer having an OS that has likely migrated from RHEL 6 in the past encountered an error while trying to symlink /etc/rhsm-host inside the container.

The real error happened before, systemd-nspawn returning an exit code which is *supposedly* the one of the command that is executed inside the container, unless it fails itself, which is apparently impossible to distinguish:

$ grep mknod 0100-leapp-debug.tgz/var/log/leapp/leapp-preupgrade.log 
2023-06-13 20:44:17.600 DEBUG    PID: 26010 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: mknod(/var/lib/leapp/scratch/mounts/root_/system_overlay/dev/null) failed: File exists

This messages comes from systemd-219/src/nspawn/nspawn.c, mknod()'ing this device descriptor is required to setup the console:

1370 static int copy_devnodes(const char *dest) {
1371 
1372         static const char devnodes[] =
1373                 "null\0"
1374                 "zero\0"
:
1389         NULSTR_FOREACH(d, devnodes) {
:
1393                 from = strappend("/dev/", d);
1394                 to = strjoin(dest, "/dev/", d, NULL);
:
1415                     if (mknod(to, st.st_mode, st.st_rdev) < 0)
1416                             return log_error_errno(errno, "mknod(%s) failed: %m", to);
:
:
3595 int main(int argc, char *argv[]) {
:
3980                         if (copy_devnodes(arg_directory) < 0)
3981                                 _exit(EXIT_FAILURE);


Fstab entries:

$ grep -v -e ^# -e xfs etc/fstab
devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=3992740k,nr_inodes=998185,mode=755 0 0
tmpfs /run tmpfs rw,seclabel,nosuid,nodev,mode=755 0 0
sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,noexec 0 0
tmpfs /sys/fs/cgroup tmpfs ro,seclabel,nosuid,nodev,noexec,mode=755 0 0

/dev is bind-mounted into the container:
2023-06-13 20:44:17.273 DEBUG    PID: 26010 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has started: ['mount', '-o', 'bind', '/var/lib/leapp/sc
ratch/mounts/root_dev/root_dev', u'/var/lib/leapp/scratch/mounts/root_/system_overlay/dev']
2023-06-13 20:44:17.285 DEBUG    PID: 26010 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has finished: ['mount', '-o', 'bind', '/var/lib/leapp/s
cratch/mounts/root_dev/root_dev', u'/var/lib/leapp/scratch/mounts/root_/system_overlay/dev']

causing the issue raised by systemd-nspawn:
95629 12:05:04.022718 mknod("/var/lib/leapp/scratch/mounts/root_/system_overlay/dev/null", S_IFCHR|0666, makedev(1, 3)) = -1 EEXIST (File exists) <0.000030>

Version-Release number of selected component (if applicable):
leapp-upgrade-el7toel8-0.18.0-1.el7_9.noarch

How reproducible:
Always

Steps to Reproduce:
1. Add a devtmpfs entry in fstab
devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=3992740k,nr_inodes=998185,mode=755 0 0
2. leapp preupgrade

Actual results:
Risk Factor: high
Title: Cannot set the container mode for the subscription-manager.
Summary: 
Key: 06583cd4c6f9e1629a9c1937643c3897618f33ef

Expected results:
An inhibitor

Additional info:
* should apply to RHEL 8>9 too
* a way to distinguish the systemd-nspawn's return code would help...