Bug 1946971

osbuild internally is not using systemd-nspawn anymore and I tried to verify that the problem still exists. On a Fedora 34 machine:

TARGET                       SOURCE               FSTYPE     OPTIONS
/                            /dev/mapper/f34-root xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
├─/proc                      proc                 proc       rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/fs/binfmt_misc systemd-1            autofs     rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=823
├─/sys                       sysfs                sysfs      rw,nosuid,nodev,noexec,relatime,seclabel
 ...
├─/dev                       devtmpfs             devtmpfs   rw,nosuid,seclabel,size=1987880k,nr_inodes=496970,mode=755,inode64
│ ├─/dev/shm                 tmpfs                tmpfs      rw,nosuid,nodev,seclabel,inode64
│ ├─/dev/pts                 devpts               devpts     rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000
│ ├─/dev/hugepages           hugetlbfs            hugetlbfs  rw,relatime,seclabel,pagesize=2M
│ └─/dev/mqueue              mqueue               mqueue     rw,nosuid,nodev,noexec,relatime,seclabel
├─/run                       tmpfs                tmpfs      rw,nosuid,nodev,seclabel,size=802620k,nr_inodes=819200,mode=755,inode64
│ └─/run/user/1000           tmpfs                tmpfs      rw,nosuid,nodev,relatime,seclabel,size=401308k,nr_inodes=100327,mode=700,uid=1000,gid=1000,inode64
├─/tmp                       tmpfs                tmpfs      rw,nosuid,nodev,seclabel,size=2006544k,nr_inodes=409600,inode64
├─/boot                      /dev/sda1            xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
├─/var                       /dev/mapper/f34-var  xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
│ └─/var/lib/nfs/rpc_pipefs  sunrpc               rpc_pipefs rw,relatime
...


I did 

 sudo osbuild test/data/manifests/fedora-boot.json --store /var/cache/osbuild --output-directory /var/tmp/osbuild

which worked fine. I tried to reproduce with composer but that is broken on F34 right now. I wonder if that issues was solved by switching to bubblewrap from systemd-nspawn.

Unfortunately, it seems that the use of bwrap leads to the same issue.
Using the latest el 8.4 pkgs:

# rpm -qa | grep osbuild
osbuild-composer-core-28.4-1.el8.x86_64
osbuild-ostree-27.2-1.el8.noarch
osbuild-composer-28.4-1.el8.x86_64
python3-osbuild-27.2-1.el8.noarch
osbuild-27.2-1.el8.noarch
osbuild-composer-worker-28.4-1.el8.x86_64
osbuild-selinux-27.2-1.el8.noarch

# composer-cli compose log 4e289c8b-66a1-4b9d-9455-2df8e6774c9c | tail -n4
Output:
bwrap: execvp /run/osbuild/lib/runners/org.osbuild.rhel84: No such file or directory

I didn't test with a 8.5 nightly build but I guess the issue remains the same.

@Christophe, thanks for checking. Could you post the mount flags for `/var`? I wonder what is different from my tests.

I just made a fresh 8.4 install, defaults are used for both XFS partitions.

# grep -e rhel-var -e rhel-usr /etc/fstab 
/dev/mapper/rhel-usr    /usr                    xfs     defaults        0 0
/dev/mapper/rhel-var    /var                    xfs     defaults        0 0

# mount | grep -e rhel-var -e rhel-usr
/dev/mapper/rhel-usr on /usr type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/rhel-var on /var type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)

# blkid | grep -e rhel-var -e rhel-usr
/dev/mapper/rhel-usr: UUID="37ab8d46-fd5d-4d95-9ac6-43b97dccb0a9" BLOCK_SIZE="512" TYPE="xfs"
/dev/mapper/rhel-var: UUID="80d91e34-84df-4464-ba1d-6e1f370228e2" BLOCK_SIZE="512" TYPE="xfs"

Ah, `/usr` and `/var` are on separate partitions. I tried with just `/var` on a 8.4 install and that worked fine. Let me retry with both separate.

We use a read only bind mount to pass the host's root to the build root setup code, and use `--bind` but *not* the recursive version (`--rbind`), this we get the root mount but not the `/usr` sub-mount. Either we just pass the root directly or if we want to keep that read only "protection" we can use the following patch:

modified   osbuild/objectstore.py
@@ -19,14 +19,15 @@ __all__ = [
 
 def mount(source, target, bind=True, ro=True, private=True, mode="0755"):
     options = []
-    if bind:
-        options += ["bind"]
     if ro:
         options += ["ro"]
     if mode:
         options += [mode]
 
     args = []
+
+    if bind:
+        args += ["--rbind"]
     if private:
         args += ["--make-private"]
     if options:

I will file a PR tomorrow. Sorry it took so long to look at this.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (osbuild bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4273