RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1946971 - osbuild-composer fails on a system with separate /var and /usr partitions
Summary: osbuild-composer fails on a system with separate /var and /usr partitions
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: osbuild
Version: 8.3
Hardware: All
OS: Linux
medium
high
Target Milestone: beta
: ---
Assignee: Christian Kellner
QA Contact: Release Test Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-07 11:55 UTC by Christophe Besson
Modified: 2021-11-10 01:16 UTC (History)
7 users (show)

Fixed In Version: osbuild-30-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-11-09 18:46:58 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github osbuild osbuild pull 713 0 None open objectstore: use recursive bind mounts 2021-07-09 09:17:16 UTC
Red Hat Knowledge Base (Solution) 5948711 0 None None None 2021-04-08 14:22:02 UTC
Red Hat Product Errata RHBA-2021:4273 0 None None None 2021-11-09 18:47:14 UTC

Description Christophe Besson 2021-04-07 11:55:13 UTC
Description of problem:
The reported error causing the issue is:
~~~
osbuild-worker[39770]: execv(/run/osbuild/lib/runners/org.osbuild.rhel83) failed: No such file or directory
~~~

That means osbuild-worker is not able to find the mentioned executable file OR the interpreter found in its shebang (namely /usr/libexec/platform-python).

-> systemd-nspawn is launched with "--bind-ro=/usr/lib/osbuild:/run/osbuild/lib":
~~~
30458 08:18:26.342290 execve("/usr/bin/systemd-nspawn", ["systemd-nspawn", "--quiet", "--register=no", "--keep-unit", "--as-pid2", "--link-journal=no", "--capability=CAP_MAC_ADMIN", "--directory=/run/osbuild/osbuild-buildroot-i93xziyi", "--setenv=PYTHONPATH=/run/osbuild/lib", "--bind-ro=/run/osbuild/osbuild-api-ge92jkzx:/run/osbuild/api", "--bind-ro=/usr/lib/osbuild:/run/osbuild/lib", "--bind-ro=/usr/lib/python3.6/site-packages/osbuild:/run/osbuild/lib/osbuild", "--bind=/var/cache/osbuild-worker/osbuild-store/tmp/tmp1phtrzqcobject/tmpd5jhxsb2-writer:/run/osbuild/tree", "--bind-ro=/var/cache/osbuild-worker/osbuild-store/osbuild-sources-output-7iss6km4:/run/osbuild/sources", "/run/osbuild/lib/runners/org.osbuild.rhel83", "/run/osbuild/lib/stages/org.osbuild.rpm"], ["LANG=en_US.UTF-8", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin", "INVOCATION_ID=2f6d28daad64421da680350288befd00", "JOURNAL_STREAM=9:28815", "CACHE_DIRECTORY=/var/cache/osbuild-worker"] <unfinished ...>
 :
30458 08:18:26.343814 <... execve resumed>) = 0 <0.001381>
~~~

-> The directory is created and then ro-bind mounted inside the container buildroot, operation is successful:
~~~
30460 08:18:26.957158 mkdir("/run/osbuild/osbuild-buildroot-i93xziyi/run/osbuild/lib", 0755) = 0 <0.000182>
30460 08:18:26.957453 mount("/usr/lib/osbuild", "/run/osbuild/osbuild-buildroot-i93xziyi/run/osbuild/lib", NULL, MS_BIND|MS_REC, NULL) = 0 <0.000077>
 :
30460 08:18:26.961791 mount(NULL, "/run/osbuild/osbuild-buildroot-i93xziyi/run/osbuild/lib", NULL, MS_RDONLY|MS_NODEV|MS_REMOUNT|MS_BIND, NULL) = 0 <0.000050>
~~~

-> Later the execve() call returns an ENOENT (No such file or directory):
~~~
30462 08:18:27.179575 execve("/run/osbuild/lib/runners/org.osbuild.rhel83", ["/run/osbuild/lib/runners/org.osbuild.rhel83", "/run/osbuild/lib/stages/org.osbuild.rpm"], ["PATH=/usr/local/sbin
:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "container=systemd-nspawn", "HOME=/root", "USER=root", "LOGNAME=root", "container_uuid=d6d2e999-ea30-4f36-8f98-da174162d4a3", "NOTIFY_SOCKET=/
run/systemd/nspawn/notify", "PYTHONPATH=/run/osbuild/lib"] <unfinished ...>
30462 08:18:27.180047 <... execve resumed>) = -1 ENOENT (No such file or directory) <0.000361>
 :
30462 08:18:27.181970 +++ exited with 1 +++
~~~

By investigating further with systemtap (SIGSTOP raised when the failed execve() occurred), we noticed the following issue:
~~~
# ls -l /proc/64104/root/run/osbuild/lib/runners/org.osbuild.rhel83
-rwxr-xr-x. 1 root root 2042 Jun 23  2020 /proc/64104/root/run/osbuild/lib/runners/org.osbuild.rhel83
# ls -l /proc/64104/root/run/usr/libexec/platform-python*
ls: cannot access '/proc/64104/root/run/usr/libexec/platform-python*': No such file or directory
~~~

Most of the OS is not present, not only python bins & libs.
Currently, I don't see any workaround except reinstalling a system with /var and /usr merged into the same rootfs.

Version-Release number of selected component (if applicable):
osbuild-18-3.el8
osbuild-composer-20.1-1.el8
osbuild-composer-worker-20.1-1.el8
python3-osbuild-18-3.el8

How reproducible:
100%

Steps to Reproduce:
1. Create a fresh RHEL 8.3 with a separate /usr and /var
2. Create a minimal blueprint and try to create an image.

Comment 2 Christian Kellner 2021-06-19 14:41:29 UTC
osbuild internally is not using systemd-nspawn anymore and I tried to verify that the problem still exists. On a Fedora 34 machine:

TARGET                       SOURCE               FSTYPE     OPTIONS
/                            /dev/mapper/f34-root xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
├─/proc                      proc                 proc       rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/fs/binfmt_misc systemd-1            autofs     rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=823
├─/sys                       sysfs                sysfs      rw,nosuid,nodev,noexec,relatime,seclabel
 ...
├─/dev                       devtmpfs             devtmpfs   rw,nosuid,seclabel,size=1987880k,nr_inodes=496970,mode=755,inode64
│ ├─/dev/shm                 tmpfs                tmpfs      rw,nosuid,nodev,seclabel,inode64
│ ├─/dev/pts                 devpts               devpts     rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000
│ ├─/dev/hugepages           hugetlbfs            hugetlbfs  rw,relatime,seclabel,pagesize=2M
│ └─/dev/mqueue              mqueue               mqueue     rw,nosuid,nodev,noexec,relatime,seclabel
├─/run                       tmpfs                tmpfs      rw,nosuid,nodev,seclabel,size=802620k,nr_inodes=819200,mode=755,inode64
│ └─/run/user/1000           tmpfs                tmpfs      rw,nosuid,nodev,relatime,seclabel,size=401308k,nr_inodes=100327,mode=700,uid=1000,gid=1000,inode64
├─/tmp                       tmpfs                tmpfs      rw,nosuid,nodev,seclabel,size=2006544k,nr_inodes=409600,inode64
├─/boot                      /dev/sda1            xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
├─/var                       /dev/mapper/f34-var  xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota
│ └─/var/lib/nfs/rpc_pipefs  sunrpc               rpc_pipefs rw,relatime
...


I did 

 sudo osbuild test/data/manifests/fedora-boot.json --store /var/cache/osbuild --output-directory /var/tmp/osbuild

which worked fine. I tried to reproduce with composer but that is broken on F34 right now. I wonder if that issues was solved by switching to bubblewrap from systemd-nspawn.

Comment 6 Christophe Besson 2021-06-21 10:14:31 UTC
Unfortunately, it seems that the use of bwrap leads to the same issue.
Using the latest el 8.4 pkgs:

# rpm -qa | grep osbuild
osbuild-composer-core-28.4-1.el8.x86_64
osbuild-ostree-27.2-1.el8.noarch
osbuild-composer-28.4-1.el8.x86_64
python3-osbuild-27.2-1.el8.noarch
osbuild-27.2-1.el8.noarch
osbuild-composer-worker-28.4-1.el8.x86_64
osbuild-selinux-27.2-1.el8.noarch

# composer-cli compose log 4e289c8b-66a1-4b9d-9455-2df8e6774c9c | tail -n4
Output:
bwrap: execvp /run/osbuild/lib/runners/org.osbuild.rhel84: No such file or directory

I didn't test with a 8.5 nightly build but I guess the issue remains the same.

Comment 7 Christian Kellner 2021-06-21 10:22:14 UTC
@Christophe, thanks for checking. Could you post the mount flags for `/var`? I wonder what is different from my tests.

Comment 8 Christophe Besson 2021-06-21 10:33:11 UTC
I just made a fresh 8.4 install, defaults are used for both XFS partitions.

# grep -e rhel-var -e rhel-usr /etc/fstab 
/dev/mapper/rhel-usr    /usr                    xfs     defaults        0 0
/dev/mapper/rhel-var    /var                    xfs     defaults        0 0

# mount | grep -e rhel-var -e rhel-usr
/dev/mapper/rhel-usr on /usr type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/mapper/rhel-var on /var type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)

# blkid | grep -e rhel-var -e rhel-usr
/dev/mapper/rhel-usr: UUID="37ab8d46-fd5d-4d95-9ac6-43b97dccb0a9" BLOCK_SIZE="512" TYPE="xfs"
/dev/mapper/rhel-var: UUID="80d91e34-84df-4464-ba1d-6e1f370228e2" BLOCK_SIZE="512" TYPE="xfs"

Comment 9 Christian Kellner 2021-06-21 17:02:33 UTC
Ah, `/usr` and `/var` are on separate partitions. I tried with just `/var` on a 8.4 install and that worked fine. Let me retry with both separate.

Comment 10 Christian Kellner 2021-06-21 21:17:48 UTC
We use a read only bind mount to pass the host's root to the build root setup code, and use `--bind` but *not* the recursive version (`--rbind`), this we get the root mount but not the `/usr` sub-mount. Either we just pass the root directly or if we want to keep that read only "protection" we can use the following patch:

modified   osbuild/objectstore.py
@@ -19,14 +19,15 @@ __all__ = [
 
 def mount(source, target, bind=True, ro=True, private=True, mode="0755"):
     options = []
-    if bind:
-        options += ["bind"]
     if ro:
         options += ["ro"]
     if mode:
         options += [mode]
 
     args = []
+
+    if bind:
+        args += ["--rbind"]
     if private:
         args += ["--make-private"]
     if options:

I will file a PR tomorrow. Sorry it took so long to look at this.

Comment 17 errata-xmlrpc 2021-11-09 18:46:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (osbuild bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4273


Note You need to log in before you can comment on or make changes to this bug.