Bug 1935922
| Summary: | Unable to run multi-stage builds with rootless buildah container | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Devon <dshumake> | ||||
| Component: | podman | Assignee: | Jindrich Novy <jnovy> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Yuhui Jiang <yujiang> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 8.3 | CC: | bbaude, ddarrah, dornelas, dwalsh, jligon, jnovy, lfriedma, lsm5, mheon, pthomas, salanis, umohnani, ypu, yujiang | ||||
| Target Milestone: | rc | Flags: | yujiang:
needinfo-
|
||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | podman-3.0.1-6.module+el8.4.0+10398+842aaf04 or newer | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-05-18 15:34:31 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1186913, 1823899 | ||||||
| Attachments: |
|
||||||
I can reproduce this with podman-2.2.1-7.module+el8.3.1+9857+68fb1526 but not with podman-3.0.1-3.module+el8.4.0+10198+36d1d0e3
In both tests I used registry.redhat.io/rhel8/buildah:8.3-24 (which has buildah-1.16.7-4.module+el8.3.1+9857+68fb1526) and the following Containerfile:
FROM registry.access.redhat.com/ubi8 AS test
RUN mkdir -p /test/testdir/
FROM test AS test2
RUN touch /test/testdir/testfile
Reproducer:
[bob@podman2 ~]$ rpm -q podman
podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64
[bob@podman2 ~]$ podman run --pull=always --name buildah -it registry.redhat.io/rhel8/buildah /bin/bash
Trying to pull registry.redhat.io/rhel8/buildah:latest...
Getting image source signatures
Copying blob fdb393d8227c done
Copying blob 01635593bb47 done
Copying blob 6b536614e8f8 done
Copying config dab33f863b done
Writing manifest to image destination
Storing signatures
[root@2fbcdb81b9c3 /]# buildah bud -f Containerfile -t mytestimage .
STEP 1: FROM registry.access.redhat.com/ubi8 AS test
Getting image source signatures
Copying blob fdb393d8227c done
Copying blob 6b536614e8f8 done
Copying config 4199acc83c done
Writing manifest to image destination
Storing signatures
STEP 2: RUN mkdir -p /test/testdir/
error running subprocess: error remounting /var/tmp/buildah821045532/mnt/rootfs/sys/fs/cgroup/systemd in mount namespace read-only: permission denied
error building at STEP "RUN mkdir -p /test/testdir/": exit status 1
ERRO exit status 1
Fixed in podman 3.0.1:
[user1@rhel84 ~]$ rpm -q podman
podman-3.0.1-3.module+el8.4.0+10198+36d1d0e3.x86_64
[user1@rhel84 ~]$ podman info
host:
arch: amd64
buildahVersion: 1.19.4
cgroupManager: cgroupfs
cgroupVersion: v1
conmon:
package: conmon-2.0.26-1.module+el8.4.0+10198+36d1d0e3.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.0.26, commit: 0a5175681bdd52b99f1f0f442cbba8f8c126a1c9'
cpus: 2
distribution:
distribution: '"rhel"'
version: "8.4"
eventLogger: file
hostname: rhel84
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 4.18.0-293.el8.x86_64
linkmode: dynamic
memFree: 3181555712
memTotal: 4116942848
ociRuntime:
name: runc
package: runc-1.0.0-70.rc92.module+el8.4.0+10198+36d1d0e3.x86_64
path: /usr/bin/runc
version: 'runc version spec: 1.0.2-dev'
os: linux
remoteSocket:
path: /run/user/1000/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
selinuxEnabled: true
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.1.8-1.module+el8.4.0+10198+36d1d0e3.x86_64
version: |-
slirp4netns version 1.1.8
commit: d361001f495417b880f20329121e3aa431a8f90f
libslirp: 4.3.1
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.1
swapFree: 2147205120
swapTotal: 2147479552
uptime: 69h 18m 30.02s (Approximately 2.88 days)
registries:
search:
- registry.access.redhat.com
- registry.redhat.io
- docker.io
store:
configFile: /home/user1/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.mount_program:
Executable: /usr/bin/fuse-overlayfs
Package: fuse-overlayfs-1.4.0-2.module+el8.4.0+10198+36d1d0e3.x86_64
Version: |-
fusermount3 version: 3.2.1
fuse-overlayfs: version 1.4
FUSE library version 3.2.1
using FUSE kernel interface version 7.26
graphRoot: /home/user1/.local/share/containers/storage
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 0
runRoot: /run/user/1000/containers
volumePath: /home/user1/.local/share/containers/storage/volumes
version:
APIVersion: 3.0.0
Built: 1614697806
BuiltTime: Tue Mar 2 10:10:06 2021
GitCommit: ""
GoVersion: go1.15.7
OsArch: linux/amd64
Version: 3.0.2-dev
$ podman run --pull=always --name buildah -it registry.redhat.io/rhel8/buildah /bin/bash
Trying to pull registry.redhat.io/rhel8/buildah:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 01635593bb47 done
Copying blob fdb393d8227c done
Copying blob 6b536614e8f8 done
Copying config dab33f863b done
Writing manifest to image destination
Storing signatures
[root@a5ba756ca5c6 /]# buildah info
{
"host": {
"CgroupVersion": "v1",
"Distribution": {
"distribution": "\"rhel\"",
"version": "8.3"
},
"MemFree": 2631962624,
"MemTotal": 4116942848,
"OCIRuntime": "runc",
"SwapFree": 2147205120,
"SwapTotal": 2147479552,
"arch": "amd64",
"cpus": 2,
"hostname": "a5ba756ca5c6",
"kernel": "4.18.0-293.el8.x86_64",
"os": "linux",
"rootless": true,
"uptime": "69h 20m 24.65s (Approximately 2.88 days)"
},
"store": {
"ContainerStore": {
"number": 0
},
"GraphDriverName": "vfs",
"GraphOptions": null,
"GraphRoot": "/var/lib/containers/storage",
"GraphStatus": {},
"ImageStore": {
"number": 0
},
"RunRoot": "/var/run/containers/storage"
}
}
[root@a5ba756ca5c6 /]# buildah bud -f Containerfile -t mytestimage .
STEP 1: FROM registry.access.redhat.com/ubi8 AS test
Getting image source signatures
Copying blob fdb393d8227c done
Copying blob 6b536614e8f8 done
Copying config 4199acc83c done
Writing manifest to image destination
Storing signatures
STEP 2: RUN mkdir -p /test/testdir/
Getting image source signatures
Copying blob 72e7d306c279 skipped: already exists
Copying blob 9624be4353eb skipped: already exists
Copying blob ddb6730e4e70 done
Copying config e5d073da38 done
Writing manifest to image destination
Storing signatures
--> e5d073da38d
STEP 3: FROM e5d073da38d33a9d95062e05c06d0079fd155ef9c08338c7b5882b02b957f9e5 AS test2
STEP 4: RUN touch /test/testdir/testfile
STEP 5: COMMIT mytestimage
Getting image source signatures
Copying blob 72e7d306c279 skipped: already exists
Copying blob 9624be4353eb skipped: already exists
Copying blob ddb6730e4e70 skipped: already exists
Copying blob 1a1a3245e9ce done
Copying config ebfc256b43 done
Writing manifest to image destination
Storing signatures
--> ebfc256b434
ebfc256b434c921a7e65001a1fcb00c16d6ef8d0f011c1c7742ba404873325d3
Devon, are you able to confirm it is fixed in podman-3.0.1-3 or newer - for which case it will be addressed in 8.4.0? The latest podman build for 8.4.0 is attached in the following advisory: https://errata.devel.redhat.com/advisory/65330/builds Looks to be working with podman-3.0.1-6.module+el8.4.0+10398+842aaf04.x86_64 exactly as Derrick was showing:
[root@d8 ~]# rpm -q podman
podman-2.2.1-8.el8.x86_64
[test@d8 ~]$ podman run --name buildah -it registry.redhat.io/rhel8/buildah /bin/bash
[root@2c9fd4dab46a /]# vi multibuild
FROM registry.access.redhat.com/ubi8 AS test
RUN mkdir -p /test/testdir/
FROM test AS test2
RUN touch /test/testdir/testfile
[root@2c9fd4dab46a /]# buildah bud -f multibuild -t mytestimage .
STEP 1: FROM registry.access.redhat.com/ubi8 AS test
Getting image source signatures
Copying blob fdb393d8227c done
Copying blob 6b536614e8f8 done
Copying config 4199acc83c done
Writing manifest to image destination
Storing signatures
STEP 2: RUN mkdir -p /test/testdir/
error running subprocess: error remounting /var/tmp/buildah517723801/mnt/rootfs/sys/fs/cgroup/systemd in mount namespace read-only: permission denied
error building at STEP "RUN mkdir -p /test/testdir/": exit status 1
ERRO exit status 1
[root@d8 ~]# rpm -q podman
podman-3.0.1-6.module+el8.4.0+10398+842aaf04.x86_64
[test@d8 ~]$ podman run --name buildah -it registry.redhat.io/rhel8/buildah /bin/bash
[root@30d5c77cb800 /]# vi multibuild
FROM registry.access.redhat.com/ubi8 AS test
RUN mkdir -p /test/testdir/
FROM test AS test2
RUN touch /test/testdir/testfile
[root@30d5c77cb800 /]# buildah bud -f multibuild -t mytestimage .
STEP 1: FROM registry.access.redhat.com/ubi8 AS test
Getting image source signatures
Copying blob fdb393d8227c done
Copying blob 6b536614e8f8 done
Copying config 4199acc83c done
Writing manifest to image destination
Storing signatures
STEP 2: RUN mkdir -p /test/testdir/
Getting image source signatures
Copying blob 72e7d306c279 skipped: already exists
Copying blob 9624be4353eb skipped: already exists
Copying blob 85a612a1e1da done
Copying config 3b6ab8ac7f done
Writing manifest to image destination
Storing signatures
--> 3b6ab8ac7f0
STEP 3: FROM 3b6ab8ac7f0c04ac5c3140e095bead202bb2cacc9de83f7ca2aa25cfae3016d1 AS test2
STEP 4: RUN touch /test/testdir/testfile
STEP 5: COMMIT mytestimage
Getting image source signatures
Copying blob 72e7d306c279 skipped: already exists
Copying blob 9624be4353eb skipped: already exists
Copying blob 85a612a1e1da skipped: already exists
Copying blob 77200b4587ba done
Copying config 04ecbf6a62 done
Writing manifest to image destination
Storing signatures
--> 04ecbf6a62f
04ecbf6a62fa182bfd89e4ccfb3bc0eda38e1ce8d62417f542792cebe866214c
Let me know if you need anything else from me.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:1796 |
Created attachment 1761033 [details] multi-stage go build output Description of problem: Customer is trying to set up an environment using rootless buildah that allows them to utilize a rootless podman run of the build image and then start a multi-stage build process from within that running buildah container. However it looks like we might be hitting an issue in creating that new namespace within the buildah container when running that multi-stage build but I am still not entirely sure that is the issue or if this is an actual limitation of the buildah container somewhere else. I will go ahead and attach the build logs but this is the section of this build that leads me to believe this is the case. DEBU bind mounted "/var/lib/containers/storage/vfs/dir/e3c2965ad501f55d81c44eef53283de185cdfc69eaf58c8864267e95e9c1af65" to "/var/tmp/buildah999829878/mnt/rootfs" DEBU bind mounted "/var/lib/containers/storage/vfs-containers/2d76ab537ed83c5c5306faf872c277561a42e01e75402f5c67059b29ccac4dd2/userdata/run/secrets" to "/var/tmp/buildah999829878/mnt/buildah-bind-target-0" DEBU bind mounted "/dev" to "/var/tmp/buildah999829878/mnt/rootfs/dev" DEBU bind mounted "/proc" to "/var/tmp/buildah999829878/mnt/rootfs/proc" WARN could not bind mount "/sys/kernel/security", skipping: no such file or directory WARN error unmounting "/var/tmp/buildah999829878/mnt/rootfs/sys/fs/cgroup": permission denied DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/debug" is not present(?), skipping DEBU "/var/tmp/buildah999829878/mnt/rootfs/sys/fs/selinux" is apparently not really mounted, skipping DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/config" is not present(?), skipping DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/tracing" is not present(?), skipping WARN error unmounting "/var/tmp/buildah999829878/mnt/rootfs/sys/fs/cgroup": permission denied DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/security" is not present(?), skipping WARN error unmounting "/var/tmp/buildah999829878/mnt/rootfs/sys": permission denied DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/debug" is not present(?), skipping DEBU "/var/tmp/buildah999829878/mnt/rootfs/sys/fs/selinux" is apparently not really mounted, skipping DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/config" is not present(?), skipping DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/tracing" is not present(?), skipping WARN error unmounting "/var/tmp/buildah999829878/mnt/rootfs/sys/fs/cgroup": permission denied DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/security" is not present(?), skipping WARN error unmounting "/var/tmp/buildah999829878/mnt/rootfs/sys": permission denied WARN error unmounting "/var/tmp/buildah999829878/mnt/rootfs/proc": permission denied error running subprocess: error remounting /var/tmp/buildah999829878/mnt/rootfs/sys/fs/cgroup/systemd in mount namespace read-only: permission denied Version-Release number of selected component (if applicable): # podman run registry.redhat.io/rhel8/buildah buildah version Version: 1.15.1 Go Version: go1.14.7 Image Spec: 1.0.1-dev Runtime Spec: 1.0.2-dev CNI Spec: 0.4.0 libcni Version: image Version: 5.5.1 Git Commit: Built: Thu Jan 1 00:00:00 1970 OS/Arch: linux/amd64 How reproducible: Every time Steps to Reproduce: 1. Run rootless buildah container 2. Start multi-stage build process of a container image 3. Hit issues as shown above remounting into the new namespace that buildah creates Actual results: Errors above Expected results: Complete the build process Additional info: