RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1935922 - Unable to run multi-stage builds with rootless buildah container
Summary: Unable to run multi-stage builds with rootless buildah container
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: podman
Version: 8.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Jindrich Novy
QA Contact: Yuhui Jiang
URL:
Whiteboard:
Depends On:
Blocks: 1186913 1823899
TreeView+ depends on / blocked
 
Reported: 2021-03-05 19:47 UTC by Devon
Modified: 2022-11-02 16:15 UTC (History)
14 users (show)

Fixed In Version: podman-3.0.1-6.module+el8.4.0+10398+842aaf04 or newer
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-18 15:34:31 UTC
Type: Bug
Target Upstream Version:
Embargoed:
yujiang: needinfo-

Attachments (Terms of Use)
multi-stage go build output (23.32 KB, text/plain)
2021-03-05 19:47 UTC, Devon 		no flags Details
View All

Description Devon 2021-03-05 19:47:58 UTC 
Created attachment 1761033 [details]
multi-stage go build output

Description of problem:

Customer is trying to set up an environment using rootless buildah that allows them to utilize a rootless podman run of the build image and then start a multi-stage build process from within that running buildah container.

However it looks like we might be hitting an issue in creating that new namespace within the buildah container when running that multi-stage build but I am still not entirely sure that is the issue or if this is an actual limitation of the buildah container somewhere else.

I will go ahead and attach the build logs but this is the section of this build that leads me to believe this is the case.

DEBU bind mounted "/var/lib/containers/storage/vfs/dir/e3c2965ad501f55d81c44eef53283de185cdfc69eaf58c8864267e95e9c1af65" to "/var/tmp/buildah999829878/mnt/rootfs"
DEBU bind mounted "/var/lib/containers/storage/vfs-containers/2d76ab537ed83c5c5306faf872c277561a42e01e75402f5c67059b29ccac4dd2/userdata/run/secrets" to "/var/tmp/buildah999829878/mnt/buildah-bind-target-0"
DEBU bind mounted "/dev" to "/var/tmp/buildah999829878/mnt/rootfs/dev"
DEBU bind mounted "/proc" to "/var/tmp/buildah999829878/mnt/rootfs/proc"
WARN could not bind mount "/sys/kernel/security", skipping: no such file or directory
WARN error unmounting "/var/tmp/buildah999829878/mnt/rootfs/sys/fs/cgroup": permission denied
DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/debug" is not present(?), skipping
DEBU "/var/tmp/buildah999829878/mnt/rootfs/sys/fs/selinux" is apparently not really mounted, skipping
DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/config" is not present(?), skipping
DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/tracing" is not present(?), skipping
WARN error unmounting "/var/tmp/buildah999829878/mnt/rootfs/sys/fs/cgroup": permission denied
DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/security" is not present(?), skipping
WARN error unmounting "/var/tmp/buildah999829878/mnt/rootfs/sys": permission denied
DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/debug" is not present(?), skipping
DEBU "/var/tmp/buildah999829878/mnt/rootfs/sys/fs/selinux" is apparently not really mounted, skipping
DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/config" is not present(?), skipping
DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/tracing" is not present(?), skipping
WARN error unmounting "/var/tmp/buildah999829878/mnt/rootfs/sys/fs/cgroup": permission denied
DEBU mountpoint "/var/tmp/buildah999829878/mnt/rootfs/sys/kernel/security" is not present(?), skipping
WARN error unmounting "/var/tmp/buildah999829878/mnt/rootfs/sys": permission denied
WARN error unmounting "/var/tmp/buildah999829878/mnt/rootfs/proc": permission denied
error running subprocess: error remounting /var/tmp/buildah999829878/mnt/rootfs/sys/fs/cgroup/systemd in mount namespace read-only: permission denied


Version-Release number of selected component (if applicable):

# podman run registry.redhat.io/rhel8/buildah buildah version
Version:         1.15.1
Go Version:      go1.14.7
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:
image Version:   5.5.1
Git Commit:
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64


How reproducible:

Every time

Steps to Reproduce:
1. Run rootless buildah container
2. Start multi-stage build process of a container image
3. Hit issues as shown above remounting into the new namespace that buildah creates

Actual results:

Errors above

Expected results:

Complete the build process

Additional info:
Comment 1 Derrick Ornelas 2021-03-19 20:52:40 UTC 
I can reproduce this with podman-2.2.1-7.module+el8.3.1+9857+68fb1526 but not with podman-3.0.1-3.module+el8.4.0+10198+36d1d0e3
In both tests I used registry.redhat.io/rhel8/buildah:8.3-24 (which has buildah-1.16.7-4.module+el8.3.1+9857+68fb1526) and the following Containerfile:

  FROM registry.access.redhat.com/ubi8 AS test
  RUN mkdir -p /test/testdir/
  FROM test AS test2
  RUN touch /test/testdir/testfile


Reproducer:

[bob@podman2 ~]$ rpm -q podman
podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64


[bob@podman2 ~]$ podman run --pull=always --name buildah -it registry.redhat.io/rhel8/buildah /bin/bash
Trying to pull registry.redhat.io/rhel8/buildah:latest...
Getting image source signatures
Copying blob fdb393d8227c done  
Copying blob 01635593bb47 done  
Copying blob 6b536614e8f8 done  
Copying config dab33f863b done  
Writing manifest to image destination
Storing signatures


[root@2fbcdb81b9c3 /]# buildah bud -f Containerfile -t mytestimage .
STEP 1: FROM registry.access.redhat.com/ubi8 AS test
Getting image source signatures
Copying blob fdb393d8227c done  
Copying blob 6b536614e8f8 done  
Copying config 4199acc83c done  
Writing manifest to image destination
Storing signatures
STEP 2: RUN mkdir -p /test/testdir/
error running subprocess: error remounting /var/tmp/buildah821045532/mnt/rootfs/sys/fs/cgroup/systemd in mount namespace read-only: permission denied
                                                                                                                                                     error building at STEP "RUN mkdir -p /test/testdir/": exit status 1
ERRO exit status 1                                




Fixed in podman 3.0.1:

[user1@rhel84 ~]$ rpm -q podman
podman-3.0.1-3.module+el8.4.0+10198+36d1d0e3.x86_64


[user1@rhel84 ~]$ podman info
host:
  arch: amd64
  buildahVersion: 1.19.4
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.26-1.module+el8.4.0+10198+36d1d0e3.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: 0a5175681bdd52b99f1f0f442cbba8f8c126a1c9'
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "8.4"
  eventLogger: file
  hostname: rhel84
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-293.el8.x86_64
  linkmode: dynamic
  memFree: 3181555712
  memTotal: 4116942848
  ociRuntime:
    name: runc
    package: runc-1.0.0-70.rc92.module+el8.4.0+10198+36d1d0e3.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.2-dev'
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.4.0+10198+36d1d0e3.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 2147205120
  swapTotal: 2147479552
  uptime: 69h 18m 30.02s (Approximately 2.88 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/user1/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-2.module+el8.4.0+10198+36d1d0e3.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.4
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/user1/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/user1/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 1614697806
  BuiltTime: Tue Mar  2 10:10:06 2021
  GitCommit: ""
  GoVersion: go1.15.7
  OsArch: linux/amd64
  Version: 3.0.2-dev


$ podman run --pull=always --name buildah -it registry.redhat.io/rhel8/buildah  /bin/bash
Trying to pull registry.redhat.io/rhel8/buildah:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 01635593bb47 done  
Copying blob fdb393d8227c done  
Copying blob 6b536614e8f8 done  
Copying config dab33f863b done  
Writing manifest to image destination
Storing signatures

[root@a5ba756ca5c6 /]# buildah info
{
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "\"rhel\"",
            "version": "8.3"
        },
        "MemFree": 2631962624,
        "MemTotal": 4116942848,
        "OCIRuntime": "runc",
        "SwapFree": 2147205120,
        "SwapTotal": 2147479552,
        "arch": "amd64",
        "cpus": 2,
        "hostname": "a5ba756ca5c6",
        "kernel": "4.18.0-293.el8.x86_64",
        "os": "linux",
        "rootless": true,
        "uptime": "69h 20m 24.65s (Approximately 2.88 days)"
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "vfs",
        "GraphOptions": null,
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {},
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/var/run/containers/storage"
    }
}


[root@a5ba756ca5c6 /]# buildah bud -f Containerfile -t mytestimage .
STEP 1: FROM registry.access.redhat.com/ubi8 AS test
Getting image source signatures
Copying blob fdb393d8227c done  
Copying blob 6b536614e8f8 done  
Copying config 4199acc83c done  
Writing manifest to image destination
Storing signatures
STEP 2: RUN mkdir -p /test/testdir/
Getting image source signatures
Copying blob 72e7d306c279 skipped: already exists  
Copying blob 9624be4353eb skipped: already exists  
Copying blob ddb6730e4e70 done  
Copying config e5d073da38 done  
Writing manifest to image destination
Storing signatures
--> e5d073da38d
STEP 3: FROM e5d073da38d33a9d95062e05c06d0079fd155ef9c08338c7b5882b02b957f9e5 AS test2
STEP 4: RUN touch /test/testdir/testfile
STEP 5: COMMIT mytestimage
Getting image source signatures
Copying blob 72e7d306c279 skipped: already exists  
Copying blob 9624be4353eb skipped: already exists  
Copying blob ddb6730e4e70 skipped: already exists  
Copying blob 1a1a3245e9ce done  
Copying config ebfc256b43 done  
Writing manifest to image destination
Storing signatures
--> ebfc256b434
ebfc256b434c921a7e65001a1fcb00c16d6ef8d0f011c1c7742ba404873325d3
Comment 2 Jindrich Novy 2021-03-22 07:43:25 UTC 
Devon, are you able to confirm it is fixed in podman-3.0.1-3 or newer - for which case it will be addressed in 8.4.0?

The latest podman build for 8.4.0 is attached in the following advisory: 

https://errata.devel.redhat.com/advisory/65330/builds
Comment 3 Devon 2021-03-25 18:57:01 UTC 
Looks to be working with podman-3.0.1-6.module+el8.4.0+10398+842aaf04.x86_64 exactly as Derrick was showing:

[root@d8 ~]# rpm -q podman
podman-2.2.1-8.el8.x86_64
[test@d8 ~]$  podman run  --name buildah -it registry.redhat.io/rhel8/buildah /bin/bash
[root@2c9fd4dab46a /]# vi multibuild
FROM registry.access.redhat.com/ubi8 AS test
RUN mkdir -p /test/testdir/
FROM test AS test2
RUN touch /test/testdir/testfile
[root@2c9fd4dab46a /]# buildah bud -f multibuild -t mytestimage .
STEP 1: FROM registry.access.redhat.com/ubi8 AS test
Getting image source signatures
Copying blob fdb393d8227c done
Copying blob 6b536614e8f8 done
Copying config 4199acc83c done
Writing manifest to image destination
Storing signatures
STEP 2: RUN mkdir -p /test/testdir/
error running subprocess: error remounting /var/tmp/buildah517723801/mnt/rootfs/sys/fs/cgroup/systemd in mount namespace read-only: permission denied
                                                                                                                                                     error building at STEP "RUN mkdir -p /test/testdir/": exit status 1
ERRO exit status 1




[root@d8 ~]# rpm -q podman
podman-3.0.1-6.module+el8.4.0+10398+842aaf04.x86_64
[test@d8 ~]$ podman run  --name buildah -it registry.redhat.io/rhel8/buildah /bin/bash
[root@30d5c77cb800 /]# vi multibuild
FROM registry.access.redhat.com/ubi8 AS test
RUN mkdir -p /test/testdir/
FROM test AS test2
RUN touch /test/testdir/testfile
[root@30d5c77cb800 /]# buildah bud -f multibuild -t mytestimage .
STEP 1: FROM registry.access.redhat.com/ubi8 AS test
Getting image source signatures
Copying blob fdb393d8227c done
Copying blob 6b536614e8f8 done
Copying config 4199acc83c done
Writing manifest to image destination
Storing signatures
STEP 2: RUN mkdir -p /test/testdir/
Getting image source signatures
Copying blob 72e7d306c279 skipped: already exists
Copying blob 9624be4353eb skipped: already exists
Copying blob 85a612a1e1da done
Copying config 3b6ab8ac7f done
Writing manifest to image destination
Storing signatures
--> 3b6ab8ac7f0
STEP 3: FROM 3b6ab8ac7f0c04ac5c3140e095bead202bb2cacc9de83f7ca2aa25cfae3016d1 AS test2
STEP 4: RUN touch /test/testdir/testfile
STEP 5: COMMIT mytestimage
Getting image source signatures
Copying blob 72e7d306c279 skipped: already exists
Copying blob 9624be4353eb skipped: already exists
Copying blob 85a612a1e1da skipped: already exists
Copying blob 77200b4587ba done
Copying config 04ecbf6a62 done
Writing manifest to image destination
Storing signatures
--> 04ecbf6a62f
04ecbf6a62fa182bfd89e4ccfb3bc0eda38e1ce8d62417f542792cebe866214c

Let me know if you need anything else from me.
Comment 15 errata-xmlrpc 2021-05-18 15:34:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1796
Note You need to log in before you can comment on or make changes to this bug.