Bug 1802907
Summary: | useradd and groupadd fail under rootless Buildah and podman | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 8 | Reporter: | andrew |
Component: | fuse-overlayfs | Assignee: | Jindrich Novy <jnovy> |
Status: | CLOSED ERRATA | QA Contact: | atomic-bugs <atomic-bugs> |
Severity: | medium | Docs Contact: | |
Priority: | unspecified | ||
Version: | 8.2 | CC: | adingman, ajia, andrew, andrew.white, castedo, c.handel, ddarrah, dgallowa, dornelas, dwalsh, ewout.ros, haegele, jnovy, jonathan.a.callen, kryadov, lsm5, mheon, mjtrangoni, nnachefski, ocasalsa, timo.sandmann, tsweeney |
Target Milestone: | rc | Keywords: | ZStream |
Target Release: | 8.2 | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | container-tools-rhel8-8020020200219144344.0d58ad57 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2020-04-28 15:53:22 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1803492, 1803493, 1803494, 1803495, 1803496 | ||
Bug Blocks: | 1186913, 1734579 |
Description
andrew
2020-02-14 04:01:35 UTC
Same thing happens just trying to use useradd or groupadd in podman: $ podman unshare cat /proc/self/uid_map 0 575900000 1 1 100000000 65537 $ podman run -it --name userbad --rm ubi8/ubi /bin/bash -l [root@9cb056aa6668 /]# ls -l /etc/passwd.lock ls: cannot access '/etc/passwd.lock': No such file or directory [root@9cb056aa6668 /]# useradd -r myservice useradd: /etc/passwd.19: lock file already used useradd: cannot lock /etc/passwd; try again later. [root@9cb056aa6668 /]# cat /etc/passwd.lock 19[root@9cb056aa6668 /]# [root@9cb056aa6668 /]# rm -f /etc/passwd.lock [root@9cb056aa6668 /]# useradd user useradd: /etc/passwd.22: lock file already used useradd: cannot lock /etc/passwd; try again later. [root@9cb056aa6668 /]# rm -f /etc/passwd.lock [root@9cb056aa6668 /]# ls -l /etc/group.lock ls: cannot access '/etc/group.lock': No such file or directory [root@9cb056aa6668 /]# groupadd -r myservice groupadd: /etc/group.25: lock file already used groupadd: cannot lock /etc/group; try again later. [root@9cb056aa6668 /]# cat /etc/group.lock 25[root@9cb056aa6668 /]# [root@9cb056aa6668 /]# rm /etc/group.lock rm: remove regular file '/etc/group.lock'? y [root@9cb056aa6668 /]# groupadd myservice groupadd: /etc/group.28: lock file already used groupadd: cannot lock /etc/group; try again later. [root@9cb056aa6668 /]# rm -f /etc/group.lock [root@9cb056aa6668 /]# logout [root@kvm-04-guest01 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.1 (Ootpa) [root@kvm-04-guest01 ~]# yum module install container-tools:rhel8 ...ignore... [root@kvm-04-guest01 ~]# rpm -q buildah podman slirp4netns buildah-1.11.6-4.module+el8.1.1+5259+bcdd613a.x86_64 podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64 slirp4netns-0.4.2-2.git21fdece.module+el8.1.1+5460+3ac089c3.x86_64 [root@kvm-04-guest01 ~]# buildah bud -t sleepy:rhel8.1 sleepy-container STEP 1: FROM registry.access.redhat.com/ubi8/ubi Getting image source signatures Copying blob ff6f434a470a done Copying blob eae5d284042d done Copying config fd73e6738a done Writing manifest to image destination Storing signatures STEP 2: USER root STEP 3: ENV SRVUSR=sleepy STEP 4: ENV SRVGRP=sleepy STEP 5: LABEL maintainer="Me <me>" STEP 6: LABEL description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman" STEP 7: LABEL io.k8s.description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman" STEP 8: LABEL io.k8s.dispaly-name="User fail on RHEL UBI 8" STEP 9: ENV YUM="yum -y --disablerepo=rhel*" STEP 10: ADD sleepy /usr/local/bin STEP 11: ADD sleepy.conf /etc STEP 12: RUN groupadd -r ${SRVGRP} || true STEP 13: RUN useradd -r -g ${SRVGRP} ${SRVUSR} || true STEP 14: RUN chmod +x /usr/local/bin/sleepy || true STEP 15: RUN chown root:${SRVGRP} /etc/sleepy.conf ; chmod 640 /etc/sleepy.conf || true STEP 16: RUN mkdir /var/local/sleepy STEP 17: RUN chmod 2750 /var/local/sleepy && chown ${SRVUSR}:${SRVGRP} /var/local/sleepy || true STEP 18: RUN ${YUM} install procps-ng iproute nmap-ncat Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode. This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. No repository match: rhel* Red Hat Universal Base Image 8 (RPMs) - BaseOS 1.0 MB/s | 760 kB 00:00 Red Hat Universal Base Image 8 (RPMs) - AppStream 4.3 MB/s | 3.3 MB 00:00 Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder 12 kB/s | 9.1 kB 00:00 Dependencies resolved. =================================================================================================================== Package Architecture Version Repository Size =================================================================================================================== Installing: iproute x86_64 4.18.0-15.el8 ubi-8-baseos 616 k procps-ng x86_64 3.3.15-1.el8 ubi-8-baseos 328 k nmap-ncat x86_64 2:7.70-5.el8 ubi-8-appstream 237 k Installing dependencies: libmnl x86_64 1.0.4-6.el8 ubi-8-baseos 30 k Transaction Summary =================================================================================================================== Install 4 Packages Total download size: 1.2 M Installed size: 3.7 M Downloading Packages: (1/4): libmnl-1.0.4-6.el8.x86_64.rpm 71 kB/s | 30 kB 00:00 (2/4): nmap-ncat-7.70-5.el8.x86_64.rpm 1.6 MB/s | 237 kB 00:00 (3/4): procps-ng-3.3.15-1.el8.x86_64.rpm 523 kB/s | 328 kB 00:00 (4/4): iproute-4.18.0-15.el8.x86_64.rpm 857 kB/s | 616 kB 00:00 ------------------------------------------------------------------------------------------------------------------- Total 1.6 MB/s | 1.2 MB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : libmnl-1.0.4-6.el8.x86_64 1/4 Running scriptlet: libmnl-1.0.4-6.el8.x86_64 1/4 Installing : iproute-4.18.0-15.el8.x86_64 2/4 Installing : nmap-ncat-2:7.70-5.el8.x86_64 3/4 Running scriptlet: nmap-ncat-2:7.70-5.el8.x86_64 3/4 Installing : procps-ng-3.3.15-1.el8.x86_64 4/4 Running scriptlet: procps-ng-3.3.15-1.el8.x86_64 4/4 Verifying : iproute-4.18.0-15.el8.x86_64 1/4 Verifying : libmnl-1.0.4-6.el8.x86_64 2/4 Verifying : procps-ng-3.3.15-1.el8.x86_64 3/4 Verifying : nmap-ncat-2:7.70-5.el8.x86_64 4/4 Installed products updated. Installed: iproute-4.18.0-15.el8.x86_64 procps-ng-3.3.15-1.el8.x86_64 nmap-ncat-2:7.70-5.el8.x86_64 libmnl-1.0.4-6.el8.x86_64 Complete! STEP 19: USER ${SRVUSR} STEP 20: ENTRYPOINT ["/usr/local/bin/sleepy"] STEP 21: COMMIT sleepy:rhel8.1 Getting image source signatures Copying blob 1295eae54c9d skipped: already exists Copying blob 85f69e555a1b skipped: already exists Copying blob 9284486bbbf7 done Copying config 8a4801573a done Writing manifest to image destination Storing signatures 8a4801573a6320b6d425595f8e2bdbb62987230b3e4a9e3c35f9cd470cdda3b2 8a4801573a6320b6d425595f8e2bdbb62987230b3e4a9e3c35f9cd470cdda3b2 [root@kvm-04-guest01 ~]# podman run -dt --rm --name sleepy sleepy:rhel8.1 2f842de8facf62f64567f296a16ab267fae4aa593c6046afb66c7844ce63a196 [root@kvm-04-guest01 ~]# podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2f842de8facf localhost/sleepy:rhel8.1 4 minutes ago Up 4 minutes ago sleepy [root@kvm-04-guest01 ~]# podman exec -i 2f842de8facf ps auxf USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND sleepy 123 0.0 0.0 43956 3420 ? Rs 08:20 0:00 ps auxf sleepy 1 0.0 0.0 11888 2904 pts/0 Ss+ 08:15 0:00 /bin/bash /usr/local/bin/sleepy sleepy 122 0.0 0.0 23028 1312 pts/0 S+ 08:20 0:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 5 (In reply to Alex Jia from comment #2) > > [root@kvm-04-guest01 ~]# buildah bud -t sleepy:rhel8.1 sleepy-container Yes. But the reproduction steps call for doing that as non-root. It works fine as root. Seeing the component change: $ rpm -q fuse-overlayfs fuse-overlayfs-0.7.2-1.module+el8.1.1+5259+bcdd613a.x86_64 it seems there is an issue in FUSE on RHEL 8.1. I've opened a PR to workaround the issue: https://github.com/containers/fuse-overlayfs/pull/184 I think we need to backport the patch as soon as possible. *** Bug 1804782 has been marked as a duplicate of this bug. *** Verified in fuse-overlayfs-0.7.2-2.module+el8.2.0+5768+3759792f.x86_64 w/ buildah-1.11.6-6.module+el8.2.0+5764+2729184f.x86_64 and podman-1.6.4-5.module+el8.2.0+5795+9bd98c8c.x86_64. [ajia@hpe-dl380pgen8-02-vm-5 ~]$ cat /etc/redhat-release Red Hat Enterprise Linux release 8.2 Beta (Ootpa) [ajia@hpe-dl380pgen8-02-vm-5 ~]$ rpm -q podman buildah fuse-overlayfs podman-1.6.4-5.module+el8.2.0+5795+9bd98c8c.x86_64 buildah-1.11.6-6.module+el8.2.0+5764+2729184f.x86_64 fuse-overlayfs-0.7.2-2.module+el8.2.0+5768+3759792f.x86_64 [ajia@hpe-dl380pgen8-02-vm-5 ~]$ id uid=1001(ajia) gid=1001(ajia) groups=1001(ajia) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [ajia@hpe-dl380pgen8-02-vm-5 ~]$ podman unshare cat /proc/self/uid_map 0 1001 1 1 165536 65536 [ajia@hpe-dl380pgen8-02-vm-5 ~]$ git clone https://gitlab.com/acdingman/sleepy-container.git Cloning into 'sleepy-container'... remote: Enumerating objects: 15, done. remote: Counting objects: 100% (15/15), done. remote: Compressing objects: 100% (11/11), done. remote: Total 15 (delta 3), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (15/15), done. [ajia@hpe-dl380pgen8-02-vm-5 ~]$ buildah bud -t sleepy:rhel8.1 sleepy-container STEP 1: FROM registry.access.redhat.com/ubi8/ubi Getting image source signatures Copying blob eae5d284042d done Copying blob ff6f434a470a done Copying config fd73e6738a done Writing manifest to image destination Storing signatures STEP 2: USER root STEP 3: ENV SRVUSR=sleepy STEP 4: ENV SRVGRP=sleepy STEP 5: LABEL maintainer="Me <me>" STEP 6: LABEL description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman" STEP 7: LABEL io.k8s.description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman" STEP 8: LABEL io.k8s.dispaly-name="User fail on RHEL UBI 8" STEP 9: ENV YUM="yum -y --disablerepo=rhel*" STEP 10: ADD sleepy /usr/local/bin STEP 11: ADD sleepy.conf /etc STEP 12: RUN groupadd -r ${SRVGRP} || true STEP 13: RUN useradd -r -g ${SRVGRP} ${SRVUSR} || true STEP 14: RUN chmod +x /usr/local/bin/sleepy || true STEP 15: RUN chown root:${SRVGRP} /etc/sleepy.conf ; chmod 640 /etc/sleepy.conf || true STEP 16: RUN mkdir /var/local/sleepy STEP 17: RUN chmod 2750 /var/local/sleepy && chown ${SRVUSR}:${SRVGRP} /var/local/sleepy || true STEP 18: RUN ${YUM} install procps-ng iproute nmap-ncat Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode. This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. No repository match: rhel* Red Hat Universal Base Image 8 (RPMs) - BaseOS 738 kB/s | 760 kB 00:01 Red Hat Universal Base Image 8 (RPMs) - AppStream 2.6 MB/s | 3.3 MB 00:01 Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder 20 kB/s | 9.1 kB 00:00 Dependencies resolved. ============================================================================================================================================================================================================================================== Package Architecture Version Repository Size ============================================================================================================================================================================================================================================== Installing: iproute x86_64 4.18.0-15.el8 ubi-8-baseos 616 k procps-ng x86_64 3.3.15-1.el8 ubi-8-baseos 328 k nmap-ncat x86_64 2:7.70-5.el8 ubi-8-appstream 237 k Installing dependencies: libmnl x86_64 1.0.4-6.el8 ubi-8-baseos 30 k Transaction Summary ============================================================================================================================================================================================================================================== Install 4 Packages Total download size: 1.2 M Installed size: 3.7 M Downloading Packages: (1/4): libmnl-1.0.4-6.el8.x86_64.rpm 162 kB/s | 30 kB 00:00 (2/4): nmap-ncat-7.70-5.el8.x86_64.rpm 1.7 MB/s | 237 kB 00:00 (3/4): procps-ng-3.3.15-1.el8.x86_64.rpm 720 kB/s | 328 kB 00:00 (4/4): iproute-4.18.0-15.el8.x86_64.rpm 1.2 MB/s | 616 kB 00:00 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 2.4 MB/s | 1.2 MB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : libmnl-1.0.4-6.el8.x86_64 1/4 Running scriptlet: libmnl-1.0.4-6.el8.x86_64 1/4 Installing : iproute-4.18.0-15.el8.x86_64 2/4 Installing : nmap-ncat-2:7.70-5.el8.x86_64 3/4 Running scriptlet: nmap-ncat-2:7.70-5.el8.x86_64 3/4 Installing : procps-ng-3.3.15-1.el8.x86_64 4/4 Running scriptlet: procps-ng-3.3.15-1.el8.x86_64 4/4 Verifying : iproute-4.18.0-15.el8.x86_64 1/4 Verifying : libmnl-1.0.4-6.el8.x86_64 2/4 Verifying : procps-ng-3.3.15-1.el8.x86_64 3/4 Verifying : nmap-ncat-2:7.70-5.el8.x86_64 4/4 Installed products updated. Installed: iproute-4.18.0-15.el8.x86_64 procps-ng-3.3.15-1.el8.x86_64 nmap-ncat-2:7.70-5.el8.x86_64 libmnl-1.0.4-6.el8.x86_64 Complete! STEP 19: USER ${SRVUSR} STEP 20: ENTRYPOINT ["/usr/local/bin/sleepy"] STEP 21: COMMIT sleepy:rhel8.1 Getting image source signatures Copying blob 1295eae54c9d skipped: already exists Copying blob 85f69e555a1b skipped: already exists Copying blob 60cf732502e3 done Copying config c65d27c13a done Writing manifest to image destination Storing signatures c65d27c13ac51ab055d715eddf26175b3435dc5c5c2609ba8a56b4b263508545 c65d27c13ac51ab055d715eddf26175b3435dc5c5c2609ba8a56b4b263508545 [ajia@hpe-dl380pgen8-02-vm-5 ~]$ podman run -dt --rm --name sleepy sleepy:rhel8.1 b206faf5514dc33590cb0ff3dd295511ed534c4223f379575c397f2688ed48b1 So the fix for this is in fuse-overlayfs-0.7.2-2 ? Any word on when this package will be available for RHEL7? Is it possible that you can direct me to a dev package i can test with? This is blocking a tekton/openshift-pipelines project i am working on. Latest package for RHEL7.7 is fuse-overlayfs-0.7.2-1.el7.x86_64 Jindrich, can you answer the question from Nicholas (https://bugzilla.redhat.com/show_bug.cgi?id=1802907#c14) please? Confirm that this problem still exists in Openshift 4.4 pre-release. RHCOS images = rhcos-4.4.0-0.nightly-2020-02-25-155201-x86_64 OCP release = 4.4.0-0.nightly-2020-03-06-030852 Lokesh or Jindrich, can you confirm that the fix was in the releases noted by Nicholas in https://bugzilla.redhat.com/show_bug.cgi?id=1802907#c16 Also verified in fuse-overlayfs-0.7.2-4.module+el8.2.0+5949+6277b64f.x86_64. [ajia@kvm-07-guest27 ~]$ rpm -q fuse-overlayfs buildah podman slirp4netns fuse-overlayfs-0.7.2-4.module+el8.2.0+5949+6277b64f.x86_64 buildah-1.11.6-6.module+el8.2.0+5855+8192c413.x86_64 podman-1.6.4-9.module+el8.2.0+5951+eb56bde6.x86_64 slirp4netns-0.4.2-3.git21fdece.module+el8.2.0+5655+72205bd1.x86_64 [ajia@kvm-07-guest27 ~]$ id uid=1001(ajia) gid=1001(ajia) groups=1001(ajia) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 [ajia@kvm-07-guest27 ~]$ buildah unshare cat /proc/self/uid_map 0 1001 1 1 165536 65536 [ajia@kvm-07-guest27 ~]$ buildah bud -t sleepy:rhel8.1 sleepy-container ...ignore... STEP 10: ADD sleepy /usr/local/bin STEP 11: ADD sleepy.conf /etc STEP 12: RUN groupadd -r ${SRVGRP} || true STEP 13: RUN useradd -r -g ${SRVGRP} ${SRVUSR} || true STEP 14: RUN chmod +x /usr/local/bin/sleepy || true ...ignore... [ajia@kvm-07-guest27 ~]$ podman run -dt --rm --name sleepy sleepy:rhel8.1 a8d97f5ed211d0468f585ab4b92c3e0cea87171068be8b21e74dd301ed161d23 [ajia@kvm-07-guest27 ~]$ podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a8d97f5ed211 localhost/sleepy:rhel8.1 3 seconds ago Up 3 seconds ago sleepy Also verified in buildah-1.11.6-7.module+el8.2.0+5856+b8046c6d.x86_64. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:1650 *** Bug 1807972 has been marked as a duplicate of this bug. *** |