Bug 1802907
| Summary: | useradd and groupadd fail under rootless Buildah and podman | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | andrew |
| Component: | fuse-overlayfs | Assignee: | Jindrich Novy <jnovy> |
| Status: | CLOSED ERRATA | QA Contact: | atomic-bugs <atomic-bugs> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.2 | CC: | adingman, ajia, andrew, andrew.white, castedo, c.handel, ddarrah, dgallowa, dornelas, dwalsh, ewout.ros, haegele, jnovy, jonathan.a.callen, kryadov, lsm5, mheon, mjtrangoni, nnachefski, ocasalsa, timo.sandmann, tsweeney |
| Target Milestone: | rc | Keywords: | ZStream |
| Target Release: | 8.2 | Flags: | pm-rhel:
mirror+
|
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | container-tools-rhel8-8020020200219144344.0d58ad57 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-04-28 15:53:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1803492, 1803493, 1803494, 1803495, 1803496 | ||
| Bug Blocks: | 1186913, 1734579 | ||
|
Description
andrew
2020-02-14 04:01:35 UTC
Same thing happens just trying to use useradd or groupadd in podman:
$ podman unshare cat /proc/self/uid_map
0 575900000 1
1 100000000 65537
$ podman run -it --name userbad --rm ubi8/ubi /bin/bash -l
[root@9cb056aa6668 /]# ls -l /etc/passwd.lock
ls: cannot access '/etc/passwd.lock': No such file or directory
[root@9cb056aa6668 /]# useradd -r myservice
useradd: /etc/passwd.19: lock file already used
useradd: cannot lock /etc/passwd; try again later.
[root@9cb056aa6668 /]# cat /etc/passwd.lock
19[root@9cb056aa6668 /]#
[root@9cb056aa6668 /]# rm -f /etc/passwd.lock
[root@9cb056aa6668 /]# useradd user
useradd: /etc/passwd.22: lock file already used
useradd: cannot lock /etc/passwd; try again later.
[root@9cb056aa6668 /]# rm -f /etc/passwd.lock
[root@9cb056aa6668 /]# ls -l /etc/group.lock
ls: cannot access '/etc/group.lock': No such file or directory
[root@9cb056aa6668 /]# groupadd -r myservice
groupadd: /etc/group.25: lock file already used
groupadd: cannot lock /etc/group; try again later.
[root@9cb056aa6668 /]# cat /etc/group.lock
25[root@9cb056aa6668 /]#
[root@9cb056aa6668 /]# rm /etc/group.lock
rm: remove regular file '/etc/group.lock'? y
[root@9cb056aa6668 /]# groupadd myservice
groupadd: /etc/group.28: lock file already used
groupadd: cannot lock /etc/group; try again later.
[root@9cb056aa6668 /]# rm -f /etc/group.lock
[root@9cb056aa6668 /]# logout
[root@kvm-04-guest01 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.1 (Ootpa)
[root@kvm-04-guest01 ~]# yum module install container-tools:rhel8
...ignore...
[root@kvm-04-guest01 ~]# rpm -q buildah podman slirp4netns
buildah-1.11.6-4.module+el8.1.1+5259+bcdd613a.x86_64
podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64
slirp4netns-0.4.2-2.git21fdece.module+el8.1.1+5460+3ac089c3.x86_64
[root@kvm-04-guest01 ~]# buildah bud -t sleepy:rhel8.1 sleepy-container
STEP 1: FROM registry.access.redhat.com/ubi8/ubi
Getting image source signatures
Copying blob ff6f434a470a done
Copying blob eae5d284042d done
Copying config fd73e6738a done
Writing manifest to image destination
Storing signatures
STEP 2: USER root
STEP 3: ENV SRVUSR=sleepy
STEP 4: ENV SRVGRP=sleepy
STEP 5: LABEL maintainer="Me <me>"
STEP 6: LABEL description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman"
STEP 7: LABEL io.k8s.description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman"
STEP 8: LABEL io.k8s.dispaly-name="User fail on RHEL UBI 8"
STEP 9: ENV YUM="yum -y --disablerepo=rhel*"
STEP 10: ADD sleepy /usr/local/bin
STEP 11: ADD sleepy.conf /etc
STEP 12: RUN groupadd -r ${SRVGRP} || true
STEP 13: RUN useradd -r -g ${SRVGRP} ${SRVUSR} || true
STEP 14: RUN chmod +x /usr/local/bin/sleepy || true
STEP 15: RUN chown root:${SRVGRP} /etc/sleepy.conf ; chmod 640 /etc/sleepy.conf || true
STEP 16: RUN mkdir /var/local/sleepy
STEP 17: RUN chmod 2750 /var/local/sleepy && chown ${SRVUSR}:${SRVGRP} /var/local/sleepy || true
STEP 18: RUN ${YUM} install procps-ng iproute nmap-ncat
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
No repository match: rhel*
Red Hat Universal Base Image 8 (RPMs) - BaseOS 1.0 MB/s | 760 kB 00:00
Red Hat Universal Base Image 8 (RPMs) - AppStream 4.3 MB/s | 3.3 MB 00:00
Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder 12 kB/s | 9.1 kB 00:00
Dependencies resolved.
===================================================================================================================
Package Architecture Version Repository Size
===================================================================================================================
Installing:
iproute x86_64 4.18.0-15.el8 ubi-8-baseos 616 k
procps-ng x86_64 3.3.15-1.el8 ubi-8-baseos 328 k
nmap-ncat x86_64 2:7.70-5.el8 ubi-8-appstream 237 k
Installing dependencies:
libmnl x86_64 1.0.4-6.el8 ubi-8-baseos 30 k
Transaction Summary
===================================================================================================================
Install 4 Packages
Total download size: 1.2 M
Installed size: 3.7 M
Downloading Packages:
(1/4): libmnl-1.0.4-6.el8.x86_64.rpm 71 kB/s | 30 kB 00:00
(2/4): nmap-ncat-7.70-5.el8.x86_64.rpm 1.6 MB/s | 237 kB 00:00
(3/4): procps-ng-3.3.15-1.el8.x86_64.rpm 523 kB/s | 328 kB 00:00
(4/4): iproute-4.18.0-15.el8.x86_64.rpm 857 kB/s | 616 kB 00:00
-------------------------------------------------------------------------------------------------------------------
Total 1.6 MB/s | 1.2 MB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : libmnl-1.0.4-6.el8.x86_64 1/4
Running scriptlet: libmnl-1.0.4-6.el8.x86_64 1/4
Installing : iproute-4.18.0-15.el8.x86_64 2/4
Installing : nmap-ncat-2:7.70-5.el8.x86_64 3/4
Running scriptlet: nmap-ncat-2:7.70-5.el8.x86_64 3/4
Installing : procps-ng-3.3.15-1.el8.x86_64 4/4
Running scriptlet: procps-ng-3.3.15-1.el8.x86_64 4/4
Verifying : iproute-4.18.0-15.el8.x86_64 1/4
Verifying : libmnl-1.0.4-6.el8.x86_64 2/4
Verifying : procps-ng-3.3.15-1.el8.x86_64 3/4
Verifying : nmap-ncat-2:7.70-5.el8.x86_64 4/4
Installed products updated.
Installed:
iproute-4.18.0-15.el8.x86_64 procps-ng-3.3.15-1.el8.x86_64 nmap-ncat-2:7.70-5.el8.x86_64
libmnl-1.0.4-6.el8.x86_64
Complete!
STEP 19: USER ${SRVUSR}
STEP 20: ENTRYPOINT ["/usr/local/bin/sleepy"]
STEP 21: COMMIT sleepy:rhel8.1
Getting image source signatures
Copying blob 1295eae54c9d skipped: already exists
Copying blob 85f69e555a1b skipped: already exists
Copying blob 9284486bbbf7 done
Copying config 8a4801573a done
Writing manifest to image destination
Storing signatures
8a4801573a6320b6d425595f8e2bdbb62987230b3e4a9e3c35f9cd470cdda3b2
8a4801573a6320b6d425595f8e2bdbb62987230b3e4a9e3c35f9cd470cdda3b2
[root@kvm-04-guest01 ~]# podman run -dt --rm --name sleepy sleepy:rhel8.1
2f842de8facf62f64567f296a16ab267fae4aa593c6046afb66c7844ce63a196
[root@kvm-04-guest01 ~]# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2f842de8facf localhost/sleepy:rhel8.1 4 minutes ago Up 4 minutes ago sleepy
[root@kvm-04-guest01 ~]# podman exec -i 2f842de8facf ps auxf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
sleepy 123 0.0 0.0 43956 3420 ? Rs 08:20 0:00 ps auxf
sleepy 1 0.0 0.0 11888 2904 pts/0 Ss+ 08:15 0:00 /bin/bash /usr/local/bin/sleepy
sleepy 122 0.0 0.0 23028 1312 pts/0 S+ 08:20 0:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 5
(In reply to Alex Jia from comment #2) > > [root@kvm-04-guest01 ~]# buildah bud -t sleepy:rhel8.1 sleepy-container Yes. But the reproduction steps call for doing that as non-root. It works fine as root. Seeing the component change: $ rpm -q fuse-overlayfs fuse-overlayfs-0.7.2-1.module+el8.1.1+5259+bcdd613a.x86_64 it seems there is an issue in FUSE on RHEL 8.1. I've opened a PR to workaround the issue: https://github.com/containers/fuse-overlayfs/pull/184 I think we need to backport the patch as soon as possible. *** Bug 1804782 has been marked as a duplicate of this bug. *** Verified in fuse-overlayfs-0.7.2-2.module+el8.2.0+5768+3759792f.x86_64 w/
buildah-1.11.6-6.module+el8.2.0+5764+2729184f.x86_64 and
podman-1.6.4-5.module+el8.2.0+5795+9bd98c8c.x86_64.
[ajia@hpe-dl380pgen8-02-vm-5 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.2 Beta (Ootpa)
[ajia@hpe-dl380pgen8-02-vm-5 ~]$ rpm -q podman buildah fuse-overlayfs
podman-1.6.4-5.module+el8.2.0+5795+9bd98c8c.x86_64
buildah-1.11.6-6.module+el8.2.0+5764+2729184f.x86_64
fuse-overlayfs-0.7.2-2.module+el8.2.0+5768+3759792f.x86_64
[ajia@hpe-dl380pgen8-02-vm-5 ~]$ id
uid=1001(ajia) gid=1001(ajia) groups=1001(ajia) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[ajia@hpe-dl380pgen8-02-vm-5 ~]$ podman unshare cat /proc/self/uid_map
0 1001 1
1 165536 65536
[ajia@hpe-dl380pgen8-02-vm-5 ~]$ git clone https://gitlab.com/acdingman/sleepy-container.git
Cloning into 'sleepy-container'...
remote: Enumerating objects: 15, done.
remote: Counting objects: 100% (15/15), done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 15 (delta 3), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (15/15), done.
[ajia@hpe-dl380pgen8-02-vm-5 ~]$ buildah bud -t sleepy:rhel8.1 sleepy-container
STEP 1: FROM registry.access.redhat.com/ubi8/ubi
Getting image source signatures
Copying blob eae5d284042d done
Copying blob ff6f434a470a done
Copying config fd73e6738a done
Writing manifest to image destination
Storing signatures
STEP 2: USER root
STEP 3: ENV SRVUSR=sleepy
STEP 4: ENV SRVGRP=sleepy
STEP 5: LABEL maintainer="Me <me>"
STEP 6: LABEL description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman"
STEP 7: LABEL io.k8s.description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman"
STEP 8: LABEL io.k8s.dispaly-name="User fail on RHEL UBI 8"
STEP 9: ENV YUM="yum -y --disablerepo=rhel*"
STEP 10: ADD sleepy /usr/local/bin
STEP 11: ADD sleepy.conf /etc
STEP 12: RUN groupadd -r ${SRVGRP} || true
STEP 13: RUN useradd -r -g ${SRVGRP} ${SRVUSR} || true
STEP 14: RUN chmod +x /usr/local/bin/sleepy || true
STEP 15: RUN chown root:${SRVGRP} /etc/sleepy.conf ; chmod 640 /etc/sleepy.conf || true
STEP 16: RUN mkdir /var/local/sleepy
STEP 17: RUN chmod 2750 /var/local/sleepy && chown ${SRVUSR}:${SRVGRP} /var/local/sleepy || true
STEP 18: RUN ${YUM} install procps-ng iproute nmap-ncat
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
No repository match: rhel*
Red Hat Universal Base Image 8 (RPMs) - BaseOS 738 kB/s | 760 kB 00:01
Red Hat Universal Base Image 8 (RPMs) - AppStream 2.6 MB/s | 3.3 MB 00:01
Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder 20 kB/s | 9.1 kB 00:00
Dependencies resolved.
==============================================================================================================================================================================================================================================
Package Architecture Version Repository Size
==============================================================================================================================================================================================================================================
Installing:
iproute x86_64 4.18.0-15.el8 ubi-8-baseos 616 k
procps-ng x86_64 3.3.15-1.el8 ubi-8-baseos 328 k
nmap-ncat x86_64 2:7.70-5.el8 ubi-8-appstream 237 k
Installing dependencies:
libmnl x86_64 1.0.4-6.el8 ubi-8-baseos 30 k
Transaction Summary
==============================================================================================================================================================================================================================================
Install 4 Packages
Total download size: 1.2 M
Installed size: 3.7 M
Downloading Packages:
(1/4): libmnl-1.0.4-6.el8.x86_64.rpm 162 kB/s | 30 kB 00:00
(2/4): nmap-ncat-7.70-5.el8.x86_64.rpm 1.7 MB/s | 237 kB 00:00
(3/4): procps-ng-3.3.15-1.el8.x86_64.rpm 720 kB/s | 328 kB 00:00
(4/4): iproute-4.18.0-15.el8.x86_64.rpm 1.2 MB/s | 616 kB 00:00
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 2.4 MB/s | 1.2 MB 00:00
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : libmnl-1.0.4-6.el8.x86_64 1/4
Running scriptlet: libmnl-1.0.4-6.el8.x86_64 1/4
Installing : iproute-4.18.0-15.el8.x86_64 2/4
Installing : nmap-ncat-2:7.70-5.el8.x86_64 3/4
Running scriptlet: nmap-ncat-2:7.70-5.el8.x86_64 3/4
Installing : procps-ng-3.3.15-1.el8.x86_64 4/4
Running scriptlet: procps-ng-3.3.15-1.el8.x86_64 4/4
Verifying : iproute-4.18.0-15.el8.x86_64 1/4
Verifying : libmnl-1.0.4-6.el8.x86_64 2/4
Verifying : procps-ng-3.3.15-1.el8.x86_64 3/4
Verifying : nmap-ncat-2:7.70-5.el8.x86_64 4/4
Installed products updated.
Installed:
iproute-4.18.0-15.el8.x86_64 procps-ng-3.3.15-1.el8.x86_64 nmap-ncat-2:7.70-5.el8.x86_64 libmnl-1.0.4-6.el8.x86_64
Complete!
STEP 19: USER ${SRVUSR}
STEP 20: ENTRYPOINT ["/usr/local/bin/sleepy"]
STEP 21: COMMIT sleepy:rhel8.1
Getting image source signatures
Copying blob 1295eae54c9d skipped: already exists
Copying blob 85f69e555a1b skipped: already exists
Copying blob 60cf732502e3 done
Copying config c65d27c13a done
Writing manifest to image destination
Storing signatures
c65d27c13ac51ab055d715eddf26175b3435dc5c5c2609ba8a56b4b263508545
c65d27c13ac51ab055d715eddf26175b3435dc5c5c2609ba8a56b4b263508545
[ajia@hpe-dl380pgen8-02-vm-5 ~]$ podman run -dt --rm --name sleepy sleepy:rhel8.1
b206faf5514dc33590cb0ff3dd295511ed534c4223f379575c397f2688ed48b1
So the fix for this is in fuse-overlayfs-0.7.2-2 ? Any word on when this package will be available for RHEL7? Is it possible that you can direct me to a dev package i can test with? This is blocking a tekton/openshift-pipelines project i am working on. Latest package for RHEL7.7 is fuse-overlayfs-0.7.2-1.el7.x86_64 Jindrich, can you answer the question from Nicholas (https://bugzilla.redhat.com/show_bug.cgi?id=1802907#c14) please? Confirm that this problem still exists in Openshift 4.4 pre-release. RHCOS images = rhcos-4.4.0-0.nightly-2020-02-25-155201-x86_64 OCP release = 4.4.0-0.nightly-2020-03-06-030852 Lokesh or Jindrich, can you confirm that the fix was in the releases noted by Nicholas in https://bugzilla.redhat.com/show_bug.cgi?id=1802907#c16 Also verified in fuse-overlayfs-0.7.2-4.module+el8.2.0+5949+6277b64f.x86_64.
[ajia@kvm-07-guest27 ~]$ rpm -q fuse-overlayfs buildah podman slirp4netns
fuse-overlayfs-0.7.2-4.module+el8.2.0+5949+6277b64f.x86_64
buildah-1.11.6-6.module+el8.2.0+5855+8192c413.x86_64
podman-1.6.4-9.module+el8.2.0+5951+eb56bde6.x86_64
slirp4netns-0.4.2-3.git21fdece.module+el8.2.0+5655+72205bd1.x86_64
[ajia@kvm-07-guest27 ~]$ id
uid=1001(ajia) gid=1001(ajia) groups=1001(ajia) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[ajia@kvm-07-guest27 ~]$ buildah unshare cat /proc/self/uid_map
0 1001 1
1 165536 65536
[ajia@kvm-07-guest27 ~]$ buildah bud -t sleepy:rhel8.1 sleepy-container
...ignore...
STEP 10: ADD sleepy /usr/local/bin
STEP 11: ADD sleepy.conf /etc
STEP 12: RUN groupadd -r ${SRVGRP} || true
STEP 13: RUN useradd -r -g ${SRVGRP} ${SRVUSR} || true
STEP 14: RUN chmod +x /usr/local/bin/sleepy || true
...ignore...
[ajia@kvm-07-guest27 ~]$ podman run -dt --rm --name sleepy sleepy:rhel8.1
a8d97f5ed211d0468f585ab4b92c3e0cea87171068be8b21e74dd301ed161d23
[ajia@kvm-07-guest27 ~]$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a8d97f5ed211 localhost/sleepy:rhel8.1 3 seconds ago Up 3 seconds ago sleepy
Also verified in buildah-1.11.6-7.module+el8.2.0+5856+b8046c6d.x86_64. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:1650 *** Bug 1807972 has been marked as a duplicate of this bug. *** |