1802907 – useradd and groupadd fail under rootless Buildah and podman

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1802907 - useradd and groupadd fail under rootless Buildah and podman

Summary: useradd and groupadd fail under rootless Buildah and podman

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	fuse-overlayfs
Sub Component:
Version:	8.2
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	medium
Target Milestone:	rc
Target Release:	8.2
Assignee:	Jindrich Novy
QA Contact:	atomic-bugs@redhat.com
Docs Contact:
URL:
Whiteboard:
Duplicates (2):	1804782 1807972 (view as bug list)
Depends On:	1803492 1803493 1803494 1803495 1803496
Blocks:	1186913 1734579
TreeView+	depends on / blocked

Reported:	2020-02-14 04:01 UTC by andrew
Modified:	2023-09-07 21:51 UTC (History)
CC List:	22 users (show)
Fixed In Version:	container-tools-rhel8-8020020200219144344.0d58ad57
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-04-28 15:53:22 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	RHELPLAN-37455	0	None	None	None	2023-02-12 20:45:53 UTC
Red Hat Product Errata	RHSA-2020:1650	0	None	None	None	2020-04-28 15:54:06 UTC

Internal Links: 2053164

Description andrew 2020-02-14 04:01:35 UTC

Description of problem: useradd fails in rootless containers under both Buildah and Podman


Version-Release number of selected component (if applicable):
$ rpm -q buildah podman
buildah-1.11.6-4.module+el8.1.1+5259+bcdd613a.x86_64
podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64


How reproducible: always


Steps to Reproduce:
1. $ git clone https://gitlab.com/acdingman/sleepy-container.git
2. $ buildah bud -t sleepy:rhel8.1 sleepy-container
3. Note the errors thrown from groupadd, useradd, and chown commands
4. $ podman run -dt --rm --name sleepy sleepy:rhel8.1
5. Note the error trying to use an entrypoint to a non-existent user


Actual results:
$ buildah bud -t sleepy:rhel8.1 sleepy-container
STEP 1: FROM registry.access.redhat.com/ubi8/ubi
Getting image source signatures
Copying blob ff6f434a470a done
Copying blob eae5d284042d done
Copying config fd73e6738a done
Writing manifest to image destination
Storing signatures
STEP 2: USER root
STEP 3: ENV SRVUSR=sleepy
STEP 4: ENV SRVGRP=sleepy
STEP 5: LABEL maintainer="Me <me>"
STEP 6: LABEL description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman"
STEP 7: LABEL io.k8s.description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman"
STEP 8: LABEL io.k8s.dispaly-name="User fail on RHEL UBI 8"
STEP 9: ENV YUM="yum -y --disablerepo=rhel*"
STEP 10: ADD sleepy /usr/local/bin
STEP 11: ADD sleepy.conf /etc
STEP 12: RUN groupadd -r ${SRVGRP} || true
groupadd: /etc/group.7: lock file already used
groupadd: cannot lock /etc/group; try again later.
STEP 13: RUN useradd -r -g ${SRVGRP} ${SRVUSR} || true
useradd: group 'sleepy' does not exist
STEP 14: RUN chmod +x /usr/local/bin/sleepy || true
STEP 15: RUN chown root:${SRVGRP} /etc/sleepy.conf ; chmod 640 /etc/sleepy.conf || true
chown: invalid group: 'root:sleepy'
STEP 16: RUN mkdir /var/local/sleepy
STEP 17: RUN chmod 2750 /var/local/sleepy && chown ${SRVUSR}:${SRVGRP} /var/local/sleepy || true
chown: invalid user: 'sleepy:sleepy'
STEP 18: RUN ${YUM} install procps-ng iproute nmap-ncat
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
Red Hat Universal Base Image 8 (RPMs) - BaseOS                           2.6 MB/s | 760 kB     00:00    
Red Hat Universal Base Image 8 (RPMs) - AppStream                        6.0 MB/s | 3.3 MB     00:00    
Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder                 43 kB/s | 9.1 kB     00:00    
Dependencies resolved.
=========================================================================================================
 Package               Architecture       Version                      Repository                   Size
=========================================================================================================
Installing:
 iproute               x86_64             4.18.0-15.el8                ubi-8-baseos                616 k
 procps-ng             x86_64             3.3.15-1.el8                 ubi-8-baseos                328 k
 nmap-ncat             x86_64             2:7.70-5.el8                 ubi-8-appstream             237 k
Installing dependencies:
 libmnl                x86_64             1.0.4-6.el8                  ubi-8-baseos                 30 k

Transaction Summary
=========================================================================================================
Install  4 Packages

Total download size: 1.2 M
Installed size: 3.7 M
Downloading Packages:
(1/4): libmnl-1.0.4-6.el8.x86_64.rpm                                     269 kB/s |  30 kB     00:00    
(2/4): procps-ng-3.3.15-1.el8.x86_64.rpm                                 2.0 MB/s | 328 kB     00:00    
(3/4): nmap-ncat-7.70-5.el8.x86_64.rpm                                   4.9 MB/s | 237 kB     00:00    
(4/4): iproute-4.18.0-15.el8.x86_64.rpm                                  3.4 MB/s | 616 kB     00:00    
---------------------------------------------------------------------------------------------------------
Total                                                                    6.6 MB/s | 1.2 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                 1/1 
  Installing       : libmnl-1.0.4-6.el8.x86_64                                                       1/4 
  Running scriptlet: libmnl-1.0.4-6.el8.x86_64                                                       1/4 
  Installing       : iproute-4.18.0-15.el8.x86_64                                                    2/4 
  Installing       : nmap-ncat-2:7.70-5.el8.x86_64                                                   3/4 
  Running scriptlet: nmap-ncat-2:7.70-5.el8.x86_64                                                   3/4 
  Installing       : procps-ng-3.3.15-1.el8.x86_64                                                   4/4 
  Running scriptlet: procps-ng-3.3.15-1.el8.x86_64                                                   4/4 
  Verifying        : iproute-4.18.0-15.el8.x86_64                                                    1/4 
  Verifying        : libmnl-1.0.4-6.el8.x86_64                                                       2/4 
  Verifying        : procps-ng-3.3.15-1.el8.x86_64                                                   3/4 
  Verifying        : nmap-ncat-2:7.70-5.el8.x86_64                                                   4/4 
Installed products updated.

Installed:
  iproute-4.18.0-15.el8.x86_64      procps-ng-3.3.15-1.el8.x86_64      nmap-ncat-2:7.70-5.el8.x86_64     
  libmnl-1.0.4-6.el8.x86_64        

Complete!
STEP 19: USER ${SRVUSR}
STEP 20: ENTRYPOINT ["/usr/local/bin/sleepy"]
STEP 21: COMMIT sleepy:rhel8.1
Getting image source signatures
Copying blob 1295eae54c9d skipped: already exists
Copying blob 85f69e555a1b skipped: already exists
Copying blob af653e3312c9 done
Copying config f3945fa772 done
Writing manifest to image destination
Storing signatures
f3945fa7720370230cea4be754d777466e02b5f360879038542a6e2bb7adda9c
f3945fa7720370230cea4be754d777466e02b5f360879038542a6e2bb7adda9c
$ podman run -dt --rm --name sleepy sleepy:rhel8.1
Error: unable to find user sleepy: no matching entries in passwd file
$ podman ps -a
CONTAINER ID  IMAGE                     COMMAND  CREATED         STATUS   PORTS  NAMES
16b74bca0aca  localhost/sleepy:rhel8.1           22 seconds ago  Created         sleepy
$ podman rm sleepy
16b74bca0acae61c707c224b3a703414391b8a42c3e2fcf33a53bfdfb4c6ca19
$ podman run -dtu root --rm --name sleepy sleepy:rhel8.1
4fc60299906f465cda02fb22ff621b1953902d6ac9b9209b5ddfb6ab46808750
[andrew@kallirhoe-rhel8 ~]$ podman exec -it sleepy ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  1.8  0.1  11888  2816 pts/0    Ss+  03:30   0:00 /bin/bash /usr/local/bin/sleepy
root        13  0.0  0.2  43956  3312 pts/1    Rs+  03:31   0:00 ps aux
root        20  0.0  0.0  23028  1328 pts/0    S+   03:31   0:00 /usr/bin/coreutils --coreutils-prog-sheb

Expected results:
$ buildah bud -t sleepy:f31 sleepy
STEP 1: FROM registry.access.redhat.com/ubi8/ubi
STEP 2: USER root
STEP 3: ENV SRVUSR=sleepy
STEP 4: ENV SRVGRP=sleepy
STEP 5: LABEL maintainer="Me <me>"
STEP 6: LABEL description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman"
STEP 7: LABEL io.k8s.description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman"
STEP 8: LABEL io.k8s.dispaly-name="User fail on RHEL UBI 8"
STEP 9: ENV YUM="yum -y --disablerepo=rhel*"
STEP 10: ADD sleepy /usr/local/bin
STEP 11: ADD sleepy.conf /etc
STEP 12: RUN groupadd -r ${SRVGRP} || true
STEP 13: RUN useradd -r -g ${SRVGRP} ${SRVUSR} || true
STEP 14: RUN chmod +x /usr/local/bin/sleepy || true
STEP 15: RUN chown root:${SRVGRP} /etc/sleepy.conf ; chmod 640 /etc/sleepy.conf || true
STEP 16: RUN mkdir /var/local/sleepy
STEP 17: RUN chmod 2750 /var/local/sleepy && chown ${SRVUSR}:${SRVGRP} /var/local/sleepy || true
STEP 18: RUN ${YUM} install procps-ng iproute nmap-ncat
Updating Subscription Management repositories.
Unable to read consumer identity
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
No repository match: rhel*
Red Hat Universal Base Image 8 (RPMs) - BaseOS                           1.3 MB/s | 760 kB     00:00    
Red Hat Universal Base Image 8 (RPMs) - AppStream                        4.9 MB/s | 3.3 MB     00:00    
Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder                 11 kB/s | 9.1 kB     00:00    
Dependencies resolved.
=========================================================================================================
 Package               Architecture       Version                      Repository                   Size
=========================================================================================================
Installing:
 iproute               x86_64             4.18.0-15.el8                ubi-8-baseos                616 k
 procps-ng             x86_64             3.3.15-1.el8                 ubi-8-baseos                328 k
 nmap-ncat             x86_64             2:7.70-5.el8                 ubi-8-appstream             237 k
Installing dependencies:
 libmnl                x86_64             1.0.4-6.el8                  ubi-8-baseos                 30 k

Transaction Summary
=========================================================================================================
Install  4 Packages

Total download size: 1.2 M
Installed size: 3.7 M
Downloading Packages:
(1/4): libmnl-1.0.4-6.el8.x86_64.rpm                                      89 kB/s |  30 kB     00:00    
(2/4): iproute-4.18.0-15.el8.x86_64.rpm                                  1.6 MB/s | 616 kB     00:00    
(3/4): procps-ng-3.3.15-1.el8.x86_64.rpm                                 833 kB/s | 328 kB     00:00    
(4/4): nmap-ncat-7.70-5.el8.x86_64.rpm                                   2.3 MB/s | 237 kB     00:00    
---------------------------------------------------------------------------------------------------------
Total                                                                    2.7 MB/s | 1.2 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                 1/1 
  Installing       : libmnl-1.0.4-6.el8.x86_64                                                       1/4 
  Running scriptlet: libmnl-1.0.4-6.el8.x86_64                                                       1/4 
  Installing       : iproute-4.18.0-15.el8.x86_64                                                    2/4 
  Installing       : nmap-ncat-2:7.70-5.el8.x86_64                                                   3/4 
  Running scriptlet: nmap-ncat-2:7.70-5.el8.x86_64                                                   3/4 
  Installing       : procps-ng-3.3.15-1.el8.x86_64                                                   4/4 
  Running scriptlet: procps-ng-3.3.15-1.el8.x86_64                                                   4/4 
  Verifying        : iproute-4.18.0-15.el8.x86_64                                                    1/4 
  Verifying        : libmnl-1.0.4-6.el8.x86_64                                                       2/4 
  Verifying        : procps-ng-3.3.15-1.el8.x86_64                                                   3/4 
  Verifying        : nmap-ncat-2:7.70-5.el8.x86_64                                                   4/4 
Installed products updated.

Installed:
  iproute-4.18.0-15.el8.x86_64      procps-ng-3.3.15-1.el8.x86_64      nmap-ncat-2:7.70-5.el8.x86_64     
  libmnl-1.0.4-6.el8.x86_64        

Complete!
STEP 19: USER ${SRVUSR}
STEP 20: ENTRYPOINT ["/usr/local/bin/sleepy"]
STEP 21: COMMIT sleepy:f31
Getting image source signatures
Copying blob 1295eae54c9d skipped: already exists
Copying blob 85f69e555a1b skipped: already exists
Copying blob 7838c87b9405 done
Copying config 9e9fb9e524 done
Writing manifest to image destination
Storing signatures
9e9fb9e5244f7569bafd56abc2375986b524b98918a2fbb3ad104bbfdae0b0ce
9e9fb9e5244f7569bafd56abc2375986b524b98918a2fbb3ad104bbfdae0b0ce
$ podman run -dt --rm --name sleepy sleepy:f31
dfd25514a1ad862359714159c25b591bb44cf668698650c99d5e9877220d7ff9
$ podman exec -it sleepy ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
sleepy         1  0.1  0.0  11884  2848 pts/0    Ss+  03:25   0:00 /bin/bash /usr/local/bin/sleepy
sleepy         6  0.0  0.0  23024  1388 pts/0    S+   03:25   0:00 /usr/bin/coreutils --coreutils-prog-sh
sleepy         7  0.0  0.0  43952  3328 pts/1    Rs+  03:25   0:00 ps aux


Additional info:
Expected results are copied and pasted from a Fedora 31 system, but a very similar Dockerfile was working on my RHEL 8 system before I updated to 8.1 on Wednesday. I can not share the actual Dockerfiles for those, but they impact service user creation in three container image builds for two proprietary applications and one Apache Software Foundation application.

Comment 1 andrew 2020-02-14 04:16:49 UTC

Same thing happens just trying to use useradd or groupadd in podman:

$ podman unshare cat /proc/self/uid_map 
         0  575900000          1
         1  100000000      65537
$ podman run -it --name userbad --rm ubi8/ubi /bin/bash -l
[root@9cb056aa6668 /]# ls -l /etc/passwd.lock
ls: cannot access '/etc/passwd.lock': No such file or directory
[root@9cb056aa6668 /]# useradd -r myservice
useradd: /etc/passwd.19: lock file already used
useradd: cannot lock /etc/passwd; try again later.
[root@9cb056aa6668 /]# cat /etc/passwd.lock
19[root@9cb056aa6668 /]# 
[root@9cb056aa6668 /]# rm -f /etc/passwd.lock
[root@9cb056aa6668 /]# useradd user
useradd: /etc/passwd.22: lock file already used
useradd: cannot lock /etc/passwd; try again later.
[root@9cb056aa6668 /]# rm -f /etc/passwd.lock
[root@9cb056aa6668 /]# ls -l /etc/group.lock
ls: cannot access '/etc/group.lock': No such file or directory
[root@9cb056aa6668 /]# groupadd -r myservice
groupadd: /etc/group.25: lock file already used
groupadd: cannot lock /etc/group; try again later.
[root@9cb056aa6668 /]# cat /etc/group.lock
25[root@9cb056aa6668 /]# 
[root@9cb056aa6668 /]# rm /etc/group.lock
rm: remove regular file '/etc/group.lock'? y
[root@9cb056aa6668 /]# groupadd myservice
groupadd: /etc/group.28: lock file already used
groupadd: cannot lock /etc/group; try again later.
[root@9cb056aa6668 /]# rm -f /etc/group.lock
[root@9cb056aa6668 /]# logout

Comment 2 Alex Jia 2020-02-14 08:56:32 UTC


[root@kvm-04-guest01 ~]# cat  /etc/redhat-release 
Red Hat Enterprise Linux release 8.1 (Ootpa)

[root@kvm-04-guest01 ~]# yum module install container-tools:rhel8
...ignore...

[root@kvm-04-guest01 ~]# rpm -q buildah podman slirp4netns
buildah-1.11.6-4.module+el8.1.1+5259+bcdd613a.x86_64
podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64
slirp4netns-0.4.2-2.git21fdece.module+el8.1.1+5460+3ac089c3.x86_64

[root@kvm-04-guest01 ~]# buildah bud -t sleepy:rhel8.1 sleepy-container
STEP 1: FROM registry.access.redhat.com/ubi8/ubi
Getting image source signatures
Copying blob ff6f434a470a done
Copying blob eae5d284042d done
Copying config fd73e6738a done
Writing manifest to image destination
Storing signatures
STEP 2: USER root
STEP 3: ENV SRVUSR=sleepy
STEP 4: ENV SRVGRP=sleepy
STEP 5: LABEL maintainer="Me <me>"
STEP 6: LABEL description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman"
STEP 7: LABEL io.k8s.description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman"
STEP 8: LABEL io.k8s.dispaly-name="User fail on RHEL UBI 8"
STEP 9: ENV YUM="yum -y --disablerepo=rhel*"
STEP 10: ADD sleepy /usr/local/bin
STEP 11: ADD sleepy.conf /etc
STEP 12: RUN groupadd -r ${SRVGRP} || true
STEP 13: RUN useradd -r -g ${SRVGRP} ${SRVUSR} || true
STEP 14: RUN chmod +x /usr/local/bin/sleepy || true
STEP 15: RUN chown root:${SRVGRP} /etc/sleepy.conf ; chmod 640 /etc/sleepy.conf || true
STEP 16: RUN mkdir /var/local/sleepy
STEP 17: RUN chmod 2750 /var/local/sleepy && chown ${SRVUSR}:${SRVGRP} /var/local/sleepy || true
STEP 18: RUN ${YUM} install procps-ng iproute nmap-ncat
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
No repository match: rhel*
Red Hat Universal Base Image 8 (RPMs) - BaseOS                                     1.0 MB/s | 760 kB     00:00    
Red Hat Universal Base Image 8 (RPMs) - AppStream                                  4.3 MB/s | 3.3 MB     00:00    
Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder                           12 kB/s | 9.1 kB     00:00    
Dependencies resolved.
===================================================================================================================
 Package                  Architecture          Version                       Repository                      Size
===================================================================================================================
Installing:
 iproute                  x86_64                4.18.0-15.el8                 ubi-8-baseos                   616 k
 procps-ng                x86_64                3.3.15-1.el8                  ubi-8-baseos                   328 k
 nmap-ncat                x86_64                2:7.70-5.el8                  ubi-8-appstream                237 k
Installing dependencies:
 libmnl                   x86_64                1.0.4-6.el8                   ubi-8-baseos                    30 k

Transaction Summary
===================================================================================================================
Install  4 Packages

Total download size: 1.2 M
Installed size: 3.7 M
Downloading Packages:
(1/4): libmnl-1.0.4-6.el8.x86_64.rpm                                                71 kB/s |  30 kB     00:00    
(2/4): nmap-ncat-7.70-5.el8.x86_64.rpm                                             1.6 MB/s | 237 kB     00:00    
(3/4): procps-ng-3.3.15-1.el8.x86_64.rpm                                           523 kB/s | 328 kB     00:00    
(4/4): iproute-4.18.0-15.el8.x86_64.rpm                                            857 kB/s | 616 kB     00:00    
-------------------------------------------------------------------------------------------------------------------
Total                                                                              1.6 MB/s | 1.2 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                           1/1 
  Installing       : libmnl-1.0.4-6.el8.x86_64                                                                 1/4 
  Running scriptlet: libmnl-1.0.4-6.el8.x86_64                                                                 1/4 
  Installing       : iproute-4.18.0-15.el8.x86_64                                                              2/4 
  Installing       : nmap-ncat-2:7.70-5.el8.x86_64                                                             3/4 
  Running scriptlet: nmap-ncat-2:7.70-5.el8.x86_64                                                             3/4 
  Installing       : procps-ng-3.3.15-1.el8.x86_64                                                             4/4 
  Running scriptlet: procps-ng-3.3.15-1.el8.x86_64                                                             4/4 
  Verifying        : iproute-4.18.0-15.el8.x86_64                                                              1/4 
  Verifying        : libmnl-1.0.4-6.el8.x86_64                                                                 2/4 
  Verifying        : procps-ng-3.3.15-1.el8.x86_64                                                             3/4 
  Verifying        : nmap-ncat-2:7.70-5.el8.x86_64                                                             4/4 
Installed products updated.

Installed:
  iproute-4.18.0-15.el8.x86_64         procps-ng-3.3.15-1.el8.x86_64         nmap-ncat-2:7.70-5.el8.x86_64        
  libmnl-1.0.4-6.el8.x86_64           

Complete!
STEP 19: USER ${SRVUSR}
STEP 20: ENTRYPOINT ["/usr/local/bin/sleepy"]
STEP 21: COMMIT sleepy:rhel8.1
Getting image source signatures
Copying blob 1295eae54c9d skipped: already exists
Copying blob 85f69e555a1b skipped: already exists
Copying blob 9284486bbbf7 done
Copying config 8a4801573a done
Writing manifest to image destination
Storing signatures
8a4801573a6320b6d425595f8e2bdbb62987230b3e4a9e3c35f9cd470cdda3b2
8a4801573a6320b6d425595f8e2bdbb62987230b3e4a9e3c35f9cd470cdda3b2
[root@kvm-04-guest01 ~]# podman run -dt --rm --name sleepy sleepy:rhel8.1
2f842de8facf62f64567f296a16ab267fae4aa593c6046afb66c7844ce63a196

[root@kvm-04-guest01 ~]# podman ps
CONTAINER ID  IMAGE                     COMMAND  CREATED        STATUS            PORTS  NAMES
2f842de8facf  localhost/sleepy:rhel8.1           4 minutes ago  Up 4 minutes ago         sleepy

[root@kvm-04-guest01 ~]# podman exec -i 2f842de8facf ps auxf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
sleepy     123  0.0  0.0  43956  3420 ?        Rs   08:20   0:00 ps auxf
sleepy       1  0.0  0.0  11888  2904 pts/0    Ss+  08:15   0:00 /bin/bash /usr/local/bin/sleepy
sleepy     122  0.0  0.0  23028  1312 pts/0    S+   08:20   0:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep 5

Comment 3 andrew 2020-02-14 12:44:26 UTC

(In reply to Alex Jia from comment #2)

> 
> [root@kvm-04-guest01 ~]# buildah bud -t sleepy:rhel8.1 sleepy-container

Yes. But the reproduction steps call for doing that as non-root. It works fine as root.

Comment 4 andrew 2020-02-14 20:40:08 UTC

Seeing the component change:
$ rpm -q fuse-overlayfs 
fuse-overlayfs-0.7.2-1.module+el8.1.1+5259+bcdd613a.x86_64

Comment 5 Giuseppe Scrivano 2020-02-15 11:38:29 UTC

it seems there is an issue in FUSE on RHEL 8.1.

I've opened a PR to workaround the issue: https://github.com/containers/fuse-overlayfs/pull/184

I think we need to backport the patch as soon as possible.

Comment 8 Giuseppe Scrivano 2020-02-19 16:48:49 UTC

*** Bug 1804782 has been marked as a duplicate of this bug. ***

Comment 13 Alex Jia 2020-02-24 10:43:28 UTC

Verified in fuse-overlayfs-0.7.2-2.module+el8.2.0+5768+3759792f.x86_64 w/ 
buildah-1.11.6-6.module+el8.2.0+5764+2729184f.x86_64 and
podman-1.6.4-5.module+el8.2.0+5795+9bd98c8c.x86_64.

[ajia@hpe-dl380pgen8-02-vm-5 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.2 Beta (Ootpa)

[ajia@hpe-dl380pgen8-02-vm-5 ~]$ rpm -q podman buildah fuse-overlayfs
podman-1.6.4-5.module+el8.2.0+5795+9bd98c8c.x86_64
buildah-1.11.6-6.module+el8.2.0+5764+2729184f.x86_64
fuse-overlayfs-0.7.2-2.module+el8.2.0+5768+3759792f.x86_64

[ajia@hpe-dl380pgen8-02-vm-5 ~]$ id
uid=1001(ajia) gid=1001(ajia) groups=1001(ajia) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

[ajia@hpe-dl380pgen8-02-vm-5 ~]$ podman unshare cat /proc/self/uid_map 
         0       1001          1
         1     165536      65536

[ajia@hpe-dl380pgen8-02-vm-5 ~]$ git clone https://gitlab.com/acdingman/sleepy-container.git
Cloning into 'sleepy-container'...
remote: Enumerating objects: 15, done.
remote: Counting objects: 100% (15/15), done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 15 (delta 3), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (15/15), done.

[ajia@hpe-dl380pgen8-02-vm-5 ~]$ buildah bud -t sleepy:rhel8.1 sleepy-container
STEP 1: FROM registry.access.redhat.com/ubi8/ubi
Getting image source signatures
Copying blob eae5d284042d done
Copying blob ff6f434a470a done
Copying config fd73e6738a done
Writing manifest to image destination
Storing signatures
STEP 2: USER root
STEP 3: ENV SRVUSR=sleepy
STEP 4: ENV SRVGRP=sleepy
STEP 5: LABEL maintainer="Me <me>"
STEP 6: LABEL description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman"
STEP 7: LABEL io.k8s.description="A trivial Dockerfile to reproduce failure to add service users in rootless buildah/podman"
STEP 8: LABEL io.k8s.dispaly-name="User fail on RHEL UBI 8"
STEP 9: ENV YUM="yum -y --disablerepo=rhel*"
STEP 10: ADD sleepy /usr/local/bin
STEP 11: ADD sleepy.conf /etc
STEP 12: RUN groupadd -r ${SRVGRP} || true
STEP 13: RUN useradd -r -g ${SRVGRP} ${SRVUSR} || true
STEP 14: RUN chmod +x /usr/local/bin/sleepy || true
STEP 15: RUN chown root:${SRVGRP} /etc/sleepy.conf ; chmod 640 /etc/sleepy.conf || true
STEP 16: RUN mkdir /var/local/sleepy
STEP 17: RUN chmod 2750 /var/local/sleepy && chown ${SRVUSR}:${SRVGRP} /var/local/sleepy || true
STEP 18: RUN ${YUM} install procps-ng iproute nmap-ncat
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
No repository match: rhel*
Red Hat Universal Base Image 8 (RPMs) - BaseOS                                                                                                                                                                738 kB/s | 760 kB     00:01    
Red Hat Universal Base Image 8 (RPMs) - AppStream                                                                                                                                                             2.6 MB/s | 3.3 MB     00:01    
Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder                                                                                                                                                      20 kB/s | 9.1 kB     00:00    
Dependencies resolved.
==============================================================================================================================================================================================================================================
 Package                                                 Architecture                                         Version                                                     Repository                                                     Size
==============================================================================================================================================================================================================================================
Installing:
 iproute                                                 x86_64                                               4.18.0-15.el8                                               ubi-8-baseos                                                  616 k
 procps-ng                                               x86_64                                               3.3.15-1.el8                                                ubi-8-baseos                                                  328 k
 nmap-ncat                                               x86_64                                               2:7.70-5.el8                                                ubi-8-appstream                                               237 k
Installing dependencies:
 libmnl                                                  x86_64                                               1.0.4-6.el8                                                 ubi-8-baseos                                                   30 k

Transaction Summary
==============================================================================================================================================================================================================================================
Install  4 Packages

Total download size: 1.2 M
Installed size: 3.7 M
Downloading Packages:
(1/4): libmnl-1.0.4-6.el8.x86_64.rpm                                                                                                                                                                          162 kB/s |  30 kB     00:00    
(2/4): nmap-ncat-7.70-5.el8.x86_64.rpm                                                                                                                                                                        1.7 MB/s | 237 kB     00:00    
(3/4): procps-ng-3.3.15-1.el8.x86_64.rpm                                                                                                                                                                      720 kB/s | 328 kB     00:00    
(4/4): iproute-4.18.0-15.el8.x86_64.rpm                                                                                                                                                                       1.2 MB/s | 616 kB     00:00    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                         2.4 MB/s | 1.2 MB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                      1/1 
  Installing       : libmnl-1.0.4-6.el8.x86_64                                                                                                                                                                                            1/4 
  Running scriptlet: libmnl-1.0.4-6.el8.x86_64                                                                                                                                                                                            1/4 
  Installing       : iproute-4.18.0-15.el8.x86_64                                                                                                                                                                                         2/4 
  Installing       : nmap-ncat-2:7.70-5.el8.x86_64                                                                                                                                                                                        3/4 
  Running scriptlet: nmap-ncat-2:7.70-5.el8.x86_64                                                                                                                                                                                        3/4 
  Installing       : procps-ng-3.3.15-1.el8.x86_64                                                                                                                                                                                        4/4 
  Running scriptlet: procps-ng-3.3.15-1.el8.x86_64                                                                                                                                                                                        4/4 
  Verifying        : iproute-4.18.0-15.el8.x86_64                                                                                                                                                                                         1/4 
  Verifying        : libmnl-1.0.4-6.el8.x86_64                                                                                                                                                                                            2/4 
  Verifying        : procps-ng-3.3.15-1.el8.x86_64                                                                                                                                                                                        3/4 
  Verifying        : nmap-ncat-2:7.70-5.el8.x86_64                                                                                                                                                                                        4/4 
Installed products updated.

Installed:
  iproute-4.18.0-15.el8.x86_64                               procps-ng-3.3.15-1.el8.x86_64                               nmap-ncat-2:7.70-5.el8.x86_64                               libmnl-1.0.4-6.el8.x86_64                              

Complete!
STEP 19: USER ${SRVUSR}
STEP 20: ENTRYPOINT ["/usr/local/bin/sleepy"]
STEP 21: COMMIT sleepy:rhel8.1
Getting image source signatures
Copying blob 1295eae54c9d skipped: already exists
Copying blob 85f69e555a1b skipped: already exists
Copying blob 60cf732502e3 done
Copying config c65d27c13a done
Writing manifest to image destination
Storing signatures
c65d27c13ac51ab055d715eddf26175b3435dc5c5c2609ba8a56b4b263508545
c65d27c13ac51ab055d715eddf26175b3435dc5c5c2609ba8a56b4b263508545

[ajia@hpe-dl380pgen8-02-vm-5 ~]$ podman run -dt --rm --name sleepy sleepy:rhel8.1
b206faf5514dc33590cb0ff3dd295511ed534c4223f379575c397f2688ed48b1

Comment 14 Nicholas Nachefski 2020-03-04 17:30:00 UTC

So the fix for this is in fuse-overlayfs-0.7.2-2 ?  Any word on when this package will be available for RHEL7?  Is it possible that you can direct me to a dev package i can test with?  This is blocking a tekton/openshift-pipelines project i am working on.  Latest package for RHEL7.7 is fuse-overlayfs-0.7.2-1.el7.x86_64

Comment 15 Tom Sweeney 2020-03-04 18:01:07 UTC

Jindrich, can you answer the question from Nicholas (https://bugzilla.redhat.com/show_bug.cgi?id=1802907#c14) please?

Comment 16 Nicholas Nachefski 2020-03-06 21:27:37 UTC

Confirm that this problem still exists in Openshift 4.4 pre-release.

RHCOS images = rhcos-4.4.0-0.nightly-2020-02-25-155201-x86_64
OCP release = 4.4.0-0.nightly-2020-03-06-030852

Comment 17 Tom Sweeney 2020-03-09 21:05:10 UTC

Lokesh or Jindrich, can you confirm that the fix was in the releases noted by Nicholas in https://bugzilla.redhat.com/show_bug.cgi?id=1802907#c16

Comment 18 Alex Jia 2020-03-16 02:07:41 UTC

Also verified in fuse-overlayfs-0.7.2-4.module+el8.2.0+5949+6277b64f.x86_64.

[ajia@kvm-07-guest27 ~]$ rpm -q fuse-overlayfs buildah podman slirp4netns
fuse-overlayfs-0.7.2-4.module+el8.2.0+5949+6277b64f.x86_64
buildah-1.11.6-6.module+el8.2.0+5855+8192c413.x86_64
podman-1.6.4-9.module+el8.2.0+5951+eb56bde6.x86_64
slirp4netns-0.4.2-3.git21fdece.module+el8.2.0+5655+72205bd1.x86_64

[ajia@kvm-07-guest27 ~]$ id
uid=1001(ajia) gid=1001(ajia) groups=1001(ajia) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[ajia@kvm-07-guest27 ~]$ buildah unshare cat /proc/self/uid_map 
         0       1001          1
         1     165536      65536

[ajia@kvm-07-guest27 ~]$ buildah bud -t sleepy:rhel8.1 sleepy-container
...ignore...
STEP 10: ADD sleepy /usr/local/bin
STEP 11: ADD sleepy.conf /etc
STEP 12: RUN groupadd -r ${SRVGRP} || true
STEP 13: RUN useradd -r -g ${SRVGRP} ${SRVUSR} || true
STEP 14: RUN chmod +x /usr/local/bin/sleepy || true
...ignore...

[ajia@kvm-07-guest27 ~]$ podman run -dt --rm --name sleepy sleepy:rhel8.1
a8d97f5ed211d0468f585ab4b92c3e0cea87171068be8b21e74dd301ed161d23

[ajia@kvm-07-guest27 ~]$ podman ps
CONTAINER ID  IMAGE                     COMMAND  CREATED        STATUS            PORTS  NAMES
a8d97f5ed211  localhost/sleepy:rhel8.1           3 seconds ago  Up 3 seconds ago         sleepy

Comment 19 Alex Jia 2020-03-30 03:38:23 UTC

Also verified in buildah-1.11.6-7.module+el8.2.0+5856+b8046c6d.x86_64.

Comment 21 errata-xmlrpc 2020-04-28 15:53:22 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:1650

Comment 22 Anna Khaitovich 2020-08-10 14:52:21 UTC

*** Bug 1807972 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.

adingman
ajia
andrew
andrew.white
castedo
c.handel
ddarrah
dgallowa
dornelas
dwalsh
ewout.ros
haegele
jnovy
jonathan.a.callen
kryadov
lsm5
mheon
mjtrangoni
nnachefski
ocasalsa
timo.sandmann
tsweeney