Bug 2009266

Summary: mkdir /home/podman/.local/share/containers/storage: permission denied
Product: Red Hat Enterprise Linux 8 Reporter: Edward Shen <weshen>
Component: podman-containerAssignee: Jindrich Novy <jnovy>
Status: CLOSED ERRATA QA Contact: Edward Shen <weshen>
Severity: high Docs Contact: Gabriela Nečasová <gnecasov>
Priority: unspecified    
Version: 8.5CC: dornelas, dwalsh, gnecasov, jnovy, mheon
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2021249 (view as bug list) Environment:
Last Closed: 2022-05-10 21:27:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2021249    

Description Edward Shen 2021-09-30 09:19:57 UTC
Description of problem:
When running rootless podman in rootful or rootless podman, this issue happens.

Version-Release number of selected component (if applicable):
registry-proxy.engineering.redhat.com/rh-osbs/rhel8-podman:8.5-5

How reproducible:
always

Steps to Reproduce:
Running rootless podman in rootful podman: 
[root@hpe-dl380pgen8-02-vm-15 ~]# podman run --user podman --privileged registry-proxy.engineering.redhat.com/rh-osbs/rhel8-podman:8.5-5 podman run ubi8 echo hello
Running rootless podman in rootless podman:
[weshen@hpe-dl380pgen8-02-vm-15 ~]$ podman run --security-opt label=disable --user podman --device /dev/fuse registry-proxy.engineering.redhat.com/rh-osbs/rhel8-podman:8.5-5 podman run ubi8 echo hello


Actual results:
The same error message occurs for both rootful and rootless as below:
Error: error creating runtime static files directory: mkdir /home/podman/.local/share/containers/storage: permission denied

Expected results:
hello

Additional info:
On the host, below version of podman is installed:
[root@hpe-dl380pgen8-02-vm-15 ~]# rpm -q podman
podman-3.3.1-9.module+el8.5.0+12697+018f24d7.x86_64

Comment 1 Jindrich Novy 2021-09-30 09:41:23 UTC
Matt, can you please take a quick look at this one?

Comment 2 Matthew Heon 2021-09-30 12:17:58 UTC
The exact same message occurs, for both root and rootless? Or are the errors different for both?

Comment 3 Edward Shen 2021-10-08 03:57:26 UTC
(In reply to Matthew Heon from comment #2)
> The exact same message occurs, for both root and rootless? Or are the errors
> different for both?

The exact same message occurs for both root and rootless.
I edit the description to make it more accurate.

Comment 4 Matthew Heon 2021-10-08 12:56:46 UTC
Assugning to Dan given this is Podman-in-Podman

Comment 5 Daniel Walsh 2021-10-08 13:08:13 UTC
I don't have access to that image, could you try with quay.io/podman/stable

Have you read 

https://www.redhat.com/sysadmin/podman-inside-container

Comment 6 Edward Shen 2021-10-09 07:15:21 UTC
(In reply to Daniel Walsh from comment #5)
> I don't have access to that image, could you try with quay.io/podman/stable
> 
> Have you read 
> 
> https://www.redhat.com/sysadmin/podman-inside-container

Yes, Dan, these two steps are from the doc.

quay.io/podman/stable doesn't have this issue on rhel8.5, it works as expected.

[root@ibm-x3650m4-01-vm-07 ~]# hostnamectl
   Static hostname: ibm-x3650m4-01-vm-07.ibm2.lab.eng.bos.redhat.com
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 7727b8107459433294f6ae35064a1e82
           Boot ID: bcce25430fcd4b979e65266e18dcc431
    Virtualization: kvm
  Operating System: Red Hat Enterprise Linux 8.5 Beta (Ootpa)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:8::baseos
            Kernel: Linux 4.18.0-348.el8.x86_64
      Architecture: x86-64

[root@ibm-x3650m4-01-vm-07 ~]# podman run --user podman --privileged quay.io/podman/stable podman run ubi8 echo hello
time="2021-10-09T07:03:15Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Copying blob sha256:06038631a24a25348b51d1bfc7d0a0ee555552a8998f8328f9b657d02dd4c64c
Copying blob sha256:262268b65bd5f33784d6a61514964887bc18bc00c60c588bc62bfae7edca46f1
Copying blob sha256:06038631a24a25348b51d1bfc7d0a0ee555552a8998f8328f9b657d02dd4c64c
Copying blob sha256:262268b65bd5f33784d6a61514964887bc18bc00c60c588bc62bfae7edca46f1
Copying config sha256:53ce4390f2adb1681eb1a90ec8b48c49c015e0a8d336c197637e7f65e365fa9e
Writing manifest to image destination
Storing signatures
hello

[weshen@ibm-x3650m4-01-vm-07 ~]$ podman run --security-opt label=disable --user podman --device /dev/fuse quay.io/podman/stable podman run ubi8 echo hello
time="2021-10-09T07:09:51Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Copying blob sha256:262268b65bd5f33784d6a61514964887bc18bc00c60c588bc62bfae7edca46f1
Copying blob sha256:06038631a24a25348b51d1bfc7d0a0ee555552a8998f8328f9b657d02dd4c64c
Copying blob sha256:06038631a24a25348b51d1bfc7d0a0ee555552a8998f8328f9b657d02dd4c64c
Copying blob sha256:262268b65bd5f33784d6a61514964887bc18bc00c60c588bc62bfae7edca46f1
Copying config sha256:53ce4390f2adb1681eb1a90ec8b48c49c015e0a8d336c197637e7f65e365fa9e
Writing manifest to image destination
Storing signatures
hello

Comment 7 Daniel Walsh 2021-10-12 17:19:56 UTC
Jindrich I believe that you are in charge of that image.  Could you check the difference in the Containerfile used to build the image?

Comment 8 Jindrich Novy 2021-10-13 09:02:25 UTC
https://github.com/containers/podman/pull/11952

VOLUME needs to be declared after all permissions are set:
https://docs.docker.com/engine/reference/builder/#volume
https://devops.stackexchange.com/questions/4540/how-to-change-the-owner-of-volume-directory-in-dockerfile/4542

With the above change:

$ podman run --rm -it --user podman --privileged test:test podman run ubi8 echo hello
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 06038631a24a done  
Copying blob 262268b65bd5 done  
Copying config 53ce4390f2 done  
Writing manifest to image destination
Storing signatures
hello

Comment 9 Edward Shen 2021-11-08 03:59:37 UTC
Seems upstream is reluctant to accept this patch, but I tested it with the PR code as Jindrich asked, it works fine for 8.5. 
If we are good to have it downstream, can you please build it and attach it to errata?

[root@kvm-08-guest22 ~]# podman build -t podman-test -f .
[root@kvm-08-guest22 ~]# podman run --user podman --privileged localhost/podman-test:latest podman run ubi8 echo hello
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:47aa3ed2034c4f27622b989b26c06087de17067268a19a1b3642a7e2686cd1a3
Copying blob sha256:eac1b95df832dc9f172fd1f07e7cb50c1929b118a4249ddd02c6318a677b506a
Copying config sha256:b1e63aaae5cffb78e4af9f3a110dbad67e8013ca3de6d09f1ef496d00641e751
Writing manifest to image destination
Storing signatures
hello
[root@kvm-08-guest22 ~]# useradd weshen
[root@kvm-08-guest22 ~]# passwd weshen
[root@kvm-08-guest22 ~]# ssh weshen@localhost
[weshen@kvm-08-guest22 ~]$ podman build -t podman-test -f .
[weshen@kvm-08-guest22 ~]$ podman run --security-opt label=disable --user podman --device /dev/fuse localhost/podman-test:latest podman run ubi8 echo hello
Resolved "ubi8" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi8:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:47aa3ed2034c4f27622b989b26c06087de17067268a19a1b3642a7e2686cd1a3
Copying blob sha256:eac1b95df832dc9f172fd1f07e7cb50c1929b118a4249ddd02c6318a677b506a
Copying config sha256:b1e63aaae5cffb78e4af9f3a110dbad67e8013ca3de6d09f1ef496d00641e751
Writing manifest to image destination
Storing signatures
hello

Comment 17 Daniel Walsh 2021-11-22 20:41:27 UTC
I believe we have a fix for this.

Comment 19 Daniel Walsh 2021-11-23 13:00:23 UTC
I would say yes,  I don't even believe this needs to go through the release process, since the Dockerfile and image are not shipped by RHEL directly but stored at the registry.

Comment 31 errata-xmlrpc 2022-05-10 21:27:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rhel8/podman container image update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2158