1701922 – podman pull need a full path

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1701922 - podman pull need a full path

Summary: podman pull need a full path

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	podman
Sub Component:
Version:	7.6
Hardware:	All
OS:	Linux
Priority:	urgent
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Valentin Rothberg
QA Contact:	Martin Jenner
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1186913 1688343
TreeView+	depends on / blocked

Reported:	2019-04-22 13:18 UTC by Qian Cai
Modified:	2019-06-04 19:10 UTC (History)
CC List:	11 users (show)
Fixed In Version:	podman-1.3.0-1.git139afa7.el7_6
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-06-04 19:10:26 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2019:1355	0	None	None	None	2019-06-04 19:10:43 UTC

Description Qian Cai 2019-04-22 13:18:06 UTC

Description of problem:
# cat /etc/containers/registries.conf
[registries.search]
registries = ['registry.access.redhat.com']


# podman pull rhel7-aarch64
Trying to pull docker://rhel7-aarch64...ERRO[0001] Error pulling image ref //rhel7-aarch64:latest: Error determining manifest MIME type for docker://rhel7-aarch64:latest: Error reading manifest latest in docker.io/library/rhel7-aarch64: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
 
Failed
(0xaaaad1530cc0,0x4420c06d20)
Error: error pulling image "rhel7-aarch64": Invalid image name "rhel7-aarch64", expected colon-separated transport:reference

Version-Release number of selected component (if applicable):
podman-1.2.0-3.git3bd528e.el7.aarch64

How reproducible:
always

Comment 3 Qian Cai 2019-04-22 13:33:55 UTC

# podman --log-level debug pull rhel7-aarch64 
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay test mount with multiple lowers succeeded 
DEBU[0000] overlay test mount indicated that metacopy is not being used 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] error parsing image name "rhel7-aarch64", trying with transport "docker://": Invalid image name "rhel7-aarch64", expected colon-separated transport:reference 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/rhel7-aarch64:latest" 
Trying to pull docker://rhel7-aarch64...DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for docker.io/library/rhel7-aarch64:latest 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Frhel7-aarch64%3Apull&service=registry.docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/library/rhel7-aarch64/manifests/latest 
ERRO[0000] Error pulling image ref //rhel7-aarch64:latest: Error determining manifest MIME type for docker://rhel7-aarch64:latest: Error reading manifest latest in docker.io/library/rhel7-aarch64: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
 
Failed
(0xaaaac8170cc0,0x4420d02fe0)
ERRO[0000] error pulling image "rhel7-aarch64": Invalid image name "rhel7-aarch64", expected colon-separated transport:reference

Comment 4 Qian Cai 2019-04-22 13:37:13 UTC

docker works fine.

# docker pull rhel7-aarch64
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7-aarch64 ... 
latest: Pulling from registry.access.redhat.com/rhel7-aarch64
14cac5d95f85: Pull complete 
02c4a9699fcf: Pull complete 
Digest: sha256:8ab711b569271d12ff2d1eb99c9e912e2e9006f86096acdcf5256a0686a35e55
Status: Downloaded newer image for registry.access.redhat.com/rhel7-aarch64:latest

Comment 5 Brent Baude 2019-04-22 13:43:02 UTC

whats podman info? and what is the fully qualified image name?

Comment 6 Qian Cai 2019-04-22 13:46:21 UTC

# podman info
host:
  BuildahVersion: 1.7.2
  Conmon:
    package: podman-1.2.0-3.git3bd528e.el7.aarch64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.14.0-dev, commit: 0c604c831dee8b5e432c0600d35e292fe82ed2f6-dirty'
  Distribution:
    distribution: '"rhel"'
    version: "7.6"
  MemFree: 89860014080
  MemTotal: 102196772864
  OCIRuntime:
    package: runc-1.0.0-59.dev.git2abd837.el7.aarch64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 0
  SwapTotal: 0
  arch: arm64
  cpus: 256
  hostname: hpe-apollo-cn99xx-12.khw3.lab.eng.bos.redhat.com
  kernel: 4.14.0-115.7.1.el7a.aarch64
  os: linux
  rootless: false
  uptime: 1h 8m 29.81s (Approximately 0.04 days)
insecure registries:
  registries:
  - brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888
registries:
  registries:
  - registry.access.redhat.com
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 2
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Comment 7 Matthew Heon 2019-04-22 13:47:06 UTC

Podman's giving a Permission Denied from the registry - this doesn't look like a shortname thing so much as being completely unable to access the image. Does using the full path to the image work? Is Podman logged into the registry?

Comment 8 Qian Cai 2019-04-22 13:55:19 UTC

Yes, full path works fine,

podman pull registry.access.redhat.com/rhel7-aarch64

Comment 9 Matthew Heon 2019-04-22 14:26:58 UTC

The paths it's pinging are only 'docker.io' - I'm not seeing it try and hit the Red Hat registries, despite their being configured in registries.conf (and docker.io not being present in same file)

Comment 10 Matthew Heon 2019-04-22 14:43:21 UTC

Reproduced locally, using F29 packaged Podman 1.2.0 (podman-1.2.0-2.git3bd528e.fc29.x86_64). Contents of registries.conf don't seem to matter, it never tries anything that's not docker.io

Comment 11 Brent Baude 2019-04-22 14:45:15 UTC

i believe the underlying image parsing functions are injecting docker.io by default.

Comment 12 Matthew Heon 2019-04-22 14:45:30 UTC

Also reproduces with Skopeo, so this is in c/image

Comment 13 Daniel Walsh 2019-04-22 15:19:00 UTC

Qian, does podman run work?  Or does it blow up in the same way?

Comment 14 Qian Cai 2019-04-22 15:29:53 UTC

podman run works.

# podman run --rm  rhel7-aarch64 date
Trying to pull registry.access.redhat.com/rhel7-aarch64...Getting image source signatures
Copying blob 02c4a9699fcf done
Copying blob 14cac5d95f85 done
Copying config accd822b20 done
Writing manifest to image destination
Storing signatures
Mon Apr 22 15:29:09 UTC 2019

Comment 15 Daniel Walsh 2019-04-22 15:39:20 UTC

I would bet 
buildah pull works also.

I believe we have a lot of cruft in podman pull that needs to be cleaned up and is causing this issue.
Although I have no idea why skopeo would fail.

Comment 25 Daniel Walsh 2019-04-24 13:55:09 UTC

I agree fixing pull is way more important then all-tags.

Comment 29 Joy Pu 2019-05-24 10:08:56 UTC

Test with podman-1.3.1-1.git7210727.el7.x86_64, when pull image with --log-level debug we can find it parsed reference based on the registries.conf. So set this to verified. Details:
# podman --log-level debug pull rhel7
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay test mount with multiple lowers succeeded 
DEBU[0000] overlay test mount indicated that metacopy is not being used 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
INFO[0000] Found CNI network mynet (type=bridge) at /etc/cni/net.d/10-mynet.conf 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]registry.access.redhat.com/rhel7:latest" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/rhel7:latest" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]registry.fedoraproject.org/rhel7:latest" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/rhel7:latest" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]registry.centos.org/rhel7:latest" 
Trying to pull registry.access.redhat.com/rhel7...DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for registry.access.redhat.com/rhel7:latest 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.access.redhat.com 
DEBU[0000]  cert: /etc/docker/certs.d/registry.access.redhat.com/1397472824682930775.cert 
DEBU[0000]  key: /etc/docker/certs.d/registry.access.redhat.com/1397472824682930775.key 
DEBU[0000]  cert: /etc/docker/certs.d/registry.access.redhat.com/3205701674833865034.cert 
DEBU[0000]  key: /etc/docker/certs.d/registry.access.redhat.com/3205701674833865034.key 
DEBU[0000]  crt: /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt 
DEBU[0000] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] GET https://registry.access.redhat.com/v2/   
DEBU[0001] Ping https://registry.access.redhat.com/v2/ status 200 
DEBU[0001] GET https://registry.access.redhat.com/v2/rhel7/manifests/latest 
DEBU[0002] Source is a manifest list; copying (only) instance sha256:a5202c981262481dffc11f7e2e69e7b19126965ceeb021cbe597e19babb14275 
DEBU[0002] GET https://registry.access.redhat.com/v2/rhel7/manifests/sha256:a5202c981262481dffc11f7e2e69e7b19126965ceeb021cbe597e19babb14275 
DEBU[0003] IsRunningImageAllowed for image docker:registry.access.redhat.com/rhel7:latest 
DEBU[0003]  Using default policy section                
DEBU[0003]  Requirement 0: allowed                      
DEBU[0003] Overall: allowed                             
DEBU[0003] Downloading /v2/rhel7/blobs/sha256:5044f6040ea5535b508dcade2cbee564dae54907ed47ee6002c8cd6e39c60c3c 
DEBU[0003] GET https://registry.access.redhat.com/v2/rhel7/blobs/sha256:5044f6040ea5535b508dcade2cbee564dae54907ed47ee6002c8cd6e39c60c3c 
Getting image source signatures
DEBU[0003] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0003] ... will first try using the original manifest unmodified 
DEBU[0003] Downloading /v2/rhel7/blobs/sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 
DEBU[0003] GET https://registry.access.redhat.com/v2/rhel7/blobs/sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 
DEBU[0003] Downloading /v2/rhel7/blobs/sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 
DEBU[0003] GET https://registry.access.redhat.com/v2/rhel7/blobs/sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 
DEBU[0005] Detected compression format gzip             
DEBU[0005] Using original blob without modification     
Copying blob d69140bdce18 [=>------------------------------------] 3.4MiB / 72.3MiB
DEBU[0006] Detected compression format gzip             
Copying blob d69140bdce18 done
Copying blob a82dd37af30d done
DEBU[0061] No compression detected                      
DEBU[0061] Using original blob without modification     
Copying config 5044f6040e done
Writing manifest to image destination
Storing signatures
DEBU[0061] Applying tar in /var/lib/containers/storage/overlay/03b8aa00f0018b0d0eb70a535c71c87da3bd7810bc7d5fb1ec5237b7aaf0a0cb/diff 
DEBU[0091] Applying tar in /var/lib/containers/storage/overlay/a10198577639cbb2aee87d548cf042b025c91f47d574a5b69427dede037536f0/diff 
DEBU[0091] setting image creation date to 2019-04-16 15:35:01.957134 +0000 UTC 
DEBU[0091] created new image ID "5044f6040ea5535b508dcade2cbee564dae54907ed47ee6002c8cd6e39c60c3c" 
DEBU[0091] set names of image "5044f6040ea5535b508dcade2cbee564dae54907ed47ee6002c8cd6e39c60c3c" to [registry.access.redhat.com/rhel7:latest] 
DEBU[0091] saved image metadata "{}"                    
DEBU[0092] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]registry.access.redhat.com/rhel7:latest" 
5044f6040ea5535b508dcade2cbee564dae54907ed47ee6002c8cd6e39c60c3c

Comment 31 errata-xmlrpc 2019-06-04 19:10:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1355

Note You need to log in before you can comment on or make changes to this bug.