Bug 1701922

Summary:	podman pull need a full path
Product:	Red Hat Enterprise Linux 7	Reporter:	Qian Cai <qcai>
Component:	podman	Assignee:	Valentin Rothberg <vrothber>
Status:	CLOSED ERRATA	QA Contact:	Martin Jenner <mjenner>
Severity:	high	Docs Contact:
Priority:	urgent
Version:	7.6	CC:	ddarrah, dornelas, dwalsh, jligon, lsm5, mheon, mitr, smccarty, umohnani, vrothber, ypu
Target Milestone:	rc	Keywords:	Extras, Regression
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	podman-1.3.0-1.git139afa7.el7_6	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-04 19:10:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1186913, 1688343

Description Qian Cai 2019-04-22 13:18:06 UTC

Description of problem:
# cat /etc/containers/registries.conf
[registries.search]
registries = ['registry.access.redhat.com']


# podman pull rhel7-aarch64
Trying to pull docker://rhel7-aarch64...ERRO[0001] Error pulling image ref //rhel7-aarch64:latest: Error determining manifest MIME type for docker://rhel7-aarch64:latest: Error reading manifest latest in docker.io/library/rhel7-aarch64: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
 
Failed
(0xaaaad1530cc0,0x4420c06d20)
Error: error pulling image "rhel7-aarch64": Invalid image name "rhel7-aarch64", expected colon-separated transport:reference

Version-Release number of selected component (if applicable):
podman-1.2.0-3.git3bd528e.el7.aarch64

How reproducible:
always

Comment 3 Qian Cai 2019-04-22 13:33:55 UTC

# podman --log-level debug pull rhel7-aarch64 
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay test mount with multiple lowers succeeded 
DEBU[0000] overlay test mount indicated that metacopy is not being used 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] error parsing image name "rhel7-aarch64", trying with transport "docker://": Invalid image name "rhel7-aarch64", expected colon-separated transport:reference 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/rhel7-aarch64:latest" 
Trying to pull docker://rhel7-aarch64...DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for docker.io/library/rhel7-aarch64:latest 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io 
DEBU[0000] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] GET https://registry-1.docker.io/v2/         
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401 
DEBU[0000] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Frhel7-aarch64%3Apull&service=registry.docker.io 
DEBU[0000] GET https://registry-1.docker.io/v2/library/rhel7-aarch64/manifests/latest 
ERRO[0000] Error pulling image ref //rhel7-aarch64:latest: Error determining manifest MIME type for docker://rhel7-aarch64:latest: Error reading manifest latest in docker.io/library/rhel7-aarch64: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
 
Failed
(0xaaaac8170cc0,0x4420d02fe0)
ERRO[0000] error pulling image "rhel7-aarch64": Invalid image name "rhel7-aarch64", expected colon-separated transport:reference

Comment 4 Qian Cai 2019-04-22 13:37:13 UTC

docker works fine.

# docker pull rhel7-aarch64
Using default tag: latest
Trying to pull repository registry.access.redhat.com/rhel7-aarch64 ... 
latest: Pulling from registry.access.redhat.com/rhel7-aarch64
14cac5d95f85: Pull complete 
02c4a9699fcf: Pull complete 
Digest: sha256:8ab711b569271d12ff2d1eb99c9e912e2e9006f86096acdcf5256a0686a35e55
Status: Downloaded newer image for registry.access.redhat.com/rhel7-aarch64:latest

Comment 5 Brent Baude 2019-04-22 13:43:02 UTC

whats podman info? and what is the fully qualified image name?

Comment 6 Qian Cai 2019-04-22 13:46:21 UTC

# podman info
host:
  BuildahVersion: 1.7.2
  Conmon:
    package: podman-1.2.0-3.git3bd528e.el7.aarch64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.14.0-dev, commit: 0c604c831dee8b5e432c0600d35e292fe82ed2f6-dirty'
  Distribution:
    distribution: '"rhel"'
    version: "7.6"
  MemFree: 89860014080
  MemTotal: 102196772864
  OCIRuntime:
    package: runc-1.0.0-59.dev.git2abd837.el7.aarch64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 0
  SwapTotal: 0
  arch: arm64
  cpus: 256
  hostname: hpe-apollo-cn99xx-12.khw3.lab.eng.bos.redhat.com
  kernel: 4.14.0-115.7.1.el7a.aarch64
  os: linux
  rootless: false
  uptime: 1h 8m 29.81s (Approximately 0.04 days)
insecure registries:
  registries:
  - brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888
registries:
  registries:
  - registry.access.redhat.com
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 2
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Comment 7 Matthew Heon 2019-04-22 13:47:06 UTC

Podman's giving a Permission Denied from the registry - this doesn't look like a shortname thing so much as being completely unable to access the image. Does using the full path to the image work? Is Podman logged into the registry?

Comment 8 Qian Cai 2019-04-22 13:55:19 UTC

Yes, full path works fine,

podman pull registry.access.redhat.com/rhel7-aarch64

Comment 9 Matthew Heon 2019-04-22 14:26:58 UTC

The paths it's pinging are only 'docker.io' - I'm not seeing it try and hit the Red Hat registries, despite their being configured in registries.conf (and docker.io not being present in same file)

Comment 10 Matthew Heon 2019-04-22 14:43:21 UTC

Reproduced locally, using F29 packaged Podman 1.2.0 (podman-1.2.0-2.git3bd528e.fc29.x86_64). Contents of registries.conf don't seem to matter, it never tries anything that's not docker.io

Comment 11 Brent Baude 2019-04-22 14:45:15 UTC

i believe the underlying image parsing functions are injecting docker.io by default.

Comment 12 Matthew Heon 2019-04-22 14:45:30 UTC

Also reproduces with Skopeo, so this is in c/image

Comment 13 Daniel Walsh 2019-04-22 15:19:00 UTC

Qian, does podman run work?  Or does it blow up in the same way?

Comment 14 Qian Cai 2019-04-22 15:29:53 UTC

podman run works.

# podman run --rm  rhel7-aarch64 date
Trying to pull registry.access.redhat.com/rhel7-aarch64...Getting image source signatures
Copying blob 02c4a9699fcf done
Copying blob 14cac5d95f85 done
Copying config accd822b20 done
Writing manifest to image destination
Storing signatures
Mon Apr 22 15:29:09 UTC 2019

Comment 15 Daniel Walsh 2019-04-22 15:39:20 UTC

I would bet 
buildah pull works also.

I believe we have a lot of cruft in podman pull that needs to be cleaned up and is causing this issue.
Although I have no idea why skopeo would fail.

Comment 25 Daniel Walsh 2019-04-24 13:55:09 UTC

I agree fixing pull is way more important then all-tags.

Comment 29 Joy Pu 2019-05-24 10:08:56 UTC

Test with podman-1.3.1-1.git7210727.el7.x86_64, when pull image with --log-level debug we can find it parsed reference based on the registries.conf. So set this to verified. Details:
# podman --log-level debug pull rhel7
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay test mount with multiple lowers succeeded 
DEBU[0000] overlay test mount indicated that metacopy is not being used 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
INFO[0000] Found CNI network mynet (type=bridge) at /etc/cni/net.d/10-mynet.conf 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]registry.access.redhat.com/rhel7:latest" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/rhel7:latest" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]registry.fedoraproject.org/rhel7:latest" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]quay.io/rhel7:latest" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]registry.centos.org/rhel7:latest" 
Trying to pull registry.access.redhat.com/rhel7...DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]  No signature storage configuration found for registry.access.redhat.com/rhel7:latest 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/registry.access.redhat.com 
DEBU[0000]  cert: /etc/docker/certs.d/registry.access.redhat.com/1397472824682930775.cert 
DEBU[0000]  key: /etc/docker/certs.d/registry.access.redhat.com/1397472824682930775.key 
DEBU[0000]  cert: /etc/docker/certs.d/registry.access.redhat.com/3205701674833865034.cert 
DEBU[0000]  key: /etc/docker/certs.d/registry.access.redhat.com/3205701674833865034.key 
DEBU[0000]  crt: /etc/docker/certs.d/registry.access.redhat.com/redhat-ca.crt 
DEBU[0000] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb 
DEBU[0000] GET https://registry.access.redhat.com/v2/   
DEBU[0001] Ping https://registry.access.redhat.com/v2/ status 200 
DEBU[0001] GET https://registry.access.redhat.com/v2/rhel7/manifests/latest 
DEBU[0002] Source is a manifest list; copying (only) instance sha256:a5202c981262481dffc11f7e2e69e7b19126965ceeb021cbe597e19babb14275 
DEBU[0002] GET https://registry.access.redhat.com/v2/rhel7/manifests/sha256:a5202c981262481dffc11f7e2e69e7b19126965ceeb021cbe597e19babb14275 
DEBU[0003] IsRunningImageAllowed for image docker:registry.access.redhat.com/rhel7:latest 
DEBU[0003]  Using default policy section                
DEBU[0003]  Requirement 0: allowed                      
DEBU[0003] Overall: allowed                             
DEBU[0003] Downloading /v2/rhel7/blobs/sha256:5044f6040ea5535b508dcade2cbee564dae54907ed47ee6002c8cd6e39c60c3c 
DEBU[0003] GET https://registry.access.redhat.com/v2/rhel7/blobs/sha256:5044f6040ea5535b508dcade2cbee564dae54907ed47ee6002c8cd6e39c60c3c 
Getting image source signatures
DEBU[0003] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU[0003] ... will first try using the original manifest unmodified 
DEBU[0003] Downloading /v2/rhel7/blobs/sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 
DEBU[0003] GET https://registry.access.redhat.com/v2/rhel7/blobs/sha256:a82dd37af30d5ff9e805ceea67ea615a17dfaafba3135b12e6b2dab29ee2cff2 
DEBU[0003] Downloading /v2/rhel7/blobs/sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 
DEBU[0003] GET https://registry.access.redhat.com/v2/rhel7/blobs/sha256:d69140bdce18c2f525b2ad0cc3998a1c6f2bc0a850353b7b7feac66eca1da526 
DEBU[0005] Detected compression format gzip             
DEBU[0005] Using original blob without modification     
Copying blob d69140bdce18 [=>------------------------------------] 3.4MiB / 72.3MiB
DEBU[0006] Detected compression format gzip             
Copying blob d69140bdce18 done
Copying blob a82dd37af30d done
DEBU[0061] No compression detected                      
DEBU[0061] Using original blob without modification     
Copying config 5044f6040e done
Writing manifest to image destination
Storing signatures
DEBU[0061] Applying tar in /var/lib/containers/storage/overlay/03b8aa00f0018b0d0eb70a535c71c87da3bd7810bc7d5fb1ec5237b7aaf0a0cb/diff 
DEBU[0091] Applying tar in /var/lib/containers/storage/overlay/a10198577639cbb2aee87d548cf042b025c91f47d574a5b69427dede037536f0/diff 
DEBU[0091] setting image creation date to 2019-04-16 15:35:01.957134 +0000 UTC 
DEBU[0091] created new image ID "5044f6040ea5535b508dcade2cbee564dae54907ed47ee6002c8cd6e39c60c3c" 
DEBU[0091] set names of image "5044f6040ea5535b508dcade2cbee564dae54907ed47ee6002c8cd6e39c60c3c" to [registry.access.redhat.com/rhel7:latest] 
DEBU[0091] saved image metadata "{}"                    
DEBU[0092] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]registry.access.redhat.com/rhel7:latest" 
5044f6040ea5535b508dcade2cbee564dae54907ed47ee6002c8cd6e39c60c3c

Comment 31 errata-xmlrpc 2019-06-04 19:10:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1355