Bug 1584494

Summary:	OpenShift installation on AWS fails to pull image with relative path from AWS registry
Product:	OpenShift Container Platform	Reporter:	Ravi Sankar <rpenta>
Component:	Node	Assignee:	Seth Jennings <sjenning>
Status:	CLOSED DUPLICATE	QA Contact:	DeShuai Ma <dma>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.10.0	CC:	aos-bugs, bbennett, jokerman, mmccomas
Target Milestone:	---
Target Release:	3.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-05-31 17:48:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1572182

Description Ravi Sankar 2018-05-31 04:24:41 UTC

Description of problem:
Pod image is pulled by kubelet and it should work similar to docker pull.
When relative image path is used, 'docker pull' searches the image in all the registries and is able to download the image but kubelet fails to download the image as it is only searching from docker.io registry. Looks like some mis-configuration?

Version-Release number of selected component (if applicable):
[root@ip-172-18-1-193 ~]# oc version
oc v3.10.0-0.54.0
kubernetes v1.10.0+b81c8f8

How reproducible:
Always

Steps to Reproduce:
1. Launch openshift on AWS using ansible installer
2. Create sample pod with relative image path (eg: openshift3/ose-deployer)
3. Pod creation fails with error
'container start failed: ErrImagePull: rpc error: code = Unknown desc = repository docker.io/openshift3/ose-deployer not found: does not exist or no pull access'
4. docker pull openshift3/ose-deployer works, downloads the image from registry.reg-aws.openshift.com:443/openshift3/ose-deployer 

Actual results:
Pod fails with image pull error

Expected results:
Pod creation should succeed

Additional info:

Comment 1 Ravi Sankar 2018-05-31 04:33:31 UTC

Tried adding AWS registry to /etc/containers/registries.conf, restarted docker and atomic-openshift-node service but that did not resolve the issue. 

More details from openshift node:

[root@ip-172-18-3-185 ~]# cat /etc/containers/registries.conf
...
[registries.search]
registries = ['registry.access.redhat.com',  'registry.reg-aws.openshift.com:443']
...
[registries.insecure]
registries = ['registry.reg-aws.openshift.com:443']

-------------
[root@ip-172-18-3-185 ~]# docker info
...
Insecure Registries:
 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888
 registry.reg-aws.openshift.com:443
 virt-openshift-05.lab.eng.nay.redhat.com:5000
 virt-openshift-05.lab.eng.nay.redhat.com:5001
 asb-registry.usersys.redhat.com:5000
 127.0.0.0/8
...
Registries: registry.reg-aws.openshift.com:443 (insecure), registry.access.redhat.com (secure), registry.access.redhat.com (secure), registry.reg-aws.openshift.com:443 (insecure), docker.io (secure)

---------------
[root@ip-172-18-3-185 ~]# cat /run/containers/registries.conf
REGISTRIES="--add-registry registry.access.redhat.com --add-registry registry.reg-aws.openshift.com:443 --insecure-registry registry.reg-aws.openshift.com:443

Comment 2 Scott Dodson 2018-05-31 12:32:31 UTC

This is a change in behavior of 3.10. You need to use fully qualified image references for your pods as far as I know but moving to Image component to verify this.

Comment 3 Gabe Montero 2018-05-31 13:44:54 UTC

The image references the kubelet uses is not the domain of the Image component.

Comment 4 Ben Bennett 2018-05-31 14:46:24 UTC

I think you need to do:
  systemctl restart registries
after changing /etc/containers/registries.conf

Comment 5 Ben Bennett 2018-05-31 14:56:22 UTC

I'll validate this with @weliang and see.

Comment 6 Seth Jennings 2018-05-31 17:48:48 UTC


*** This bug has been marked as a duplicate of bug 1583500 ***