Bug 1588435

Summary: OCP 3.9.30 missing ose-pod images
Product: OpenShift Container Platform Reporter: Borja Aranda <farandac>
Component: NodeAssignee: Seth Jennings <sjenning>
Status: CLOSED DUPLICATE QA Contact: DeShuai Ma <dma>
Severity: high Docs Contact:
Priority: unspecified    
Version: 3.9.0CC: aos-bugs, cshereme, dmoessne, farandac, farid, isanchez, jokerman, lstanton, mmccomas, mzali, openshift-bugs-escalate, sgarciam, takirby, worrelborst
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-07 13:39:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Borja Aranda 2018-06-07 10:54:37 UTC
Description of problem:
When installing or upgrading to OCP 3.9.30, the kubelet is trying to pull ose-pod from docker.io, resulting in error in every deployment.

This is an installation without oreg_url, so it should default every internal component pull to registry.access.redhat.com

Also, during installation/upgrade the webconsole tries to deploy using that docker.io/openshift3/ose-pod image, which causes the deploy to fail and ending the playbook:

Warning  FailedCreatePodSandBox  15s (x16 over 4m)  kubelet, master-0.mordor.quicklab.pnq2.cee.redhat.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "openshift3/ose-pod:v3.9.30": repository docker.io/openshift3/ose-pod not found: does not exist or no pull access

- node-config looks like this:
imageConfig:
  format: openshift3/ose-${component}:${version}
  latest: False

- /etc/sysconfig/docker:
ADD_REGISTRY='--add-registry registry.access.redhat.com'

- /etc/containers/registries.conf:
[registries.search]
registries = ['registry.access.redhat.com']

But, for some reason the deploy always try to pull ose-pod from docker.io

The workaround is to tune in every node the imageConfig in /etc/origin/node/node-config.yaml, to this:

imageConfig:
  format: registry.access.redhat.com/openshift3/ose-${component}:${version}
  latest: False

After that and forcing the registry for internal components everything works fine.

Version-Release number of the following components:
3.9.30

How reproducible:

Steps to Reproduce:
1. upgrade an existing env or install a fresh 3.9.30

Actual results:
Error pulling the image, because it doesn't exist.

Expected results:
Internal componentes pulled by default from registry.access.redhat.com

Comment 1 Scott Dodson 2018-06-07 12:37:07 UTC
I cannot reproduce this. Please provide docker logs from the docker daemon experiencing the error or grant us access to the test host you've reproduced this on.

Please advise the customer to use the workaround of configuring the fully qualified format string in the mean time. That also needs to make it into /etc/origin/master/master-config.yaml Setting oreg_url to the same string would achieve the same outcome.

Comment 7 Borrelworst 2018-06-18 13:06:10 UTC
I am experiencing the exact same issue:

Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image \"openshift3/ose-pod:v3.9.30\": repository docker.io/openshift3/ose-pod not found: does not exist or no pull access

However, when using the command 'docker pull openshift3/ose-pod' manually, it does pull the docker image, however not from docker.io but from registry.access.redhat.com, so the error is a little misleading. 

When using 'docker pull ose-pod', I do get the error which I am experiencing when installing Openshift:

docker pull ose-pod
Using default tag: latest
Trying to pull repository registry.access.redhat.com/ose-pod ... 
Trying to pull repository registry.access.redhat.com/ose-pod ... 
Trying to pull repository docker.io/library/ose-pod ... 
repository docker.io/ose-pod not found: does not exist or no pull access

I think you just need to add openshift3 to the pull command, to make it 'docker pull openshift3/ose-pod', that will solve the issue.

Please update ASAP, as at this point it is impossible to deploy a new cluster.

Comment 8 Farid Saad 2018-06-18 17:58:11 UTC
Facing this issue as well. No additional useful logs to contribute, the symptoms are exactly the same for us. 

The deployment image reference is properly prefixed with openshift3/ose-pod:v3.9.30 and the docker process has --add-registry registry.access.redhat.com yet the logs show it can't pull from docker.io/openshift3/ose-pod:v3.9.30.

Comment 9 daniel 2018-06-19 04:21:55 UTC
mind

https://access.redhat.com/solutions/3480921