Description of problem: The installation of OpenShift 3.9 onto RHEL Atomic 7.5 hosts running in Amazon Web Services using the Ansible playbooks fails with the following error: PLAY RECAP ********************************************************************* ip-10-0-1-172.ec2.internal : ok=480 changed=190 unreachable=0 failed=1 ip-10-0-1-204.ec2.internal : ok=107 changed=48 unreachable=0 failed=0 ip-10-0-1-96.ec2.internal : ok=107 changed=47 unreachable=0 failed=0 localhost : ok=13 changed=0 unreachable=0 failed=0 INSTALLER STATUS *************************************************************** Initialization : Complete (0:00:43) Health Check : Complete (0:00:39) etcd Install : Complete (0:01:01) Master Install : Complete (0:02:24) Master Additional Install : Complete (0:00:38) Node Install : Complete (0:03:58) Hosted Install : Complete (0:33:22) Web Console Install : In Progress (0:11:12) This phase can be restarted by running: playbooks/openshift-web-console/config.yml Failure summary: 1. Hosts: ip-10-0-1-172.ec2.internal Play: Web Console Task: Report console errors Message: Console install failed. Adding password for user admin ip-10-0-1-172.ec2.internal | SUCCESS => { "changed": true, "checksum": "096c0c3d033fc861e10007c20b20a48284183646", "dest": "/etc/origin/master/htpasswd", "gid": 0, "group": "root", "md5sum": "8636093cf1db173ce1e33e72bacef575", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 44, "src": "/home/cloud-user/.ansible/tmp/ansible-tmp-1529004349.53-176093319923539/source", "state": "file", "uid": 0 } Version-Release number of selected component (if applicable): RHEL Atomic 7.5 latest (AWS AMI) OpenShift Enterprise 3.9 How reproducible: Install OCP on 3 node cluster (1 master, 2 nodes) on RHEL Atomic in AWS Steps to Reproduce: 1. install OCP using ansible playbooks according to advanced installation instructions 2. try to access the web console 3. Actual results: The web console returns a 502 and a blank page. On the master the following errors are seen when running journalctl -f Jun 14 20:12:42 ip-10-0-1-172.ec2.internal atomic-openshift-node[2024]: I0614 20:12:42.794414 2049 kuberuntime_manager.go:385] No sandbox for pod "webconsole-746dbc7568-zmssm_openshift-web-console(fe3d f67f-700b-11e8-bbb9-127cfc5e0428)" can be found. Need to start a new one Jun 14 20:12:42 ip-10-0-1-172.ec2.internal dockerd-current[1209]: time="2018-06-14T20:12:42.795701056Z" level=error msg="Handler for GET /v1.26/images/openshift3/ose-pod:v3.9.30/json returned error: No su ch image: openshift3/ose-pod:v3.9.30" Jun 14 20:12:42 ip-10-0-1-172.ec2.internal dockerd-current[1209]: time="2018-06-14T20:12:42.795989300Z" level=error msg="Handler for GET /v1.26/images/openshift3/ose-pod:v3.9.30/json returned error: No s$ ch image: openshift3/ose-pod:v3.9.30" Jun 14 20:12:42 ip-10-0-1-172.ec2.internal dockerd-current[1209]: time="2018-06-14T20:12:42.890103098Z" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the res$ urce is denied\nunauthorized: authentication required\n" Jun 14 20:12:42 ip-10-0-1-172.ec2.internal dockerd-current[1209]: time="2018-06-14T20:12:42.890133595Z" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required" Jun 14 20:12:42 ip-10-0-1-172.ec2.internal dockerd-current[1209]: time="2018-06-14T20:12:42.890162061Z" level=info msg="Translating \"denied: requested access to the resource is denied\" to \"repository $ ocker.io/openshift3/ose-pod not found: does not exist or no pull access\"" Jun 14 20:12:42 ip-10-0-1-172.ec2.internal atomic-openshift-node[2024]: I0614 20:12:42.890375 2049 kube_docker_client.go:348] Stop pulling image "openshift3/ose-pod:v3.9.30": "Trying to pull repositor$ docker.io/openshift3/ose-pod ... " Jun 14 20:12:42 ip-10-0-1-172.ec2.internal atomic-openshift-node[2024]: E0614 20:12:42.890459 2049 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = fai$ ed pulling image "openshift3/ose-pod:v3.9.30": repository docker.io/openshift3/ose-pod not found: does not exist or no pull access Jun 14 20:12:42 ip-10-0-1-172.ec2.internal atomic-openshift-node[2024]: E0614 20:12:42.890508 2049 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "webconsole-746dbc7568-zmssm_openshift-web-consol$ (fe3df67f-700b-11e8-bbb9-127cfc5e0428)" failed: rpc error: code = Unknown desc = failed pulling image "openshift3/ose-pod:v3.9.30": repository docker.io/openshift3/ose-pod not found: does not exist or no pull access Jun 14 20:12:42 ip-10-0-1-172.ec2.internal atomic-openshift-node[2024]: E0614 20:12:42.890528 2049 kuberuntime_manager.go:647] createPodSandbox for pod "webconsole-746dbc7568-zmssm_openshift-web-conso$ e(fe3df67f-700b-11e8-bbb9-127cfc5e0428)" failed: rpc error: code = Unknown desc = failed pulling image "openshift3/ose-pod:v3.9.30": repository docker.io/openshift3/ose-pod not found: does not exist or n$ pull access Jun 14 20:12:42 ip-10-0-1-172.ec2.internal atomic-openshift-node[2024]: E0614 20:12:42.890598 2049 pod_workers.go:186] Error syncing pod fe3df67f-700b-11e8-bbb9-127cfc5e0428 ("webconsole-746dbc7568-zm$ sm_openshift-web-console(fe3df67f-700b-11e8-bbb9-127cfc5e0428)"), skipping: failed to "CreatePodSandbox" for "webconsole-746dbc7568-zmssm_openshift-web-console(fe3df67f-700b-11e8-bbb9-127cfc5e0428)" with CreatePodSandboxError: "CreatePodSandbox for pod \"webconsole-746dbc7568-zmssm_openshift-web-console(fe3df67f-700b-11e8-bbb9-127cfc5e0428)\" failed: rpc error: code = Unknown desc = failed pulling image $ "openshift3/ose-pod:v3.9.30\": repository docker.io/openshift3/ose-pod not found: does not exist or no pull access" Expected results: The web console should be accessible. Additional info: After manually installing the missing docker image the web console becomes available but apps are not installable. BEFORE manual install of ose-pod docker image --------------------------------------------- [root@ip-10-0-1-172 ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE registry.access.redhat.com/openshift3/openvswitch v3.9.30 e37239ae2fa3 2 weeks ago 1.46 GB registry.access.redhat.com/openshift3/node v3.9.30 1fea394aac80 2 weeks ago 1.46 GB registry.access.redhat.com/openshift3/ose v3.9 6655406c26d8 2 weeks ago 1.23 GB registry.access.redhat.com/openshift3/ose v3.9.30 6655406c26d8 2 weeks ago 1.23 GB registry.access.redhat.com/rhel7/etcd latest 924412659272 5 weeks ago 256 MB [root@ip-10-0-1-172 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES da4ddb6d6144 openshift3/node:v3.9.30 "/usr/local/bin/or..." 3 minutes ago Up 3 minutes atomic-openshift-node 8698550a51ab openshift3/ose:v3.9.30 "/usr/bin/openshif..." 3 minutes ago Up 3 minutes atomic-openshift-master-controllers 97cc62e7a642 openshift3/ose:v3.9.30 "/usr/bin/openshif..." 4 minutes ago Up 3 minutes atomic-openshift-master-api 292f7d09b7ee openshift3/openvswitch:v3.9.30 "/usr/local/bin/ov..." 4 minutes ago Up 3 minutes openvswitch fc6fc5a48749 registry.access.redhat.com/rhel7/etcd "/usr/bin/etcd" 4 minutes ago Up 3 minutes etcd_container AFTER manual install of ose-pod docker image --------------------------------------------- [root@ip-10-0-1-172 ~]# docker pull openshift3/ose-pod:v3.9.30 Trying to pull repository registry.access.redhat.com/openshift3/ose-pod ... sha256:388ede198262b7fb97afd7ab04235e4cb3f841ad2e5cbe2de452a0db16a5d973: Pulling from registry.access.redhat.com/openshift3/ose-pod e0f71f706c2a: Already exists 121ab4741000: Already exists 9988e1f7ff11: Pull complete Digest: sha256:388ede198262b7fb97afd7ab04235e4cb3f841ad2e5cbe2de452a0db16a5d973 Status: Downloaded newer image for registry.access.redhat.com/openshift3/ose-pod:v3.9.30 [root@ip-10-0-1-172 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f144ca9b3e69 openshift3/ose-pod:v3.9.30 "/usr/bin/pod" 5 seconds ago Up 4 seconds k8s_POD_webconsole-746dbc7568-snfx7_openshift-web-console_73b04a4e-7012-11e8-837a-127cfc5e0428_0 da4ddb6d6144 openshift3/node:v3.9.30 "/usr/local/bin/or..." 4 minutes ago Up 4 minutes atomic-openshift-node 8698550a51ab openshift3/ose:v3.9.30 "/usr/bin/openshif..." 5 minutes ago Up 5 minutes atomic-openshift-master-controllers 97cc62e7a642 openshift3/ose:v3.9.30 "/usr/bin/openshif..." 5 minutes ago Up 5 minutes atomic-openshift-master-api 292f7d09b7ee openshift3/openvswitch:v3.9.30 "/usr/local/bin/ov..." 5 minutes ago Up 5 minutes openvswitch fc6fc5a48749 registry.access.redhat.com/rhel7/etcd "/usr/bin/etcd" 5 minutes ago Up 5 minutes etcd_container [root@ip-10-0-1-172 ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE registry.access.redhat.com/openshift3/openvswitch v3.9.30 e37239ae2fa3 2 weeks ago 1.46 GB registry.access.redhat.com/openshift3/node v3.9.30 1fea394aac80 2 weeks ago 1.46 GB registry.access.redhat.com/openshift3/ose-web-console v3.9.30 873584bc0826 2 weeks ago 466 MB registry.access.redhat.com/openshift3/ose v3.9 6655406c26d8 2 weeks ago 1.23 GB registry.access.redhat.com/openshift3/ose v3.9.30 6655406c26d8 2 weeks ago 1.23 GB registry.access.redhat.com/openshift3/ose-pod v3.9.30 de52b000bc56 2 weeks ago 214 MB registry.access.redhat.com/rhel7/etcd latest 924412659272 5 weeks ago 256 MB
> RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = fai$ ed pulling image "openshift3/ose-pod:v3.9.30": repository docker.io/openshift3/ose-pod not found: does not exist or no pull access As far as I can tell, this isn't specific to the console install tasks. It's trying to pull the ose-pod image from DockerHub, which doesn't look right. Reassigning to install component. Scott, let me know if I can help.
This is a recently introduced regression in 3.9. https://bugzilla.redhat.com/show_bug.cgi?id=1588768 *** This bug has been marked as a duplicate of bug 1588768 ***