Bug 1567647

Summary: crio unable to pull images
Product: OpenShift Container Platform Reporter: raffaele spazzoli <rspazzol>
Component: InstallerAssignee: Scott Dodson <sdodson>
Status: CLOSED CURRENTRELEASE QA Contact: Johnny Liu <jialiu>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.9.0CC: amurdaca, aos-bugs, bleanhar, jliggitt, jokerman, mmccomas, scuppett, wmeng
Target Milestone: ---   
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1614353 (view as bug list) Environment:
Last Closed: 2018-08-09 13:11:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1614353    

Description raffaele spazzoli 2018-04-15 18:27:30 UTC
Description of problem:

when a OCP is configured to run with crio as the container runtime, it looks like crio is not able to pull images. Here is the log from the node service:

Apr 15 18:20:03 env1-infranode-frbn atomic-openshift-node[24200]: E0415 18:20:03.420087   24200 remote_image.go:83] ImageStatus "openshift3/ose-deployer:v3.9.14" from image service failed: rpc error: code = Unknown desc = no registries configured while trying to pull an unqualified image
Apr 15 18:20:03 env1-infranode-frbn atomic-openshift-node[24200]: E0415 18:20:03.420135   24200 kuberuntime_image.go:86] ImageStatus for image {"openshift3/ose-deployer:v3.9.14"} failed: rpc error: code = Unknown desc = no registries configured while trying to pull an unqualified image
Apr 15 18:20:03 env1-infranode-frbn atomic-openshift-node[24200]: E0415 18:20:03.420215   24200 kuberuntime_manager.go:734] container start failed: ImageInspectError: Failed to inspect image "openshift3/ose-deployer:v3.9.14": rpc error: code = Unknown desc = no registries configured while trying to pull an unqualified image
Apr 15 18:20:03 env1-infranode-frbn atomic-openshift-node[24200]: E0415 18:20:03.420243   24200 pod_workers.go:186] Error syncing pod 7ebb607f-40d7-11e8-a282-42010a800009 ("docker-registry-1-deploy_default(7ebb607f-40d7-11e8-a282-42010a800009)"), skipping: failed to "StartContainer" for "deployment" with ImageInspectError: "Failed to inspect image \"openshift3/ose-deployer:v3.9.14\": rpc error: code = Unknown desc = no registries configured while trying to pull an unqualified image"



crio service log:

[root@env1-infranode-frbn ~]# journalctl -u crio
-- Logs begin at Sun 2018-04-15 17:16:49 UTC, end at Sun 2018-04-15 18:18:23 UTC. --
Apr 15 17:17:42 env1-infranode-frbn systemd[1]: Starting Open Container Initiative Daemon...
Apr 15 17:17:42 env1-infranode-frbn crio[9979]: time="2018-04-15 17:17:42.975389179Z" level=warning msg="hooks path: "/etc/containers/oci/hooks.d" does not exist"
Apr 15 17:17:42 env1-infranode-frbn crio[9979]: time="2018-04-15 17:17:42.976677349Z" level=info msg="CNI network crio-bridge (type=bridge) is used from /etc/cni/net.d/100-crio-bridge.conf"
Apr 15 17:17:42 env1-infranode-frbn crio[9979]: time="2018-04-15 17:17:42.976929247Z" level=info msg="CNI network crio-bridge (type=bridge) is used from /etc/cni/net.d/100-crio-bridge.conf"
Apr 15 17:17:42 env1-infranode-frbn systemd[1]: Started Open Container Initiative Daemon.
Apr 15 17:53:47 env1-infranode-frbn systemd[1]: Stopping Open Container Initiative Daemon...
Apr 15 17:53:47 env1-infranode-frbn crio[9979]: time="2018-04-15 17:53:47.588580466Z" level=error msg="Failed to start streaming server: http: Server closed"
Apr 15 17:53:47 env1-infranode-frbn systemd[1]: Starting Open Container Initiative Daemon...
Apr 15 17:53:47 env1-infranode-frbn crio[18862]: time="2018-04-15 17:53:47.838258492Z" level=warning msg="hooks path: "/etc/containers/oci/hooks.d" does not exist"
Apr 15 17:53:47 env1-infranode-frbn crio[18862]: time="2018-04-15 17:53:47.838686727Z" level=info msg="CNI network crio-bridge (type=bridge) is used from /etc/cni/net.d/100-crio-bridge.conf
Apr 15 17:53:47 env1-infranode-frbn crio[18862]: time="2018-04-15 17:53:47.838800539Z" level=info msg="CNI network crio-bridge (type=bridge) is used from /etc/cni/net.d/100-crio-bridge.conf
Apr 15 17:53:47 env1-infranode-frbn systemd[1]: Started Open Container Initiative Daemon.
Apr 15 18:04:53 env1-infranode-frbn crio[18862]: time="2018-04-15 18:04:53.932224968Z" level=info msg="Running conmon under slice kubepods-besteffort-pod7ebb607f_40d7_11e8_a282_42010a800009
Apr 15 18:04:54 env1-infranode-frbn crio[18862]: time="2018-04-15 18:04:54.044941132Z" level=info msg="Got pod network {Name:docker-registry-1-deploy Namespace:default ID:ae16d95891e21842e8
Apr 15 18:04:54 env1-infranode-frbn crio[18862]: time="2018-04-15 18:04:54.044965292Z" level=info msg="About to add CNI network cni-loopback (type=loopback)"
Apr 15 18:04:54 env1-infranode-frbn crio[18862]: time="2018-04-15 18:04:54.056499292Z" level=info msg="Got pod network {Name:docker-registry-1-deploy Namespace:default ID:ae16d95891e21842e8
Apr 15 18:04:54 env1-infranode-frbn crio[18862]: time="2018-04-15 18:04:54.056525738Z" level=info msg="About to add CNI network crio-bridge (type=bridge)"

Actual results:
unable to pull images


Expected results:
image should be pulled and container started

Additional info:
with the same settings and without crio, images are being pulled. The only thing that I added was:
openshift_use_crio: true
openshift_crio_enable_docker_gc: true  


I saw in this blog (https://medium.com/cri-o/how-to-run-cri-o-1-9-10-with-openshift-container-platform-3-9-and-red-hat-enterprise-linux-7-4-c8ecf47c66b5) that one may have to add the following:
oreg_url=registry.access.redhat.com/openshift3/ose-${component}:${version}

If that is the case then this should be turned into a doc bug.

Comment 1 Antonio Murdaca 2018-04-17 14:07:13 UTC
Yes, you actually need to setup registries during installation

Comment 2 raffaele spazzoli 2018-04-17 14:17:50 UTC
Antonio, "you actually need to setup registries during installation" what does that mean and where is that process documented?

Comment 4 Brenton Leanhardt 2018-08-09 13:11:16 UTC
Hi Raffaele,

Starting in 3.10 we now fully qualify all images by default. As that blog post mentions you should set oreg_url.  To Antonio's point about registries, all registries are configured by default in 3.10 so I'm going to close this as fixed in the current release.  I'll ask our docs team to add a "known issue" that 3.9 uses should be sure to set oreg_url when using cri-o.