Bug 1567647 - crio unable to pull images
Summary: crio unable to pull images
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.9.z
Assignee: Scott Dodson
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks: 1614353
TreeView+ depends on / blocked
 
Reported: 2018-04-15 18:27 UTC by raffaele spazzoli
Modified: 2018-08-09 13:11 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1614353 (view as bug list)
Environment:
Last Closed: 2018-08-09 13:11:16 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description raffaele spazzoli 2018-04-15 18:27:30 UTC
Description of problem:

when a OCP is configured to run with crio as the container runtime, it looks like crio is not able to pull images. Here is the log from the node service:

Apr 15 18:20:03 env1-infranode-frbn atomic-openshift-node[24200]: E0415 18:20:03.420087   24200 remote_image.go:83] ImageStatus "openshift3/ose-deployer:v3.9.14" from image service failed: rpc error: code = Unknown desc = no registries configured while trying to pull an unqualified image
Apr 15 18:20:03 env1-infranode-frbn atomic-openshift-node[24200]: E0415 18:20:03.420135   24200 kuberuntime_image.go:86] ImageStatus for image {"openshift3/ose-deployer:v3.9.14"} failed: rpc error: code = Unknown desc = no registries configured while trying to pull an unqualified image
Apr 15 18:20:03 env1-infranode-frbn atomic-openshift-node[24200]: E0415 18:20:03.420215   24200 kuberuntime_manager.go:734] container start failed: ImageInspectError: Failed to inspect image "openshift3/ose-deployer:v3.9.14": rpc error: code = Unknown desc = no registries configured while trying to pull an unqualified image
Apr 15 18:20:03 env1-infranode-frbn atomic-openshift-node[24200]: E0415 18:20:03.420243   24200 pod_workers.go:186] Error syncing pod 7ebb607f-40d7-11e8-a282-42010a800009 ("docker-registry-1-deploy_default(7ebb607f-40d7-11e8-a282-42010a800009)"), skipping: failed to "StartContainer" for "deployment" with ImageInspectError: "Failed to inspect image \"openshift3/ose-deployer:v3.9.14\": rpc error: code = Unknown desc = no registries configured while trying to pull an unqualified image"



crio service log:

[root@env1-infranode-frbn ~]# journalctl -u crio
-- Logs begin at Sun 2018-04-15 17:16:49 UTC, end at Sun 2018-04-15 18:18:23 UTC. --
Apr 15 17:17:42 env1-infranode-frbn systemd[1]: Starting Open Container Initiative Daemon...
Apr 15 17:17:42 env1-infranode-frbn crio[9979]: time="2018-04-15 17:17:42.975389179Z" level=warning msg="hooks path: "/etc/containers/oci/hooks.d" does not exist"
Apr 15 17:17:42 env1-infranode-frbn crio[9979]: time="2018-04-15 17:17:42.976677349Z" level=info msg="CNI network crio-bridge (type=bridge) is used from /etc/cni/net.d/100-crio-bridge.conf"
Apr 15 17:17:42 env1-infranode-frbn crio[9979]: time="2018-04-15 17:17:42.976929247Z" level=info msg="CNI network crio-bridge (type=bridge) is used from /etc/cni/net.d/100-crio-bridge.conf"
Apr 15 17:17:42 env1-infranode-frbn systemd[1]: Started Open Container Initiative Daemon.
Apr 15 17:53:47 env1-infranode-frbn systemd[1]: Stopping Open Container Initiative Daemon...
Apr 15 17:53:47 env1-infranode-frbn crio[9979]: time="2018-04-15 17:53:47.588580466Z" level=error msg="Failed to start streaming server: http: Server closed"
Apr 15 17:53:47 env1-infranode-frbn systemd[1]: Starting Open Container Initiative Daemon...
Apr 15 17:53:47 env1-infranode-frbn crio[18862]: time="2018-04-15 17:53:47.838258492Z" level=warning msg="hooks path: "/etc/containers/oci/hooks.d" does not exist"
Apr 15 17:53:47 env1-infranode-frbn crio[18862]: time="2018-04-15 17:53:47.838686727Z" level=info msg="CNI network crio-bridge (type=bridge) is used from /etc/cni/net.d/100-crio-bridge.conf
Apr 15 17:53:47 env1-infranode-frbn crio[18862]: time="2018-04-15 17:53:47.838800539Z" level=info msg="CNI network crio-bridge (type=bridge) is used from /etc/cni/net.d/100-crio-bridge.conf
Apr 15 17:53:47 env1-infranode-frbn systemd[1]: Started Open Container Initiative Daemon.
Apr 15 18:04:53 env1-infranode-frbn crio[18862]: time="2018-04-15 18:04:53.932224968Z" level=info msg="Running conmon under slice kubepods-besteffort-pod7ebb607f_40d7_11e8_a282_42010a800009
Apr 15 18:04:54 env1-infranode-frbn crio[18862]: time="2018-04-15 18:04:54.044941132Z" level=info msg="Got pod network {Name:docker-registry-1-deploy Namespace:default ID:ae16d95891e21842e8
Apr 15 18:04:54 env1-infranode-frbn crio[18862]: time="2018-04-15 18:04:54.044965292Z" level=info msg="About to add CNI network cni-loopback (type=loopback)"
Apr 15 18:04:54 env1-infranode-frbn crio[18862]: time="2018-04-15 18:04:54.056499292Z" level=info msg="Got pod network {Name:docker-registry-1-deploy Namespace:default ID:ae16d95891e21842e8
Apr 15 18:04:54 env1-infranode-frbn crio[18862]: time="2018-04-15 18:04:54.056525738Z" level=info msg="About to add CNI network crio-bridge (type=bridge)"

Actual results:
unable to pull images


Expected results:
image should be pulled and container started

Additional info:
with the same settings and without crio, images are being pulled. The only thing that I added was:
openshift_use_crio: true
openshift_crio_enable_docker_gc: true  


I saw in this blog (https://medium.com/cri-o/how-to-run-cri-o-1-9-10-with-openshift-container-platform-3-9-and-red-hat-enterprise-linux-7-4-c8ecf47c66b5) that one may have to add the following:
oreg_url=registry.access.redhat.com/openshift3/ose-${component}:${version}

If that is the case then this should be turned into a doc bug.

Comment 1 Antonio Murdaca 2018-04-17 14:07:13 UTC
Yes, you actually need to setup registries during installation

Comment 2 raffaele spazzoli 2018-04-17 14:17:50 UTC
Antonio, "you actually need to setup registries during installation" what does that mean and where is that process documented?

Comment 4 Brenton Leanhardt 2018-08-09 13:11:16 UTC
Hi Raffaele,

Starting in 3.10 we now fully qualify all images by default. As that blog post mentions you should set oreg_url.  To Antonio's point about registries, all registries are configured by default in 3.10 so I'm going to close this as fixed in the current release.  I'll ask our docs team to add a "known issue" that 3.9 uses should be sure to set oreg_url when using cri-o.


Note You need to log in before you can comment on or make changes to this bug.