Bug 1781019

Summary: Fail to start crio service on RHEL worker
Product: OpenShift Container Platform Reporter: Gaoyun Pei <gpei>
Component: Machine Config OperatorAssignee: Peter Hunt <pehunt>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: high Docs Contact:
Priority: high    
Version: 4.3.0CC: jialiu, kgarriso, wsun, yanyang
Target Milestone: ---Keywords: TestBlocker
Target Release: 4.3.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-23 11:18:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1782843    
Bug Blocks:    

Description Gaoyun Pei 2019-12-09 05:57:03 UTC
Description of problem:
With the new cri-o package(cri-o-1.16.1-2.dev.rhaos4.3.git7b04b62.el7.x86_64) installed on RHEL worker, crio.service failed to get started.

-- Unit crio.service has begun starting up.
Dec 09 00:49:57 ip-10-0-62-115.us-east-2.compute.internal crio[17896]: time="2019-12-09 00:49:57.366178213-05:00" level=fatal msg="runtime config: invalid hooks_dir: stat /usr/share/containers/oci/hooks.d: no such file or directory: stat /usr/share/containers/oci/hooks.d: no such file or directory"
Dec 09 00:49:57 ip-10-0-62-115.us-east-2.compute.internal systemd[1]: crio.service: main process exited, code=exited, status=1/FAILURE
Dec 09 00:49:57 ip-10-0-62-115.us-east-2.compute.internal systemd[1]: Failed to start Open Container Initiative Daemon.

Compared to cri-o-1.14, there're two directories added in the hooks_dir
[root@ip-10-0-62-115 ec2-user]# grep -A 3 hooks_dir /etc/crio/crio.conf
hooks_dir = [

[root@ip-10-0-62-115 ec2-user]# ls /usr/share/containers/oci/hooks.d
ls: cannot access /usr/share/containers/oci/hooks.d: No such file or directory

On RHCOS worker, it's provided by oci-systemd-hook package.

# rpm -qf /usr/share/containers/oci/hooks.d

Version-Release number of the following components:
openshift-ansible.noarch 0:4.3.0-201912061055.git.177.a1caf57.el7

How reproducible:

Steps to Reproduce:

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Kirsten Garrison 2019-12-10 18:20:28 UTC
@Peter  this is relevant to the crio dirs

Comment 2 Peter Hunt 2019-12-11 16:52:26 UTC
I opened a PR on master to fix in MCO

Comment 3 Peter Hunt 2019-12-12 13:33:05 UTC
4.3 backport linked

Comment 5 Johnny Liu 2019-12-13 10:46:18 UTC
Verified this bug with cri-o-1.16.1-6.dev.rhaos4.3.gitfa42e2f.el7.x86_64 and openshift-ansible-4.3.0-201912061055.git.177.a1caf57.el7.noarch, PASS.

$ cat /etc/crio/crio.conf
hooks_dir = [

crio service is started successfully.

# journalctl -f  -u crio.service 
-- Logs begin at Fri 2019-12-13 01:00:24 EST. --
Dec 13 01:00:33 xiyuan-1210-zxpm6-rhel-0 systemd[1]: Starting Open Container Initiative Daemon...
Dec 13 01:00:34 xiyuan-1210-zxpm6-rhel-0 systemd[1]: Started Open Container Initiative Daemon.
Dec 13 01:01:33 xiyuan-1210-zxpm6-rhel-0 crio[1859]: 2019-12-13T01:01:33-05:00 [verbose] Add: openshift-dns:dns-default-n2wcf:openshift-sdn:eth0 {"cniVersion":"0.3.1","interfaces":[{"name":"eth0","sandbox":"/proc/12600/ns/net"}],"ips":[{"version":"4","interface":0,"address":""}],"routes":[{"dst":"","gw":""},{"dst":""},{"dst":""}],"dns":{}}
Dec 13 01:01:34 xiyuan-1210-zxpm6-rhel-0 crio[1859]: 2019-12-13T01:01:34-05:00 [verbose] Add: openshift-image-registry:node-ca-ntbzv:openshift-sdn:eth0 {"cniVersion":"0.3.1","interfaces":[{"name":"eth0","sandbox":"/proc/12608/ns/net"}],"ips":[{"version":"4","interface":0,"address":""}],"routes":[{"dst":"","gw":""},{"dst":""},{"dst":""}],"dns":{}}

Comment 7 errata-xmlrpc 2020-01-23 11:18:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.