Bug 1777838 - crio-1.16 does not execute or read in hook description
Summary: crio-1.16 does not execute or read in hook description
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers
Version: 4.3.0
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: ---
: 4.3.0
Assignee: Peter Hunt
QA Contact: weiwei jiang
URL:
Whiteboard:
Depends On:
Blocks: 1769174 1794257
TreeView+ depends on / blocked
 
Reported: 2019-11-28 12:25 UTC by Zvonko Kosic
Modified: 2023-09-14 05:47 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-23 11:14:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github cri-o cri-o pull 3011 0 'None' closed [1.16] add default hooks dir to crio.conf 2020-11-05 12:42:32 UTC
Github openshift machine-config-operator pull 1299 0 'None' closed CRI-O: add old default hook directories 2020-11-05 12:42:32 UTC
Github openshift machine-config-operator pull 1305 0 'None' closed Bug 1777838: [release-4.3] CRI-O: add old default hook directories 2020-11-05 12:42:32 UTC

Description Zvonko Kosic 2019-11-28 12:25:52 UTC
Description of problem:

Looks like the crio version ( 1.16.0-0.4.dev.rhaos4.3.giteed6aa1.el8-rc2) I have in my 4.3 cluster is not executing any hooks, I added log-level=debug and I cannot see that crio is trying to read any hooks from /usr/share/containers/oci/hooks.d or /etc/containers/oci/hooks.d  and I do not see any log of "AddHook", 

Version-Release number of selected component (if applicable):

crio version 1.16.0-0.4.dev.rhaos4.3.giteed6aa1.el8-rc2
commit: "c64f75c08173ed16b4aa09252eb07f669c2c5c39-dirty"

How reproducible:


Steps to Reproduce:

Install a recent 4.3 cluster enable log-level=debug and look for any mention of hook in the logs. 



Actual results:

# journalctl -u -f crio | grep hook
#EMPTY


Expected results:
# journalctl -u -f crio | grep hook

DEBU[2019-11-27 20:59:13.286157549Z] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[2019-11-27 20:59:13.286295598Z] added hook /usr/share/containers/oci/hooks.d/oci-systemd-hook.json 
DEBU[2019-11-27 20:59:13.286339362Z] reading hooks from /etc/containers/oci/hooks.d 
17:00
DEBU[2019-11-27 20:59:13.544742728Z] monitoring "/usr/share/containers/oci/hooks.d" for hooks 
DEBU[2019-11-27 20:59:13.544770728Z] monitoring "/etc/containers/oci/hooks.d" for hooks 


Additional info:

Ran crio-1.14 on the 4.3 node and i got the desired output.

Comment 1 Valentin Rothberg 2019-11-28 12:54:09 UTC
The behaviour w.r.t. hooks has changed a bit with 1.16 (see https://github.com/cri-o/cri-o/pull/2731). The `hooks_dir` option must now be set explicitly in the crio.conf.

Can you check if it's set in the crio.conf and retest?

Comment 3 Zvonko Kosic 2019-11-28 13:46:00 UTC
Setting the hooks_dir = [] works only if I restart crio, SIGHUP didn't to anything. 

I do not want to restart crio at all cost. 


Is the oci-systemd-hook.json optional? Shouldn't this thing be activated per default?

Comment 4 Zvonko Kosic 2019-11-28 13:48:08 UTC
Looking at the PR, all the things I need to make NVIDIA work was removed from cri-o, this is bad, see this issue https://github.com/cri-o/cri-o/pull/1943 and the folllowing PR https://github.com/cri-o/cri-o/pull/1943

if config.HooksDir == nil {
		for _, hooksDir := range []string{hooks.DefaultDir, hooks.OverrideDir} {
			_, err = os.Stat(hooksDir)
			if err == nil {
				hookDirectories = append(hookDirectories, hooksDir)
				logrus.Warnf("implicit hook directories are deprecated; set --hooks-dir=%q explicitly to continue to load hooks from this directory", hooksDir)
			}
		}
	}

this was added ^^ and the new PR removed it

/etc/container/oci/hooks.d is the only place where custom hooks can be placed on RHCOS this is needed by several customers.

Comment 5 Zvonko Kosic 2019-11-28 13:56:47 UTC
From an OpenShift, Operator perspective I do not want to/can restart crio, edit files on the Node, send signals to processes (which means I need elevated privileges). 

How crio worked before was a desired state I needed for several customers (NVIDIA, SolarFlare, Intel) so they can easily leverage the hooks in crio.

Comment 6 Valentin Rothberg 2019-11-28 15:21:18 UTC
I think that the breaking upstream change was not reflected in the packages or configurator to properly set the hooks_dir. We can either revert the upstream change or set the hooks_dir.

Mrunal, Peter, what do you think?

In general, I really don't like breaking changes such as with hooks_dir. It's always tricky to catch them, either in package updates or with configurators.

Comment 7 Peter Hunt 2019-12-02 16:24:12 UTC
So I've opened a PR upstream for including a default hooks dir in the crio.conf. That prevents any fixes in the machine config operator, which I think is good. 

Zvonko, are the desired hooks in `/usr/share/containers/oci/hooks.d` ?

Comment 9 Zvonko Kosic 2019-12-02 17:38:18 UTC
Peter, 

it is /etc/containers/oci/hooks.d, the /usr/... is not writable when running OCP with RHCOS.

Comment 10 Zvonko Kosic 2019-12-02 17:40:53 UTC
Peter, 

oci-systemd-hook.json is in /usr/share/containers/oci/hooks.d is this hook optional?

Comment 11 Peter Hunt 2019-12-02 18:07:26 UTC
Yes this hook is optional. In fact, I think it's even deprecated. I'll open a PR in MCO to point towards /etc/containers/oci/hook.d

Comment 16 weiwei jiang 2019-12-10 02:42:16 UTC
Checked with cri-o-1.16.1-2.dev.rhaos4.3.git7b04b62.el8


sh-4.4# journalctl  -u crio |grep -i hook                                                                                                                                                     
                                                
Dec 09 06:50:31 ip-10-0-72-197 crio[447394]: time="2019-12-09 06:50:31.354430625Z" level=debug msg="reading hooks from /etc/containers/oci/hooks.d" file="hooks/read.go:65"
Dec 09 06:50:31 ip-10-0-72-197 crio[447394]: time="2019-12-09 06:50:31.354491956Z" level=debug msg="reading hooks from /usr/share/containers/oci/hooks.d" file="hooks/read.go:65"
Dec 09 06:50:31 ip-10-0-72-197 crio[447394]: time="2019-12-09 06:50:31.354646242Z" level=debug msg="added hook /usr/share/containers/oci/hooks.d/oci-systemd-hook.json" file="hooks/read.go:91
"
Dec 09 06:50:33 ip-10-0-72-197 crio[447394]: time="2019-12-09 06:50:33.170015801Z" level=debug msg="monitoring \"/etc/containers/oci/hooks.d\" for hooks" file="hooks/monitor.go:43"
Dec 09 06:50:33 ip-10-0-72-197 crio[447394]: time="2019-12-09 06:50:33.171674411Z" level=debug msg="monitoring \"/usr/share/containers/oci/hooks.d\" for hooks" file="hooks/monitor.go:43"
Dec 09 06:51:01 ip-10-0-72-197 crio[447394]: time="2019-12-09 06:51:01.634261654Z" level=debug msg="hook oci-systemd-hook.json did not match" file="hooks/hooks.go:135"
sh-4.4# rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a8d26fe5744bc92205a6df0677b0666cc08fb36f26184aacfc9e748cbe38c9d
              CustomOrigin: Managed by machine-config-operator
                   Version: 43.81.201912081754.0 (2019-12-08T17:59:29Z)

  ostree://e884477421640d1285c07a6dd9aaf01c9e125038ebbe6290a5e341eb3695a4d1
                   Version: 43.81.201911221453.0 (2019-11-22T14:58:44Z)

Comment 18 errata-xmlrpc 2020-01-23 11:14:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062

Comment 19 Red Hat Bugzilla 2023-09-14 05:47:45 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.