Bug 1809906

Description of problem:
The conmon version is conmon-2.0.8-1.el7.x86_64.rpm in the rhel7_extra we use, it is installed but conmon-2.0.8-1.el7.x86_64.rpm don't provide /usr/libexec/crio/conmon file so crio failed to start. This file can be found in conmon-2.0.8-2.el7.x86_64.rpm


Version-Release number of selected component (if applicable):
OCP 4.3.2
rhel_repo: http://download.eng.bos.redhat.com/rhel-7/rel-eng/RHEL-7/RHEL-7.8-20200225.1/compose/Server/x86_64/os
rhel_optional_repo: http://download.eng.bos.redhat.com/rhel-7/rel-eng/RHEL-7/RHEL-7.8-20200225.1/compose/Server-optional/x86_64/os

How reproducible:
Always

Steps to Reproduce:
1.Set up an OCP cluster at version 4.3.2, then add RHEL 78 RC worker nodes 


Actual results:
1. crio service on RHEL 78 rc worker nodes can't be started, here are some logs and messages
$ systemctl status crio
● crio.service - Open Container Initiative Daemon
   Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2020-03-04 01:24:27 EST; 3min 4s ago
     Docs: https://github.com/cri-o/cri-o
  Process: 10693 ExecStart=/usr/bin/crio $CRIO_STORAGE_OPTIONS $CRIO_NETWORK_OPTIONS $CRIO_METRICS_OPTIONS (code=exited, status=1/FAILURE)
 Main PID: 10693 (code=exited, status=1/FAILURE)

Mar 04 01:24:27 qe-lpt-4613-k6nlr-rhel-3 systemd[1]: Starting Open Container Initiative Daemon...
Mar 04 01:24:27 qe-lpt-4613-k6nlr-rhel-3 crio[10693]: time="2020-03-04 01:24:27.618774609-05:00" level=fatal msg="runtime config: ...ctory"
Mar 04 01:24:27 qe-lpt-4613-k6nlr-rhel-3 systemd[1]: crio.service: main process exited, code=exited, status=1/FAILURE
Mar 04 01:24:27 qe-lpt-4613-k6nlr-rhel-3 systemd[1]: Failed to start Open Container Initiative Daemon.
Mar 04 01:24:27 qe-lpt-4613-k6nlr-rhel-3 systemd[1]: Unit crio.service entered failed state.
Mar 04 01:24:27 qe-lpt-4613-k6nlr-rhel-3 systemd[1]: crio.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

$ journalctl -a | grep crio
Mar 04 01:24:19 qe-lpt-4613-k6nlr-rhel-3 ansible-systemd[10394]: Invoked with no_block=False force=None name=crio daemon_reexec=False enabled=True daemon_reload=False state=None masked=None scope=None user=None
Mar 04 01:24:25 qe-lpt-4613-k6nlr-rhel-3 ansible-systemd[10665]: Invoked with no_block=False force=None name=crio daemon_reexec=False enabled=None daemon_reload=False state=restarted masked=None scope=None user=None
Mar 04 01:24:27 qe-lpt-4613-k6nlr-rhel-3 crio[10670]: version file /var/lib/crio/version not found: open /var/lib/crio/version: no such file or directory. Triggering wipe
Mar 04 01:24:27 qe-lpt-4613-k6nlr-rhel-3 crio[10693]: time="2020-03-04 01:24:27.618774609-05:00" level=fatal msg="runtime config: conmon validation: invalid conmon path: stat /usr/libexec/crio/conmon: no such file or directory"
Mar 04 01:24:27 qe-lpt-4613-k6nlr-rhel-3 systemd[1]: crio.service: main process exited, code=exited, status=1/FAILURE
Mar 04 01:24:27 qe-lpt-4613-k6nlr-rhel-3 systemd[1]: Unit crio.service entered failed state.
Mar 04 01:24:27 qe-lpt-4613-k6nlr-rhel-3 systemd[1]: crio.service failed.

$ rpm -qa | grep conmon
conmon-2.0.8-1.el7.x86_64

$ rpm -qi conmon-2.0.8-1.el7.x86_64
Name        : conmon
Epoch       : 2
Version     : 2.0.8
Release     : 1.el7
Architecture: x86_64
Install Date: Wed Mar  4 01:23:04 2020
Group       : Unspecified
Size        : 82409
License     : ASL 2.0
Signature   : RSA/SHA256, Fri Jan 17 10:14:59 2020, Key ID 199e2f91fd431d51
Source RPM  : conmon-2.0.8-1.el7.src.rpm
Build Date  : Mon Dec 16 08:01:19 2019
Build Host  : x86-vm-27.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : https://github.com/containers/conmon
Summary     : OCI container runtime monitor
Description :
OCI container runtime monitor.

$ repoquery -i conmon-2.0.8-1.el7.x86_64
Failed to set locale, defaulting to C

Name        : conmon
Version     : 2.0.8
Release     : 1.el7
Architecture: x86_64
Size        : 82409
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Group       : Unspecified
URL         : https://github.com/containers/conmon
Repository  : rhel7-extra
Summary     : OCI container runtime monitor
Source      : conmon-2.0.8-1.el7.src.rpm
Description :
OCI container runtime monitor.

$ cat /etc/yum.repos.d/qe_additional.repo 
[rhel7]
name=rhel7
baseurl=http://download.eng.bos.redhat.com/rhel-7/rel-eng/RHEL-7/RHEL-7.8-20200225.1/compose/Server/x86_64/os
metadata_expire=1
enabled=1
gpgcheck=0
[rhel7-extra]
name=rhel7-extra
baseurl=http://download.eng.bos.redhat.com/rhel-7/rel-eng/EXTRAS-7/latest-EXTRAS-7.8-RHEL-7/compose/Server/x86_64/os
metadata_expire=1
enabled=1
gpgcheck=0
[rhel7-optional]
name=rhel7-optional
baseurl=http://download.eng.bos.redhat.com/rhel-7/rel-eng/RHEL-7/RHEL-7.8-20200225.1/compose/Server-optional/x86_64/os
metadata_expire=1
enabled=1
gpgcheck=0
[aos-v4-install]
name=aos-v4-install
baseurl=http://download.eng.bos.redhat.com/rcm-guest/puddles/RHAOS/AtomicOpenShift/4.3/latest/x86_64/os/
metadata_expire=1
enabled=1
gpgcheck=0


Expected results:
1. crio service should be started and running successfully

Additional info:
Adding rhel77 worker nodes works as expected, also adding rhel 78 works nodes to OCP 4.1 & OCP 4.2 work well too