Bug 1615884

Summary:	Ability to install a cluster with a mix of Docker and CRI-O nodes
Product:	OpenShift Container Platform	Reporter:	Ryan Howe <rhowe>
Component:	Installer	Assignee:	Russell Teague <rteague>
Installer sub component:	openshift-ansible	QA Contact:	Johnny Liu <jialiu>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	unspecified	CC:	aos-bugs, dmoessne, dornelas, jokerman, mmccomas, nschuetz, pdwyer, rbohne, rhowe, scuppett, wmeng, xtian
Version:	3.10.0
Target Milestone:	---
Target Release:	3.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	To facilitate using cri-o as the container-runtime for OpenShift, the node-config must be updated with the correct endpoint settings. The openshift_node_groups defaults have been extended to include cri-o variants for each of the existing default node groups. To use the cri-o runtime for a group of compute nodes, the following inventory variables should be used. openshift_use_crio=True openshift_node_group_name="node-config-compute-crio" Additionally, to deploy the docker garbage collector (docker gc), the following variable must be set to True. This variable has been changed from its previous default of True to False. openshift_crio_enable_docker_gc=True	Story Points:	---
Clone Of:
Clones:	1638519 (view as bug list)		Environment:
Last Closed:	2018-11-20 03:10:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1638519

Description Ryan Howe 2018-08-14 13:17:11 UTC

Description of problem:

With the installer I should be able to install some nodes with Docker as the runtime and some with CRI-O as the runtime. 

Version-Release number of the following components:

openshift-ansible-3.10.21-1.git.0.6446011.el7.noarc

Steps to Reproduce:
Host file with openshift_use_crio=True set on one node.

```
[nodes]
10.10.95.224 openshift_node_group_name="node-config-master"
10.10.95.225 openshift_node_group_name="node-config-infra"
10.10.95.226 openshift_node_group_name="node-config-compute" openshift_use_crio=True
10.10.95.228 openshift_node_group_name="node-config-infra"
```

Actual results:

Installer fails to install and configure crio, leaving us with a node that fails to join the cluster. 

Expected results:

The specified node or node group should be configured with crio. 

Additional info:

I should be able to specify nodes or node groups to run with my choice of runtime. 

In one cluster I should be able to install nodes with different configurations for runtime
- Docker Build Node using CRIO runtime 
- Pure Docker Node 
- Pure CRI-O Node 


Documentation should follow showing what is needed.

Comment 2 Russell Teague 2018-08-16 20:56:11 UTC

I was unable to reproduce this in a test environment with the latest code.  There have been several cri-o related fixes merged since 3.10.21 which could have caused the issue presented here.  Please attempt to reproduce this with the latest build and see if the problem persists.

release-3.10 branch tested at:
openshift-ansible-3.10.29-1-13-gcd7507e14

Comment 3 Scott Dodson 2018-08-23 13:55:00 UTC

We'll add crio variants to all of our default configmaps so that configuring a node for cri-o is as simple as openshift_use_crio=true on the node and openshift_node_group_name='node-config-compute-crio'

Comment 5 Russell Teague 2018-08-23 16:05:18 UTC

Proposed (master): https://github.com/openshift/openshift-ansible/pull/9732

Comment 8 Johnny Liu 2018-09-06 18:17:01 UTC

Verified this bug with openshift-ansible-3.11.0-0.28.0.git.0.730d4be.el7.noarch, and PASS.


[nodes]
master-node  openshift_node_group_name='node-config-master'
infra-node openshift_node_group_name='node-config-infra'
pure-docker-node openshift_use_crio=False  openshift_node_group_name='node-config-compute'
pure-crio-node openshift_use_crio=True openshift_use_crio_only=True  openshift_node_group_name='node-config-compute-crio'
both-docker-crio-node openshift_use_crio=True openshift_use_crio_only=False  openshift_node_group_name='node-config-compute-crio'

Installation is completed successfully.
# oc get po -o wide
NAME                       READY     STATUS    RESTARTS   AGE       IP            NODE                              NOMINATED NODE
docker-registry-1-c96ff    1/1       Running   0          46m       10.131.0.4    qe-jialiu311test1-node-infra-1    <none>
dockergc-6jvdv             1/1       Running   0          45m       10.130.0.2    qe-jialiu311test1-nd-1            <none>
dockergc-d4sp5             1/1       Running   0          45m       10.128.2.2    qe-jialiu311test1-np-2            <none>
registry-console-1-gpwds   1/1       Running   0          46m       10.128.0.3    qe-jialiu311test1-master-etcd-1   <none>
router-1-pgk8r             1/1       Running   0          46m       10.240.0.15   qe-jialiu311test1-node-infra-1    <none>


pure-docker-node:
kubeletArguments:
  bootstrap-kubeconfig:
  - /etc/origin/node/bootstrap.kubeconfig
  cert-dir:
  - /etc/origin/node/certificates
  cloud-config:
  - /etc/origin/cloudprovider/gce.conf
  cloud-provider:
  - gce
  enable-controller-attach-detach:
  - 'true'
  feature-gates:
  - RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true
  node-labels:
  - node-role.kubernetes.io/compute=true
  pod-manifest-path:
  - /etc/origin/node/pods
  rotate-certificates:
  - 'true'


pure-crio-node:
kubeletArguments:
  bootstrap-kubeconfig:
  - /etc/origin/node/bootstrap.kubeconfig
  cert-dir:
  - /etc/origin/node/certificates
  cloud-config:
  - /etc/origin/cloudprovider/gce.conf
  cloud-provider:
  - gce
  container-runtime:
  - remote
  container-runtime-endpoint:
  - unix:///var/run/crio/crio.sock
  enable-controller-attach-detach:
  - 'true'
  feature-gates:
  - RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true
  image-service-endpoint:
  - unix:///var/run/crio/crio.sock
  node-labels:
  - node-role.kubernetes.io/compute=true,runtime=qe-crio
  pod-manifest-path:
  - /etc/origin/node/pods
  rotate-certificates:
  - 'true'
  runtime-request-timeout:
  - 10m

both-docker-crio-node:
kubeletArguments:
  bootstrap-kubeconfig:
  - /etc/origin/node/bootstrap.kubeconfig
  cert-dir:
  - /etc/origin/node/certificates
  cloud-config:
  - /etc/origin/cloudprovider/gce.conf
  cloud-provider:
  - gce
  container-runtime:
  - remote
  container-runtime-endpoint:
  - unix:///var/run/crio/crio.sock
  enable-controller-attach-detach:
  - 'true'
  feature-gates:
  - RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true
  image-service-endpoint:
  - unix:///var/run/crio/crio.sock
  node-labels:
  - node-role.kubernetes.io/compute=true,runtime=qe-crio
  pod-manifest-path:
  - /etc/origin/node/pods
  rotate-certificates:
  - 'true'
  runtime-request-timeout:
  - 10m

Labeling pure-docker-node with "a=b" as a S2I build node. And modify master-config file to make sure S2I builder pod running on the node.
admissionConfig:
  pluginConfig:
    BuildDefaults:
      configuration:
        apiVersion: v1
        env: []
        kind: BuildDefaultsConfig
        nodeSelector:
          a: b

S2I build is completed successfully.

# oc get po -n install-test -o wide
NAME                             READY     STATUS      RESTARTS   AGE       IP           NODE                     NOMINATED NODE
mongodb-1-w9lvn                  1/1       Running     0          16m       10.129.0.5   qe-jialiu311test1-np-1   <none>
nodejs-mongodb-example-1-build   0/1       Completed   0          16m       10.129.0.4   qe-jialiu311test1-np-1   <none>
nodejs-mongodb-example-1-r2v5z   1/1       Running     0          5m        10.128.2.4   qe-jialiu311test1-np-2   <none>

Comment 11 errata-xmlrpc 2018-11-20 03:10:43 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3537