Description of problem: Using openstack playbooks I'm trying to have an environment such as: * Masters with cri-o only * App nodes with cri-o only * Infra nodes with docker only So I have: $ grep osm_use_cockpit all.yml osm_use_cockpit: false $ cat inventory/group_vars/masters.yml openshift_use_crio_only: true openshift_use_crio: true openshift_node_group_name: node-config-master-crio $ cat inventory/group_vars/openstack_compute_nodes.yml openshift_use_crio_only: true openshift_use_crio: true openshift_node_group_name: node-config-compute-crio $ cat inventory/group_vars/openstack_infra_nodes.yml openshift_use_crio_only: false openshift_use_crio: false openshift_node_group_name: node-config-infra Also, I need to 'patch' the roles/openshift_node/defaults/main.yml until this is released https://github.com/openshift/openshift-ansible/pull/10501/files The installer skips docker installation in cri-o nodes but when creating the node-config.yaml file for infra nodes, it edits the node-config-infra to add the cri-o settings even if it shouldn't. Logs: ----8<---- 2018-11-07 06:48:33,684 p=18330 u=cloud-user | TASK [openshift_node_group : fetch node configmap] ***************************** 2018-11-07 06:48:33,684 p=18330 u=cloud-user | Wednesday 07 November 2018 06:48:33 -0500 (0:00:00.071) 1:14:20.993 **** 2018-11-07 06:48:35,090 p=18330 u=cloud-user | ok: [master-0.shiftstack.automated.lan] 2018-11-07 06:48:35,111 p=18330 u=cloud-user | TASK [openshift_node_group : debug node config] ******************************** 2018-11-07 06:48:35,111 p=18330 u=cloud-user | Wednesday 07 November 2018 06:48:35 -0500 (0:00:01.427) 1:14:22.421 **** 2018-11-07 06:48:35,155 p=18330 u=cloud-user | ok: [master-0.shiftstack.automated.lan] => { "configout": { "changed": false, "failed": false, "results": { "cmd": "/bin/oc get configmap node-config-infra -o json -n openshift-node", "results": [ {} ], "returncode": 0, "stderr": "Error from server (NotFound): configmaps \"node-config-infra\" not found\n", "stdout": "" }, "state": "list" } } 2018-11-07 06:48:35,177 p=18330 u=cloud-user | TASK [openshift_node_group : create a temp dir for this work] ****************** 2018-11-07 06:48:35,177 p=18330 u=cloud-user | Wednesday 07 November 2018 06:48:35 -0500 (0:00:00.065) 1:14:22.487 **** 2018-11-07 06:48:35,875 p=18330 u=cloud-user | changed: [master-0.shiftstack.automated.lan] 2018-11-07 06:48:35,899 p=18330 u=cloud-user | TASK [openshift_node_group : create node config template] ********************** 2018-11-07 06:48:35,899 p=18330 u=cloud-user | Wednesday 07 November 2018 06:48:35 -0500 (0:00:00.721) 1:14:23.209 **** 2018-11-07 06:48:38,791 p=18330 u=cloud-user | changed: [master-0.shiftstack.automated.lan] 2018-11-07 06:48:38,815 p=18330 u=cloud-user | TASK [openshift_node_group : lay down the config from the existing configmap] *** 2018-11-07 06:48:38,816 p=18330 u=cloud-user | Wednesday 07 November 2018 06:48:38 -0500 (0:00:02.916) 1:14:26.125 **** 2018-11-07 06:48:38,836 p=18330 u=cloud-user | skipping: [master-0.shiftstack.automated.lan] 2018-11-07 06:48:38,859 p=18330 u=cloud-user | TASK [openshift_node_group : specialize the generated configs for node-config-infra] *** 2018-11-07 06:48:38,859 p=18330 u=cloud-user | Wednesday 07 November 2018 06:48:38 -0500 (0:00:00.043) 1:14:26.169 **** 2018-11-07 06:48:39,712 p=18330 u=cloud-user | changed: [master-0.shiftstack.automated.lan] 2018-11-07 06:48:39,737 p=18330 u=cloud-user | TASK [openshift_node_group : show the yeditout debug var] ********************** 2018-11-07 06:48:39,737 p=18330 u=cloud-user | Wednesday 07 November 2018 06:48:39 -0500 (0:00:00.877) 1:14:27.047 **** 2018-11-07 06:48:39,786 p=18330 u=cloud-user | ok: [master-0.shiftstack.automated.lan] => { "yeditout": { "changed": true, "failed": false, "result": [ { "edit": { "apiVersion": "v1", "authConfig": { "authenticationCacheSize": 1000, "authenticationCacheTTL": "5m", "authorizationCacheSize": 1000, "authorizationCacheTTL": "5m" }, "dnsBindAddress": "127.0.0.1:53", "dnsDomain": "cluster.local", "dnsIP": "0.0.0.0", "dnsNameservers": null, "dnsRecursiveResolvConf": "/etc/origin/node/resolv.conf", "dockerConfig": { "dockerShimRootDirectory": "/var/lib/dockershim", "dockerShimSocket": "/var/run/dockershim.sock", "execHandlerName": "native" }, "enableUnidling": true, "imageConfig": { "format": "registry.redhat.io/openshift3/ose-${component}:${version}", "latest": false }, "iptablesSyncPeriod": "30s", "kind": "NodeConfig", "kubeletArguments": { "bootstrap-kubeconfig": [ "/etc/origin/node/bootstrap.kubeconfig" ], "cert-dir": [ "/etc/origin/node/certificates" ], "cloud-config": [ "/etc/origin/cloudprovider/openstack.conf" ], "cloud-provider": [ "openstack" ], "container-runtime": [ "remote" ], ], "container-runtime-endpoint": [ "/var/run/crio/crio.sock" ], "enable-controller-attach-detach": [ "true" ], "feature-gates": [ "RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true" ], "image-service-endpoint": [ "/var/run/crio/crio.sock" ], "node-labels": [ "node-role.kubernetes.io/infra=true" ], "pod-manifest-path": [ "/etc/origin/node/pods" ], "rotate-certificates": [ "true" ], "runtime-request-timeout": [ "10m" ] }, "masterClientConnectionOverrides": { "acceptContentTypes": "application/vnd.kubernetes.protobuf,application/json", "burst": 40, "contentType": "application/vnd.kubernetes.protobuf", "qps": 20 }, "masterKubeConfig": "node.kubeconfig", "networkConfig": { "mtu": 1450, "networkPluginName": "redhat/openshift-ovs-subnet" }, "servingInfo": { "bindAddress": "0.0.0.0:10250", "bindNetwork": "tcp4", "clientCA": "client-ca.crt" }, "volumeConfig": { "localQuota": { "perFSGroup": null } }, "volumeDirectory": "/var/lib/origin/openshift.local.volumes" }, "key": "kubeletArguments.node-labels" } ], "state": "present" } } ---->8---- Version-Release number of the following components: $ rpm -q openshift-ansible openshift-ansible-3.11.16-1.git.0.4ac6f81.el7.noarch $ rpm -q ansible ansible-2.5.7-1.el7ae.noarch $ ansible --version ansible 2.5.7 config file = /home/cloud-user/ansible.cfg configured module search path = [u'/home/cloud-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] How reproducible: See above. Steps to Reproduce: 1. 2. 3. Actual results: See above. Expected results: node-config-infra configmap is not modified Additional info: BOOTSTRAP_CONFIG_NAME in /etc/sysconfig/atomic-openshift-node is properly set to 'node-config-infra', the issue is that the node-config is modified and it shouldn't.
I've tested the same scenario but setting per host variables instead per group with same result. * Masters with cri-o only * App nodes with cri-o only * Infra nodes with docker only $ grep osm_use_cockpit all.yml osm_use_cockpit: false $ cat inventory/group_vars/masters.yml openshift_use_crio_only: true openshift_use_crio: true openshift_node_group_name: node-config-master-crio $ cat inventory/group_vars/openstack_compute_nodes.yml openshift_use_crio_only: true openshift_use_crio: true openshift_node_group_name: node-config-compute-crio $ cat inventory/host_vars/infra-node-0.shiftstack.automated.lan.yml openshift_use_crio_only: false openshift_use_crio: false openshift_node_group_name: node-config-infra $ cat inventory/host_vars/infra-node-1.shiftstack.automated.lan.yml openshift_use_crio_only: false openshift_use_crio: false openshift_node_group_name: node-config-infra $ cat inventory/host_vars/infra-node-2.shiftstack.automated.lan.yml openshift_use_crio_only: false openshift_use_crio: false openshift_node_group_name: node-config-infra
I apologize as I copy/pasted the ansible version wrong. Those are the proper values: $ rpm -q openshift-ansible openshift-ansible-3.11.16-1.git.0.4ac6f81.el7.noarch $ rpm -q ansible ansible-2.6.7-1.el7ae.noarch $ ansible --version ansible 2.6.7 config file = /home/cloud-user/ansible.cfg configured module search path = [u'/home/cloud-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
I've been looking into this and found there is a possible issue with the oc_configmap module. When it attempts to retrieve the configmaps, it fails, and then uses the default configmap template from the openshift_node_group role. However, the template is processed as if it was on the master, which is set for use_crio, and therefore ends up including the crio settings. Looking into a fix.
WIP Proposed: https://github.com/openshift/openshift-ansible/pull/10645
Reproduce this bug with openshift-ansible-3.11.44-1.git.0.11d174e.el7.noarch. [nodes] master-node openshift_use_crio=True openshift_use_crio_only=True openshift_node_group_name='node-config-master-crio' infra-node openshift_node_group_name='node-config-infra' pure-crio-node openshift_use_crio=True openshift_use_crio_only=True openshift_node_group_name='node-config-compute-crio' [root@qe-jialiu311-mep-1 ~]# oc get node NAME STATUS ROLES AGE VERSION qe-jialiu311-mep-1 Ready master 26m v1.11.0+d4cacc0 qe-jialiu311-node-infra-1 NotReady infra 23m v1.11.0+d4cacc0 qe-jialiu311-node-pure-crio-runtime-1 Ready compute 23m v1.11.0+d4cacc0 [root@qe-jialiu311-node-infra-1 ~]# journalctl -f -u atomic-openshift-node.service -- Logs begin at Mon 2018-11-19 01:00:43 EST. -- Nov 19 01:42:00 qe-jialiu311-node-infra-1 atomic-openshift-node[46639]: I1119 01:42:00.919032 46639 kubelet.go:299] Watching apiserver Nov 19 01:42:00 qe-jialiu311-node-infra-1 atomic-openshift-node[46639]: W1119 01:42:00.925647 46639 util_unix.go:75] Using "/var/run/crio/crio.sock" as endpoint is deprecated, please consider using full url format "unix:///var/run/crio/crio.sock". Nov 19 01:42:00 qe-jialiu311-node-infra-1 atomic-openshift-node[46639]: W1119 01:42:00.925718 46639 util_unix.go:75] Using "/var/run/crio/crio.sock" as endpoint is deprecated, please consider using full url format "unix:///var/run/crio/crio.sock". Nov 19 01:42:00 qe-jialiu311-node-infra-1 atomic-openshift-node[46639]: E1119 01:42:00.926073 46639 remote_runtime.go:69] Version from runtime service failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable Nov 19 01:42:00 qe-jialiu311-node-infra-1 atomic-openshift-node[46639]: E1119 01:42:00.926147 46639 kuberuntime_manager.go:172] Get runtime version failed: rpc error: code = Unavailable desc = grpc: the connection is unavailable Nov 19 01:42:00 qe-jialiu311-node-infra-1 atomic-openshift-node[46639]: F1119 01:42:00.926160 46639 server.go:262] failed to run Kubelet: failed to create kubelet: rpc error: code = Unavailable desc = grpc: the connection is unavailable Nov 19 01:42:00 qe-jialiu311-node-infra-1 systemd[1]: atomic-openshift-node.service: main process exited, code=exited, status=255/n/a Nov 19 01:42:00 qe-jialiu311-node-infra-1 systemd[1]: Failed to start OpenShift Node. Nov 19 01:42:00 qe-jialiu311-node-infra-1 systemd[1]: Unit atomic-openshift-node.service entered failed state. Nov 19 01:42:00 qe-jialiu311-node-infra-1 systemd[1]: atomic-openshift-node.service failed. Added one more TC to cover it.
Waiting for build $ git tag --contains 63e84e757e781a19da8b8cdac151c78922ae4ebc
Fixed. openshift-ansible-3.11.51-1.git.0.51c90a3.el7.noarch node-config are correct for crio container-runtime nodes and docker container-runtime nodes. Kernel Version: 3.10.0-957.1.3.el7.x86_64 Operating System: Red Hat Enterprise Linux Server 7.6 (Maipo)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:3743
I'm not sure if I should re-open this bug but it is still not working for me. Env: * OCP on OSP using openshift-ansible * 3 Masters running cri-o only * 3 Infra nodes running docker only * 3 App nodes running cri-o only * OCP 3.11.51 ---8<--- [openshift@master-0 ~]$ oc version oc v3.11.51 kubernetes v1.11.0+d4cacc0 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://10.240.0.11:8443 openshift v3.11.51 kubernetes v1.11.0+d4cacc0 --->8--- Inventory: ---8<--- $ cat inventory/group_vars/masters.yml openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_master_group_name: node-config-master-crio openshift_node_group_name: node-config-master-crio $ cat inventory/group_vars/openstack_compute_nodes.yml openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_compute_group_name: node-config-compute-crio openshift_node_group_name: node-config-compute-crio $ cat inventory/group_vars/openstack_infra_nodes.yml openshift_use_crio: false $ grep -c -i cri inventory/group_vars/all.yml 0 $ grep -c -i cri inventory/group_vars/OSEv3.yml 0 --->8--- Results: ---8<--- $ ssh -i openstack.key openshift.automated.lan grep BOOTSTRAP_CONFIG_NAME /etc/sysconfig/atomic-openshift-node BOOTSTRAP_CONFIG_NAME=node-config-master $ ssh -i openstack.key openshift.automated.lan grep BOOTSTRAP_CONFIG_NAME /etc/sysconfig/atomic-openshift-node BOOTSTRAP_CONFIG_NAME=node-config-compute $ ssh -i openstack.key openshift.automated.lan grep BOOTSTRAP_CONFIG_NAME /etc/sysconfig/atomic-openshift-node BOOTSTRAP_CONFIG_NAME=node-config-infra --->8--- Ansible log: ---8<--- ... 2018-12-13 06:31:02,714 p=32616 u=cloud-user | TASK [openshift_node_group : Update bootstrap-node-config.yaml to use crio] **** <--- this seems to be OK, it only changes the hosts needed 2018-12-13 06:31:02,715 p=32616 u=cloud-user | Thursday 13 December 2018 06:31:02 -0500 (0:00:07.044) 0:57:59.095 ***** 2018-12-13 06:31:03,048 p=32616 u=cloud-user | skipping: [infra-node-0.shiftstack.automated.lan] 2018-12-13 06:31:03,166 p=32616 u=cloud-user | skipping: [infra-node-2.shiftstack.automated.lan] 2018-12-13 06:31:03,242 p=32616 u=cloud-user | skipping: [infra-node-1.shiftstack.automated.lan] 2018-12-13 06:31:04,103 p=32616 u=cloud-user | changed: [master-1.shiftstack.automated.lan] 2018-12-13 06:31:04,407 p=32616 u=cloud-user | changed: [master-0.shiftstack.automated.lan] 2018-12-13 06:31:04,438 p=32616 u=cloud-user | changed: [app-node-2.shiftstack.automated.lan] 2018-12-13 06:31:04,505 p=32616 u=cloud-user | changed: [app-node-0.shiftstack.automated.lan] 2018-12-13 06:31:04,576 p=32616 u=cloud-user | changed: [master-2.shiftstack.automated.lan] 2018-12-13 06:31:04,668 p=32616 u=cloud-user | changed: [app-node-1.shiftstack.automated.lan] ... 2018-12-13 06:31:07,419 p=32616 u=cloud-user | TASK [openshift_node_group : Update the sysconfig to group "node-config-master"] *** <--- now it seems to change everything 2018-12-13 06:31:07,419 p=32616 u=cloud-user | Thursday 13 December 2018 06:31:07 -0500 (0:00:02.725) 0:58:03.800 ***** 2018-12-13 06:31:08,715 p=32616 u=cloud-user | changed: [master-1.shiftstack.automated.lan] 2018-12-13 06:31:08,789 p=32616 u=cloud-user | changed: [infra-node-0.shiftstack.automated.lan] 2018-12-13 06:31:08,817 p=32616 u=cloud-user | changed: [app-node-0.shiftstack.automated.lan] 2018-12-13 06:31:08,855 p=32616 u=cloud-user | changed: [app-node-2.shiftstack.automated.lan] 2018-12-13 06:31:08,883 p=32616 u=cloud-user | changed: [master-2.shiftstack.automated.lan] 2018-12-13 06:31:08,974 p=32616 u=cloud-user | changed: [app-node-1.shiftstack.automated.lan] 2018-12-13 06:31:08,988 p=32616 u=cloud-user | changed: [infra-node-1.shiftstack.automated.lan] 2018-12-13 06:31:09,101 p=32616 u=cloud-user | changed: [infra-node-2.shiftstack.automated.lan] 2018-12-13 06:31:09,121 p=32616 u=cloud-user | changed: [master-0.shiftstack.automated.lan] ... 2018-12-13 06:43:43,163 p=32616 u=cloud-user | TASK [openshift_node_group : create node config template] ********************** 2018-12-13 06:43:43,163 p=32616 u=cloud-user | Thursday 13 December 2018 06:43:43 -0500 (0:00:00.104) 1:10:39.544 ***** 2018-12-13 06:43:48,366 p=32616 u=cloud-user | changed: [master-0.shiftstack.automated.lan] 2018-12-13 06:43:49,016 p=32616 u=cloud-user | changed: [master-1.shiftstack.automated.lan] 2018-12-13 06:43:49,625 p=32616 u=cloud-user | changed: [master-2.shiftstack.automated.lan] 2018-12-13 06:43:49,650 p=32616 u=cloud-user | TASK [openshift_node_group : Update bootstrap-node-config.yaml to use crio] **** 2018-12-13 06:43:49,650 p=32616 u=cloud-user | Thursday 13 December 2018 06:43:49 -0500 (0:00:06.486) 1:10:46.031 ***** 2018-12-13 06:43:50,950 p=32616 u=cloud-user | changed: [master-0.shiftstack.automated.lan] 2018-12-13 06:43:51,035 p=32616 u=cloud-user | changed: [master-2.shiftstack.automated.lan] 2018-12-13 06:43:51,098 p=32616 u=cloud-user | changed: [master-1.shiftstack.automated.lan] ... 2018-12-13 06:51:58,244 p=32616 u=cloud-user | TASK [openshift_node_group : Build node config maps] *************************** 2018-12-13 06:51:58,244 p=32616 u=cloud-user | Thursday 13 December 2018 06:51:58 -0500 (0:00:00.171) 1:18:54.625 ***** 2018-12-13 06:51:58,537 p=32616 u=cloud-user | included: /usr/share/ansible/openshift-ansible/roles/openshift_node_group/tasks/create_config.yml for master-2.shiftstack.automated.lan 2018-12-13 06:51:58,549 p=32616 u=cloud-user | included: /usr/share/ansible/openshift-ansible/roles/openshift_node_group/tasks/create_config.yml for master-2.shiftstack.automated.lan 2018-12-13 06:51:58,561 p=32616 u=cloud-user | included: /usr/share/ansible/openshift-ansible/roles/openshift_node_group/tasks/create_config.yml for master-2.shiftstack.automated.lan 2018-12-13 06:51:58,573 p=32616 u=cloud-user | included: /usr/share/ansible/openshift-ansible/roles/openshift_node_group/tasks/create_config.yml for master-2.shiftstack.automated.lan 2018-12-13 06:51:58,587 p=32616 u=cloud-user | included: /usr/share/ansible/openshift-ansible/roles/openshift_node_group/tasks/create_config.yml for master-2.shiftstack.automated.lan 2018-12-13 06:51:58,705 p=32616 u=cloud-user | included: /usr/share/ansible/openshift-ansible/roles/openshift_node_group/tasks/create_config.yml for master-2.shiftstack.automated.lan 2018-12-13 06:51:58,718 p=32616 u=cloud-user | included: /usr/share/ansible/openshift-ansible/roles/openshift_node_group/tasks/create_config.yml for master-2.shiftstack.automated.lan 2018-12-13 06:51:58,730 p=32616 u=cloud-user | included: /usr/share/ansible/openshift-ansible/roles/openshift_node_group/tasks/create_config.yml for master-2.shiftstack.automated.lan 2018-12-13 06:51:58,742 p=32616 u=cloud-user | included: /usr/share/ansible/openshift-ansible/roles/openshift_node_group/tasks/create_config.yml for master-2.shiftstack.automated.lan 2018-12-13 06:51:58,754 p=32616 u=cloud-user | included: /usr/share/ansible/openshift-ansible/roles/openshift_node_group/tasks/create_config.yml for master-2.shiftstack.automated.lan 2018-12-13 06:51:58,778 p=32616 u=cloud-user | TASK [openshift_node_group : include_tasks] ************************************ 2018-12-13 06:51:58,778 p=32616 u=cloud-user | Thursday 13 December 2018 06:51:58 -0500 (0:00:00.533) 1:18:55.158 ***** 2018-12-13 06:51:58,816 p=32616 u=cloud-user | included: /usr/share/ansible/openshift-ansible/roles/openshift_node_group/tasks/fetch_config.yml for master-2.shiftstack.automated.lan 2018-12-13 06:51:58,845 p=32616 u=cloud-user | TASK [openshift_node_group : fetch node configmap] ***************************** 2018-12-13 06:51:58,845 p=32616 u=cloud-user | Thursday 13 December 2018 06:51:58 -0500 (0:00:00.067) 1:18:55.226 ***** 2018-12-13 06:52:00,539 p=32616 u=cloud-user | ok: [master-2.shiftstack.automated.lan] 2018-12-13 06:52:00,564 p=32616 u=cloud-user | TASK [openshift_node_group : debug node config] ******************************** 2018-12-13 06:52:00,564 p=32616 u=cloud-user | Thursday 13 December 2018 06:52:00 -0500 (0:00:01.718) 1:18:56.945 ***** <---------- shouldn't this get the node-config-master-crio group??? 2018-12-13 06:52:00,605 p=32616 u=cloud-user | ok: [master-2.shiftstack.automated.lan] => { "configout": { "changed": false, "failed": false, "results": { "cmd": "/bin/oc get configmap node-config-master -o json -n openshift-node", "results": [ {} ], "returncode": 0, "stderr": "Error from server (NotFound): configmaps \"node-config-master\" not found\n", "stdout": "" }, "state": "list" } } ... --->8--- I'm attaching the whole ansible log file just in case as there are tons of openshift_node_group calls. Thanks.
I deployed a fresh new environment from scratch but this time I've set the cri-o variables PER HOST instead per group, and it worked: ---8<--- $ cat host_vars/master-?.shiftstack.automated.lan.yml openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_master_group_name: node-config-master-crio openshift_node_group_name: node-config-master-crio openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_master_group_name: node-config-master-crio openshift_node_group_name: node-config-master-crio openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_master_group_name: node-config-master-crio openshift_node_group_name: node-config-master-crio $ cat host_vars/app-node-?.shiftstack.automated.lan.yml openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_compute_group_name: node-config-compute-crio openshift_node_group_name: node-config-compute-crio openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_compute_group_name: node-config-compute-crio openshift_node_group_name: node-config-compute-crio openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_compute_group_name: node-config-compute-crio openshift_node_group_name: node-config-compute-crio --->8--- ---8<--- $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME app-node-0.shiftstack.automated.lan Ready compute 23m v1.11.0+d4cacc0 10.240.0.6 10.19.115.126 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 cri-o://1.11.10 app-node-1.shiftstack.automated.lan Ready compute 23m v1.11.0+d4cacc0 10.240.0.9 10.19.115.128 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 cri-o://1.11.10 app-node-2.shiftstack.automated.lan Ready compute 23m v1.11.0+d4cacc0 10.240.0.7 10.19.115.117 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 cri-o://1.11.10 infra-node-0.shiftstack.automated.lan Ready infra 23m v1.11.0+d4cacc0 10.240.0.33 10.19.115.139 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 docker://1.13.1 infra-node-1.shiftstack.automated.lan Ready infra 23m v1.11.0+d4cacc0 10.240.0.16 10.19.115.123 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 docker://1.13.1 infra-node-2.shiftstack.automated.lan Ready infra 23m v1.11.0+d4cacc0 10.240.0.8 10.19.115.127 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 docker://1.13.1 master-0.shiftstack.automated.lan Ready master 17h v1.11.0+d4cacc0 10.240.0.19 10.19.115.125 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 cri-o://1.11.10 master-1.shiftstack.automated.lan Ready master 17h v1.11.0+d4cacc0 10.240.0.30 10.19.115.129 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 cri-o://1.11.10 master-2.shiftstack.automated.lan Ready master 17h v1.11.0+d4cacc0 10.240.0.24 10.19.115.130 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 cri-o://1.11.10 $ oc get nodes -o wide --show-labels NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME LABELS app-node-0.shiftstack.automated.lan Ready compute 24m v1.11.0+d4cacc0 10.240.0.6 10.19.115.126 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 cri-o://1.11.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=1470ffe1-aea0-4806-a1be-e24c83c08e5f,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=app-node-0.shiftstack.automated.lan,node-role.kubernetes.io/compute=true,runtime=cri-o app-node-1.shiftstack.automated.lan Ready compute 24m v1.11.0+d4cacc0 10.240.0.9 10.19.115.128 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 cri-o://1.11.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=1470ffe1-aea0-4806-a1be-e24c83c08e5f,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=app-node-1.shiftstack.automated.lan,node-role.kubernetes.io/compute=true,runtime=cri-o app-node-2.shiftstack.automated.lan Ready compute 24m v1.11.0+d4cacc0 10.240.0.7 10.19.115.117 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 cri-o://1.11.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=1470ffe1-aea0-4806-a1be-e24c83c08e5f,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=app-node-2.shiftstack.automated.lan,node-role.kubernetes.io/compute=true,runtime=cri-o infra-node-0.shiftstack.automated.lan Ready infra 24m v1.11.0+d4cacc0 10.240.0.33 10.19.115.139 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 docker://1.13.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=1470ffe1-aea0-4806-a1be-e24c83c08e5f,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=infra-node-0.shiftstack.automated.lan,node-role.kubernetes.io/infra=true infra-node-1.shiftstack.automated.lan Ready infra 24m v1.11.0+d4cacc0 10.240.0.16 10.19.115.123 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 docker://1.13.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=1470ffe1-aea0-4806-a1be-e24c83c08e5f,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=infra-node-1.shiftstack.automated.lan,node-role.kubernetes.io/infra=true infra-node-2.shiftstack.automated.lan Ready infra 24m v1.11.0+d4cacc0 10.240.0.8 10.19.115.127 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 docker://1.13.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=1470ffe1-aea0-4806-a1be-e24c83c08e5f,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=infra-node-2.shiftstack.automated.lan,node-role.kubernetes.io/infra=true master-0.shiftstack.automated.lan Ready master 17h v1.11.0+d4cacc0 10.240.0.19 10.19.115.125 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 cri-o://1.11.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=1470ffe1-aea0-4806-a1be-e24c83c08e5f,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=master-0.shiftstack.automated.lan,node-role.kubernetes.io/master=true,runtime=cri-o master-1.shiftstack.automated.lan Ready master 17h v1.11.0+d4cacc0 10.240.0.30 10.19.115.129 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 cri-o://1.11.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=1470ffe1-aea0-4806-a1be-e24c83c08e5f,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=master-1.shiftstack.automated.lan,node-role.kubernetes.io/master=true,runtime=cri-o master-2.shiftstack.automated.lan Ready master 17h v1.11.0+d4cacc0 10.240.0.24 10.19.115.130 Red Hat Enterprise Linux Server 7.6 (Maipo) 3.10.0-957.1.3.el7.x86_64 cri-o://1.11.10 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=1470ffe1-aea0-4806-a1be-e24c83c08e5f,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/zone=nova,kubernetes.io/hostname=master-2.shiftstack.automated.lan,node-role.kubernetes.io/master=true,runtime=cri-o --->8---