Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1626223

Summary: OpenShift upgrade fails on openshift_node_group: fail task when node-config-master-crio configmap not present
Product: OpenShift Container Platform Reporter: Matt Bruzek <mbruzek>
Component: Cluster Version OperatorAssignee: Russell Teague <rteague>
Status: CLOSED CURRENTRELEASE QA Contact: liujia <jiajliu>
Severity: high Docs Contact:
Priority: high    
Version: 3.11.0CC: aos-bugs, cshereme, jack.ottofaro, jiajliu, jkaur, jokerman, mifiedle, mmccomas, pgozart, rteague
Target Milestone: ---   
Target Release: 3.11.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
The configmap check is no longer necessary once a cluster is running 3.10. As new default node groups are added, the check would fail on upgrades unnecessarily.
Story Points: ---
Clone Of:
: 1650694 (view as bug list) Environment:
Last Closed: 2018-12-21 15:23:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1650694    

Description Matt Bruzek 2018-09-06 18:48:51 UTC
Description of problem:

While performing a 3.10 to 3.11 upgrade I got an error that the node-config-master-crio ConfigMap does not exist. This system does not use crio and it fails the early check for the upgrade.

Version-Release number of the following components:
# rpm -q openshift-ansible
openshift-ansible-3.11.0-0.28.0.git.0.730d4be.el7.noarch
# rpm -q ansible
ansible-2.6.2-1.el7ae.noarch
# ansible --version
ansible 2.6.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Jul 16 2018, 19:52:45) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

How reproducible: Always.

Steps to Reproduce:
1. Install a non crio 3.10 system.
2. Attempt an upgrade to 3.11 system.
3. Notice the upgrade fails with this error.

Actual results:

Failure summary:


  1. Hosts:    ec2-18-237-118-157.us-west-2.compute.amazonaws.com
     Play:     Ensure essential node configmaps are present
     Task:     openshift_node_group : fail
     Message:  Configmap for node-config-master-crio must be present. Please see https://docs.openshift.com/container-platform/3.10/upgrading/automated_upgrades.html for details.

Expected results:

For non crio systems I would not expect this error.

Additional info:

Comment 2 Mike Fiedler 2018-09-06 18:58:27 UTC
2018-09-06 18:15:37,697 p=29725 u=root |  ok: [ec2-18-237-118-157.us-west-2.compute.amazonaws.com] => {                                                                                                                                                                                   
    "changed": false,                                                                                                                                                                                                                                                                     
    "invocation": {                                                                                                                                                                                                                                                                       
        "module_args": {                                                                                                                                                                                                                                                                  
            "debug": false,                                                                                                                                                                                                                                                               
            "from_file": null,                                                                                                                                                                                                                                                            
            "from_literal": null,                                                                                                                                                                                                                                                         
            "kubeconfig": "/etc/origin/master/admin.kubeconfig",                                                                                                                                                                                                                          
            "name": "node-config-master-crio",                                                                                                                                                                                                                                            
            "namespace": "openshift-node",                                                                                                                                                                                                                                                
            "state": "list"                                                                                                                                                                                                                                                               
        }                                                                                                                                                                                                                                                                                 
    },                                                                                                                                                                                                                                                                                    
    "results": {                                                                                                                                                                                                                                                                          
        "cmd": "/usr/bin/oc get configmap node-config-master-crio -o json -n openshift-node",                                                                                                                                                                                             
        "results": [                                                                                                                                                                                                                                                                      
            {}                                                                                                                                                                                                                                                                            
        ],                                                                                                                                                                                                                                                                                
        "returncode": 0,                                                                                                                                                                                                                                                                  
        "stderr": "Error from server (NotFound): configmaps \"node-config-master-crio\" not found\n",                                                                                                                                                                                     
        "stdout": ""                                                                                                                                                                                                                                                                      
    },                                                                                                                                                                                                                                                                                    
    "state": "list"                                                                                                                                                                                                                                                                       
}

Comment 5 Russell Teague 2018-09-07 19:21:31 UTC
openshift-ansible-3.11.0-0.32.0

Comment 6 liujia 2018-09-12 09:22:49 UTC
Got a mistaken person. Set to default one back.

Comment 7 liujia 2018-09-13 03:08:17 UTC
Re-produce on v3.11.0-0.28.0

1. Trigger v3.10 installation with default node group(no node-config-master-crio group).
[root@ip-172-18-4-4 ~]# oc get cm -n openshift-node
NAME                       DATA      AGE
node-config-all-in-one     1         26m
node-config-compute        1         26m
node-config-infra          1         26m
node-config-master         1         26m
node-config-master-infra   1         26m

2. Enable v3.11 repos and run upgrade against above ocp

Upgrade failed.
TASK [openshift_node_group : fail] *********************************************
task path: /usr/share/ansible/openshift-ansible/roles/openshift_node_group/tasks/check_for_config.yml:5
Thursday 13 September 2018  03:03:28 +0000 (0:00:00.109)       0:00:31.894 **** 
fatal: [x]: FAILED! => {"changed": false, "msg": "Configmap for node-config-master-crio must be present. Please see https://docs.openshift.com/container-platform/3.10/upgrading/automated_upgrades.html for details."}

Comment 8 liujia 2018-09-13 05:35:16 UTC
Verified on openshift-ansible-3.11.0-0.32.0.git.0.b27b349.el7.noarch

Added test case OCP-19925

Comment 12 Luke Meyer 2018-12-21 15:23:22 UTC
Closing bugs that were verified and targeted for GA but for some reason were not picked up by errata. This bug fix should be present in current 3.11 release content.