Bug 1725834

Summary: Scale up playbook tries to run and fails if openshift_node_group_name points to a non-existing group
Product: OpenShift Container Platform Reporter: Pablo Alonso Rodriguez <palonsor>
Component: InstallerAssignee: Russell Teague <rteague>
Installer sub component: openshift-ansible QA Contact: Gaoyun Pei <gpei>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: low    
Version: 3.11.0   
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-12-16 11:57:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pablo Alonso Rodriguez 2019-07-01 14:16:48 UTC 
Description of problem:

If a user modifies openshift_node_groups to add a new group (but does not reflect that change on the cluster) and tries to scale up to add a node of that group, scale up playbook tries the scale up anyway and fails (because the node points to a non-existing configmap).

Scale up playbooks should either update the config maps on openshift-node project or fail early during prechecks.

Version-Release number of the following components:
rpm -q openshift-ansible

openshift-ansible-3.11.98-1.git.0.3cfa7c3.el7.noarch

rpm -q ansible

ansible-2.6.16-1.el7ae.noarch

ansible --version

ansible 2.6.16
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/cloud-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Mar 26 2019, 22:13:06) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

How reproducible:

Always

Steps to Reproduce:
1. Install a cluster with custom openshift_node_groups
2. Modify the inventory to add a new group to openshift_node_groups but do not modify configmaps at openshift-node project (so that configmap for the new group does not exist)
3. Try to scale up to add a new node on the new group

Actual results:

Scale up playbooks try to actually perform the scale up and fail (as one would expect). New configmap for new group does not exist in the openshift-node project.

Expected results:

Scale up playbook should take one of these actions, either:
- Reconcile the current value of openshift_node_groups with the one at the inventory BEFORE trying to scale up
- Fail early during pre-check phase and let user know how to fix it (maybe by running other playbook...)

Additional info:

This was easily workarounded by manually fixing the configmaps on openshift-node project.
Comment 1 Russell Teague 2019-07-01 14:28:57 UTC 
Please attach Ansible log.
Comment 3 Pablo Alonso Rodriguez 2019-07-01 15:09:13 UTC 
Logs attached privately. However, this should be easy to reproduce
Comment 6 Gaoyun Pei 2019-12-03 10:26:53 UTC 
Could reproduce this bug with openshift-ansible-3.11.98-1.git.0.3cfa7c3.el7.noarch

Add a new node host to [new_nodes] group, set a new openshift_node_group_name to it, but without such node_group configmap under openshift-node project. 
The new node scale-up failed at TASK [openshift_manage_node : Wait for sync DS to set annotations on all nodes] ***

Works well with openshift-ansible-3.11.156-1.git.0.5f2cfa4.el7.noarch.rpm.

The scale-up playbook will create the missing configmap and add the node to the cluster successfully.
Comment 8 errata-xmlrpc 2019-12-16 11:57:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4050