Bug 1661232

Summary: [openstack] node-groups variables missmatch when using groups
Product: OpenShift Container Platform Reporter: Eduardo Minguez <eminguez>
Component: InstallerAssignee: Russell Teague <rteague>
Installer sub component: openshift-ansible QA Contact: Johnny Liu <jialiu>
Status: CLOSED WONTFIX Docs Contact:
Severity: unspecified    
Priority: unspecified CC: aos-bugs, eduen, gpei, jokerman, mmccomas, pablo.iranzo, wmeng
Version: 3.11.0   
Target Milestone: ---   
Target Release: 3.11.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-18 17:07:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Eduardo Minguez 2018-12-20 14:04:31 UTC
Description of problem:
Related to https://bugzilla.redhat.com/show_bug.cgi?id=1647516

The scenario is trying to deploy an HA environment (see more details in that BZs) with cri-o for masters and app-nodes but no in infra-nodes.

It only seems to work when specifying the node-group variables per host basis instead per group basis.

Version-Release number of the following components:
(shiftstack) [cloud-user@shiftstack-ansibler ~]$ rpm -q openshift-ansible
openshift-ansible-3.11.51-2.git.0.51c90a3.el7.noarch
(shiftstack) [cloud-user@shiftstack-ansibler ~]$ rpm -q ansible
ansible-2.5.14-1.el7ae.noarch
(shiftstack) [cloud-user@shiftstack-ansibler ~]$ ansible --version
ansible 2.5.14
  config file = /home/cloud-user/ansible.cfg
  configured module search path = [u'/home/cloud-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]

How reproducible:

NOTE: I've attached the 

# ENVIRONMENT WITH DOCKER ONLY (NO CRIO)

$ find inventory/
inventory/
inventory/group_vars
inventory/group_vars/OSEv3.yml
inventory/group_vars/all.yml
inventory/host_vars

ansible-inventory -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory/ --list --yaml > no_crio.yml

# CRI-O PER GROUP OPENSHIFT_OPENSTACK_*_GROUP_NAME

$ find inventory/
inventory/
inventory/group_vars
inventory/group_vars/OSEv3.yml
inventory/group_vars/all.yml
inventory/group_vars/masters.yml
inventory/group_vars/openstack_compute_nodes.yml
inventory/group_vars/openstack_infra_nodes.yml
inventory/host_vars

$ for i in masters openstack_compute_nodes openstack_infra_nodes; do echo $i; cat inventory/group_vars/$i.yml; done
masters
openshift_use_crio_only: true
openshift_use_crio: true
#openshift_node_group_name: node-config-master-crio
openshift_openstack_master_group_name: node-config-master-crio

openstack_compute_nodes
openshift_use_crio_only: true
openshift_use_crio: true
#openshift_node_group_name: node-config-compute-crio
openshift_openstack_compute_group_name: node-config-compute-crio

openstack_infra_nodes
openshift_use_crio_only: false
openshift_use_crio: false
#openshift_node_group_name: node-config-infra
openshift_openstack_infra_group_name: node-config-infra

$ ansible-inventory -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory/ --list --yaml > crio_per_group_openstack_openshift_group_node.yml


# CRI-O PER GROUP OPENSHIFT GROUP NAME

$ find inventory/
inventory/
inventory/group_vars
inventory/group_vars/OSEv3.yml
inventory/group_vars/all.yml
inventory/group_vars/masters.yml
inventory/group_vars/openstack_compute_nodes.yml
inventory/group_vars/openstack_infra_nodes.yml
inventory/host_vars

$ for i in masters openstack_compute_nodes openstack_infra_nodes; do echo $i; cat inventory/group_vars/$i.yml; done
masters
openshift_use_crio_only: true
openshift_use_crio: true
openshift_node_group_name: node-config-master-crio
#openshift_openstack_master_group_name: node-config-master-crio

openstack_compute_nodes
openshift_use_crio_only: true
openshift_use_crio: true
openshift_node_group_name: node-config-compute-crio
#openshift_openstack_compute_group_name: node-config-compute-crio

openstack_infra_nodes
openshift_use_crio_only: false
openshift_use_crio: false
openshift_node_group_name: node-config-infra
#openshift_openstack_infra_group_name: node-config-infra

$ ansible-inventory -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory/ --list --yaml > crio_per_group_node_group.yml

# CRI-O PER GROUP BOTH OPENSHIFT_OPENSTACK_*_GROUP_NAME AND OPENSHIFT GROUP NAME

$ find inventory/
inventory/
inventory/group_vars
inventory/group_vars/OSEv3.yml
inventory/group_vars/all.yml
inventory/group_vars/masters.yml
inventory/group_vars/openstack_compute_nodes.yml
inventory/group_vars/openstack_infra_nodes.yml
inventory/host_vars

$ for i in masters openstack_compute_nodes openstack_infra_nodes; do echo $i; cat inventory/group_vars/$i.yml; done
masters
openshift_use_crio_only: true
openshift_use_crio: true
openshift_node_group_name: node-config-master-crio
openshift_openstack_master_group_name: node-config-master-crio

openstack_compute_nodes
openshift_use_crio_only: true
openshift_use_crio: true
openshift_node_group_name: node-config-compute-crio
openshift_openstack_compute_group_name: node-config-compute-crio

openstack_infra_nodes
openshift_use_crio_only: false
openshift_use_crio: false
openshift_node_group_name: node-config-infra
openshift_openstack_infra_group_name: node-config-infra

$ ansible-inventory -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory/ --list --yaml > crio_per_group_both.yml

# CRI-O PER HOST

$ find inventory/
inventory/
inventory/group_vars
inventory/group_vars/OSEv3.yml
inventory/group_vars/all.yml
inventory/host_vars
inventory/host_vars/app-node-0.shiftstack.automated.lan.yml
inventory/host_vars/app-node-1.shiftstack.automated.lan.yml
inventory/host_vars/app-node-2.shiftstack.automated.lan.yml
inventory/host_vars/infra-node-0.shiftstack.automated.lan.yml
inventory/host_vars/infra-node-1.shiftstack.automated.lan.yml
inventory/host_vars/infra-node-2.shiftstack.automated.lan.yml
inventory/host_vars/master-0.shiftstack.automated.lan.yml
inventory/host_vars/master-1.shiftstack.automated.lan.yml
inventory/host_vars/master-2.shiftstack.automated.lan.yml

$ for i in inventory/host_vars/*; do echo $i; cat $i; done
inventory/host_vars/app-node-0.shiftstack.automated.lan.yml
openshift_use_crio_only: true
openshift_use_crio: true
openshift_openstack_compute_group_name: node-config-compute-crio
openshift_node_group_name: node-config-compute-crio

inventory/host_vars/app-node-1.shiftstack.automated.lan.yml
openshift_use_crio_only: true
openshift_use_crio: true
openshift_openstack_compute_group_name: node-config-compute-crio
openshift_node_group_name: node-config-compute-crio

inventory/host_vars/app-node-2.shiftstack.automated.lan.yml
openshift_use_crio_only: true
openshift_use_crio: true
openshift_openstack_compute_group_name: node-config-compute-crio
openshift_node_group_name: node-config-compute-crio

inventory/host_vars/infra-node-0.shiftstack.automated.lan.yml
openshift_use_crio: false

inventory/host_vars/infra-node-1.shiftstack.automated.lan.yml
openshift_use_crio: false

inventory/host_vars/infra-node-2.shiftstack.automated.lan.yml
openshift_use_crio: false

inventory/host_vars/master-0.shiftstack.automated.lan.yml
openshift_use_crio_only: true
openshift_use_crio: true
openshift_openstack_master_group_name: node-config-master-crio
openshift_node_group_name: node-config-master-crio

inventory/host_vars/master-1.shiftstack.automated.lan.yml
openshift_use_crio_only: true
openshift_use_crio: true
openshift_openstack_master_group_name: node-config-master-crio
openshift_node_group_name: node-config-master-crio

inventory/host_vars/master-2.shiftstack.automated.lan.yml
openshift_use_crio_only: true
openshift_use_crio: true
openshift_openstack_master_group_name: node-config-master-crio
openshift_node_group_name: node-config-master-crio


$ ansible-inventory -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory/ --list --yaml > crio_per_host.yml

Results anonymized by:

sed -i '/password/d' ./*
sed -i '/rhsub_*/d' ./*
sed -i '/secret/d' ./*


Steps to Reproduce:
1.
2.
3.

Actual results:
Attached inventory files generated where there are differences, but mainly:

$ grep openshift_node_group_name *
crio_per_group_both.yml:          openshift_node_group_name: node-config-compute
crio_per_group_both.yml:          openshift_node_group_name: node-config-compute
crio_per_group_both.yml:          openshift_node_group_name: node-config-compute
crio_per_group_both.yml:          openshift_node_group_name: node-config-infra
crio_per_group_both.yml:          openshift_node_group_name: node-config-infra
crio_per_group_both.yml:          openshift_node_group_name: node-config-infra
crio_per_group_both.yml:          openshift_node_group_name: node-config-master
crio_per_group_both.yml:          openshift_node_group_name: node-config-master
crio_per_group_both.yml:          openshift_node_group_name: node-config-master
crio_per_group_node_group.yml:          openshift_node_group_name: node-config-compute
crio_per_group_node_group.yml:          openshift_node_group_name: node-config-compute
crio_per_group_node_group.yml:          openshift_node_group_name: node-config-compute
crio_per_group_node_group.yml:          openshift_node_group_name: node-config-infra
crio_per_group_node_group.yml:          openshift_node_group_name: node-config-infra
crio_per_group_node_group.yml:          openshift_node_group_name: node-config-infra
crio_per_group_node_group.yml:          openshift_node_group_name: node-config-master
crio_per_group_node_group.yml:          openshift_node_group_name: node-config-master
crio_per_group_node_group.yml:          openshift_node_group_name: node-config-master
crio_per_group_openstack_openshift_group_node.yml:          openshift_node_group_name: node-config-compute
crio_per_group_openstack_openshift_group_node.yml:          openshift_node_group_name: node-config-compute
crio_per_group_openstack_openshift_group_node.yml:          openshift_node_group_name: node-config-compute
crio_per_group_openstack_openshift_group_node.yml:          openshift_node_group_name: node-config-infra
crio_per_group_openstack_openshift_group_node.yml:          openshift_node_group_name: node-config-infra
crio_per_group_openstack_openshift_group_node.yml:          openshift_node_group_name: node-config-infra
crio_per_group_openstack_openshift_group_node.yml:          openshift_node_group_name: node-config-master
crio_per_group_openstack_openshift_group_node.yml:          openshift_node_group_name: node-config-master
crio_per_group_openstack_openshift_group_node.yml:          openshift_node_group_name: node-config-master
crio_per_host.yml:          openshift_node_group_name: node-config-compute-crio
crio_per_host.yml:          openshift_node_group_name: node-config-compute-crio
crio_per_host.yml:          openshift_node_group_name: node-config-compute-crio
crio_per_host.yml:          openshift_node_group_name: node-config-infra
crio_per_host.yml:          openshift_node_group_name: node-config-infra
crio_per_host.yml:          openshift_node_group_name: node-config-infra
crio_per_host.yml:          openshift_node_group_name: node-config-master-crio
crio_per_host.yml:          openshift_node_group_name: node-config-master-crio
crio_per_host.yml:          openshift_node_group_name: node-config-master-crio
no_crio.yml:          openshift_node_group_name: node-config-compute
no_crio.yml:          openshift_node_group_name: node-config-compute
no_crio.yml:          openshift_node_group_name: node-config-compute
no_crio.yml:          openshift_node_group_name: node-config-infra
no_crio.yml:          openshift_node_group_name: node-config-infra
no_crio.yml:          openshift_node_group_name: node-config-infra
no_crio.yml:          openshift_node_group_name: node-config-master
no_crio.yml:          openshift_node_group_name: node-config-master
no_crio.yml:          openshift_node_group_name: node-config-master


Expected results:
Applying openshift_openstack_master_group_name or openshift_node_group_name per group should work.

Additional info:
Attached inventory files.

Comment 2 Eduardo Minguez 2018-12-20 14:11:16 UTC
I *THINK* the issue is located here https://github.com/openshift/openshift-ansible/blob/master/playbooks/openstack/resources.py#L105-L106

Comment 3 Eduardo Minguez 2018-12-20 15:45:31 UTC
I think this is somehow related to the heat stack...

I've tried to run the provision playbook and canceling it while the "Check if the stack exists" and browsing the generated stack in the /tmp/XXX folder:

# Per group (both openshift_node_group_name and openshift_openstack_*_group_name)

$ grep -B2 node_group stack.yaml 
                cluster_id: shiftstack.automated.lan
          type:        master
          openshift_node_group_name: node-config-master
--
          type:        node
          subtype:     app
          openshift_node_group_name: node-config-compute
--
          type:        node
          subtype:     infra
          openshift_node_group_name: node-config-infra
--
                cluster_id: shiftstack.automated.lan
          type:        cns
          openshift_node_group_name: node-config-compute
          
# Per host

$ grep -B2 node_group stack.yaml 
                cluster_id: shiftstack.automated.lan
          type:        master
          openshift_node_group_name: node-config-master
--
          type:        node
          subtype:     app
          openshift_node_group_name: node-config-compute
--
          type:        node
          subtype:     infra
          openshift_node_group_name: node-config-infra
--
                cluster_id: shiftstack.automated.lan
          type:        cns
          openshift_node_group_name: node-config-compute

Comment 4 Eduardo Minguez 2018-12-20 16:13:47 UTC
I think the issue is the stack is created without taking into account the group_vars/{masters,openstack_*}.yml (as the nodes doesn't exist yet) but only the all.yml file... in this case, I've added the openshift_openstack_*_group_name to the all.yml file and now it seems to work:

...
          type:        node
          subtype:     app
          openshift_node_group_name: node-config-compute-crio
          image:       rhel-server-7.6
...

So, I think the node_group_name should be specified in the all.yml to be taken into account for heat, but the other variables should be added in their corresponding group:


all.yml:
...
# Node group settings
openshift_openstack_compute_group_name: node-config-compute-crio
openshift_openstack_master_group_name: node-config-master-crio
openshift_openstack_infra_group_name: node-config-infra

$ for i in masters openstack_compute_nodes openstack_infra_nodes; do echo $i; cat inventory/group_vars/$i.yml; done
masters
openshift_use_crio_only: true
openshift_use_crio: true

openstack_compute_nodes
openshift_use_crio_only: true
openshift_use_crio: true

openstack_infra_nodes
openshift_use_crio_only: false
openshift_use_crio: false

I'm running a deployment with this settings now and will notify here the result.

Comment 5 Tzu-Mainn Chen 2018-12-20 18:23:17 UTC
I can confirm that theory! Oddly enough, looking at the documentation you may have been the one to add the documentation that the openshift_openstack_*_group_name parameters would have to  be specified in all.yml :)

Comment 6 Eduardo Minguez 2018-12-21 09:42:38 UTC
(In reply to Tzu-Mainn Chen from comment #5)
> I can confirm that theory! Oddly enough, looking at the documentation you
> may have been the one to add the documentation that the
> openshift_openstack_*_group_name parameters would have to  be specified in
> all.yml :)

PEBKAC... this scenario worked:


all.yml:
...
# Node group settings
openshift_openstack_compute_group_name: node-config-compute-crio
openshift_openstack_master_group_name: node-config-master-crio
openshift_openstack_infra_group_name: node-config-infra

$ for i in masters openstack_compute_nodes openstack_infra_nodes; do echo $i; cat inventory/group_vars/$i.yml; done
masters
openshift_use_crio_only: true
openshift_use_crio: true

openstack_compute_nodes
openshift_use_crio_only: true
openshift_use_crio: true

openstack_infra_nodes
openshift_use_crio_only: false
openshift_use_crio: false

I did created a PR to clarify it and to be merged in the release-3.11 branch here https://github.com/openshift/openshift-ansible/pull/10915

Thanks!