Description of problem: Related to https://bugzilla.redhat.com/show_bug.cgi?id=1647516 The scenario is trying to deploy an HA environment (see more details in that BZs) with cri-o for masters and app-nodes but no in infra-nodes. It only seems to work when specifying the node-group variables per host basis instead per group basis. Version-Release number of the following components: (shiftstack) [cloud-user@shiftstack-ansibler ~]$ rpm -q openshift-ansible openshift-ansible-3.11.51-2.git.0.51c90a3.el7.noarch (shiftstack) [cloud-user@shiftstack-ansibler ~]$ rpm -q ansible ansible-2.5.14-1.el7ae.noarch (shiftstack) [cloud-user@shiftstack-ansibler ~]$ ansible --version ansible 2.5.14 config file = /home/cloud-user/ansible.cfg configured module search path = [u'/home/cloud-user/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, Sep 12 2018, 05:31:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] How reproducible: NOTE: I've attached the # ENVIRONMENT WITH DOCKER ONLY (NO CRIO) $ find inventory/ inventory/ inventory/group_vars inventory/group_vars/OSEv3.yml inventory/group_vars/all.yml inventory/host_vars ansible-inventory -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory/ --list --yaml > no_crio.yml # CRI-O PER GROUP OPENSHIFT_OPENSTACK_*_GROUP_NAME $ find inventory/ inventory/ inventory/group_vars inventory/group_vars/OSEv3.yml inventory/group_vars/all.yml inventory/group_vars/masters.yml inventory/group_vars/openstack_compute_nodes.yml inventory/group_vars/openstack_infra_nodes.yml inventory/host_vars $ for i in masters openstack_compute_nodes openstack_infra_nodes; do echo $i; cat inventory/group_vars/$i.yml; done masters openshift_use_crio_only: true openshift_use_crio: true #openshift_node_group_name: node-config-master-crio openshift_openstack_master_group_name: node-config-master-crio openstack_compute_nodes openshift_use_crio_only: true openshift_use_crio: true #openshift_node_group_name: node-config-compute-crio openshift_openstack_compute_group_name: node-config-compute-crio openstack_infra_nodes openshift_use_crio_only: false openshift_use_crio: false #openshift_node_group_name: node-config-infra openshift_openstack_infra_group_name: node-config-infra $ ansible-inventory -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory/ --list --yaml > crio_per_group_openstack_openshift_group_node.yml # CRI-O PER GROUP OPENSHIFT GROUP NAME $ find inventory/ inventory/ inventory/group_vars inventory/group_vars/OSEv3.yml inventory/group_vars/all.yml inventory/group_vars/masters.yml inventory/group_vars/openstack_compute_nodes.yml inventory/group_vars/openstack_infra_nodes.yml inventory/host_vars $ for i in masters openstack_compute_nodes openstack_infra_nodes; do echo $i; cat inventory/group_vars/$i.yml; done masters openshift_use_crio_only: true openshift_use_crio: true openshift_node_group_name: node-config-master-crio #openshift_openstack_master_group_name: node-config-master-crio openstack_compute_nodes openshift_use_crio_only: true openshift_use_crio: true openshift_node_group_name: node-config-compute-crio #openshift_openstack_compute_group_name: node-config-compute-crio openstack_infra_nodes openshift_use_crio_only: false openshift_use_crio: false openshift_node_group_name: node-config-infra #openshift_openstack_infra_group_name: node-config-infra $ ansible-inventory -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory/ --list --yaml > crio_per_group_node_group.yml # CRI-O PER GROUP BOTH OPENSHIFT_OPENSTACK_*_GROUP_NAME AND OPENSHIFT GROUP NAME $ find inventory/ inventory/ inventory/group_vars inventory/group_vars/OSEv3.yml inventory/group_vars/all.yml inventory/group_vars/masters.yml inventory/group_vars/openstack_compute_nodes.yml inventory/group_vars/openstack_infra_nodes.yml inventory/host_vars $ for i in masters openstack_compute_nodes openstack_infra_nodes; do echo $i; cat inventory/group_vars/$i.yml; done masters openshift_use_crio_only: true openshift_use_crio: true openshift_node_group_name: node-config-master-crio openshift_openstack_master_group_name: node-config-master-crio openstack_compute_nodes openshift_use_crio_only: true openshift_use_crio: true openshift_node_group_name: node-config-compute-crio openshift_openstack_compute_group_name: node-config-compute-crio openstack_infra_nodes openshift_use_crio_only: false openshift_use_crio: false openshift_node_group_name: node-config-infra openshift_openstack_infra_group_name: node-config-infra $ ansible-inventory -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory/ --list --yaml > crio_per_group_both.yml # CRI-O PER HOST $ find inventory/ inventory/ inventory/group_vars inventory/group_vars/OSEv3.yml inventory/group_vars/all.yml inventory/host_vars inventory/host_vars/app-node-0.shiftstack.automated.lan.yml inventory/host_vars/app-node-1.shiftstack.automated.lan.yml inventory/host_vars/app-node-2.shiftstack.automated.lan.yml inventory/host_vars/infra-node-0.shiftstack.automated.lan.yml inventory/host_vars/infra-node-1.shiftstack.automated.lan.yml inventory/host_vars/infra-node-2.shiftstack.automated.lan.yml inventory/host_vars/master-0.shiftstack.automated.lan.yml inventory/host_vars/master-1.shiftstack.automated.lan.yml inventory/host_vars/master-2.shiftstack.automated.lan.yml $ for i in inventory/host_vars/*; do echo $i; cat $i; done inventory/host_vars/app-node-0.shiftstack.automated.lan.yml openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_compute_group_name: node-config-compute-crio openshift_node_group_name: node-config-compute-crio inventory/host_vars/app-node-1.shiftstack.automated.lan.yml openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_compute_group_name: node-config-compute-crio openshift_node_group_name: node-config-compute-crio inventory/host_vars/app-node-2.shiftstack.automated.lan.yml openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_compute_group_name: node-config-compute-crio openshift_node_group_name: node-config-compute-crio inventory/host_vars/infra-node-0.shiftstack.automated.lan.yml openshift_use_crio: false inventory/host_vars/infra-node-1.shiftstack.automated.lan.yml openshift_use_crio: false inventory/host_vars/infra-node-2.shiftstack.automated.lan.yml openshift_use_crio: false inventory/host_vars/master-0.shiftstack.automated.lan.yml openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_master_group_name: node-config-master-crio openshift_node_group_name: node-config-master-crio inventory/host_vars/master-1.shiftstack.automated.lan.yml openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_master_group_name: node-config-master-crio openshift_node_group_name: node-config-master-crio inventory/host_vars/master-2.shiftstack.automated.lan.yml openshift_use_crio_only: true openshift_use_crio: true openshift_openstack_master_group_name: node-config-master-crio openshift_node_group_name: node-config-master-crio $ ansible-inventory -i /usr/share/ansible/openshift-ansible/playbooks/openstack/inventory.py -i inventory/ --list --yaml > crio_per_host.yml Results anonymized by: sed -i '/password/d' ./* sed -i '/rhsub_*/d' ./* sed -i '/secret/d' ./* Steps to Reproduce: 1. 2. 3. Actual results: Attached inventory files generated where there are differences, but mainly: $ grep openshift_node_group_name * crio_per_group_both.yml: openshift_node_group_name: node-config-compute crio_per_group_both.yml: openshift_node_group_name: node-config-compute crio_per_group_both.yml: openshift_node_group_name: node-config-compute crio_per_group_both.yml: openshift_node_group_name: node-config-infra crio_per_group_both.yml: openshift_node_group_name: node-config-infra crio_per_group_both.yml: openshift_node_group_name: node-config-infra crio_per_group_both.yml: openshift_node_group_name: node-config-master crio_per_group_both.yml: openshift_node_group_name: node-config-master crio_per_group_both.yml: openshift_node_group_name: node-config-master crio_per_group_node_group.yml: openshift_node_group_name: node-config-compute crio_per_group_node_group.yml: openshift_node_group_name: node-config-compute crio_per_group_node_group.yml: openshift_node_group_name: node-config-compute crio_per_group_node_group.yml: openshift_node_group_name: node-config-infra crio_per_group_node_group.yml: openshift_node_group_name: node-config-infra crio_per_group_node_group.yml: openshift_node_group_name: node-config-infra crio_per_group_node_group.yml: openshift_node_group_name: node-config-master crio_per_group_node_group.yml: openshift_node_group_name: node-config-master crio_per_group_node_group.yml: openshift_node_group_name: node-config-master crio_per_group_openstack_openshift_group_node.yml: openshift_node_group_name: node-config-compute crio_per_group_openstack_openshift_group_node.yml: openshift_node_group_name: node-config-compute crio_per_group_openstack_openshift_group_node.yml: openshift_node_group_name: node-config-compute crio_per_group_openstack_openshift_group_node.yml: openshift_node_group_name: node-config-infra crio_per_group_openstack_openshift_group_node.yml: openshift_node_group_name: node-config-infra crio_per_group_openstack_openshift_group_node.yml: openshift_node_group_name: node-config-infra crio_per_group_openstack_openshift_group_node.yml: openshift_node_group_name: node-config-master crio_per_group_openstack_openshift_group_node.yml: openshift_node_group_name: node-config-master crio_per_group_openstack_openshift_group_node.yml: openshift_node_group_name: node-config-master crio_per_host.yml: openshift_node_group_name: node-config-compute-crio crio_per_host.yml: openshift_node_group_name: node-config-compute-crio crio_per_host.yml: openshift_node_group_name: node-config-compute-crio crio_per_host.yml: openshift_node_group_name: node-config-infra crio_per_host.yml: openshift_node_group_name: node-config-infra crio_per_host.yml: openshift_node_group_name: node-config-infra crio_per_host.yml: openshift_node_group_name: node-config-master-crio crio_per_host.yml: openshift_node_group_name: node-config-master-crio crio_per_host.yml: openshift_node_group_name: node-config-master-crio no_crio.yml: openshift_node_group_name: node-config-compute no_crio.yml: openshift_node_group_name: node-config-compute no_crio.yml: openshift_node_group_name: node-config-compute no_crio.yml: openshift_node_group_name: node-config-infra no_crio.yml: openshift_node_group_name: node-config-infra no_crio.yml: openshift_node_group_name: node-config-infra no_crio.yml: openshift_node_group_name: node-config-master no_crio.yml: openshift_node_group_name: node-config-master no_crio.yml: openshift_node_group_name: node-config-master Expected results: Applying openshift_openstack_master_group_name or openshift_node_group_name per group should work. Additional info: Attached inventory files.
I *THINK* the issue is located here https://github.com/openshift/openshift-ansible/blob/master/playbooks/openstack/resources.py#L105-L106
I think this is somehow related to the heat stack... I've tried to run the provision playbook and canceling it while the "Check if the stack exists" and browsing the generated stack in the /tmp/XXX folder: # Per group (both openshift_node_group_name and openshift_openstack_*_group_name) $ grep -B2 node_group stack.yaml cluster_id: shiftstack.automated.lan type: master openshift_node_group_name: node-config-master -- type: node subtype: app openshift_node_group_name: node-config-compute -- type: node subtype: infra openshift_node_group_name: node-config-infra -- cluster_id: shiftstack.automated.lan type: cns openshift_node_group_name: node-config-compute # Per host $ grep -B2 node_group stack.yaml cluster_id: shiftstack.automated.lan type: master openshift_node_group_name: node-config-master -- type: node subtype: app openshift_node_group_name: node-config-compute -- type: node subtype: infra openshift_node_group_name: node-config-infra -- cluster_id: shiftstack.automated.lan type: cns openshift_node_group_name: node-config-compute
I think the issue is the stack is created without taking into account the group_vars/{masters,openstack_*}.yml (as the nodes doesn't exist yet) but only the all.yml file... in this case, I've added the openshift_openstack_*_group_name to the all.yml file and now it seems to work: ... type: node subtype: app openshift_node_group_name: node-config-compute-crio image: rhel-server-7.6 ... So, I think the node_group_name should be specified in the all.yml to be taken into account for heat, but the other variables should be added in their corresponding group: all.yml: ... # Node group settings openshift_openstack_compute_group_name: node-config-compute-crio openshift_openstack_master_group_name: node-config-master-crio openshift_openstack_infra_group_name: node-config-infra $ for i in masters openstack_compute_nodes openstack_infra_nodes; do echo $i; cat inventory/group_vars/$i.yml; done masters openshift_use_crio_only: true openshift_use_crio: true openstack_compute_nodes openshift_use_crio_only: true openshift_use_crio: true openstack_infra_nodes openshift_use_crio_only: false openshift_use_crio: false I'm running a deployment with this settings now and will notify here the result.
I can confirm that theory! Oddly enough, looking at the documentation you may have been the one to add the documentation that the openshift_openstack_*_group_name parameters would have to be specified in all.yml :)
(In reply to Tzu-Mainn Chen from comment #5) > I can confirm that theory! Oddly enough, looking at the documentation you > may have been the one to add the documentation that the > openshift_openstack_*_group_name parameters would have to be specified in > all.yml :) PEBKAC... this scenario worked: all.yml: ... # Node group settings openshift_openstack_compute_group_name: node-config-compute-crio openshift_openstack_master_group_name: node-config-master-crio openshift_openstack_infra_group_name: node-config-infra $ for i in masters openstack_compute_nodes openstack_infra_nodes; do echo $i; cat inventory/group_vars/$i.yml; done masters openshift_use_crio_only: true openshift_use_crio: true openstack_compute_nodes openshift_use_crio_only: true openshift_use_crio: true openstack_infra_nodes openshift_use_crio_only: false openshift_use_crio: false I did created a PR to clarify it and to be merged in the release-3.11 branch here https://github.com/openshift/openshift-ansible/pull/10915 Thanks!