Bug 1578482

Description of problem:

When running the openshift-ansible/playbooks/openshift-etcd/scaleup.yml playbook to add 2 etcds to an existing OCP 3.10.0-0.38.0 HA CRI-O cluster (rpm install), we get the following error:

TASK [etcd : Add new etcd members to cluster] ***************************************************************************************************
task path: /root/openshift-ansible/roles/etcd/tasks/add_new_member.yml:5
fatal: [<new_etcd_hostname_fqdn>]: FAILED! => {
    "msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'etcd_ip'\n\nThe error appears to have been in '/root/openshift-ansible/roles/etcd/tasks/add_new_member.yml': line 5, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Add new etcd members to cluster\n  ^ here\n\nexception type: <class 'ansible.errors.AnsibleUndefinedVariable'>\nexception: 'dict object' has no attribute 'etcd_ip'"
}

The scaleup.yaml playbook was executed from a cloned openshift-ansible repo, on one of the master/etcd hosts.  This is on AWS EC2, the HA cluster has 1 load-balancer, 3 master/etcds nodes, 2 infra nodes, 4 compute nodes, and 2 newly added master nodes which were added successfully with the openshift-master/scaleup.yaml playbook.  The 2 etcds I am trying to add are on the 2 newly added master node instances.
   
Version-Release number of selected component (if applicable):
 ~/openshift-ansible # git log --oneline -1
df9f5ed Merge pull request #8299 from mgleung/upgrade-calico-master

# openshift version
openshift v3.10.0-0.38.0
kubernetes v1.10.0+b81c8f8
etcd 3.2.16

Container Runtime Version:          cri-o://1.10.1

# rpm -qva | grep cri-o
cri-o-1.10.1-2.git728df92.el7.x86_64

# rpm -q openshift-ansible
openshift-ansible-3.10.0-0.38.0.git.7.848b045.el7.noarch

# rpm -q ansible
ansible-2.4.3.0-1.el7ae.noarch

# ansible --version
ansible 2.4.3.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, May  4 2018, 09:38:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-34)]


How reproducible:
Always

Steps to Reproduce:
1. Install an OCP 3.10.0-0.38.0 HA CRI-O (rpm install) cluster (1 load-balancer, 3 master/etcds nodes, 2 infra nodes, 4 compute nodes) in AWS EC2 with openshift-ansible/ with inventory attached:

  Run:
    - ansible-playbook -i inv openshift-ansible/playbooks/prerequisites.yml
    - ansible-playbook -i inv run openshift-ansible/playbooks/deploy_cluster.yml

2.Create two new instances in AWS EC2

3. Update inventory to add two master nodes (attached "master_scaleup_inventory").

4. On one master/etcd host, run:
   ansible-playbook -i master_scaleup_inv openshift-ansible/playbooks/prerequisites.yml

5. ansible-playbook -i master_scaleup_inv openshift-ansible/playbooks/openshift-master/scaleup.yml

6. Verify cluster is up and 2 newly added masters are running as static pods:
   oc get pods -n kube-system

7. Create etcd_scaleup inventory with the 2 newly added master hostnames

8. Run etcd_scaleup inventory
   ansible-playbook -i etcd_scaleup_inv openshift-ansible/playbooks/openshift-etcd/scaleup.yml

9.  I have also tried to re-run the prerequistes.yaml playbook before the openshift-etcd/scaleup.yml playbook and I am getting the same error in the same task

Actual results:

Error in task:
TASK [etcd : Add new etcd members to cluster] 
(see "Description of problem" above)

Expected results:
Playbook successful completion and 2 added etcds running as static pods on the two newly added master nodes


Additional info:
Inventory files and logs from ansible-playbook with the -vvv flag are in next comment