Bug 1385449

Description of problem:

In ansible hosts file there are 6 hosts, but actually they are multihomed 3 hosts. The m1 == e1 and they have different IP addresses.

[masters]
m1
m2
m3

[etcd]
e1
e2
e3

In this case, 2 facts processes are executed on the same host and going to race, and the following error happens in save_local_facts() method:

TASK [openshift_facts : Gather Cluster facts and set is_containerized if needed]
Traceback (most recent call last):
  File "/tmp/ansible_0cs8t6/ansible_module_openshift_facts.py", line 1331, in save_local_facts
    "Could not create fact file: %s, error: %s" % (filename, ex)
__main__.OpenShiftFactsFileWriteError: Could not create fact file: /etc/ansible/facts.d/openshift.fact, error: [Errno 17] File exists: '/etc/ansible/facts.d'

Both 2 processes pass the exists == false and one makedirs and another failed, it makes the whole installation failed.

We can probably retry the exists/makedir if it failed.

In /usr/share/ansible/openshift-ansible/roles/openshift_facts/library/openshift_facts.py:

def save_local_facts(filename, facts):
    """ Save local facts

        Args:
            filename (str): local facts file
            facts (dict): facts to set
    """
    try:
        fact_dir = os.path.dirname(filename)
        if not os.path.exists(fact_dir):
            os.makedirs(fact_dir)
        with open(filename, 'w') as fact_file:
            fact_file.write(module.jsonify(facts))
        os.chmod(filename, 0o600)
    except (IOError, OSError) as ex:
        raise OpenShiftFactsFileWriteError(
            "Could not create fact file: %s, error: %s" % (filename, ex)
        )

Version-Release number of selected component (if applicable):

OpenShift Container Platform 3.3

How reproducible:

Rare, timing issue

Steps to Reproduce:
1. perform install to hosts with multihomed master/etcd
2.
3.

Actual results:

Sometimes failed with:
__main__.OpenShiftFactsFileWriteError: Could not create fact file: /etc/ansible/facts.d/openshift.fact, error: [Errno 17] File exists: '/etc/ansible/facts.d'

Expected results:

No failure

Additional info: