Description of problem:

When upgrading from OSP15 to OSP16, at the time of running the converge step (executes the whole deployment underneath) the following task failed:


2020-04-07 14:40:05,075 p=48174 u=mistral |  TASK [tripleo-ceph-common : set calling_ansible_environment_variables] *********
2020-04-07 14:40:05,076 p=48174 u=mistral |  Tuesday 07 April 2020  14:40:05 +0000 (0:00:00.134)       0:06:57.979 *********
2020-04-07 14:40:05,134 p=48174 u=mistral |  skipping: [undercloud] => {"changed": false, "skip_reason": "Conditional result was False"}
2020-04-07 14:40:05,194 p=48174 u=mistral |  TASK [create ceph-ansible working direcotry] ***********************************
2020-04-07 14:40:05,195 p=48174 u=mistral |  Tuesday 07 April 2020  14:40:05 +0000 (0:00:00.118)       0:06:58.098 *********
2020-04-07 14:40:05,470 p=48174 u=mistral |  TASK [tripleo-ceph-work-dir : create ceph-ansible temp dirs] *******************
2020-04-07 14:40:05,471 p=48174 u=mistral |  Tuesday 07 April 2020  14:40:05 +0000 (0:00:00.275)       0:06:58.374 *********
2020-04-07 14:40:05,832 p=48174 u=mistral |  changed: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible) => {"ansible_loop_var": "item", "changed": true, "gid": 42430, "group": "42430", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible", "secontext": "system_u:object_r:container_file_t:s0", "size": 205, "state": "directory", "uid": 1002}
2020-04-07 14:40:06,137 p=48174 u=mistral |  changed: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible/group_vars) => {"ansible_loop_var": "item", "changed": true, "gid": 42430, "group": "42430", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/group_vars", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/group_vars", "secontext": "system_u:object_r:container_file_t:s0", "size": 88, "state": "directory", "uid": 1002}
2020-04-07 14:40:06,390 p=48174 u=mistral |  changed: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible/host_vars) => {"ansible_loop_var": "item", "changed": true, "gid": 42430, "group": "42430", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/host_vars", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/host_vars", "secontext": "system_u:object_r:container_file_t:s0", "size": 174, "state": "directory", "uid": 1002}
2020-04-07 14:40:06,646 p=48174 u=mistral |  changed: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible/fetch_dir) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/fetch_dir", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/fetch_dir", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 6, "state": "directory", "uid": 1002}
2020-04-07 14:40:06,729 p=48174 u=mistral |  TASK [tripleo-ceph-work-dir : symbolic link to tripleo inventory from ceph-ansible work directory] ***
2020-04-07 14:40:06,729 p=48174 u=mistral |  Tuesday 07 April 2020  14:40:06 +0000 (0:00:01.258)       0:06:59.632 *********
2020-04-07 14:40:07,040 p=48174 u=mistral |  fatal: [undercloud]: FAILED! => {"changed": false, "gid": 42430, "group": "42430", "mode": "0664", "msg": "refusing to convert from file to symlink for /var/lib/mistral/qe-Cloud-0/ceph-ansible/inventory.yml", "owner": "42430", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/inventory.yml", "secontext": "system_u:object_r:container_file_t:s0", "size": 1303, "state": "file", "uid": 42430}
2020-04-07 14:40:07,041 p=48174 u=mistral |  NO MORE HOSTS LEFT *************************************************************
2020-04-07 14:40:07,042 p=48174 u=mistral |  PLAY RECAP *********************************************************************
2020-04-07 14:40:07,042 p=48174 u=mistral |  ceph-0                     : ok=103  changed=21   unreachable=0    failed=0    skipped=55   rescued=0    ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral |  ceph-1                     : ok=99   changed=21   unreachable=0    failed=0    skipped=55   rescued=0    ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral |  ceph-2                     : ok=99   changed=21   unreachable=0    failed=0    skipped=55   rescued=0    ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral |  compute-0                  : ok=128  changed=37   unreachable=0    failed=0    skipped=71   rescued=0    ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral |  compute-1                  : ok=127  changed=37   unreachable=0    failed=0    skipped=71   rescued=0    ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral |  controller-0               : ok=151  changed=51   unreachable=0    failed=0    skipped=73   rescued=0    ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral |  controller-1               : ok=148  changed=50   unreachable=0    failed=0    skipped=73   rescued=0    ignored=0
2020-04-07 14:40:07,043 p=48174 u=mistral |  controller-2               : ok=148  changed=50   unreachable=0    failed=0    skipped=73   rescued=0    ignored=0
2020-04-07 14:40:07,044 p=48174 u=mistral |  undercloud                 : ok=12   changed=3    unreachable=0    failed=1    skipped=4    rescued=0    ignored=0
2020-04-07 14:40:07,044 p=48174 u=mistral |  Tuesday 07 April 2020  14:40:07 +0000 (0:00:00.314)       0:06:59.947 *********
2020-04-07 14:40:07,044 p=48174 u=mistral |  ===============================================================================

After some debugging and local testing, I did try adding force: yes to the task:

- name: symbolic link to tripleo inventory from ceph-ansible work directory
  # If we call ceph-ansible with the same inventory as the calling
  # playbook, then config-download/groups_vars will be used instead
  # of config-download/ceph-ansible/group_vars.
  file:
    src: "{{ inventory_file }}"
    dest: "{{ playbook_dir }}/ceph-ansible/inventory.yml"
    state: link
    force: true
  tags:
    - run_uuid_ansible
    - run_ceph_ansible


Re-ran the same converge and then the task successfuly executed:

TASK [tripleo-ceph-work-dir : create ceph-ansible temp dirs] *******************
Tuesday 07 April 2020  16:14:32 +0000 (0:00:00.254)       0:07:43.964 *********
ok: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible) => {"ansible_loop_var": "item", "changed": false, "gid": 42430, "group": "42430", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible", "secontext": "system_u:object_r:container_file_t:s0", "size": 222, "state": "directory", "uid": 1002}
ok: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible/group_vars) => {"ansible_loop_var": "item", "changed": false, "gid": 42430, "group": "42430", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/group_vars", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/group_vars", "secontext": "system_u:object_r:container_file_t:s0", "size": 88, "state": "directory", "uid": 1002}
ok: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible/host_vars) => {"ansible_loop_var": "item", "changed": false, "gid": 42430, "group": "42430", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/host_vars", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/host_vars", "secontext": "system_u:object_r:container_file_t:s0", "size": 174, "state": "directory", "uid": 1002}
ok: [undercloud] => (item=/var/lib/mistral/qe-Cloud-0/ceph-ansible/fetch_dir) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/fetch_dir", "mode": "0755", "owner": "tripleo-admin", "path": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/fetch_dir", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 6, "state": "directory", "uid": 1002}

TASK [tripleo-ceph-work-dir : symbolic link to tripleo inventory from ceph-ansible work directory] ***
Tuesday 07 April 2020  16:14:33 +0000 (0:00:01.312)       0:07:45.277 *********
changed: [undercloud] => {"changed": true, "dest": "/var/lib/mistral/qe-Cloud-0/ceph-ansible/inventory.yml", "gid": 1003, "group": "tripleo-admin", "mode": "0777", "owner": "tripleo-admin", "secontext": "unconfined_u:object_r:container_file_t:s0", "size": 58, "src": "/var/lib/mistral/qe-Cloud-0/tripleo-ansible-inventory.yaml", "state": "link", "uid": 1002}

Checking the upstreame code, it looks like  that line was missed to backport to tripleo-ansible stable/train: https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ceph_work_dir/tasks/prepare.yml#L37 


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: