Description of problem:
-----------------------
Re-running overcloud deploy after upgrade failed:

openstack overcloud deploy \
  --timeout 100 \
  --templates /usr/share/openstack-tripleo-heat-templates \
  --libvirt-type kvm \
  --stack overcloud \
  -r /home/stack/composable_roles/roles/roles_data.yaml \
  -e /home/stack/composable_roles/roles/nodes.yaml \
  -e /home/stack/composable_roles/internal.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /home/stack/composable_roles/network/network-environment.yaml \
  -e /home/stack/composable_roles/enable-tls.yaml \
  -e /home/stack/composable_roles/inject-trust-anchor.yaml \
  -e /home/stack/composable_roles/public_vip.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
  -e /home/stack/composable_roles/hostnames.yaml \
  -e /home/stack/composable_roles/debug.yaml \
  -e /home/stack/composable_roles/config_heat.yaml \
  -e /home/stack/composable_roles/docker-images.yaml 2>&1
...
2018-05-03 05:57:54Z [overcloud-AllNodesDeploySteps-75rmqeap2l7f.WorkflowTasks_Step2_Execution]: UPDATE_IN_PROGRESS  state changed
2018-05-03 05:57:55Z [overcloud-AllNodesDeploySteps-75rmqeap2l7f.WorkflowTasks_Step2_Execution]: UPDATE_COMPLETE  The Resource WorkflowTasks_Step2_Execution requires replacement.
2018-05-03 05:57:56Z [overcloud-AllNodesDeploySteps-75rmqeap2l7f.WorkflowTasks_Step2_Execution]: CREATE_IN_PROGRESS  state changed
2018-05-03 06:01:56Z [overcloud-AllNodesDeploySteps-75rmqeap2l7f.WorkflowTasks_Step2_Execution]: CREATE_FAILED  resources.WorkflowTasks_Step2_Execution: ERROR
2018-05-03 06:01:56Z [overcloud-AllNodesDeploySteps-75rmqeap2l7f]: UPDATE_FAILED  Resource CREATE failed: resources.WorkflowTasks_Step2_Execution: ERROR
2018-05-03 06:01:56Z [AllNodesDeploySteps]: UPDATE_FAILED  resources.AllNodesDeploySteps: Resource CREATE failed: resources.WorkflowTasks_Step2_Execution: ERROR
2018-05-03 06:01:56Z [overcloud]: UPDATE_FAILED  Resource UPDATE failed: resources.AllNodesDeploySteps: Resource CREATE failed: resources.WorkflowTasks_Step2_Execution: ERROR

 Stack overcloud UPDATE_FAILED 

overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution:
  resource_type: OS::TripleO::WorkflowSteps
  physical_resource_id: 5347177f-ceaf-4a36-974b-6d7c81328afa
  status: CREATE_FAILED
  status_reason: |
    resources.WorkflowTasks_Step2_Execution: ERROR


From /var/log/mistral/ceph-install-workflow.log
...
2018-05-03 02:01:47,648 p=29504 u=mistral |  failed: [192.168.24.6] (item={u'rule_name': u'', u'pg_num': 32, u'name': u'metrics'}) => {"changed": false, "cmd": ["docker", "exec", "ceph-mon-controller-2", "ceph", "--cluster", "ceph", "osd", "pool", "application", "enable", "metrics", "rbd"], "delta": "0:00:00.453216", "end": "2018-05-03 06:01:47.207562", "item": {"name": "metrics", "pg_num": 32, "rule_name": ""}, "msg": "non-zero return code", "rc": 1, "start": "2018-05-03 06:01:46.754346", "stderr": "Error EPERM: Are you SURE? Pool 'metrics' already has an enabled application; pass --yes-i-really-mean-it to proceed anyway", "stderr_lines": ["Error EPERM: Are you SURE? Pool 'metrics' already has an enabled application; pass --yes-i-really-mean-it to proceed anyway"], "stdout": "", "stdout_lines": []}
2018-05-03 02:01:48,969 p=29504 u=mistral |  ok: [192.168.24.6] => (item={u'rule_name': u'', u'pg_num': 32, u'name': u'backups'}) => {"changed": false, "cmd": ["docker", "exec", "ceph-mon-controller-2", "ceph", "--cluster", "ceph", "osd", "pool", "application", "enable", "backups", "rbd"], "delta": "0:00:01.069515", "end": "2018-05-03 06:01:48.527271", "item": {"name": "backups", "pg_num": 32, "rule_name": ""}, "rc": 0, "start": "2018-05-03 06:01:47.457756", "stderr": "enabled application 'rbd' on pool 'backups'", "stderr_lines": ["enabled application 'rbd' on pool 'backups'"], "stdout": "", "stdout_lines": []}
2018-05-03 02:01:49,968 p=29504 u=mistral |  ok: [192.168.24.6] => (item={u'rule_name': u'', u'pg_num': 32, u'name': u'vms'}) => {"changed": false, "cmd": ["docker", "exec", "ceph-mon-controller-2", "ceph", "--cluster", "ceph", "osd", "pool", "application", "enable", "vms", "rbd"], "delta": "0:00:00.736113", "end": "2018-05-03 06:01:49.529686", "item": {"name": "vms", "pg_num": 32, "rule_name": ""}, "rc": 0, "start": "2018-05-03 06:01:48.793573", "stderr": "enabled application 'rbd' on pool 'vms'", "stderr_lines": ["enabled application 'rbd' on pool 'vms'"], "stdout": "", "stdout_lines": []}
2018-05-03 02:01:50,984 p=29504 u=mistral |  ok: [192.168.24.6] => (item={u'rule_name': u'', u'pg_num': 32, u'name': u'volumes'}) => {"changed": false, "cmd": ["docker", "exec", "ceph-mon-controller-2", "ceph", "--cluster", "ceph", "osd", "pool", "application", "enable", "volumes", "rbd"], "delta": "0:00:00.650049", "end": "2018-05-03 06:01:50.547516", "item": {"name": "volumes", "pg_num": 32, "rule_name": ""}, "rc": 0, "start": "2018-05-03 06:01:49.897467", "stderr": "enabled application 'rbd' on pool 'volumes'", "stderr_lines": ["enabled application 'rbd' on pool 'volumes'"], "stdout": "", "stdout_lines": []}
2018-05-03 02:01:50,987 p=29504 u=mistral |  PLAY RECAP *********************************************************************
2018-05-03 02:01:50,988 p=29504 u=mistral |  192.168.24.11              : ok=3    changed=0    unreachable=0    failed=0   
2018-05-03 02:01:50,988 p=29504 u=mistral |  192.168.24.13              : ok=3    changed=0    unreachable=0    failed=0   
2018-05-03 02:01:50,988 p=29504 u=mistral |  192.168.24.14              : ok=58   changed=4    unreachable=0    failed=0   
2018-05-03 02:01:50,988 p=29504 u=mistral |  192.168.24.17              : ok=4    changed=0    unreachable=0    failed=0   
2018-05-03 02:01:50,988 p=29504 u=mistral |  192.168.24.21              : ok=4    changed=0    unreachable=0    failed=0   
2018-05-03 02:01:50,988 p=29504 u=mistral |  192.168.24.24              : ok=58   changed=6    unreachable=0    failed=0   
2018-05-03 02:01:50,988 p=29504 u=mistral |  192.168.24.6               : ok=62   changed=5    unreachable=0    failed=1   
2018-05-03 02:01:50,988 p=29504 u=mistral |  192.168.24.7               : ok=3    changed=0    unreachable=0    failed=0   



Version-Release number of selected component (if applicable):
-------------------------------------------------------------
ceph-ansible-3.1.0-0.1.beta8.el7cp.noarch
puppet-ceph-2.5.0-1.el7ost.noarch
openstack-tripleo-heat-templates-8.0.2-4.el7ost.noarch

Steps to Reproduce:
-------------------
1. Upgrade UC from 12 to 13
2. Upgrade OC from 12 to 13
3. Run converge step
4. Upgrade ceph
openstack overcloud ceph-upgrade run \
    --templates /usr/share/openstack-tripleo-heat-templates \
        -e /home/stack/composable_roles/roles/nodes.yaml \
        -e /home/stack/composable_roles/internal.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
        -e /home/stack/composable_roles/network/network-environment.yaml \
        -e /home/stack/composable_roles/enable-tls.yaml \
        -e /home/stack/composable_roles/inject-trust-anchor.yaml \
        -e /home/stack/composable_roles/public_vip.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
        -e /home/stack/composable_roles/hostnames.yaml \
        -e /home/stack/composable_roles/debug.yaml \
        -e /home/stack/composable_roles/config_heat.yaml \
        -e /home/stack/composable_roles/docker-images.yaml \
        --container-registry-file /home/stack/composable_roles/docker-images.yaml \
    --roles-file /home/stack/composable_roles/roles/roles_data.yaml

5. Re-run overcloud deploy

Actual results:
---------------
Re-running deploy fails

Additional info:
----------------
Virtual setup: 3controllers + 3messaging + 3database + 3ceph + 2networker + 2compute