Description of problem: OSP13 minor update: common_deploy_steps_tasks.yaml runs twice for each role which results in a long minor update duration caused by redundant task. Let's take as an example updating a simple Networker role which runs just a few services: openstack overcloud update run --nodes Networker --playbook all we can see this results in the following ansible-playbook commands being run: First one: [root@undercloud-0 stack]# cat /tmp/ansible-mistral-action3UP8t2/ansible-playbook-command.sh #!/bin/bash HOME="/tmp/ansible-mistral-action3UP8t2" ANSIBLE_LOG_PATH="/var/log/mistral/package_update.log" ANSIBLE_HOST_KEY_CHECKING="False" ANSIBLE_CONFIG="/tmp/ansible-mistral-action3UP8t2/ansible.cfg" ansible-playbook -v /var/lib/mistral/204b7661-006f-4398-908f-076939b895ca/update_steps_playbook.yaml --limit Networker --module-path /usr/share/ansible-modules --user heat-admin --become --become-user root --inventory-file /tmp/ansible-mistral-action3UP8t2/inventory.yaml --private-key /tmp/ansible-mistral-action3UP8t2/ssh_private_key "$@" and Second one: [root@undercloud-0 stack]# cat /tmp/ansible-mistral-actionwy3dr5/ansible-playbook-command.sh #!/bin/bash HOME="/tmp/ansible-mistral-actionwy3dr5" ANSIBLE_LOG_PATH="/var/log/mistral/package_update.log" ANSIBLE_HOST_KEY_CHECKING="False" ANSIBLE_CONFIG="/tmp/ansible-mistral-actionwy3dr5/ansible.cfg" ansible-playbook -v /var/lib/mistral/3289d21e-8672-420c-8039-5d981f8925d7/deploy_steps_playbook.yaml --limit Networker --module-path /usr/share/ansible-modules --user heat-admin --become --become-user root --inventory-file /tmp/ansible-mistral-actionwy3dr5/inventory.yaml --private-key /tmp/ansible-mistral-actionwy3dr5/ssh_private_key "$@" Checking the actual playbooks we can spot that update_steps_playbook.yaml includes both update_steps_tasks.yaml and common_deploy_steps_tasks.yaml: [root@undercloud-0 ~]# cat /var/lib/mistral/204b7661-006f-4398-908f-076939b895ca/update_steps_playbook.yaml - hosts: undercloud name: Gather facts undercloud gather_facts: yes become: false - hosts: overcloud name: Gather facts overcloud gather_facts: yes - hosts: all name: Load global variables gather_facts: no tasks: - include_vars: global_vars.yaml - hosts: overcloud name: Run update serial: 1 gather_facts: no tasks: - include: update_steps_tasks.yaml with_sequence: start=0 end=5 loop_control: loop_var: step - include: common_deploy_steps_tasks.yaml with_sequence: start=1 end=5 loop_control: loop_var: step Now checking the contents of deploy_steps_playbook.yaml we can see that it also includes common_deploy_steps_tasks.yaml so these are duplicating the ones in update_steps_playbook.yaml: [root@undercloud-0 ~]# cat /var/lib/mistral/3289d21e-8672-420c-8039-5d981f8925d7/deploy_steps_playbook.yaml - hosts: undercloud name: Gather facts undercloud gather_facts: yes become: false tags: - facts - hosts: overcloud name: Gather facts overcloud gather_facts: yes tags: - facts - hosts: all name: Load global variables gather_facts: no tasks: - include_vars: global_vars.yaml tags: - always - hosts: overcloud name: Bootstrap TripleO servers gather_facts: no any_errors_fatal: yes roles: - tripleo-bootstrap tags: - bootstrap - hosts: overcloud name: Server deployments gather_facts: no any_errors_fatal: yes tasks: - include: ControllerOpenstack/deployments.yaml vars: force: false when: role_name == 'ControllerOpenstack' with_items: "{{ ControllerOpenstack_pre_deployments|default([]) }}" - include: Database/deployments.yaml vars: force: false when: role_name == 'Database' with_items: "{{ Database_pre_deployments|default([]) }}" - include: Messaging/deployments.yaml vars: force: false when: role_name == 'Messaging' with_items: "{{ Messaging_pre_deployments|default([]) }}" - include: Networker/deployments.yaml vars: force: false when: role_name == 'Networker' with_items: "{{ Networker_pre_deployments|default([]) }}" - include: CephStorage/deployments.yaml vars: force: false when: role_name == 'CephStorage' with_items: "{{ CephStorage_pre_deployments|default([]) }}" - include: Compute/deployments.yaml vars: force: false when: role_name == 'Compute' with_items: "{{ Compute_pre_deployments|default([]) }}" tags: - overcloud - pre_deploy_steps - hosts: overcloud name: Host prep steps gather_facts: no any_errors_fatal: yes vars: bootstrap_server_id: 2001f017-6bb9-435b-8b46-ceb2da0908a3 become: true tasks: - include: ControllerOpenstack/host_prep_tasks.yaml when: role_name == 'ControllerOpenstack' - include: Database/host_prep_tasks.yaml when: role_name == 'Database' - include: Messaging/host_prep_tasks.yaml when: role_name == 'Messaging' - include: Networker/host_prep_tasks.yaml when: role_name == 'Networker' - include: CephStorage/host_prep_tasks.yaml when: role_name == 'CephStorage' - include: Compute/host_prep_tasks.yaml when: role_name == 'Compute' tags: - overcloud - host_prep_steps - hosts: undercloud name: External deployment step 1 gather_facts: no any_errors_fatal: yes become: false vars: step: '1' tasks: - include: external_deploy_steps_tasks.yaml tags: - external - external_deploy_steps - hosts: overcloud name: Overcloud deploy step tasks for 1 gather_facts: no any_errors_fatal: yes # FIXME(shardy) - it would be nice to use strategy: free to # allow the tasks per-step to run in parallel on each role, # but that doesn't work with any_errors_fatal: yes vars: bootstrap_server_id: 2001f017-6bb9-435b-8b46-ceb2da0908a3 step: '1' tasks: - include: ControllerOpenstack/deploy_steps_tasks.yaml when: role_name == 'ControllerOpenstack' - include: Database/deploy_steps_tasks.yaml when: role_name == 'Database' - include: Messaging/deploy_steps_tasks.yaml when: role_name == 'Messaging' - include: Networker/deploy_steps_tasks.yaml when: role_name == 'Networker' - include: CephStorage/deploy_steps_tasks.yaml when: role_name == 'CephStorage' - include: Compute/deploy_steps_tasks.yaml when: role_name == 'Compute' tags: - overcloud - deploy_steps - hosts: overcloud name: Overcloud common deploy step tasks 1 gather_facts: no any_errors_fatal: yes vars: bootstrap_server_id: 2001f017-6bb9-435b-8b46-ceb2da0908a3 step: '1' tasks: - include: common_deploy_steps_tasks.yaml tags: - overcloud - deploy_steps - hosts: undercloud name: External deployment step 2 gather_facts: no any_errors_fatal: yes become: false vars: step: '2' tasks: - include: external_deploy_steps_tasks.yaml tags: - external - external_deploy_steps - hosts: overcloud name: Overcloud deploy step tasks for 2 gather_facts: no any_errors_fatal: yes # FIXME(shardy) - it would be nice to use strategy: free to # allow the tasks per-step to run in parallel on each role, # but that doesn't work with any_errors_fatal: yes vars: bootstrap_server_id: 2001f017-6bb9-435b-8b46-ceb2da0908a3 step: '2' tasks: - include: ControllerOpenstack/deploy_steps_tasks.yaml when: role_name == 'ControllerOpenstack' - include: Database/deploy_steps_tasks.yaml when: role_name == 'Database' - include: Messaging/deploy_steps_tasks.yaml when: role_name == 'Messaging' - include: Networker/deploy_steps_tasks.yaml when: role_name == 'Networker' - include: CephStorage/deploy_steps_tasks.yaml when: role_name == 'CephStorage' - include: Compute/deploy_steps_tasks.yaml when: role_name == 'Compute' tags: - overcloud - deploy_steps - hosts: overcloud name: Overcloud common deploy step tasks 2 gather_facts: no any_errors_fatal: yes vars: bootstrap_server_id: 2001f017-6bb9-435b-8b46-ceb2da0908a3 step: '2' tasks: - include: common_deploy_steps_tasks.yaml tags: - overcloud - deploy_steps - hosts: undercloud name: External deployment step 3 gather_facts: no any_errors_fatal: yes become: false vars: step: '3' tasks: - include: external_deploy_steps_tasks.yaml tags: - external - external_deploy_steps - hosts: overcloud name: Overcloud deploy step tasks for 3 gather_facts: no any_errors_fatal: yes # FIXME(shardy) - it would be nice to use strategy: free to # allow the tasks per-step to run in parallel on each role, # but that doesn't work with any_errors_fatal: yes vars: bootstrap_server_id: 2001f017-6bb9-435b-8b46-ceb2da0908a3 step: '3' tasks: - include: ControllerOpenstack/deploy_steps_tasks.yaml when: role_name == 'ControllerOpenstack' - include: Database/deploy_steps_tasks.yaml when: role_name == 'Database' - include: Messaging/deploy_steps_tasks.yaml when: role_name == 'Messaging' - include: Networker/deploy_steps_tasks.yaml when: role_name == 'Networker' - include: CephStorage/deploy_steps_tasks.yaml when: role_name == 'CephStorage' - include: Compute/deploy_steps_tasks.yaml when: role_name == 'Compute' tags: - overcloud - deploy_steps - hosts: overcloud name: Overcloud common deploy step tasks 3 gather_facts: no any_errors_fatal: yes vars: bootstrap_server_id: 2001f017-6bb9-435b-8b46-ceb2da0908a3 step: '3' tasks: - include: common_deploy_steps_tasks.yaml tags: - overcloud - deploy_steps - hosts: undercloud name: External deployment step 4 gather_facts: no any_errors_fatal: yes become: false vars: step: '4' tasks: - include: external_deploy_steps_tasks.yaml tags: - external - external_deploy_steps - hosts: overcloud name: Overcloud deploy step tasks for 4 gather_facts: no any_errors_fatal: yes # FIXME(shardy) - it would be nice to use strategy: free to # allow the tasks per-step to run in parallel on each role, # but that doesn't work with any_errors_fatal: yes vars: bootstrap_server_id: 2001f017-6bb9-435b-8b46-ceb2da0908a3 step: '4' tasks: - include: ControllerOpenstack/deploy_steps_tasks.yaml when: role_name == 'ControllerOpenstack' - include: Database/deploy_steps_tasks.yaml when: role_name == 'Database' - include: Messaging/deploy_steps_tasks.yaml when: role_name == 'Messaging' - include: Networker/deploy_steps_tasks.yaml when: role_name == 'Networker' - include: CephStorage/deploy_steps_tasks.yaml when: role_name == 'CephStorage' - include: Compute/deploy_steps_tasks.yaml when: role_name == 'Compute' tags: - overcloud - deploy_steps - hosts: overcloud name: Overcloud common deploy step tasks 4 gather_facts: no any_errors_fatal: yes vars: bootstrap_server_id: 2001f017-6bb9-435b-8b46-ceb2da0908a3 step: '4' tasks: - include: common_deploy_steps_tasks.yaml tags: - overcloud - deploy_steps - hosts: undercloud name: External deployment step 5 gather_facts: no any_errors_fatal: yes become: false vars: step: '5' tasks: - include: external_deploy_steps_tasks.yaml tags: - external - external_deploy_steps - hosts: overcloud name: Overcloud deploy step tasks for 5 gather_facts: no any_errors_fatal: yes # FIXME(shardy) - it would be nice to use strategy: free to # allow the tasks per-step to run in parallel on each role, # but that doesn't work with any_errors_fatal: yes vars: bootstrap_server_id: 2001f017-6bb9-435b-8b46-ceb2da0908a3 step: '5' tasks: - include: ControllerOpenstack/deploy_steps_tasks.yaml when: role_name == 'ControllerOpenstack' - include: Database/deploy_steps_tasks.yaml when: role_name == 'Database' - include: Messaging/deploy_steps_tasks.yaml when: role_name == 'Messaging' - include: Networker/deploy_steps_tasks.yaml when: role_name == 'Networker' - include: CephStorage/deploy_steps_tasks.yaml when: role_name == 'CephStorage' - include: Compute/deploy_steps_tasks.yaml when: role_name == 'Compute' tags: - overcloud - deploy_steps - hosts: overcloud name: Overcloud common deploy step tasks 5 gather_facts: no any_errors_fatal: yes vars: bootstrap_server_id: 2001f017-6bb9-435b-8b46-ceb2da0908a3 step: '5' tasks: - include: common_deploy_steps_tasks.yaml tags: - overcloud - deploy_steps - hosts: overcloud name: Server Post Deployments gather_facts: no any_errors_fatal: yes tasks: - include: ControllerOpenstack/deployments.yaml vars: force: false when: role_name == 'ControllerOpenstack' with_items: "{{ ControllerOpenstack_post_deployments|default([]) }}" - include: Database/deployments.yaml vars: force: false when: role_name == 'Database' with_items: "{{ Database_post_deployments|default([]) }}" - include: Messaging/deployments.yaml vars: force: false when: role_name == 'Messaging' with_items: "{{ Messaging_post_deployments|default([]) }}" - include: Networker/deployments.yaml vars: force: false when: role_name == 'Networker' with_items: "{{ Networker_post_deployments|default([]) }}" - include: CephStorage/deployments.yaml vars: force: false when: role_name == 'CephStorage' with_items: "{{ CephStorage_post_deployments|default([]) }}" - include: Compute/deployments.yaml vars: force: false when: role_name == 'Compute' with_items: "{{ Compute_post_deployments|default([]) }}" tags: - overcloud - post_deploy_steps - hosts: undercloud name: External deployment Post Deploy tasks gather_facts: no any_errors_fatal: yes become: false tasks: - include: external_post_deploy_steps_tasks.yaml tags: - external - external_deploy_steps Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-8.0.2-33.el7ost.noarch openstack-tripleo-common-8.6.1-20.el7ost.noarch python-tripleoclient-9.2.1-12.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy OSP13 2. Run minor update procedure 3. Check the resulting ansible playbook Actual results: Duplicated common_deploy_steps_tasks.yaml are included. Expected results: No unnecessary tasks are run. Additional info: This results in long minor update times, to give an example, to update *only* 3 controller nodes it takes 1h and 30minutes. To update 2 networker nodes it takes 23 minutes. If I try to re-run the update on the networker nodes after minor update(basically a noop since everything is up to date) it takes 13 minutes which is a lot imo just for a noop for 2 simple nodes.
Created attachment 1450738 [details] controller minor update output Attaching ControllerOpenstack role minor update output
It's being addressed as bug 1589346, i gave the other bug a more descriptive name than it had until now, i'll close this bug as duplicate. *** This bug has been marked as a duplicate of bug 1589346 ***