Description of problem: In the scale lab the OSP Perf Team is doing a 750 node compute scale with ceph (3 ceph nodes). To expedite deploys we are utilizing the "--limit" flag in our deploy which greatly reduces deploys. However we have noticed that "ceph_ansible_command.sh" isn't passed the limiting nodes and repeats the ceph tasks for all nodes which increases deploy times exponentially. Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. Deploy overcloud with controllers,ceph, and compute 2. scale up overcloud with computes utilizing --limit 3. verify ceph ansible replayed tasks on all nodes John F. did some initial work on this in 16.0 on our last scale with https://bugzilla.redhat.com/show_bug.cgi?id=1798781. Not sure if it was supposed to make it into this release Expected results: Ceph-ansible enforces --limit and provisioned nodes are skipped
For example, the config-download command is openstack overcloud deploy --templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-ha.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/containers-prepare-parameter.yaml \ -e /home/stack/templates/network-environment.yaml \ -e /home/stack/templates/deploy.yaml \ -e /home/stack/templates/ceph.yaml \ -e /home/stack/templates/networking_ovn_hotfix_template.yaml \ -r /home/stack/templates/roles_data.yaml \ --ntp-server clock1.rdu2.redhat.com,clock.redhat.com,clock2.redhat.com \ --validation-warnings-fatal --config-download-only --config-download-timeout 960 --limit Undercloud,Controller,overcloud-fc640compute-10 which results in the right ansible-playbook-command.sh being generated in /var/lib/mistral/overcloud/ansible-playbook-command.sh ansible-playbook-3 -v /var/lib/mistral/overcloud/deploy_steps_playbook.yaml --limit Undercloud:Controller:overcloud-fc640compute-10 --become --timeout 600 --inventory-file /var/lib/mistral/overcloud/tripleo-ansible-inventory.yaml --skip-tags opendev-validation "$@" However, the ceph-ansible command does not have the --limit option and basically runs on all the existing nodes (several 100s) which ends up taking 2.5+ hrs just for ceph ansible even to scale up computes by 1 node (undercloud) [stack@f17-h23-000-1029p ceph-ansible]$ cat ceph_ansible_command.sh #!/usr/bin/env bash set -e echo "Running $0" >> /var/lib/mistral/overcloud/ceph-ansible/ceph_ansible_command.log ANSIBLE_ACTION_PLUGINS=/usr/share/ceph-ansible/plugins/actions/ ANSIBLE_CALLBACK_PLUGINS=/usr/share/ceph-ansible/plugins/callback/ ANSIBLE_FILTER_PLUGINS=/usr/share/ceph-ansible/plugins/filter/ ANSIBLE_ROLES_PATH=/usr/share/ceph-ansible/roles/ ANSIBLE_LOG_PATH="/var/lib/mistral/overcloud/ceph-ansible/ceph_ansible_command.log" ANSIBLE_LIBRARY=/usr/share/ceph-ansible/library/ ANSIBLE_CONFIG=/usr/share/ceph-ansible/ansible.cfg ANSIBLE_REMOTE_TEMP="/tmp/ceph_ansible_tmp" ANSIBLE_FORKS=25 ANSIBLE_GATHER_TIMEOUT=60 ANSIBLE_CALLBACK_WHITELIST=profile_tasks ANSIBLE_STDOUT_CALLBACK=default ansible-playbook --private-key /var/lib/mistral/overcloud/ssh_private_key -e ansible_python_interpreter=/usr/bin/python3 -v --skip-tags package-install,with_pkg --extra-vars @/var/lib/mistral/overcloud/ceph-ansible/extra_vars.yml -i /var/lib/mistral/overcloud/ceph-ansible/inventory.yml /usr/share/ceph-ansible/site-container.yml.sample 2>&1
As per the below reference bz update, I guess ceph client has already enhanced with --limit option from the ceph-ansible-4.0.15 package. But we experienced this issue in the latest ceph-ansible package. https://bugzilla.redhat.com/show_bug.cgi?id=1798781#c2 $ rpm -qa |grep ceph ceph-ansible-4.0.23-1.el8cp.noarch puppet-ceph-3.1.2-0.20200603075505.3b8ab1f.el8ost.noarch Red Hat OpenStack Platform release 16.1.0 RC (Train) OSP Puddle: 16.1_20200625.1 Luke, It seems {{ ansible_limit }} variable not work in OSP16.1. please suggest, if there any additional arguments required in overcloud deploy command.
We currently *do not* set the --limit option for ceph-ansible from {{ ansible_limit }}, it is set from the {{ ceph_ansible_limit }} var [1] which can be set passing -e to ansible-playbook 1. https://github.com/openstack/tripleo-ansible/blob/stable/train/tripleo_ansible/roles/tripleo-ceph-run-ansible/tasks/main.yml#L55
So what would be the workflow for someone kicking off a deploy using "openstack overcloud deploy"(In reply to Giulio Fidente from comment #3) > We currently *do not* set the --limit option for ceph-ansible from {{ > ansible_limit }}, it is set from the {{ ceph_ansible_limit }} var [1] which > can be set passing -e to ansible-playbook Which "naislbe-playbook" is this? Is it the ansible-playbook-command.sh or ceph_ansible_command.sh > > 1. > https://github.com/openstack/tripleo-ansible/blob/stable/train/ > tripleo_ansible/roles/tripleo-ceph-run-ansible/tasks/main.yml#L55 What would the workflow of the user be when driving this from the "openstack overcloud deploy" command?
Looks like it it is by passing -e ceph_ansible_limit to ansible-playbook-command.sh? Even then, it is not clear how a user would drive this from the openstack cli during deploy without needing to tweak the playbook manually.
(In reply to Sai Sindhur Malleni from comment #5) > Looks like it it is by passing -e ceph_ansible_limit to > ansible-playbook-command.sh? Even then, it is not clear how a user would > drive this from the openstack cli during deploy without needing to tweak the > playbook manually. agreed, that is just a workaround; I triaged the bug and we'll work on it in a 16.1 zstream to make the ceph-ansible call reuse {{ ansible_limit }} value
(In reply to Sai Sindhur Malleni from comment #5) > Looks like it it is by passing -e ceph_ansible_limit to > ansible-playbook-command.sh? Even then, it is not clear how a user would > drive this from the openstack cli during deploy without needing to tweak the > playbook manually. alternatively, can you try the "DeploymentServerBlacklist" THT parameter? [1] blacklisted hostnames will *not* end in the ceph-ansible inventory at all, basically implementing something close to what --limit would do 1. https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/ceph-ansible/ceph-base.yaml#L53
Ack. Thanks for clarifying. I will give that a try. Just for future reference, the time taken for ansible config-download run when scaling up the number of compute nodes from 471 to 472 using --limit passed to the openstack overcloud deploy command openstack overcloud deploy --templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-ha.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/containers-prepare-parameter.yaml \ -e /home/stack/templates/network-environment.yaml \ -e /home/stack/templates/deploy.yaml \ -e /home/stack/templates/ceph.yaml \ -e /home/stack/templates/networking_ovn_hotfix_template.yaml \ -r /home/stack/templates/roles_data.yaml \ --ntp-server clock1.rdu2.redhat.com,clock.redhat.com,clock2.redhat.com \ --validation-warnings-fatal --config-download-only --config-download-timeout 960 --limit Undercloud,Controller,overcloud-fc640compute-10 =============================================================================== tripleo-ceph-run-ansible : run ceph-ansible ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 8830.51s Wait for containers to start for step 2 using paunch -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 152.24s Wait for containers to start for step 3 using paunch --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 84.41s Pre-fetch all the containers --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 69.84s Wait for container-puppet tasks (generate config) to finish -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 55.32s Wait for containers to start for step 5 using paunch --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 45.45s tripleo-ceph-uuid : run nodes-uuid command ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 42.59s Run tripleo-container-image-prepare logged to: /var/log/tripleo-container-image-prepare.log ------------------------------------------------------------------------------------------------------------------------------------------ 38.03s Run NetworkConfig script ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 36.75s Write kolla config json files -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 23.23s Creating container startup configs for step_4 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 20.58s Wait for containers to start for step 4 using paunch --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 19.62s tripleo-ceph-run-ansible : search output of ceph-ansible run(s) non-zero return codes ------------------------------------------------------------------------------------------------------------------------------------------------ 18.16s Wait for container-puppet tasks (bootstrap tasks) for step 4 to finish --------------------------------------------------------------------------------------------------------------------------------------------------------------- 16.41s tripleo-hosts-entries : Render out the hosts entries --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 15.68s Wait for puppet host configuration to finish ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 13.34s Wait for puppet host configuration to finish ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 13.31s Wait for container-puppet tasks (bootstrap tasks) for step 3 to finish --------------------------------------------------------------------------------------------------------------------------------------------------------------- 13.27s Wait for puppet host configuration to finish ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 13.18s Wait for puppet host configuration to finish ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 13.13s We can see that by default majority of time is spent in ceph-ansible as it runs against all the nodes.
(In reply to Sai Sindhur Malleni from comment #8) > tripleo-ceph-run-ansible : run ceph-ansible > ----------------------------------------------------------------------------- > ----------------------------------------------------------------------------- > ------------------------------ 8830.51s > > We can see that by default majority of time is spent in ceph-ansible as it > runs against all the nodes. yes, thanks for collecting this data! if you can please try the DeploymentServerBlacklist param and see if it works as intended
(In reply to Giulio Fidente from comment #9) > (In reply to Sai Sindhur Malleni from comment #8) > > tripleo-ceph-run-ansible : run ceph-ansible > > ----------------------------------------------------------------------------- > > ----------------------------------------------------------------------------- > > ------------------------------ 8830.51s > > > > We can see that by default majority of time is spent in ceph-ansible as it > > runs against all the nodes. > > yes, thanks for collecting this data! if you can please try the > DeploymentServerBlacklist param and see if it works as intended So looking at that, it looks like I have to list out every node I don't want ansible to run on? So that's 470+ nodes in my case. Is that right or am I missing something?
(In reply to Sai Sindhur Malleni from comment #10) > (In reply to Giulio Fidente from comment #9) > > (In reply to Sai Sindhur Malleni from comment #8) > > > tripleo-ceph-run-ansible : run ceph-ansible > > > ----------------------------------------------------------------------------- > > > ----------------------------------------------------------------------------- > > > ------------------------------ 8830.51s > > > > > > We can see that by default majority of time is spent in ceph-ansible as it > > > runs against all the nodes. > > > > yes, thanks for collecting this data! if you can please try the > > DeploymentServerBlacklist param and see if it works as intended > So looking at that, it looks like I have to list out every node I don't want > ansible to run on? So that's 470+ nodes in my case. Is that right or am I > missing something? correct, it's a blacklist so you have to list all nodes which don't want to be reconfigured ... the only good news is names should follow all the same pattern when deployed by director
(In reply to Giulio Fidente from comment #11) > (In reply to Sai Sindhur Malleni from comment #10) > > (In reply to Giulio Fidente from comment #9) > > > (In reply to Sai Sindhur Malleni from comment #8) > > > > tripleo-ceph-run-ansible : run ceph-ansible > > > > ----------------------------------------------------------------------------- > > > > ----------------------------------------------------------------------------- > > > > ------------------------------ 8830.51s > > > > > > > > We can see that by default majority of time is spent in ceph-ansible as it > > > > runs against all the nodes. > > > > > > yes, thanks for collecting this data! if you can please try the > > > DeploymentServerBlacklist param and see if it works as intended > > So looking at that, it looks like I have to list out every node I don't want > > ansible to run on? So that's 470+ nodes in my case. Is that right or am I > > missing something? > > correct, it's a blacklist so you have to list all nodes which don't want to > be reconfigured ... the only good news is names should follow all the same > pattern when deployed by director Ack, Just fyi, I have different composable roles and different naming schemes for all my compute nodes :-). I guess, I'm just trying to figure out the best path for me currently as you already triaged this and are going to fix it, so not worried about that. Quick question, is the -e ceph_ansible_limit expected to be passed to ansible-playboook-command.sh or the ceph_ansible_command.sh
Something like ansible-playbook-3 -vvv /var/lib/mistral/overcloud/deploy_steps_playbook.yaml --limit Undercloud:Controller:overcloud-p1029compute-54 --become --timeout 600 --inventory-file /var/lib/mistral/overcloud/tripleo-ansible-inventory.yaml --skip-tags opendev-validation "$@" -e "ceph_ansible_limit=Undercloud:Controller:overcloud-p1029compute-54" seems to be working.
(In reply to Sai Sindhur Malleni from comment #12) > Ack, Just fyi, I have different composable roles and different naming > schemes for all my compute nodes :-). I guess, I'm just trying to figure out > the best path for me currently as you already triaged this and are going to > fix it, so not worried about that. Quick question, is the -e > ceph_ansible_limit expected to be passed to ansible-playboook-command.sh or > the ceph_ansible_command.sh /var/lib/mistral/config-download-latest/ceph-ansible/ceph_ansible_command.sh it will just be appended to the command line for ansible-playbook like it already happens in [1] 1. https://github.com/openstack/tripleo-ansible/blob/stable/train/tripleo_ansible/roles/tripleo-ceph-run-ansible/tasks/main.yml#L55
(In reply to Pradipta Kumar Sahoo from comment #2) > As per the below reference bz update, I guess ceph client has already > enhanced with --limit option from the ceph-ansible-4.0.15 package. But we > experienced this issue in the latest ceph-ansible package. > > https://bugzilla.redhat.com/show_bug.cgi?id=1798781#c2 > > $ rpm -qa |grep ceph > ceph-ansible-4.0.23-1.el8cp.noarch > puppet-ceph-3.1.2-0.20200603075505.3b8ab1f.el8ost.noarch > > Red Hat OpenStack Platform release 16.1.0 RC (Train) > OSP Puddle: 16.1_20200625.1 > > Luke, It seems {{ ansible_limit }} variable not work in OSP16.1. please > suggest, if there any additional arguments required in overcloud deploy > command. I think this question is answered by comment #15
As per Luke: Ansible has a "magic" variable for limit called `ansible_limit`. Whatever is passed as --limit will be saved to it. We can probably re-use that with the Ceph bits.
*** Bug 1856965 has been marked as a duplicate of this bug. ***
Using Ansible --limit with ceph-ansible --------------------------------------- When using config-download to configure Ceph, if Ansible's `--limit` option is used, then it is passed to the execution of ceph-ansible too. This is the case for Train and newer. In the previous section an example was provided where Ceph was deployed with TripleO. The examples below show how to update the deployment and pass the `--limit` option. If oc0-cephstorage-0 had a disk failure and a factory clean disk was put in place of the failed disk, then the following could be run so that the new disk is used to bring up the missing OSD and so that ceph-ansible is only run on the nodes where it needs to be run. This is useful to reduce the time it takes to update the deployment:: openstack overcloud deploy --templates -r /home/stack/roles_data.yaml -n /usr/share/openstack-tripleo-heat-templates/network_data_dashboard.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml -e ~/my-ceph-settings.yaml --limit oc0-controller-0:oc0-controller-2:oc0-controller-1:oc0-cephstorage-0:undercloud If config-download has generated a `ansible-playbook-command.sh` script, then that script may also be run with the `--limit` option and it will be passed to ceph-ansible:: ./ansible-playbook-command.sh --limit oc0-controller-0:oc0-controller-2:oc0-controller-1:oc0-cephstorage-0:undercloud In the above example the controllers are included because the Ceph Mons need Ansible to change their OSD definitions. Both commands above would do the same thing. The former would only be needed if there were Heat environment file updates. After either of the above has run the `~/config-download/config-download-latest/ceph-ansible/ceph_ansible_command.sh` file should contain the `--limit` option. .. warning:: You must always include the undercloud in the limit list or ceph-ansible will not be executed when using `--limit`. This is necessary because the ceph-ansible execution happens through the external_deploy_steps_tasks playbook and that playbook only runs on the undercloud.
Docbug for this feature: bz 1871177
Verified on tripleo-ansible-0.5.1-1.20200914163922.902c3c8.el8ost.noarch cat /var/lib/mistral/overcloud/ceph-ansible/ceph_ansible_command.sh #!/usr/bin/env bash set -e echo "Running $0" >> /var/lib/mistral/overcloud/ceph-ansible/ceph_ansible_command.log ANSIBLE_ACTION_PLUGINS=/usr/share/ceph-ansible/plugins/actions/ ANSIBLE_CALLBACK_PLUGINS=/usr/share/ceph-ansible/plugins/callback/ ANSIBLE_FILTER_PLUGINS=/usr/share/ceph-ansible/plugins/filter/ ANSIBLE_ROLES_PATH=/usr/share/ceph-ansible/roles/ ANSIBLE_LOG_PATH="/var/lib/mistral/overcloud/ceph-ansible/ceph_ansible_command.log" ANSIBLE_SSH_CONTROL_PATH_DIR="/tmp/ceph_ansible_control_path" ANSIBLE_LIBRARY=/usr/share/ceph-ansible/library/ ANSIBLE_CONFIG=/usr/share/ceph-ansible/ansible.cfg ANSIBLE_REMOTE_TEMP="/tmp/ceph_ansible_tmp" ANSIBLE_FORKS=25 ANSIBLE_GATHER_TIMEOUT=60 ANSIBLE_CALLBACK_WHITELIST=profile_tasks ANSIBLE_STDOUT_CALLBACK=default ansible-playbook --private-key /var/lib/mistral/overcloud/ssh_private_key -e ansible_python_interpreter=/usr/bin/python3 -vv --skip-tags package-install,with_pkg --extra-vars @/var/lib/mistral/overcloud/ceph-ansible/extra_vars.yml --limit Undercloud:Controller:compute-1 -i /var/lib/mistral/overcloud/ceph-ansible/inventory.yml /usr/share/ceph-ansible/site-container.yml.sample 2>&1
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:4284