Description of problem: Overcloud deployment has failed with this message: 2018-10-19 14:46:51 | "2018-10-19 18:34:10,726 ERROR: 19339 -- Failed running docker-puppet.py for crond", 2018-10-19 14:46:51 | "2018-10-19 18:34:10,726 ERROR: 19339 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-crond\" is already in use by container 622001fb6d5d2756fac0cc99b5a79720e1ac44062eedbf32bce9345cfb75d518. You have to remove (or rename) that container to be able to reuse that name..", 2018-10-19 14:46:51 | "See '/usr/bin/docker-current run --help'.", while processing compute-0 node. Deployment command line: openstack overcloud deploy \ --templates /usr/share/openstack-tripleo-heat-templates \ --libvirt-type kvm --control-flavor baremetal --compute-flavor baremetal --ceph-storage-flavor baremetal --block-storage-flavor oooq_blockstorage --swift-storage-flavor oooq_objectstorage --timeout 90 --ntp-server 10.5.26.10 -e /home/stack/cloud-names.yaml -e /home/stack/neutronl3ha.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml -e /home/stack/containers-default-parameters.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /home/stack/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml -e /home/stack/inject-trust-anchor.yaml --validation-warnings-fatal -e /usr/share/openstack-tripleo-heat-templates/environments/config-debug.yaml There's no trace of any container in every node (both controllers and computes). The "docker ps" and "docker ps -a" commands output are empty. Version-Release number of selected component (if applicable): Puddle 2018-10-17.2 How reproducible: Still trying to verify if this is consistent. Steps to Reproduce: 1. Deploy with the command line mentioned above. Actual results: Deployment failure. Expected results: Successful deployment. Additional info: The sosreports are on their way and will be linked here.
A new deployment with the same Puddle (2018-10-17.2) failed again during the overcloud deploy step. This time the conflicts are multiple compared to the original failure: [stack@undercloud ~]$ grep ERROR overcloud_deploy.log 2018-10-22 12:57:48 | "2018-10-22 16:44:17,999 ERROR: 28138 -- Failed running docker-puppet.py for clustercheck", 2018-10-22 12:57:48 | "2018-10-22 16:44:17,999 ERROR: 28138 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-clustercheck\" is already in use by container d6cac5f55471dc2f72e7e5107a7862674caa45debd34ea57da8e51921e65757d. You have to remove (or rename) that container to be able to reuse that name..", 2018-10-22 12:57:48 | "2018-10-22 16:45:30,764 ERROR: 28137 -- Failed running docker-puppet.py for swift_ringbuilder", 2018-10-22 12:57:48 | "2018-10-22 16:45:30,764 ERROR: 28137 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-swift_ringbuilder\" is already in use by container 3ab1de1ff90e318d90f6f3418dfe62f428330f53299d075f6fbda99d36c41367. You have to remove (or rename) that container to be able to reuse that name..", 2018-10-22 12:57:48 | "2018-10-22 16:45:37,106 ERROR: 28136 -- Failed running docker-puppet.py for nova_placement", 2018-10-22 12:57:48 | "2018-10-22 16:45:37,106 ERROR: 28136 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-nova_placement\" is already in use by container 9a86ea58bb9e1a2198ee7e2b4a01a01e9da17d746f89a17b3f83f55c2f4dff28. You have to remove (or rename) that container to be able to reuse that name..", 2018-10-22 12:57:48 | "2018-10-22 16:46:26,081 ERROR: 28138 -- Failed running docker-puppet.py for nova", 2018-10-22 12:57:48 | "2018-10-22 16:46:26,081 ERROR: 28138 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-nova\" is already in use by container 01e10a99502b1678f3f6cfaf25e2fa136b1a91949194e72f959cc4b30560e546. You have to remove (or rename) that container to be able to reuse that name..", 2018-10-22 12:57:48 | "2018-10-22 16:48:42,126 ERROR: 28137 -- Failed running docker-puppet.py for keystone", 2018-10-22 12:57:48 | "2018-10-22 16:48:42,126 ERROR: 28137 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-keystone\" is already in use by container 486434fee1fe492784bace0bcadaf8502475efe3c1283a68f71abbf33355931b. You have to remove (or rename) that container to be able to reuse that name..", 2018-10-22 12:57:48 | "2018-10-22 16:50:58,768 ERROR: 28136 -- Failed running docker-puppet.py for heat", 2018-10-22 12:57:48 | "2018-10-22 16:50:58,768 ERROR: 28136 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-heat\" is already in use by container bc76487ba0236119de44d954e85c2f7ab9865c3961405d0fe909d4dffc85fa3b. You have to remove (or rename) that container to be able to reuse that name..", 2018-10-22 12:57:48 | "2018-10-22 16:51:11,677 ERROR: 28136 -- Failed running docker-puppet.py for crond", 2018-10-22 12:57:48 | "2018-10-22 16:51:11,677 ERROR: 28136 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-crond\" is already in use by container 1d184630731aee73d7a2abad17162a0bbf2705d88c7f1b3fc3acb8d95051a248. You have to remove (or rename) that container to be able to reuse that name..", 2018-10-22 12:57:48 | "2018-10-22 16:54:55,685 ERROR: 28136 -- Failed running docker-puppet.py for horizon", 2018-10-22 12:57:48 | "2018-10-22 16:54:55,685 ERROR: 28136 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-horizon\" is already in use by container 9aa18a8cf2db94229c4af1b448833508aeb845947f6442057804795a77c9015d. You have to remove (or rename) that container to be able to reuse that name..", 2018-10-22 12:57:48 | "2018-10-22 16:55:14,899 ERROR: 28138 -- Failed running docker-puppet.py for neutron", 2018-10-22 12:57:48 | "2018-10-22 16:55:14,899 ERROR: 28138 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-neutron\" is already in use by container d0b1c395f1dba120df1cfd37b5a5e43bc384762a7598b80cc992f1dcab6f453c. You have to remove (or rename) that container to be able to reuse that name..", 2018-10-22 12:57:48 | "2018-10-22 16:56:12,157 ERROR: 28135 -- ERROR configuring nova_placement", 2018-10-22 12:57:48 | "2018-10-22 16:56:12,157 ERROR: 28135 -- ERROR configuring swift_ringbuilder", 2018-10-22 12:57:48 | "2018-10-22 16:56:12,157 ERROR: 28135 -- ERROR configuring clustercheck", 2018-10-22 12:57:48 | "2018-10-22 16:56:12,157 ERROR: 28135 -- ERROR configuring nova", 2018-10-22 12:57:48 | "2018-10-22 16:56:12,157 ERROR: 28135 -- ERROR configuring keystone", 2018-10-22 12:57:48 | "2018-10-22 16:56:12,157 ERROR: 28135 -- ERROR configuring heat", 2018-10-22 12:57:48 | "2018-10-22 16:56:12,157 ERROR: 28135 -- ERROR configuring crond", 2018-10-22 12:57:48 | "2018-10-22 16:56:12,157 ERROR: 28135 -- ERROR configuring neutron", 2018-10-22 12:57:48 | "2018-10-22 16:56:12,157 ERROR: 28135 -- ERROR configuring horizon" If needed I have the environment still available.
I've raised an upstream bug. I think containers created by docker-puppet.py should have unique names, and auto-delete on terminate.
Hello we fail currently in this issue when we try to deploy a new compute node on out openstack 13. (undercloud) [stack@Labsup-Undercloud-13-1 ~]$ openstack stack resource list overcloud --filter status=FAILED +---------------------+--------------------------------------+------------------------------+-----------------+----------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | +---------------------+--------------------------------------+------------------------------+-----------------+----------------------+ | AllNodesDeploySteps | cbe40440-cacb-484d-aa72-78acfab927a8 | OS::TripleO::PostDeploySteps | UPDATE_FAILED | 2018-12-19T23:36:37Z | +---------------------+--------------------------------------+------------------------------+-----------------+----------------------+ (undercloud) [stack@Labsup-Undercloud-13-1 ~]$ openstack stack failures list --long overcloud overcloud.AllNodesDeploySteps.ComputeDeployment_Step1.1: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 89a69d40-f79c-4fc5-baca-b40de132b2a8 status: CREATE_FAILED status_reason: | Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2 deploy_stdout: | [...] TASK [Run docker-puppet tasks (generate config) during step 1] ***************** ok: [localhost] TASK [Debug output for task which failed: Run docker-puppet tasks (generate config) during step 1] *** fatal: [localhost]: FAILED! => { "failed_when_result": true, "outputs.stdout_lines|default([])|union(outputs.stderr_lines|default([]))": [ "2018-12-20 00:38:23,949 INFO: 977685 -- Running docker-puppet", "2018-12-20 00:38:23,950 INFO: 977685 -- Service compilation completed.", "2018-12-20 00:38:23,950 INFO: 977685 -- Starting multiprocess configuration steps. Using 3 processes.", "2018-12-20 00:38:23,962 INFO: 977688 -- Starting configuration of crond using image 10.14.53.1:8787/rhosp13/openstack-cron:13.0-60", "2018-12-20 00:38:23,962 INFO: 977687 -- Starting configuration of nova_libvirt using image 10.14.53.1:8787/rhosp13/openstack-nova-compute:13.0-62.1", "2018-12-20 00:38:23,962 INFO: 977686 -- Starting configuration of ceilometer using image 10.14.53.1:8787/rhosp13/openstack-ceilometer-central:13.0-53", "2018-12-20 00:38:23,964 INFO: 977688 -- Removing container: docker-puppet-crond", "2018-12-20 00:38:23,964 INFO: 977687 -- Removing container: docker-puppet-nova_libvirt", "2018-12-20 00:38:23,964 INFO: 977686 -- Removing container: docker-puppet-ceilometer", "2018-12-20 00:38:23,988 INFO: 977686 -- Pulling image: 10.14.53.1:8787/rhosp13/openstack-ceilometer-central:13.0-53", "2018-12-20 00:38:23,989 INFO: 977687 -- Pulling image: 10.14.53.1:8787/rhosp13/openstack-nova-compute:13.0-62.1", "2018-12-20 00:38:23,990 INFO: 977688 -- Pulling image: 10.14.53.1:8787/rhosp13/openstack-cron:13.0-60", "2018-12-20 00:38:24,048 ERROR: 977688 -- Failed running docker-puppet.py for crond", "2018-12-20 00:38:24,048 ERROR: 977688 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-crond\" is already in use by container af2203347b835c4b80dfe05448cd336b65bcb64ec827b21805f169b1d0727124. You have to remove (or rename) that container to be able to reuse that name..", "See '/usr/bin/docker-current run --help'.", "", "2018-12-20 00:38:24,048 INFO: 977688 -- Finished processing puppet configs for crond", "2018-12-20 00:38:24,049 INFO: 977688 -- Starting configuration of neutron using image 10.14.53.1:8787/rhosp13/openstack-neutron-server:13.0-58", "2018-12-20 00:38:24,049 ERROR: 977686 -- Failed running docker-puppet.py for ceilometer", "2018-12-20 00:38:24,049 INFO: 977688 -- Removing container: docker-puppet-neutron", "2018-12-20 00:38:24,049 ERROR: 977686 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-ceilometer\" is already in use by container 50848130d58d2f70d45d683323796213b2e53db26247a8249438bedaa0a3a622. You have to remove (or rename) that container to be able to reuse that name..", "2018-12-20 00:38:24,049 INFO: 977686 -- Finished processing puppet configs for ceilometer", "2018-12-20 00:38:24,049 INFO: 977686 -- Starting configuration of iscsid using image 10.14.53.1:8787/rhosp13/openstack-iscsid:13.0-54", "2018-12-20 00:38:24,050 INFO: 977686 -- Removing container: docker-puppet-iscsid", "2018-12-20 00:38:24,054 ERROR: 977687 -- Failed running docker-puppet.py for nova_libvirt", "2018-12-20 00:38:24,055 ERROR: 977687 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-nova_libvirt\" is already in use by container ad43cef92e5edc71ca0f37ef93b5b15f8d06c7b4a63bce7efef3185d6381f669. You have to remove (or rename) that container to be able to reuse that name..", "2018-12-20 00:38:24,055 INFO: 977687 -- Finished processing puppet configs for nova_libvirt", "2018-12-20 00:38:24,067 INFO: 977688 -- Pulling image: 10.14.53.1:8787/rhosp13/openstack-neutron-server:13.0-58", "2018-12-20 00:38:24,067 INFO: 977686 -- Pulling image: 10.14.53.1:8787/rhosp13/openstack-iscsid:13.0-54", "2018-12-20 00:38:24,123 ERROR: 977686 -- Failed running docker-puppet.py for iscsid", "2018-12-20 00:38:24,123 ERROR: 977686 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-iscsid\" is already in use by container 9c2a2a7328fea54343f81e35e96f22073334bbb11852b55167ca915fa3a13047. You have to remove (or rename) that container to be able to reuse that name..", "2018-12-20 00:38:24,123 INFO: 977686 -- Finished processing puppet configs for iscsid", "2018-12-20 00:38:24,132 ERROR: 977688 -- Failed running docker-puppet.py for neutron", "2018-12-20 00:38:24,132 ERROR: 977688 -- /usr/bin/docker-current: Error response from daemon: Conflict. The container name \"/docker-puppet-neutron\" is already in use by container dd31d1dac20bd533b150af48c24bf24f2f8d97887e2bde66afd69c0ac9037622. You have to remove (or rename) that container to be able to reuse that name..", "2018-12-20 00:38:24,132 INFO: 977688 -- Finished processing puppet configs for neutron", "2018-12-20 00:38:24,132 ERROR: 977685 -- ERROR configuring ceilometer", "2018-12-20 00:38:24,133 ERROR: 977685 -- ERROR configuring nova_libvirt", "2018-12-20 00:38:24,133 ERROR: 977685 -- ERROR configuring crond", "2018-12-20 00:38:24,133 ERROR: 977685 -- ERROR configuring neutron", "2018-12-20 00:38:24,133 ERROR: 977685 -- ERROR configuring iscsid" ] } to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/62b49199-ace4-4300-820a-25dd8c9f1e75_playbook.retry PLAY RECAP ********************************************************************* localhost : ok=24 changed=6 unreachable=0 failed=1 deploy_stderr: | We are in redhat openstack 13 version. Regards
What is the version of the used openstack-tripleo-heat-templates package? It should be not earlier than openstack-tripleo-heat-templates-9.0.1-0.20181013060870.ffbe879.el7ost
Oh right, that's for OSP13 and https://review.openstack.org/#/q/I4386b155a4bdba430dc350914db7a6b6fdf92ac0 shows a backport is missing for Queens upstream as well. I'll clone this bug to track it for OSP13 then.
I cannot verify it using your command, because I don't see output for specific yaml files and etc, but for regular deployments it was pass openstack-tripleo-heat-templates-9.0.1-0.20181013060907.el7ost.noarch.rpm
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045