Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1713615

Summary: OSP 14->15: Upgrade data transfer step: "/usr/bin/python: No such file or directory"
Product: Red Hat OpenStack Reporter: Jiri Stransky <jstransk>
Component: openstack-tripleo-heat-templatesAssignee: Jiri Stransky <jstransk>
Status: CLOSED ERRATA QA Contact: Ronnie Rasouli <rrasouli>
Severity: high Docs Contact:
Priority: high    
Version: 15.0 (Stein)CC: jfrancoa, lbezdick, mburns
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-10.6.1-0.20190909163923.999c846.el8ost.noarch Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-05 11:59:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1727807    

Description Jiri Stransky 2019-05-24 09:15:52 UTC 
When running:

openstack overcloud external-upgrade run \
    --tags system_upgrade_transfer_data

We get:

TASK [tripleo-transfer : create the archive] ***********************************
Friday 24 May 2019  08:56:45 +0000 (0:00:00.616)       0:00:47.934 ************ 
[DEPRECATION WARNING]: evaluating mysql_upgrade_transfer as a bare variable, 
this behaviour will go away and you might need to add |bool to the expression 
in the future. Also see CONDITIONAL_BARE_VARS configuration toggle.. This 
feature will be removed in version 2.12. Deprecation warnings can be disabled 
by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Consider using the unarchive module rather than running 'tar'.  If
you need to use command because unarchive is insufficient you can add 'warn:
false' to this command task or set 'command_warnings=False' in ansible.cfg to
get rid of this message.
changed: [undercloud] => {"changed": true, "cmd": "tar --transform \"s|^mysql|mysql|\" -czf \"/tmp/ansible.tripleo-transfer.9jcfQq\" -C \"/var/lib\" \"mysql\"\n", "delta": "0:00:08.632559", "end": "2019-05-24 08
:56:55.144521", "rc": 0, "start": "2019-05-24 08:56:46.511962", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [tripleo-transfer : fetch the archive] ************************************
Friday 24 May 2019  08:56:54 +0000 (0:00:09.002)       0:00:56.937 ************ 
[DEPRECATION WARNING]: evaluating mysql_upgrade_transfer as a bare variable, 
this behaviour will go away and you might need to add |bool to the expression 
in the future. Also see CONDITIONAL_BARE_VARS configuration toggle.. This 
feature will be removed in version 2.12. Deprecation warnings can be disabled 
by setting deprecation_warnings=False in ansible.cfg.
changed: [undercloud] => {"changed": true, "checksum": "7320ea8a4a03a90e04cdaa4ed85ba6def4ec0525", "dest": "/var/lib/mistral/tripleo-transfer/controller-0/var/lib/mysql.tar.gz", "md5sum": "02c206a940060174633667
a5530a54db", "remote_checksum": "7320ea8a4a03a90e04cdaa4ed85ba6def4ec0525", "remote_md5sum": null}

TASK [tripleo-transfer : remove tempfile] **************************************
Friday 24 May 2019  08:57:02 +0000 (0:00:07.506)       0:01:04.443 ************ 
[DEPRECATION WARNING]: evaluating mysql_upgrade_transfer as a bare variable, 
this behaviour will go away and you might need to add |bool to the expression 
in the future. Also see CONDITIONAL_BARE_VARS configuration toggle.. This 
feature will be removed in version 2.12. Deprecation warnings can be disabled 
by setting deprecation_warnings=False in ansible.cfg.
changed: [undercloud] => {"changed": true, "path": "/tmp/ansible.tripleo-transfer.9jcfQq", "state": "absent"}

TASK [tripleo-transfer : wipe the destination directory] ***********************
Friday 24 May 2019  08:57:02 +0000 (0:00:00.377)       0:01:04.821 ************ 
[DEPRECATION WARNING]: evaluating mysql_upgrade_transfer as a bare variable, 
this behaviour will go away and you might need to add |bool to the expression 
in the future. Also see CONDITIONAL_BARE_VARS configuration toggle.. This 
feature will be removed in version 2.12. Deprecation warnings can be disabled 
by setting deprecation_warnings=False in ansible.cfg.
fatal: [undercloud]: FAILED! => {"changed": false, "module_stderr": "/bin/sh: /usr/bin/python: No such file or directory\n", "module_stdout": "", "msg": "The module failed to execute correctly, you probably need
 to set the interpreter.\nSee stdout/stderr for the exact error", "rc": 127}
Comment 3 Jiri Stransky 2019-05-24 09:35:01 UTC 
I can think of two possible causes:

* We don't have Ansible 2.8.0 GA installed yet, but just rc1. Ansible 2.8 features python interpreter autodetection, i'm not sure if that's in rc1 too.

(undercloud) [stack@undercloud-0 ~]$ sudo podman exec mistral_executor ansible --version
ansible 2.8.0rc1
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/var/lib/mistral/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.8 (default, Jan 11 2019, 02:17:16) [GCC 8.2.1 20180905 (Red Hat 8.2.1-3)]

  If this is correct, the fix would be installing Ansible 2.8.

* Alternatively, if we do have the python interpreter detection in that version of Ansible already, it may be that it's broken by use of delegate_to on the task which is failing.
Comment 4 Lon Hohberger 2019-05-24 14:18:33 UTC 
We should be using Ansible 2.8 GA bits at this point. The compose/containers/etc. are all switched over, but I can doublecheck.
Comment 6 Jiri Stransky 2019-05-24 14:50:10 UTC 
Thanks, we'll retry with fresh containers.
Comment 7 Jiri Stransky 2019-05-27 15:50:11 UTC 
(undercloud) [stack@undercloud-0 ~]$ sudo podman exec -ti mistral_executor ansible --version
ansible 2.8.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/var/lib/mistral/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.8 (default, Apr  3 2019, 17:26:03) [GCC 8.2.1 20180905 (Red Hat 8.2.1-3)]


(In reply to Jiri Stransky from comment #3)
> * Alternatively, if we do have the python interpreter detection in that
> version of Ansible already, it may be that it's broken by use of delegate_to
> on the task which is failing.

^ This is the case. I just retested with Ansible 2.8, and it is still broken. `delegate_to` must be confusing the autodetection. We'll need to try working around it with the way Sergii mentioned, setting the ansible_python_interpreter explicitly to "/usr/libexec/platform-python".
Comment 13 Shelley Dunne 2019-09-19 18:29:50 UTC 
Re-setting Target Milestone z1 to --- to begin the 15z1 Maintenance Release.
Comment 17 Jiri Stransky 2019-11-04 12:25:02 UTC 
The fix has merged some time ago, fixing the BZ fields.
Comment 19 Jose Luis Franco 2019-12-13 10:07:57 UTC 
Working on THT package:
(undercloud) [stack@undercloud-0 ~]$ rpm -qa | grep tripleo-heat-templates                                                                                                   openstack-tripleo-heat-templates-10.6.2-0.20191202200455.41d9f8a.el8ost.noarch

2019-12-12 19:53:50 | [Thu Dec 12 19:53:50 EST 2019] Started system upgrade transfer data for controller-0 role
2019-12-12 19:53:51 | 2019-12-12 19:53:51.702 131397 INFO osc_lib.shell [-] command: overcloud external-upgrade run -> tripleoclient.v1.overcloud_external_upgrade.ExternalUpgradeRun (auth=True)^[[00m
2019-12-12 19:53:51 | 2019-12-12 19:53:51.703 131397 INFO osc_lib.clientmanager [-] Using auth plugin: password^[[00m
2019-12-12 19:53:51 | 2019-12-12 19:53:51.704 131397 DEBUG osc_lib.clientmanager [-] Using parameters {'auth_url': 'https://192.168.24.2:13000', 'project_name': 'admin', 'project_domain_name': 'Default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'} setup_auth /usr/lib/python3.6/site-packages/osc_lib/clientmanager.py:157^[[00m
2019-12-12 19:53:51 | 2019-12-12 19:53:51.707 131397 DEBUG osc_lib.clientmanager [-] Get auth_ref auth_ref /usr/lib/python3.6/site-packages/osc_lib/clientmanager.py:201^[[00m
2019-12-12 19:53:56 | 2019-12-12 19:53:56.211 131397 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout.^[[00m
2019-12-12 19:54:33 | Using /var/lib/mistral/5b5a8e46-795d-40c5-8db2-70b6df1429d9/ansible.cfg as config file
2019-12-12 19:54:33 |
2019-12-12 19:54:33 | PLAY [Gather facts from undercloud] ********************************************
2019-12-12 19:54:33 |
2019-12-12 19:54:33 | PLAY [Gather facts from overcloud] *********************************************
2019-12-12 19:54:33 |
2019-12-12 19:54:33 | PLAY [Load global variables] ***************************************************
2019-12-12 19:54:33 | Thursday 12 December 2019  19:54:05 -0500 (0:00:00.121)       0:00:00.121 *****
2019-12-12 19:54:33 |
2019-12-12 19:54:33 | TASK [include_vars] ************************************************************
2019-12-12 19:54:33 | ok: [undercloud] => {"ansible_facts": {"deploy_steps_max": 6, "ssh_known_hosts": {"compute-0": "[172.17.1.18]*,[compute-0.redhat.local]*,[compute-0]*,[172.17.3.31]*,[compute-0.storage.redhat.local]*,[compute-0.storage]*,[172.17.1.18]*,[compute-0.internalapi.redhat.local]*,[compute-0.internalapi]*,[172.17.2.31]*,[compute-0.tenant.redhat.local]*,[compute-0.tenant]*,[192.168.24.6]*,[compute-0.ctlplane.redhat.local]*,[compute-0.ctlplane]*", "compute-1": "[172.17.1.24]*,[compute-1.redhat.local]*,[compute-1]*,[172.17.3.13]*,[compute-1.storage.redhat.local]*,[compute-1.storage]*,[172.17.1.24]*,[compute-1.internalapi.redhat.local]*,[compute-1.internalapi]*,[172.17.2.17]*,[compute-1.tenant.redhat.local]*,[compute-1.tenant]*,[192.168.24.9]*,[compute-1.ctlplane.redhat.local]*,[compute-1.ctlplane]*", "controller-0": "[172.17.1.10]*,[controller-0.redhat.local]*,[controller-0]*,[172.17.3.19]*,[controller-0.storage.redhat.local]*,[controller-0.storage]*,[172.17.4.17]*,[controller-0.storagemgmt.redhat.local]*,[controller-0.storagemgmt]*,[172.17.1.10]*,[controller-0.internalapi.redhat.local]*,[controller-0.internalapi]*,[172.17.2.36]*,[controller-0.tenant.redhat.local]*,[controller-0.tenant]*,[10.0.0.103]*,[controller-0.external.redhat.local]*,[controller-0.external]*,[192.168.24.16]*,[controller-0.ctlplane.redhat.local]*,[controller-0.ctlplane]*", "controller-1": "[172.17.1.16]*,[controller-1.redhat.local]*,[controller-1]*,[172.17.3.20]*,[controller-1.storage.redhat.local]*,[controller-1.storage]*,[172.17.4.15]*,[controller-1.storagemgmt.redhat.local]*,[controller-1.storagemgmt]*,[172.17.1.16]*,[controller-1.internalapi.redhat.local]*,[controller-1.internalapi]*,[172.17.2.15]*,[controller-1.tenant.redhat.local]*,[controller-1.tenant]*,[10.0.0.106]*,[controller-1.external.redhat.local]*,[controller-1.external]*,[192.168.24.23]*,[controller-1.ctlplane.redhat.local]*,[controller-1.ctlplane]*", "controller-2": "[172.17.1.28]*,[controller-2.redhat.local]*,[controller-2]*,[172.17.3.12]*,[controller-2.storage.redhat.local]*,[controller-2.storage]*,[172.17.4.25]*,[controller-2.storagemgmt.redhat.local]*,[controller-2.storagemgmt]*,[172.17.1.28]*,[controller-2.internalapi.redhat.local]*,[controller-2.internalapi]*,[172.17.2.13]*,[controller-2.tenant.redhat.local]*,[controller-2.tenant]*,[10.0.0.113]*,[controller-2.external.redhat.local]*,[controller-2.external]*,[192.168.24.20]*,[controller-2.ctlplane.redhat.local]*,[controller-2.ctlplane]*"}}, "ansible_included_var_files": ["/var/lib/mistral/5b5a8e46-795d-40c5-8db2-70b6df1429d9/global_vars.yaml"], "changed": false}
2019-12-12 19:54:33 | ok: [controller-0] => {"ansible_facts": {"deploy_steps_max": 6, "ssh_known_hosts": {"compute-0": "[172.17.1.18]*,[compute-0.redhat.local]*,[compute-0]*,[172.17.3.31]*,[compute-0.storage.redhat.local]*,[compute-0.storage]*,[172.17.1.18]*,[compute-0.internalapi.redhat.local]*,[compute-0.internalapi]*,[172.17.2.31]*,[compute-0.tenant.redhat.local]*,[compute-0.tenant]*,[192.168.24.6]*,[compute-0.ctlplane.redhat.local]*,[compute-0.ctlplane]*", "compute-1": "[172.17.1.24]*,[compute-1.redhat.local]*,[compute-1]*,[172.17.3.13]*,[compute-1.storage.redhat.local]*,[compute-1.storage]*,[172.17.1.24]*,[compute-1.internalapi.redhat.local]*,[compute-1.internalapi]*,[172.17.2.17]*,[compute-1.tenant.redhat.local]*,[compute-1.tenant]*,[192.168.24.9]*,[compute-1.ctlplane.redhat.local]*,[compute-1.ctlplane]*", "controller-0": "[172.17.1.10]*,[controller-0.redhat.local]*,[controller-0]*,[172.17.3.19]*,[controller-0.storage.redhat.local]*,[controller-0.storage]*,[172.17.4.17]*,[controller-0.storagemgmt.redhat.local]*,[controller-0.storagemgmt]*,[172.17.1.10]*,[controller-0.internalapi.redhat.local]*,[controller-0.internalapi]*,[172.17.2.36]*,[controller-0.tenant.redhat.local]*,[controller-0.tenant]*,[10.0.0.103]*,[controller-0.external.redhat.local]*,[controller-0.external]*,[192.168.24.16]*,[controller-0.ctlplane.redhat.local]*,[controller-0.ctlplane]*", "controller-1": "[172.17.1.16]*,[controller-1.redhat.local]*,[controller-1]*,[172.17.3.20]*,[controller-1.storage.redhat.local]*,[controller-1.storage]*,[172.17.4.15]*,[controller-1.storagemgmt.redhat.local]*,[controller-1.storagemgmt]*,[172.17.1.16]*,[controller-1.internalapi.redhat.local]*,[controller-1.internalapi]*,[172.17.2.15]*,[controller-1.tenant.redhat.local]*,[controller-1.tenant]*,[10.0.0.106]*,[controller-1.external.redhat.local]*,[controller-1.external]*,[192.168.24.23]*,[controller-1.ctlplane.redhat.local]*,[controller-1.ctlplane]*", "controller-2": "[172.17.1.28]*,[controller-2.redhat.local]*,[controller-2]*,[172.17.3.12]*,[controller-2.storage.redhat.local]*,[controller-2.storage]*,[172.17.4.25]*,[controller-2.storagemgmt.redhat.local]*,[controller-2.storagemgmt]*,[172.17.1.28]*,[controller-2.internalapi.redhat.local]*,[controller-2.internalapi]*,[172.17.2.13]*,[controller-2.tenant.redhat.local]*,[controller-2.tenant]*,[10.0.0.113]*,[controller-2.external.redhat.local]*,[controller-2.external]*,[192.168.24.20]*,[controller-2.ctlplane.redhat.local]*,[controller-2.ctlplane]*"}}, "ansible_included_var_files": ["/var/lib/mistral/5b5a8e46-795d-40c5-8db2-70b6df1429d9/global_vars.yaml"], "changed": false}
2019-12-12 19:54:33 | ok: [controller-1] => {"ansible_facts": {"deploy_steps_max": 6, "ssh_known_hosts": {"compute-0": "[172.17.1.18]*,[compute-0.redhat.local]*,[compute-0]*,[172.17.3.31]*,[compute-0.storage.redhat.local]*,[compute-0.storage]*,[172.17.1.18]*,[compute-0.internalapi.redhat.local]*,[compute-0.internalapi]*,[172.17.2.31]*,[compute-0.tenant.redhat.local]*,[compute-0.tenant]*,[192.168.24.6]*,[compute-0.ctlplane.redhat.local]*,[compute-0.ctlplane]*", "compute-1": "[172.17.1.24]*,[compute-1.redhat.local]*,[compute-1]*,[172.17.3.13]*,[compute-1.storage.redhat.local]*,[compute-1.storage]*,[172.17.1.24]*,[compute-1.internalapi.redhat.local]*,[compute-1.internalapi]*,[172.17.2.17]*,[compute-1.tenant.redhat.local]*,[compute-1.tenant]*,[192.168.24.9]*,[compute-1.ctlplane.redhat.local]*,[compute-1.ctlplane]*", "controller-0": "[172.17.1.10]*,[controller-0.redhat.local]*,[controller-0]*,[172.17.3.19]*,[controller-0.storage.redhat.local]*,[controller-0.storage]*,[172.17.4.17]*,[controller-0.storagemgmt.redhat.local]*,[controller-0.storagemgmt]*,[172.17.1.10]*,[controller-0.internalapi.redhat.local]*,[controller-0.internalapi]*,[172.17.2.36]*,[controller-0.tenant.redhat.local]*,[controller-0.tenant]*,[10.0.0.103]*,[controller-0.external.redhat.local]*,[controller-0.external]*,[192.168.24.16]*,[controller-0.ctlplane.redhat.local]*,[controller-0.ctlplane]*", "controller-1": "[172.17.1.16]*,[controller-1.redhat.local]*,[controller-1]*,[172.17.3.20]*,[controller-1.storage.redhat.local]*,[controller-1.storage]*,[172.17.4.15]*,[controller-1.storagemgmt.redhat.local]*,[controller-1.storagemgmt]*,[172.17.1.16]*,[controller-1.internalapi.redhat.local]*,[controller-1.internalapi]*,[172.17.2.15]*,[controller-1.tenant.redhat.local]*,[controller-1.tenant]*,[10.0.0.106]*,[controller-1.external.redhat.local]*,[controller-1.external]*,[192.168.24.23]*,[controller-1.ctlplane.redhat.local]*,[controller-1.ctlplane]*", "controller-2": "[172.17.1.28]*,[controller-2.redhat.local]*,[controller-2]*,[172.17.3.12]*,[controller-2.storage.redhat.local]*,[controller-2.storage]*,[172.17.4.25]*,[controller-2.storagemgmt.redhat.local]*,[controller-2.storagemgmt]*,[172.17.1.28]*,[controller-2.internalapi.redhat.local]*,[controller-2.internalapi]*,[172.17.2.13]*,[controller-2.tenant.redhat.local]*,[controller-2.tenant]*,[10.0.0.113]*,[controller-2.external.redhat.local]*,[controller-2.external]*,[192.168.24.20]*,[controller-2.ctlplane.redhat.local]*,[controller-2.ctlplane]*"}}, "ansible_included_var_files": ["/var/lib/mistral/5b5a8e46-795d-40c5-8db2-70b6df1429d9/global_vars.yaml"], "changed": false}
2019-12-12 19:54:33 | ok: [controller-2] => {"ansible_facts": {"deploy_steps_max": 6, "ssh_known_hosts": {"compute-0": "[172.17.1.18]*,[compute-0.redhat.local]*,[compute-0]*,[172.17.3.31]*,[compute-0.storage.redhat.local]*,[compute-0.storage]*,[172.17.1.18]*,[compute-0.internalapi.redhat.local]*,[compute-0.internalapi]*,[172.17.2.31]*,[compute-0.tenant.redhat.local]*,[compute-0.tenant]*,[192.168.24.6]*,[compute-0.ctlplane.redhat.local]*,[compute-0.ctlplane]*", "compute-1": "[172.17.1.24]*,[compute-1.redhat.local]*,[compute-1]*,[172.17.3.13]*,[compute-1.storage.redhat.local]*,[compute-1.storage]*,[172.17.1.24]*,[compute-1.internalapi.redhat.local]*,[compute-1.internalapi]*,[172.17.2.17]*,[compute-1.tenant.redhat.local]*,[compute-1.tenant]*,[192.168.24.9]*,[compute-1.ctlplane.redhat.local]*,[compute-1.ctlplane]*", "controller-0": "[172.17.1.10]*,[controller-0.redhat.local]*,[controller-0]*,[172.17.3.19]*,[controller-0.storage.redhat.local]*,[controller-0.storage]*,[172.17.4.17]*,[controller-0.storagemgmt.redhat.local]*,[controller-0.storagemgmt]*,[172.17.1.10]*,[controller-0.internalapi.redhat.local]*,[controller-0.internalapi]*,[172.17.2.36]*,[controller-0.tenant.redhat.local]*,[controller-0.tenant]*,[10.0.0.103]*,[controller-0.external.redhat.local]*,[controller-0.external]*,[192.168.24.16]*,[controller-0.ctlplane.redhat.local]*,[controller-0.ctlplane]*", "controller-1": "[172.17.1.16]*,[controller-1.redhat.local]*,[controller-1]*,[172.17.3.20]*,[controller-1.storage.redhat.local]*,[controller-1.storage]*,[172.17.4.15]*,[controller-1.storagemgmt.redhat.local]*,[controller-1.storagemgmt]*,[172.17.1.16]*,[controller-1.internalapi.redhat.local]*,[controller-1.internalapi]*,[172.17.2.15]*,[controller-1.tenant.redhat.local]*,[controller-1.tenant]*,[10.0.0.106]*,[controller-1.external.redhat.local]*,[controller-1.external]*,[192.168.24.23]*,[controller-1.ctlplane.redhat.local]*,[controller-1.ctlplane]*", "controller-2": "[172.17.1.28]*,[controller-2.redhat.local]*,[controller-2]*,[172.17.3.12]*,[controller-2.storage.redhat.local]*,[controller-2.storage]*,[172.17.4.25]*,[controller-2.storagemgmt.redhat.local]*,[controller-2.storagemgmt]*,[172.17.1.28]*,[controller-2.internalapi.redhat.local]*,[controller-2.internalapi]*,[172.17.2.13]*,[controller-2.tenant.redhat.local]*,[controller-2.tenant]*,[10.0.0.113]*,[controller-2.external.redhat.local]*,[controller-2.external]*,[192.168.24.20]*,[controller-2.ctlplane.redhat.local]*,[controller-2.ctlplane]*"}}, "ansible_included_var_files": ["/var/lib/mistral/5b5a8e46-795d-40c5-8db2-70b6df1429d9/global_vars.yaml"], "changed": false}
2019-12-12 19:54:33 | ok: [compute-0] => {"ansible_facts": {"deploy_steps_max": 6, "ssh_known_hosts": {"compute-0": "[172.17.1.18]*,[compute-0.redhat.local]*,[compute-0]*,[172.17.3.31]*,[compute-0.storage.redhat.local]*,[compute-0.storage]*,[172.17.1.18]*,[compute-0.internalapi.redhat.local]*,[compute-0.internalapi]*,[172.17.2.31]*,[compute-0.tenant.redhat.local]*,[compute-0.tenant]*,[192.168.24.6]*,[compute-0.ctlplane.redhat.local]*,[compute-0.ctlplane]*", "compute-1": "[172.17.1.24]*,[compute-1.redhat.local]*,[compute-1]*,[172.17.3.13]*,[compute-1.storage.redhat.local]*,[compute-1.storage]*,[172.17.1.24]*,[compute-1.internalapi.redhat.local]*,[compute-1.internalapi]*,[172.17.2.17]*,[compute-1.tenant.redhat.local]*,[compute-1.tenant]*,[192.168.24.9]*,[compute-1.ctlplane.redhat.local]*,[compute-1.ctlplane]*", "controller-0": "[172.17.1.10]*,[controller-0.redhat.local]*,[controller-0]*,[172.17.3.19]*,[controller-0.storage.redhat.local]*,[controller-0.storage]*,[172.17.4.17]*,[controller-0.storagemgmt.redhat.local]*,[controller-0.storagemgmt]*,[172.17.1.10]*,[controller-0.internalapi.redhat.local]*,[controller-0.internalapi]*,[172.17.2.36]*,[controller-0.tenant.redhat.local]*,[controller-0.tenant]*,[10.0.0.103]*,[controller-0.external.redhat.local]*,[controller-0.external]*,[192.168.24.16]*,[controller-0.ctlplane.redhat.local]*,[controller-0.ctlplane]*", "controller-1": "[172.17.1.16]*,[controller-1.redhat.local]*,[controller-1]*,[172.17.3.20]*,[controller-1.storage.redhat.local]*,[controller-1.storage]*,[172.17.4.15]*,[controller-1.storagemgmt.redhat.local]*,[controller-1.storagemgmt]*,[172.17.1.16]*,[controller-1.internalapi.redhat.local]*,[controller-1.internalapi]*,[172.17.2.15]*,[controller-1.tenant.redhat.local]*,[controller-1.tenant]*,[10.0.0.106]*,[controller-1.external.redhat.local]*,[controller-1.external]*,[192.168.24.23]*,[controller-1.ctlplane.redhat.local]*,[controller-1.ctlplane]*", "controller-2": "[172.17.1.28]*,[controller-2.redhat.local]*,[controller-2]*,[172.17.3.12]*,[controller-2.storage.redhat.local]*,[controller-2.storage]*,[172.17.4.25]*,[controller-2.storagemgmt.redhat.local]*,[controller-2.storagemgmt]*,[172.17.1.28]*,[controller-2.internalapi.redhat.local]*,[controller-2.internalapi]*,[172.17.2.13]*,[controller-2.tenant.redhat.local]*,[controller-2.tenant]*,[10.0.0.113]*,[controller-2.external.redhat.local]*,[controller-2.external]*,[192.168.24.20]*,[controller-2.ctlplane.redhat.local]*,[controller-2.ctlplane]*"}}, "ansible_included_var_files": ["/var/lib/mistral/5b5a8e46-795d-40c5-8db2-70b6df1429d9/global_vars.yaml"], "changed": false}
2019-12-12 19:54:33 | ok: [compute-1] => {"ansible_facts": {"deploy_steps_max": 6, "ssh_known_hosts": {"compute-0": "[172.17.1.18]*,[compute-0.redhat.local]*,[compute-0]*,[172.17.3.31]*,[compute-0.storage.redhat.local]*,[compute-0.storage]*,[172.17.1.18]*,[compute-0.internalapi.redhat.local]*,[compute-0.internalapi]*,[172.17.2.31]*,[compute-0.tenant.redhat.local]*,[compute-0.tenant]*,[192.168.24.6]*,[compute-0.ctlplane.redhat.local]*,[compute-0.ctlplane]*", "compute-1": "[172.17.1.24]*,[compute-1.redhat.local]*,[compute-1]*,[172.17.3.13]*,[compute-1.storage.redhat.local]*,[compute-1.storage]*,[172.17.1.24]*,[compute-1.internalapi.redhat.local]*,[compute-1.internalapi]*,[172.17.2.17]*,[compute-1.tenant.redhat.local]*,[compute-1.tenant]*,[192.168.24.9]*,[compute-1.ctlplane.redhat.local]*,[compute-1.ctlplane]*", "controller-0": "[172.17.1.10]*,[controller-0.redhat.local]*,[controller-0]*,[172.17.3.19]*,[controller-0.storage.redhat.local]*,[controller-0.storage]*,[172.17.4.17]*,[controller-0.storagemgmt.redhat.local]*,[controller-0.storagemgmt]*,[172.17.1.10]*,[controller-0.internalapi.redhat.local]*,[controller-0.internalapi]*,[172.17.2.36]*,[controller-0.tenant.redhat.local]*,[controller-0.tenant]*,[10.0.0.103]*,[controller-0.external.redhat.local]*,[controller-0.external]*,[192.168.24.16]*,[controller-0.ctlplane.redhat.local]*,[controller-0.ctlplane]*", "controller-1": "[172.17.1.16]*,[controller-1.redhat.local]*,[controller-1]*,[172.17.3.20]*,[controller-1.storage.redhat.local]*,[controller-1.storage]*,[172.17.4.15]*,[controller-1.storagemgmt.redhat.local]*,[controller-1.storagemgmt]*,[172.17.1.16]*,[controller-1.internalapi.redhat.local]*,[controller-1.internalapi]*,[172.17.2.15]*,[controller-1.tenant.redhat.local]*,[controller-1.tenant]*,[10.0.0.106]*,[controller-1.external.redhat.local]*,[controller-1.external]*,[192.168.24.23]*,[controller-1.ctlplane.redhat.local]*,[controller-1.ctlplane]*", "controller-2": "[172.17.1.28]*,[controller-2.redhat.local]*,[controller-2]*,[172.17.3.12]*,[controller-2.storage.redhat.local]*,[controller-2.storage]*,[172.17.4.25]*,[controller-2.storagemgmt.redhat.local]*,[controller-2.storagemgmt]*,[172.17.1.28]*,[controller-2.internalapi.redhat.local]*,[controller-2.internalapi]*,[172.17.2.13]*,[controller-2.tenant.redhat.local]*,[controller-2.tenant]*,[10.0.0.113]*,[controller-2.external.redhat.local]*,[controller-2.external]*,[192.168.24.20]*,[controller-2.ctlplane.redhat.local]*,[controller-2.ctlplane]*"}}, "ansible_included_var_files": ["/var/lib/mistral/5b5a8e46-795d-40c5-8db2-70b6df1429d9/global_vars.yaml"], "changed": false}
2019-12-12 19:54:33 |
2019-12-12 19:54:33 | PLAY [External upgrade step 0] *************************************************
2019-12-12 19:54:33 | Thursday 12 December 2019  19:54:06 -0500 (0:00:00.526)       0:00:00.647 *****
2019-12-12 19:54:33 | Thursday 12 December 2019  19:54:06 -0500 (0:00:00.091)       0:00:00.739 *****
2019-12-12 19:54:33 | Thursday 12 December 2019  19:54:06 -0500 (0:00:00.091)       0:00:00.831 *****
2019-12-12 19:54:33 | Thursday 12 December 2019  19:54:06 -0500 (0:00:00.091)       0:00:00.922 *****
2019-12-12 19:54:33 | Thursday 12 December 2019  19:54:06 -0500 (0:00:00.104)       0:00:01.027 *****
2019-12-12 19:54:33 | PLAY [External upgrade step 1] *************************************************
2019-12-12 19:54:33 | Thursday 12 December 2019  19:54:09 -0500 (0:00:00.131)       0:00:03.936 *****
2019-12-12 19:54:33 | included: /usr/share/ansible/roles/tripleo-container-stop/tasks/container_stop.yaml for undercloud
2019-12-12 19:54:33 | Thursday 12 December 2019  19:54:10 -0500 (0:00:00.373)       0:00:04.310 *****
2019-12-12 19:54:33 |
2019-12-12 19:54:33 | TASK [tripleo-container-stop : Make sure the container is stopped even if container_cli do not match] ***
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.16] => (item=controller-0) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_api /bin/true; then\n    systemctl stop tripleo_aodh_api.service\nfi\nif type docker &> /dev/null && docker exec aodh_api /bin/true; then\n    docker stop aodh_api\nfi\n", "delta": "0:00:00.003134", "end": "2019-12-13 00:54:10.717540", "rc": 0, "start": "2019-12-13 00:54:10.714406", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": [], "tripleo_delegate_to_item": "controller-0"}
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.23] => (item=controller-1) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_api /bin/true; then\n    systemctl stop tripleo_aodh_api.service\nfi\nif type docker &> /dev/null && docker exec aodh_api /bin/true; then\n    docker stop aodh_api\nfi\n", "delta": "0:00:00.429473", "end": "2019-12-13 00:54:11.693939", "rc": 0, "start": "2019-12-13 00:54:11.264466", "stderr": "", "stderr_lines": [], "stdout": "aodh_api", "stdout_lines": ["aodh_api"], "tripleo_delegate_to_item": "controller-1"}
2019-12-12 19:54:33 | changed: [undercloud -> 192.168.24.20] => (item=controller-2) => {"ansible_loop_var": "tripleo_delegate_to_item", "changed": true, "cmd": "# We need to make sure that containers are stopped\n# as we might have different CLIs to interact with\n# them. I.e the container_cli might be setted to be podman\n# but we might have the containers running with docker.\nset -eu\nif command -v podman && podman exec aodh_api /bin/true; then\n    systemctl stop tripleo_aodh_api.service\nfi\nif type docker &> /dev/null && docker exec aodh_api /bin/true; then\n    docker stop aodh_api\nfi\n", "delta": "0:00:00.444288", "end": "2019-12-13 00:54:12.688377", "rc": 0, "start": "2019-12-13 00:54:12.244089", "stderr": "", "stderr_lines": [], "stdout": "aodh_api", "stdout_lines": ["aodh_api"], "tripleo_delegate_to_item": "controller-2"}
Comment 20 Alex McLeod 2020-02-19 12:44:18 UTC 
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.
Comment 23 errata-xmlrpc 2020-03-05 11:59:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0643