Bug 1655764

R
2018-12-04 01:30:01 | 2018-12-04 01:30:01.159 118787 ERROR openstack [-] Update failed with: {u'status': u'FAILED', u'message': u'Failed to update nodes - Controller, please see the logs.', u'execution': {u'name': u'tripleo.package_update.v1.update_nodes', u'created_at': u'2018-12-04 05:34:58', u'updated_at': u'2018-12-04 05:34:58', u'id': u'45635779-b654-4847-97b3-601b224719c0', u'params': {u'namespace': u'', u'env': {}}, u'input': {u'inventory_file': u'Undercloud:\n  hosts:\n    undercloud: {}\n  vars:\n    ansible_connection: local\n    ansible_host: localhost\n    ansible_remote_tmp: /tmp/ansible-${USER}\n    auth_url: http://192.168.24.1:5000/\n    cacert: null\n    os_auth_token: gAAAAABcBgwlI2U2Ai8gUBUFS2xT_3pRKpZXhXrtJGSpGsuZHnxkscU32eJIGQKY0sJqxXV7_SfbP_dIZsupw4yonw_eotT-E_94Q27U2wSKQyTIIFlevlnm965vF2eN-YaJ6E6A8OhKCFcADkeGcGtg14VdQXUbXAGBmkF1CjQo0wvVlyHEPLU\n    overcloud_admin_password: HrYyvRVT3cbRpx7HwyeyD93x6\n    overcloud_horizon_url: http://10.0.0.118:80/dashboard\n    overcloud_keystone_url: http://10.0.0.118:5000\n    plan: overcloud\n    project_name: admin\n    undercloud_service_list: [openstack-nova-compute, openstack-heat-engine, openstack-ironic-conductor,\n      openstack-swift-container, openstack-swift-object, openstack-mistral-engine]\n    undercloud_swift_url: http://192.168.24.1:8080/v1/AUTH_fb94786894ec4fb984ed4c2486f69ca1\n    username: admin\nCephStorage:\n  hosts:\n    overcloud-cephstorage-0:\n      ansible_host: 192.168.24.7\n      ctlplane_ip: 192.168.24.7\n      deploy_server_id: 7bc385c6-fc07-4193-9f5a-85090f2bbb21\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.7\n      internal_api_ip: 192.168.24.7\n      management_ip: 192.168.24.7\n      storage_ip: 172.17.3.17\n      storage_mgmt_ip: 172.17.4.18\n      tenant_ip: 192.168.24.7\n    overcloud-cephstorage-1:\n      ansible_host: 192.168.24.13\n      ctlplane_ip: 192.168.24.13\n      deploy_server_id: 74e4d64f-18a0-4f7a-8b13-d549a5fb5f1a\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.13\n      internal_api_ip: 192.168.24.13\n      management_ip: 192.168.24.13\n      storage_ip: 172.17.3.26\n      storage_mgmt_ip: 172.17.4.15\n      tenant_ip: 192.168.24.13\n    overcloud-cephstorage-2:\n      ansible_host: 192.168.24.8\n      ctlplane_ip: 192.168.24.8\n      deploy_server_id: f085b801-83f1-49d3-b73f-fa16a8f3dd3e\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.8\n      internal_api_ip: 192.168.24.8\n      management_ip: 192.168.24.8\n      storage_ip: 172.17.3.14\n      storage_mgmt_ip: 172.17.4.14\n      tenant_ip: 192.168.24.8\n  vars: {ansible_ssh_user: tripleo-admin, bootstrap_server_id: 5406ba48-e416-4f03-bb6b-4e55dbbcf4f6,\n    tripleo_role_name: CephStorage}\nController:\n  hosts:\n    overcloud-controller-0:\n      ansible_host: 192.168.24.15\n      ctlplane_ip: 192.168.24.15\n      deploy_server_id: 5406ba48-e416-4f03-bb6b-4e55dbbcf4f6\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 10.0.0.107\n      internal_api_ip: 172.17.1.12\n      management_ip: 192.168.24.15\n      storage_ip: 172.17.3.12\n      storage_mgmt_ip: 172.17.4.17\n      tenant_ip: 172.17.2.30\n    overcloud-controller-1:\n      ansible_host: 192.168.24.18\n      ctlplane_ip: 192.168.24.18\n      deploy_server_id: cfba54b3-6f9d-4474-a904-e2c50accd9c0\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 10.0.0.124\n      internal_api_ip: 172.17.1.17\n      management_ip: 192.168.24.18\n      storage_ip: 172.17.3.27\n      storage_mgmt_ip: 172.17.4.12\n      tenant_ip: 172.17.2.11\n    overcloud-controller-2:\n      ansible_host: 192.168.24.24\n      ctlplane_ip: 192.168.24.24\n      deploy_server_id: 51f5f03b-8641-476a-bc8a-2663d52c3493\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 10.0.0.110\n      internal_api_ip: 172.17.1.18\n      management_ip: 192.168.24.24\n      storage_ip: 172.17.3.15\n      storage_mgmt_ip: 172.17.4.11\n      tenant_ip: 172.17.2.37\n  vars: {ansible_ssh_user: tripleo-admin, bootstrap_server_id: 5406ba48-e416-4f03-bb6b-4e55dbbcf4f6,\n    tripleo_role_name: Controller}\nCompute:\n  hosts:\n    overcloud-novacompute-0:\n      ansible_host: 192.168.24.9\n      ctlplane_ip: 192.168.24.9\n      deploy_server_id: 80cae74c-a00a-43bd-a6a7-c2980791aa80\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.9\n      internal_api_ip: 172.17.1.29\n      management_ip: 192.168.24.9\n      storage_ip: 172.17.3.18\n      storage_mgmt_ip: 192.168.24.9\n      tenant_ip: 172.17.2.10\n    overcloud-novacompute-1:\n      ansible_host: 192.168.24.10\n      ctlplane_ip: 192.168.24.10\n      deploy_server_id: f9132711-bed0-4d4a-96c5-d3ddca609d66\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.10\n      internal_api_ip: 172.17.1.16\n      management_ip: 192.168.24.10\n      storage_ip: 172.17.3.11\n      storage_mgmt_ip: 192.168.24.10\n      tenant_ip: 172.17.2.14\n  vars: {ansible_ssh_user: tripleo-admin, bootstrap_server_id: 5406ba48-e416-4f03-bb6b-4e55dbbcf4f6,\n    tripleo_role_name: Compute}\nIronicConductor:\n  hosts:\n    overcloud-ironic-0:\n      ansible_host: 192.168.24.11\n      ctlplane_ip: 192.168.24.11\n      deploy_server_id: 70cb2471-5353-48e4-b1f1-e8475b5d4427\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.11\n      internal_api_ip: 172.17.1.11\n      management_ip: 192.168.24.11\n      storage_ip: 192.168.24.11\n      storage_mgmt_ip: 192.168.24.11\n      tenant_ip: 192.168.24.11\n    overcloud-ironic-1:\n      ansible_host: 192.168.24.16\n      ctlplane_ip: 192.168.24.16\n      deploy_server_id: 8650d10d-b7e3-4589-a779-582ffaf19422\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.16\n      internal_api_ip: 172.17.1.19\n      management_ip: 192.168.24.16\n      storage_ip: 192.168.24.16\n      storage_mgmt_ip: 192.168.24.16\n      tenant_ip: 192.168.24.16\n  vars: {ansible_ssh_user: tripleo-admin, bootstrap_server_id: 5406ba48-e416-4f03-bb6b-4e55dbbcf4f6,\n    tripleo_role_name: IronicConductor}\novercloud:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ctlplane_vip: 192.168.24.6, external_vip: 10.0.0.118, internal_api_vip: 172.17.1.14,\n    redis_vip: 172.17.1.10, storage_mgmt_vip: 172.17.4.23, storage_vip: 172.17.3.10}\naodh_evaluator:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nkernel:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nneutron_metadata:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_conductor:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_placement:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nneutron_plugin_ml2:\n  children:\n    Compute: {}\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nsnmp:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nheat_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ncinder_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nironic_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceph_mon:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\naodh_listener:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nswift_ringbuilder:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceph_osd:\n  children:\n    CephStorage: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nneutron_dhcp:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ngnocchi_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ncontainer_image_prepare:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ntimezone:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ncertmonger_user:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceilometer_agent_central:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nironic_pxe:\n  children:\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nheat_api_cloudwatch_disabled:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\naodh_notifier:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ntripleo_firewall:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nclustercheck:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nredis:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ngnocchi_statsd:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\niscsid:\n  children:\n    Compute: {}\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nmysql_client:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_consoleauth:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nglance_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nkeystone:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ncinder_volume:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceilometer_agent_notification:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nmemcached:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\noslo_messaging_rpc:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nmongodb_disabled:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\noslo_messaging_notify:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nironic_conductor:\n  children:\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\naodh_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_metadata:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nheat_engine:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nntp:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nneutron_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_ironic:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceph_client:\n  children:\n    Compute: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_migration_target:\n  children:\n    Compute: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ncinder_scheduler:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ngnocchi_metricd:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ntripleo_packages:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_scheduler:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_compute:\n  children:\n    Compute: {}\n  vars: {ansible_ssh_user: tripleo-admin}\npacemaker:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nswift_proxy:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nlogrotate_crond:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nneutron_ovs_agent:\n  children:\n    Compute: {}\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nhaproxy:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nsshd:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceph_mgr:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nmysql:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nneutron_l3:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_libvirt:\n  children:\n    Compute: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ntuned:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\npanko_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nhorizon:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceilometer_agent_compute:\n  children:\n    Compute: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nca_certs:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nheat_api_cfn:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ndocker:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_vnc_proxy:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nswift_storage:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nglance_registry_disabled:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\n', u'work_dir': u'/var/lib/mistral', u'verbosity': 1, u'tags': u'', u'skip_tags': u'', u'playbook': u'deploy_steps_playbook.yaml', u'ansible_extra_env_variables': {u'ANSIBLE_HOST_KEY_CHECKING': u'False', u'ANSIBLE_LOG_PATH': u'/var/log/mistral/package_update.log'}, u'module_path': u'/usr/share/ansible-modules', u'nodes': u'Controller', u'node_user': u'tripleo-admin', u'ansible_queue_name': u'upgrade'}, u'spec': {u'tasks': {u'get_private_key': {u'name': u'get_private_key', u'on-success': u'node_update', u'publish': {u'private_key': u'<% task().result %>'}, u'version': u'2.0', u'action': u'tripleo.validations.get_privkey', u'type': u'direct'}, u'node_update_failed': {u'version': u'2.0', u'type': u'direct', u'name': u'node_update_failed', u'publish': {u'status': u'FAILED', u'message': u'Failed to update nodes - <% $.nodes %>, please see the logs.'}, u'on-success': u'send_message'}, u'node_update_passed': {u'version': u'2.0', u'type': u'direct', u'name': u'node_update_passed', u'publish': {u'status': u'SUCCESS', u'message': u'Updated nodes - <% $.nodes %>'}, u'on-success': u'send_message'}, u'send_message': {u'input': {u'status': u"<% $.get('status', 'SUCCESS') %>", u'message': u"<% $.get('message', '') %>", u'queue_name': u'<% $.ansible_queue_name %>', u'type': u'<% execution().name %>', u'execution': u'<% execution() %>'}, u'version': u'2.0', u'type': u'direct', u'name': u'send_message', u'workflow': u'tripleo.messaging.v1.send'}, u'node_update': {u'name': u'node_update', u'on-error': u'node_update_failed', u'on-success': [{u'node_update_passed': u'<% task().result.returncode = 0 %>'}, {u'node_update_failed': u'<% task().result.returncode != 0 %>'}], u'publish': {u'output': u'<% task().result %>'}, u'version': u'2.0', u'action': u'tripleo.ansible-playbook', u'input': {u'remote_user': u'<% $.node_user %>', u'limit_hosts': u'<% $.nodes %>', u'tags': u'<% $.tags %>', u'become_user': u'root', u'verbosity': u'<% $.verbosity %>', u'queue_name': u'<% $.ansible_queue_name %>', u'extra_env_variables': u'<% $.ansible_extra_env_variables %>', u'skip_tags': u'<% $.skip_tags %>', u'inventory': u'<% $.inventory_file %>', u'become': True, u'module_path': u'<% $.module_path %>', u'playbook': u'<% $.work_dir %>/<% execution().id %>/<% $.playbook %>', u'trash_output': True, u'execution_id': u'<% execution().id %>', u'ssh_private_key': u'<% $.private_key %>'}, u'type': u'direct'}, u'download_config': {u'name': u'download_config', u'on-error': u'node_update_failed', u'on-success': u'get_private_key', u'version': u'2.0', u'action': u'tripleo.config.download_config', u'input': {u'work_dir': u'<% $.work_dir %>/<% execution().id %>'}, u'type': u'direct'}}, u'description': u'Take a container and perform an update nodes by nodes', u'tags': [u'tripleo-common-managed'], u'version': u'2.0', u'input': [{u'node_user': u'tripleo-admin'}, u'nodes', u'playbook', u'inventory_file', {u'ansible_queue_name': u'tripleo'}, {u'module_path': u'/usr/share/ansible-modules'}, {u'ansible_extra_env_variables': {u'ANSIBLE_HOST_KEY_CHECKING': u'False', u'ANSIBLE_LOG_PATH': u'/var/log/mistral/package_update.log'}}, {u'verbosity': 1}, {u'work_dir': u'/var/lib/mistral'}, {u'tags': u''}, {u'skip_tags': u''}], u'name': u'update_nodes'}}, u'plan_name': None, u'execution_id': u'45635779-b654-4847-97b3-601b224719c0', u'deployment_status': None}: RuntimeError: Update failed with: {u'status': u'FAILED', u'message': u'Failed to update nodes - Controller, please see the logs.', u'execution': {u'name': u'tripleo.package_update.v1.update_nodes', u'created_at': u'2018-12-04 05:34:58', u'updated_at': u'2018-12-04 05:34:58', u'id': u'45635779-b654-4847-97b3-601b224719c0', u'params': {u'namespace': u'', u'env': {}}, u'input': {u'inventory_file': u'Undercloud:\n  hosts:\n    undercloud: {}\n  vars:\n    ansible_connection: local\n    ansible_host: localhost\n    ansible_remote_tmp: /tmp/ansible-${USER}\n    auth_url: http://192.168.24.1:5000/\n    cacert: null\n    os_auth_token: gAAAAABcBgwlI2U2Ai8gUBUFS2xT_3pRKpZXhXrtJGSpGsuZHnxkscU32eJIGQKY0sJqxXV7_SfbP_dIZsupw4yonw_eotT-E_94Q27U2wSKQyTIIFlevlnm965vF2eN-YaJ6E6A8OhKCFcADkeGcGtg14VdQXUbXAGBmkF1CjQo0wvVlyHEPLU\n    overcloud_admin_password: HrYyvRVT3cbRpx7HwyeyD93x6\n    overcloud_horizon_url: http://10.0.0.118:80/dashboard\n    overcloud_keystone_url: http://10.0.0.118:5000\n    plan: overcloud\n    project_name: admin\n    undercloud_service_list: [openstack-nova-compute, openstack-heat-engine, openstack-ironic-conductor,\n      openstack-swift-container, openstack-swift-object, openstack-mistral-engine]\n    undercloud_swift_url: http://192.168.24.1:8080/v1/AUTH_fb94786894ec4fb984ed4c2486f69ca1\n    username: admin\nCephStorage:\n  hosts:\n    overcloud-cephstorage-0:\n      ansible_host: 192.168.24.7\n      ctlplane_ip: 192.168.24.7\n      deploy_server_id: 7bc385c6-fc07-4193-9f5a-85090f2bbb21\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.7\n      internal_api_ip: 192.168.24.7\n      management_ip: 192.168.24.7\n      storage_ip: 172.17.3.17\n      storage_mgmt_ip: 172.17.4.18\n      tenant_ip: 192.168.24.7\n    overcloud-cephstorage-1:\n      ansible_host: 192.168.24.13\n      ctlplane_ip: 192.168.24.13\n      deploy_server_id: 74e4d64f-18a0-4f7a-8b13-d549a5fb5f1a\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.13\n      internal_api_ip: 192.168.24.13\n      management_ip: 192.168.24.13\n      storage_ip: 172.17.3.26\n      storage_mgmt_ip: 172.17.4.15\n      tenant_ip: 192.168.24.13\n    overcloud-cephstorage-2:\n      ansible_host: 192.168.24.8\n      ctlplane_ip: 192.168.24.8\n      deploy_server_id: f085b801-83f1-49d3-b73f-fa16a8f3dd3e\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.8\n      internal_api_ip: 192.168.24.8\n      management_ip: 192.168.24.8\n      storage_ip: 172.17.3.14\n      storage_mgmt_ip: 172.17.4.14\n      tenant_ip: 192.168.24.8\n  vars: {ansible_ssh_user: tripleo-admin, bootstrap_server_id: 5406ba48-e416-4f03-bb6b-4e55dbbcf4f6,\n    tripleo_role_name: CephStorage}\nController:\n  hosts:\n    overcloud-controller-0:\n      ansible_host: 192.168.24.15\n      ctlplane_ip: 192.168.24.15\n      deploy_server_id: 5406ba48-e416-4f03-bb6b-4e55dbbcf4f6\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 10.0.0.107\n      internal_api_ip: 172.17.1.12\n      management_ip: 192.168.24.15\n      storage_ip: 172.17.3.12\n      storage_mgmt_ip: 172.17.4.17\n      tenant_ip: 172.17.2.30\n    overcloud-controller-1:\n      ansible_host: 192.168.24.18\n      ctlplane_ip: 192.168.24.18\n      deploy_server_id: cfba54b3-6f9d-4474-a904-e2c50accd9c0\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 10.0.0.124\n      internal_api_ip: 172.17.1.17\n      management_ip: 192.168.24.18\n      storage_ip: 172.17.3.27\n      storage_mgmt_ip: 172.17.4.12\n      tenant_ip: 172.17.2.11\n    overcloud-controller-2:\n      ansible_host: 192.168.24.24\n      ctlplane_ip: 192.168.24.24\n      deploy_server_id: 51f5f03b-8641-476a-bc8a-2663d52c3493\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 10.0.0.110\n      internal_api_ip: 172.17.1.18\n      management_ip: 192.168.24.24\n      storage_ip: 172.17.3.15\n      storage_mgmt_ip: 172.17.4.11\n      tenant_ip: 172.17.2.37\n  vars: {ansible_ssh_user: tripleo-admin, bootstrap_server_id: 5406ba48-e416-4f03-bb6b-4e55dbbcf4f6,\n    tripleo_role_name: Controller}\nCompute:\n  hosts:\n    overcloud-novacompute-0:\n      ansible_host: 192.168.24.9\n      ctlplane_ip: 192.168.24.9\n      deploy_server_id: 80cae74c-a00a-43bd-a6a7-c2980791aa80\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.9\n      internal_api_ip: 172.17.1.29\n      management_ip: 192.168.24.9\n      storage_ip: 172.17.3.18\n      storage_mgmt_ip: 192.168.24.9\n      tenant_ip: 172.17.2.10\n    overcloud-novacompute-1:\n      ansible_host: 192.168.24.10\n      ctlplane_ip: 192.168.24.10\n      deploy_server_id: f9132711-bed0-4d4a-96c5-d3ddca609d66\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.10\n      internal_api_ip: 172.17.1.16\n      management_ip: 192.168.24.10\n      storage_ip: 172.17.3.11\n      storage_mgmt_ip: 192.168.24.10\n      tenant_ip: 172.17.2.14\n  vars: {ansible_ssh_user: tripleo-admin, bootstrap_server_id: 5406ba48-e416-4f03-bb6b-4e55dbbcf4f6,\n    tripleo_role_name: Compute}\nIronicConductor:\n  hosts:\n    overcloud-ironic-0:\n      ansible_host: 192.168.24.11\n      ctlplane_ip: 192.168.24.11\n      deploy_server_id: 70cb2471-5353-48e4-b1f1-e8475b5d4427\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.11\n      internal_api_ip: 172.17.1.11\n      management_ip: 192.168.24.11\n      storage_ip: 192.168.24.11\n      storage_mgmt_ip: 192.168.24.11\n      tenant_ip: 192.168.24.11\n    overcloud-ironic-1:\n      ansible_host: 192.168.24.16\n      ctlplane_ip: 192.168.24.16\n      deploy_server_id: 8650d10d-b7e3-4589-a779-582ffaf19422\n      enabled_networks: [management, storage, ctlplane, external, internal_api, storage_mgmt,\n        tenant]\n      external_ip: 192.168.24.16\n      internal_api_ip: 172.17.1.19\n      management_ip: 192.168.24.16\n      storage_ip: 192.168.24.16\n      storage_mgmt_ip: 192.168.24.16\n      tenant_ip: 192.168.24.16\n  vars: {ansible_ssh_user: tripleo-admin, bootstrap_server_id: 5406ba48-e416-4f03-bb6b-4e55dbbcf4f6,\n    tripleo_role_name: IronicConductor}\novercloud:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ctlplane_vip: 192.168.24.6, external_vip: 10.0.0.118, internal_api_vip: 172.17.1.14,\n    redis_vip: 172.17.1.10, storage_mgmt_vip: 172.17.4.23, storage_vip: 172.17.3.10}\naodh_evaluator:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nkernel:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nneutron_metadata:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_conductor:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_placement:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nneutron_plugin_ml2:\n  children:\n    Compute: {}\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nsnmp:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nheat_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ncinder_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nironic_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceph_mon:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\naodh_listener:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nswift_ringbuilder:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceph_osd:\n  children:\n    CephStorage: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nneutron_dhcp:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ngnocchi_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ncontainer_image_prepare:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ntimezone:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ncertmonger_user:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceilometer_agent_central:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nironic_pxe:\n  children:\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nheat_api_cloudwatch_disabled:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\naodh_notifier:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ntripleo_firewall:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nclustercheck:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nredis:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ngnocchi_statsd:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\niscsid:\n  children:\n    Compute: {}\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nmysql_client:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_consoleauth:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nglance_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nkeystone:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ncinder_volume:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceilometer_agent_notification:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nmemcached:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\noslo_messaging_rpc:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nmongodb_disabled:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\noslo_messaging_notify:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nironic_conductor:\n  children:\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\naodh_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_metadata:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nheat_engine:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nntp:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nneutron_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_ironic:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceph_client:\n  children:\n    Compute: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_migration_target:\n  children:\n    Compute: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ncinder_scheduler:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ngnocchi_metricd:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ntripleo_packages:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_scheduler:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_compute:\n  children:\n    Compute: {}\n  vars: {ansible_ssh_user: tripleo-admin}\npacemaker:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nswift_proxy:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nlogrotate_crond:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nneutron_ovs_agent:\n  children:\n    Compute: {}\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nhaproxy:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nsshd:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceph_mgr:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nmysql:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nneutron_l3:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_libvirt:\n  children:\n    Compute: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ntuned:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\npanko_api:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nhorizon:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nceilometer_agent_compute:\n  children:\n    Compute: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nca_certs:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nheat_api_cfn:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\ndocker:\n  children:\n    CephStorage: {}\n    Compute: {}\n    Controller: {}\n    IronicConductor: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nnova_vnc_proxy:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nswift_storage:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\nglance_registry_disabled:\n  children:\n    Controller: {}\n  vars: {ansible_ssh_user: tripleo-admin}\n', u'work_dir': u'/var/lib/mistral', u'verbosity': 1, u'tags': u'', u'skip_tags': u'', u'playbook': u'deploy_steps_playbook.yaml', u'ansible_extra_env_variables': {u'ANSIBLE_HOST_KEY_CHECKING': u'False', u'ANSIBLE_LOG_PATH': u'/var/log/mistral/package_update.log'}, u'module_path': u'/usr/share/ansible-modules', u'nodes': u'Controller', u'node_user': u'tripleo-admin', u'ansible_queue_name': u'upgrade'}, u'spec': {u'tasks': {u'get_private_key': {u'name': u'get_private_key', u'on-success': u'node_update', u'publish': {u'private_key': u'<% task().result %>'}, u'version': u'2.0', u'action': u'tripleo.validations.get_privkey', u'type': u'direct'}, u'node_update_failed': {u'version': u'2.0', u'type': u'direct', u'name': u'node_update_failed', u'publish': {u'status': u'FAILED', u'message': u'Failed to update nodes - <% $.nodes %>, please see the logs.'}, u'on-success': u'send_message'}, u'node_update_passed': {u'version': u'2.0', u'type': u'direct', u'name': u'node_update_passed', u'publish': {u'status': u'SUCCESS', u'message': u'Updated nodes - <% $.nodes %>'}, u'on-success': u'send_message'}, u'send_message': {u'input': {u'status': u"<% $.get('status', 'SUCCESS') %>", u'message': u"<% $.get('message', '') %>", u'queue_name': u'<% $.ansible_queue_name %>', u'type': u'<% execution().name %>', u'execution': u'<% execution() %>'}, u'version': u'2.0', u'type': u'direct', u'name': u'send_message', u'workflow': u'tripleo.messaging.v1.send'}, u'node_update': {u'name': u'node_update', u'on-error': u'node_update_failed', u'on-success': [{u'node_update_passed': u'<% task().result.returncode = 0 %>'}, {u'node_update_failed': u'<% task().result.returncode != 0 %>'}], u'publish': {u'output': u'<% task().result %>'}, u'version': u'2.0', u'action': u'tripleo.ansible-playbook', u'input': {u'remote_user': u'<% $.node_user %>', u'limit_hosts': u'<% $.nodes %>', u'tags': u'<% $.tags %>', u'become_user': u'root', u'verbosity': u'<% $.verbosity %>', u'queue_name': u'<% $.ansible_queue_name %>', u'extra_env_variables': u'<% $.ansible_extra_env_variables %>', u'skip_tags': u'<% $.skip_tags %>', u'inventory': u'<% $.inventory_file %>', u'become': True, u'module_path': u'<% $.module_path %>', u'playbook': u'<% $.work_dir %>/<% execution().id %>/<% $.playbook %>', u'trash_output': True, u'execution_id': u'<% execution().id %>', u'ssh_private_key': u'<% $.private_key %>'}, u'type': u'direct'}, u'download_config': {u'name': u'download_config', u'on-error': u'node_update_failed', u'on-success': u'get_private_key', u'version': u'2.0', u'action': u'tripleo.config.download_config', u'input': {u'work_dir': u'<% $.work_dir %>/<% execution().id %>'}, u'type': u'direct'}}, u'description': u'Take a container and perform an update nodes by nodes', u'tags': [u'tripleo-common-managed'], u'version': u'2.0', u'input': [{u'node_user': u'tripleo-admin'}, u'nodes', u'playbook', u'inventory_file', {u'ansible_queue_name': u'tripleo'}, {u'module_path': u'/usr/share/ansible-modules'}, {u'ansible_extra_env_variables': {u'ANSIBLE_HOST_KEY_CHECKING': u'False', u'ANSIBLE_LOG_PATH': u'/var/log/mistral/package_update.log'}}, {u'verbosity': 1}, {u'work_dir': u'/var/lib/mistral'}, {u'tags': u''}, {u'skip_tags': u''}], u'name': u'update_nodes'}}, u'plan_name': None, u'execution_id': u'45635779-b654-4847-97b3-601b224719c0', u'deployment_status': None}

This is caused by https://bugzilla.redhat.com/show_bug.cgi?id=1656368

Working on a fix in the resource-agents

*** This bug has been marked as a duplicate of bug 1656368 ***

Although this one is clearly a duplicate, I am keeping this one open because it effects OpenStack upgrades. When the original (RHEL) bug is closed and delivered, we will need to re-test it in OpenStack and this bug will serve as a tracker.

Fix was delivered to rhel 7.6 (see original bug) and OpenStack got it earlier in December.
Automation ran and all upgrade jobs have passed. I am closing this one.