Description of problem: During my FFWD upgrade, I ran the openstack "overcloud ffwd-upgrade run" and it failed. Debugging it I realized I had forgotten to include the ffwd.yaml to over-ride the repositories in FastForwardCustomRepoScriptContent. I updated my prepare command and when I re-executed the prepare it failed with the following error: Waiting for messages on queue 'ffwdupgrade' with no timeout. {u'config_name': u'ffwd-upgrade-prepare', u'execution': {u'created_at': u'2018-06-22 02:45:22', u'id': u'd05a82ef-c002-4f9b-9890-5d465a8a6bd2', u'input': {u'config': u'#!/bin/bash \nrm -f /usr/libexec/os-apply-config/templates/etc/os-net-config/config.json || true \n', u'config_name': u'ffwd-upgrade-prepare', u'group': u'script', u'queue_name': u'ffwdupgrade', u'server_name': u'overcloud-controller-0', u'server_uuid': u'afb6d2a8-0937-488b-85dd-157ac38ad6bf'}, u'name': u'tripleo.deployment.v1.deploy_on_server', u'params': {u'index': 1, u'namespace': u'', u'root_execution_id': u'9bbb254a-2b83-4047-8dd1-e92f672ceadd', u'task_execution_id': u'ce79401a-09e8-4b35-baed-adc5ba6dc0d4'}, u'spec': {u'input': [u'server_uuid', u'server_name', u'config', u'config_name', u'group', {u'queue_name': u'tripleo'}], u'name': u'deploy_on_server', u'tags': [u'tripleo-common-managed'], u'tasks': {u'deploy_config': {u'action': u'tripleo.deployment.config', u'input': {u'config': u'<% $.config %>', u'group': u'<% $.group %>', u'name': u'<% $.config_name %>', u'server_id': u'<% $.server_uuid %>'}, u'name': u'deploy_config', u'on-complete': u'send_message', u'publish': {u'status_code': u'<% task().result.deploy_status_code %>', u'stderr': u'<% task().result.deploy_stderr %>', u'stdout': u'<% task().result.deploy_stdout %>'}, u'publish-on-error': {u'message': u'<% task().result %>', u'status': u'FAILED'}, u'type': u'direct', u'version': u'2.0'}, u'send_message': {u'action': u'zaqar.queue_post', u'input': {u'messages': {u'body': {u'payload': {u'config_name': u'<% $.config_name %>', u'execution': u'<% execution() %>', u'message': u'<% $.get("message", "") %>', u'server_name': u'<% $.server_name %>', u'server_uuid': u'<% $.server_uuid %>', u'status': u'<% $.get("status", "SUCCESS") %>', u'status_code': u'<% $.get("status_code", "") %>', u'stderr': u'<% $.get("stderr", "") %>', u'stdout': u'<% $.get("stdout", "") %>'}, u'type': u'tripleo.deployment.v1.deploy_on_server'}}, u'queue_name': u'<% $.queue_name %>'}, u'name': u'send_message', u'on-success': [{u'fail': u'<% $.get(\'status\') = "FAILED" %>'}], u'retry': u'count=5 delay=1', u'type': u'direct', u'version': u'2.0'}}, u'version': u'2.0'}, u'updated_at': u'2018-06-22 02:45:22'}, u'message': u"Timeout for heat deployment 'ffwd-upgrade-prepare'", u'server_name': u'overcloud-controller-0', u'server_uuid': u'afb6d2a8-0937-488b-85dd-157ac38ad6bf', u'status': u'FAILED', u'status_code': u'', u'stderr': u'', u'stdout': u''} Updating the prepare statement and re-running it worked at the Brno Hackfest and in my previous attempts to upgrade OSP13. It was late so I worked around the issue by copying the corrected ffu_update_repo.sh to my overcloud nodes and storing it in ~heat-admin and then on each node, I ran the following until the "upgrade run" completed successfully: while true do /bin/cp -f ~heat-admin/ffu_update_repo.sh ~root sleep 1 done This allowed the "upgrade run" to complete successfully and I was able to continue to the next step (which is running now). Version-Release number of selected component (if applicable): 13 How reproducible: Not sure, I would have to go through the steps again. Steps to Reproduce: 1. Forget to include the YAML file with the FastForwardCustomRepoScriptContent that points to rhos-release when you run "openstack overcloud ffwd-upgrade prepare". This will complete successfully. 2. Execute "openstack overcloud ffwd-upgrade run" and it will fail since the overcloud nodes can't reach CDN. 3. Update your "openstack overcloud ffwd-upgrade prepare" to include the YAML and re-run it. It will fail. This worked at the BRNO Hackfest. Actual results: Had to implement a hack to move forward. Expected results: The prepare should update the plan accordingly and then I should be able to re execute the "upgrade run". Additional info:
Just a note here - I've seen the u'message': u"Timeout for heat deployment 'ffwd-upgrade-prepare'" failure when the overcloud nodes weren't able to reach the undercloud SSL enabled swift endpoint(unable to verify the SSL certificate). Could this have been the case here?
(In reply to Marius Cornea from comment #2) > Just a note here - I've seen the u'message': u"Timeout for heat deployment > 'ffwd-upgrade-prepare'" failure when the overcloud nodes weren't able to > reach the undercloud SSL enabled swift endpoint(unable to verify the SSL > certificate). Could this have been the case here? No, it's not likely since I was able to workaround it by forcing the repositories on the overcloud nodes and proceeding with the run. It's like some resource was locked up by the run process preventing the prepare from executing.
I have just hit the same error with slightly difference circumstances. I have included ffwd.yaml in the prepare step, however I've made an error in that file and after fixing the problem in the script I could no longer re-run prepare step. Including my sosreport.
Created attachment 1455536 [details] sosreport-undercloud
I have a supposition on what could be going on here: overcloud ffwd-upgrade run stops os-collect-config in the beginning: https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/common/deploy-steps.j2#L626 and then it starts it back again once it completes(which in this case doesn't happen as the upgrade run stops due to an error): https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/common/deploy-steps.j2#L663-L664 My guess is that ffwd-upgrade-prepare expects os-collect-config to be running on the overcloud nodes and since it's not we're seeing the timeout. A possible workaround is to start the os-collect-config service before re-running prepare with something like: ansible -i /usr/bin/tripleo-ansible-inventory -m service -a 'name=os-collect-config state=started' overcloud Note: I need to test this to validate, currently it's just a supposition.
I reproduced the issue and starting os-collect-config allowed me to move forward. So as a workaround you need to start os-collect-config before running ffwd-upgrade prepare by running the following from undercloud: ansible -i /usr/bin/tripleo-ansible-inventory -m service -a 'name=os-collect-config state=started' overcloud -b ########################################################### Running ffwd-upgrade upgrade prepare step Warning! The TripleO Fast Forward Upgrade workflow is currently considered under development. In particular invocations of the ffwd-upgrade cli should be initially limited to development/test environments. Once and if you decide to use ffwd-upgrade in production, ensure you are adequately prepared with valid backup of your current deployment state. Continuing fast forward upgrade Started Mistral Workflow tripleo.deployment.v1.deploy_on_servers. Execution ID: c89bddb7-9cd8-4f4b-ae0a-20513d403868 Waiting for messages on queue 'ffwdupgrade' with no timeout. {u'config_name': u'ffwd-upgrade-prepare', u'execution': {u'created_at': u'2018-06-29 19:15:06', u'id': u'ba1298b2-5b4f-40ee-9629-ac11429367f5', u'input': {u'config': u'#!/bin/bash \nrm -f /usr/libexec/os-apply-config/templates/etc/os-net-config/config.json || true \n', u'config_name': u'ffwd-upgrade-prepare', u'group': u'script', u'queue_name': u'ffwdupgrade', u'server_name': u'controller-r00-00', u'server_uuid': u'd0799927-d916-41ba-a2ab-fb8bf5d38b21'}, u'name': u'tripleo.deployment.v1.deploy_on_server', u'params': {u'index': 0, u'namespace': u'', u'root_execution_id': u'c89bddb7-9cd8-4f4b-ae0a-20513d403868', u'task_execution_id': u'095c0929-59e9-4d75-8799-af47b258b1c0'}, u'spec': {u'input': [u'server_uuid', u'server_name', u'config', u'config_name', u'group', {u'queue_name': u'tripleo'}], u'name': u'deploy_on_server', u'tags': [u'tripleo-common-managed'], u'tasks': {u'deploy_config': {u'action': u'tripleo.deployment.config', u'input': {u'config': u'<% $.config %>', u'group': u'<% $.group %>', u'name': u'<% $.config_name %>', u'server_id': u'<% $.server_uuid %>'}, u'name': u'deploy_config', u'on-complete': u'send_message', u'publish': {u'status_code': u'<% task().result.deploy_status_code %>', u'stderr': u'<% task().result.deploy_stderr %>', u'stdout': u'<% task().result.deploy_stdout %>'}, u'publish-on-error': {u'message': u'<% task().result %>', u'status': u'FAILED'}, u'type': u'direct', u'version': u'2.0'}, u'send_message': {u'action': u'zaqar.queue_post', u'input': {u'messages': {u'body': {u'payload': {u'config_name': u'<% $.config_name %>', u'execution': u'<% execution() %>', u'message': u'<% $.get("message", "") %>', u'server_name': u'<% $.server_name %>', u'server_uuid': u'<% $.server_uuid %>', u'status': u'<% $.get("status", "SUCCESS") %>', u'status_code': u'<% $.get("status_code", "") %>', u'stderr': u'<% $.get("stderr", "") %>', u'stdout': u'<% $.get("stdout", "") %>'}, u'type': u'tripleo.deployment.v1.deploy_on_server'}}, u'queue_name': u'<% $.queue_name %>'}, u'name': u'send_message', u'on-success': [{u'fail': u'<% $.get(\'status\') = "FAILED" %>'}], u'retry': u'count=5 delay=1', u'type': u'direct', u'version': u'2.0'}}, u'version': u'2.0'}, u'updated_at': u'2018-06-29 19:15:06'}, u'message': u"Timeout for heat deployment 'ffwd-upgrade-prepare'", u'server_name': u'controller-r00-00', u'server_uuid': u'd0799927-d916-41ba-a2ab-fb8bf5d38b21', u'status': u'FAILED', u'status_code': u'', u'stderr': u'', u'stdout': u''} (undercloud) [stack@undercloud-0 ~]$ ansible -i /usr/bin/tripleo-ansible-inventory -m service -a 'name=os-collect-config state=started' overcloud -b 192.168.0.20 | SUCCESS => { "changed": false, "name": "os-collect-config", "state": "started", "status": { "ActiveEnterTimestamp": "Fri 2018-06-29 19:02:08 UTC", "ActiveEnterTimestampMonotonic": "3021418826", "ActiveExitTimestamp": "Fri 2018-06-29 18:45:09 UTC", "ActiveExitTimestampMonotonic": "2002131847", "ActiveState": "active", "After": "basic.target cloud-final.service system.slice systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Fri 2018-06-29 19:02:08 UTC", "AssertTimestampMonotonic": "3021417641", "Before": "crond.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2018-06-29 19:02:08 UTC", "ConditionTimestampMonotonic": "3021417641", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/os-collect-config.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Collect metadata and run hook commands.", "DevicePolicy": "auto", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "227465", "ExecMainStartTimestamp": "Fri 2018-06-29 19:02:08 UTC", "ExecMainStartTimestampMonotonic": "3021418765", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/bin/os-collect-config ; argv[]=/usr/bin/os-collect-config ; ignore_errors=no ; start_time=[Fri 2018-06-29 19:02:08 UTC] ; stop_time=[n/a] ; pid=227465 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/os-collect-config.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "os-collect-config.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Fri 2018-06-29 18:45:09 UTC", "InactiveEnterTimestampMonotonic": "2002150438", "InactiveExitTimestamp": "Fri 2018-06-29 19:02:08 UTC", "InactiveExitTimestampMonotonic": "3021418826", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "process", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "47161", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "47161", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "227465", "MemoryAccounting": "no", "MemoryCurrent": "38576128", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "os-collect-config.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "on-failure", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "no", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "1", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Fri 2018-06-29 19:02:08 UTC", "WatchdogTimestampMonotonic": "3021418804", "WatchdogUSec": "0" } } 192.168.0.17 | SUCCESS => { "changed": false, "name": "os-collect-config", "state": "started", "status": { "ActiveEnterTimestamp": "Fri 2018-06-29 19:02:08 UTC", "ActiveEnterTimestampMonotonic": "3022904783", "ActiveExitTimestamp": "Fri 2018-06-29 18:45:12 UTC", "ActiveExitTimestampMonotonic": "2006328178", "ActiveState": "active", "After": "cloud-final.service systemd-journald.socket basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Fri 2018-06-29 19:02:08 UTC", "AssertTimestampMonotonic": "3022903657", "Before": "multi-user.target crond.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2018-06-29 19:02:08 UTC", "ConditionTimestampMonotonic": "3022903657", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/os-collect-config.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Collect metadata and run hook commands.", "DevicePolicy": "auto", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "35484", "ExecMainStartTimestamp": "Fri 2018-06-29 19:02:08 UTC", "ExecMainStartTimestampMonotonic": "3022904710", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/bin/os-collect-config ; argv[]=/usr/bin/os-collect-config ; ignore_errors=no ; start_time=[Fri 2018-06-29 19:02:08 UTC] ; stop_time=[n/a] ; pid=35484 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/os-collect-config.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "os-collect-config.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Fri 2018-06-29 18:45:12 UTC", "InactiveEnterTimestampMonotonic": "2006342116", "InactiveExitTimestamp": "Fri 2018-06-29 19:02:08 UTC", "InactiveExitTimestampMonotonic": "3022904783", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "process", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "31034", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "31034", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "35484", "MemoryAccounting": "no", "MemoryCurrent": "38645760", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "os-collect-config.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "on-failure", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "no", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "1", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Fri 2018-06-29 19:02:08 UTC", "WatchdogTimestampMonotonic": "3022904750", "WatchdogUSec": "0" } } 192.168.0.14 | SUCCESS => { "changed": false, "name": "os-collect-config", "state": "started", "status": { "ActiveEnterTimestamp": "Fri 2018-06-29 19:02:08 UTC", "ActiveEnterTimestampMonotonic": "3020753663", "ActiveExitTimestamp": "Fri 2018-06-29 18:45:09 UTC", "ActiveExitTimestampMonotonic": "2001013481", "ActiveState": "active", "After": "systemd-journald.socket system.slice cloud-final.service basic.target", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Fri 2018-06-29 19:02:08 UTC", "AssertTimestampMonotonic": "3020752379", "Before": "crond.service shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2018-06-29 19:02:08 UTC", "ConditionTimestampMonotonic": "3020752379", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/os-collect-config.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Collect metadata and run hook commands.", "DevicePolicy": "auto", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "225391", "ExecMainStartTimestamp": "Fri 2018-06-29 19:02:08 UTC", "ExecMainStartTimestampMonotonic": "3020753601", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/bin/os-collect-config ; argv[]=/usr/bin/os-collect-config ; ignore_errors=no ; start_time=[Fri 2018-06-29 19:02:08 UTC] ; stop_time=[n/a] ; pid=225391 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/os-collect-config.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "os-collect-config.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Fri 2018-06-29 18:45:09 UTC", "InactiveEnterTimestampMonotonic": "2001031564", "InactiveExitTimestamp": "Fri 2018-06-29 19:02:08 UTC", "InactiveExitTimestampMonotonic": "3020753663", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "process", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "47161", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "47161", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "225391", "MemoryAccounting": "no", "MemoryCurrent": "38498304", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "os-collect-config.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "on-failure", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "no", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "1", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Fri 2018-06-29 19:02:08 UTC", "WatchdogTimestampMonotonic": "3020753641", "WatchdogUSec": "0" } } 192.168.0.15 | SUCCESS => { "changed": true, "name": "os-collect-config", "state": "started", "status": { "ActiveEnterTimestamp": "Fri 2018-06-29 18:12:36 UTC", "ActiveEnterTimestampMonotonic": "50511354", "ActiveExitTimestamp": "Fri 2018-06-29 18:45:08 UTC", "ActiveExitTimestampMonotonic": "2002301047", "ActiveState": "inactive", "After": "systemd-journald.socket cloud-final.service basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Fri 2018-06-29 18:12:36 UTC", "AssertTimestampMonotonic": "50510117", "Before": "shutdown.target multi-user.target crond.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2018-06-29 18:12:36 UTC", "ConditionTimestampMonotonic": "50510117", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Collect metadata and run hook commands.", "DevicePolicy": "auto", "ExecMainCode": "2", "ExecMainExitTimestamp": "Fri 2018-06-29 18:45:08 UTC", "ExecMainExitTimestampMonotonic": "2002319280", "ExecMainPID": "4083", "ExecMainStartTimestamp": "Fri 2018-06-29 18:12:36 UTC", "ExecMainStartTimestampMonotonic": "50511274", "ExecMainStatus": "15", "ExecStart": "{ path=/usr/bin/os-collect-config ; argv[]=/usr/bin/os-collect-config ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/os-collect-config.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "os-collect-config.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Fri 2018-06-29 18:45:08 UTC", "InactiveEnterTimestampMonotonic": "2002319372", "InactiveExitTimestamp": "Fri 2018-06-29 18:12:36 UTC", "InactiveExitTimestampMonotonic": "50511354", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "process", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "47161", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "47161", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "os-collect-config.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "on-failure", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "no", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0" } } 192.168.0.13 | SUCCESS => { "changed": true, "name": "os-collect-config", "state": "started", "status": { "ActiveEnterTimestamp": "Fri 2018-06-29 18:12:15 UTC", "ActiveEnterTimestampMonotonic": "30823630", "ActiveExitTimestamp": "Fri 2018-06-29 18:45:12 UTC", "ActiveExitTimestampMonotonic": "2007101485", "ActiveState": "inactive", "After": "cloud-final.service system.slice basic.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Fri 2018-06-29 18:12:15 UTC", "AssertTimestampMonotonic": "30821871", "Before": "multi-user.target crond.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2018-06-29 18:12:15 UTC", "ConditionTimestampMonotonic": "30821871", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Collect metadata and run hook commands.", "DevicePolicy": "auto", "ExecMainCode": "2", "ExecMainExitTimestamp": "Fri 2018-06-29 18:45:12 UTC", "ExecMainExitTimestampMonotonic": "2007115641", "ExecMainPID": "2641", "ExecMainStartTimestamp": "Fri 2018-06-29 18:12:15 UTC", "ExecMainStartTimestampMonotonic": "30822937", "ExecMainStatus": "15", "ExecStart": "{ path=/usr/bin/os-collect-config ; argv[]=/usr/bin/os-collect-config ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/os-collect-config.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "os-collect-config.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Fri 2018-06-29 18:45:12 UTC", "InactiveEnterTimestampMonotonic": "2007115744", "InactiveExitTimestamp": "Fri 2018-06-29 18:12:15 UTC", "InactiveExitTimestampMonotonic": "30823630", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "process", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "31034", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "31034", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "os-collect-config.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "on-failure", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "no", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0" } } (undercloud) [stack@undercloud-0 ~]$ bash overcloud_upgrade_prepare.sh Running ffwd-upgrade upgrade prepare step Warning! The TripleO Fast Forward Upgrade workflow is currently considered under development. In particular invocations of the ffwd-upgrade cli should be initially limited to development/test environments. Once and if you decide to use ffwd-upgrade in production, ensure you are adequately prepared with valid backup of your current deployment state. Continuing fast forward upgrade Started Mistral Workflow tripleo.deployment.v1.deploy_on_servers. Execution ID: f8d333a1-f679-462c-96bd-a3455922a941 Waiting for messages on queue 'ffwdupgrade' with no timeout. Success You have not provided a container registry file. Note that none of the containers on your environment will be updated. If you want to update your containers you have to re-run this command and provide the registry file with: --container-registry-file option. Started Mistral Workflow tripleo.validations.v1.check_pre_deployment_validations. Execution ID: ebc34fc3-e110-46e6-b64e-49ecf76cd928 Waiting for messages on queue 'tripleo' with no timeout. Removing the current plan files Uploading new plan files Started Mistral Workflow tripleo.plan_management.v1.update_deployment_plan. Execution ID: dc2435f6-e6e1-4836-838a-bb0f4cc684cb Plan updated. Processing templates in the directory /tmp/tripleoclient-W6paih/tripleo-heat-templates Started Mistral Workflow tripleo.plan_management.v1.get_deprecated_parameters. Execution ID: b0d46d22-b6a8-49fd-b937-2fa98785a97f WARNING: Following parameters are deprecated and still defined. Deprecated parameters will be removed soon! NovaComputeSchedulerHints OvercloudControlFlavor NovaComputeIPs WARNING: Following parameters are defined but not used in plan. Could be possible that parameter is valid but currently not used. DockerMysqlClientConfigImage Started Mistral Workflow tripleo.package_update.v1.package_update_plan. Execution ID: 18991038-26f1-433f-864a-bb4e85913a54 Waiting for messages on queue 'tripleo' with no timeout. 2018-06-29 19:30:53Z [ServiceNetMap]: UPDATE_IN_PROGRESS state changed 2018-06-29 19:30:53Z [overcloud-ServiceNetMap-gcsamrirdbdr]: UPDATE_IN_PROGRESS Stack UPDATE started 2018-06-29 19:30:54Z [overcloud-ServiceNetMap-gcsamrirdbdr]: UPDATE_COMPLETE Stack UPDATE completed successfully 2018-06-29 19:30:54Z [ServiceNetMap]: UPDATE_COMPLETE state changed 2018-06-29 19:30:56Z [DefaultPasswords]: UPDATE_IN_PROGRESS state changed 2018-06-29 19:30:57Z [Networks]: UPDATE_IN_PROGRESS state changed 2018-06-29 19:30:58Z [overcloud-Networks-oz6qe6qetnjl]: UPDATE_IN_PROGRESS Stack UPDATE started 2018-06-29 19:30:58Z [DefaultPasswords]: UPDATE_COMPLETE state changed 2018-06-29 19:30:58Z [overcloud-Networks-oz6qe6qetnjl.ExternalNetwork]: UPDATE_IN_PROGRESS state changed 2018-06-29 19:30:58Z [overcloud-Networks-oz6qe6qetnjl-ExternalNetwork-xnherguuioqp]: UPDATE_I
*** Bug 1636001 has been marked as a duplicate of this bug. ***