Bug 1607623 - running the deployment command fails with '('The read operation timed out',)'
Summary: running the deployment command fails with '('The read operation timed out',)'
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-tripleoclient
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: beta
: 14.0 (Rocky)
Assignee: RHOS Maint
QA Contact: Artem Hrechanychenko
URL:
Whiteboard:
Depends On:
Blocks: 1644754
TreeView+ depends on / blocked
 
Reported: 2018-07-23 20:38 UTC by Alexander Chuzhoy
Modified: 2023-09-14 04:31 UTC (History)
17 users (show)

Fixed In Version: python-tripleoclient-10.6.1-0.20181010222404.8c8f259.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1644754 (view as bug list)
Environment:
Last Closed: 2019-01-11 11:50:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 613623 0 'None' MERGED Increase the deploy_plan timeout in tripleoclient 2020-02-20 17:05:02 UTC
Red Hat Product Errata RHEA-2019:0045 0 None None None 2019-01-11 11:51:03 UTC

Description Alexander Chuzhoy 2018-07-23 20:38:58 UTC
Re-running the deployment command fails with '('The read operation timed out',)'

openstack-tripleo-heat-templates-9.0.0-0.20180710202746.d2994ca.el7ost.noarch
puppet-mistral-13.1.1-0.20180702200255.5fa0de7.el7ost.noarch
python2-mistralclient-3.6.0-0.20180612110548.7451abb.el7ost.noarch
python2-mistral-lib-0.5.0-0.20180425121912.ef3482e.el7ost.noarch
instack-undercloud-9.1.1-0.20180709220206.3fc4cc7.el7ost.noarch


Steps to reproduce.

1. Run the OC deployment command.
2. Stop the deployment command (used ctrl+z as ctrl+c didn't work).
3. Re-ran the deployment command.


Result:
(undercloud) [stack@undercloud75 ~]$ bash overcloud_deploy.sh 
Removing the current plan files
Uploading new plan files
Plan updated.
Processing templates in the directory /tmp/tripleoclient-tDAJs9/tripleo-heat-templates
WARNING: Following parameter(s) are defined but not used in plan. Could be possible that parameter is valid but currently not used.
  DockerMysqlClientConfigImage
  CinderBackupBackend
Deploying templates in the directory /tmp/tripleoclient-tDAJs9/tripleo-heat-templates
Initializing overcloud plan deployment
('The read operation timed out',)
(undercloud) [stack@undercloud75 ~]$

Comment 2 Alexander Chuzhoy 2018-07-23 21:57:25 UTC
Rebooting the undercloud doesn't help.

Comment 3 Matt Wisch 2018-07-27 13:56:45 UTC
Is the Zaqar service running?

Comment 5 Steve Baker 2018-08-27 21:20:37 UTC
Can you check the status of the undercloud zaqar and mistral services? It sounds like they may be down.

Comment 6 Alexander Chuzhoy 2018-09-11 23:08:10 UTC
I reproduce it with new deployment too on BM setup...
The services are containerized in OSP14.



[root@undercloud ~]# docker ps|grep -i zaqar
4919a3fd9b0a        192.168.0.1:8787/rhosp14/openstack-zaqar:2018-09-06.1                       "kolla_start"            3 hours ago         Up 3 hours                                 zaqar_websocket
136da9aa3ab9        192.168.0.1:8787/rhosp14/openstack-zaqar:2018-09-06.1                       "kolla_start"            3 hours ago         Up 3 hours                                 zaqar
[root@undercloud ~]# docker ps|grep -i mistral
f85d47b56e64        192.168.0.1:8787/rhosp14/openstack-mistral-api:2018-09-06.1                 "kolla_start"            3 hours ago         Up 3 hours (healthy)                       mistral_api
d5a0a7dbd7d7        192.168.0.1:8787/rhosp14/openstack-mistral-engine:2018-09-06.1              "kolla_start"            3 hours ago         Up 3 hours (healthy)                       mistral_engine
5f2ab530d090        192.168.0.1:8787/rhosp14/openstack-mistral-event-engine:2018-09-06.1        "kolla_start"            3 hours ago         Up 3 hours (healthy)                       mistral_event_engine
1cc6200f10e8        192.168.0.1:8787/rhosp14/openstack-mistral-executor:2018-09-06.1            "kolla_start"            3 hours ago         Up 3 hours (healthy)                       mistral_executor
[root@undercloud ~]# 

Let me know if you want me to run a particular command against these containers.

Comment 7 Alexander Chuzhoy 2018-09-17 14:26:53 UTC
This could be an environmental issue.
Once I updated the CPU count and flags on the VM used for undercloud - the issue didn't re-appear.

Comment 8 Alex Schultz 2018-09-17 14:32:19 UTC
Seemed to be an underpowered undercloud. Feel free to reopen if it happens again.

Comment 12 James Slagle 2018-10-25 20:01:59 UTC
can you provide all the mistral logs from when the error occurred?

Comment 13 James Slagle 2018-10-25 20:10:17 UTC
the mistral logs are at /var/log/containers/mistral on the undercloud

Comment 15 James Slagle 2018-10-26 15:04:51 UTC
the full traceback is:

('The read operation timed out',)                                                                                                                                                                                                 
Traceback (most recent call last):                                                                                                                                                                                                
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand                                                                                                                                               
    result = cmd.run(parsed_args)                                                                                                                                                                                                 
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run                                                                                                                                               
    super(Command, self).run(parsed_args)                                                                                                                                                                                         
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run                                                                                                                                             
    return super(Command, self).run(parsed_args)                                                                                                                                                                                  
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run                                                                                                                                                      
    return_code = self.take_action(parsed_args) or 0                                                                                                                                                                              
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 886, in take_action                                                                                                                          
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)                                                                                                                                                                
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 359, in _deploy_tripleo_heat_templates_tmpdir                                                                                                
    new_tht_root, tht_root)                                                                                                                                                                                                       
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 452, in _deploy_tripleo_heat_templates                                                                                                       
    parsed_args.plan_environment_file)                                                                                                                                                                                            
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 467, in _try_overcloud_deploy_with_compat_yaml                                                                                               
    plan_env_file)                                                                                                                                                                                                                
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 234, in _heat_deploy                                                                                                                         
    skip_deploy_identifier=skip_deploy_identifier)                                                                                                                                                                                
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 78, in deploy_and_wait                                                                                                                      
    deploy(log, clients, **workflow_input)                                                                                                                                                                                        
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 52, in deploy                                                                                                                               
    360):                                                                                                                                                                                                                         
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/base.py", line 61, in wait_for_messages                                                                                                                          
    for payload in websocket.wait_for_messages(timeout=timeout):                                                                                                                                                                  
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 152, in wait_for_messages                                                                                                                                 
    message = self.recv()
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 126, in recv
    return json.loads(self._ws.recv())
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 348, in recv
    opcode, data = self.recv_data()
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 365, in recv_data
    opcode, frame = self.recv_data_frame(control_frame)
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 378, in recv_data_frame
    frame = self.recv_frame()
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 410, in recv_frame
    return self.frame_buffer.recv_frame()
  File "/usr/lib/python2.7/site-packages/websocket/_abnf.py", line 300, in recv_frame
    self.recv_header()
  File "/usr/lib/python2.7/site-packages/websocket/_abnf.py", line 249, in recv_header
    header = self.recv_strict(2)
  File "/usr/lib/python2.7/site-packages/websocket/_abnf.py", line 334, in recv_strict
    bytes = self.recv(min(16384, shortage))
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 476, in _recv
    return recv(self.sock, bufsize)
  File "/usr/lib/python2.7/site-packages/websocket/_socket.py", line 77, in recv
    bytes = sock.recv(bufsize)
  File "/usr/lib64/python2.7/ssl.py", line 757, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.7/ssl.py", line 651, in read
    v = self._sslobj.read(len or 1024)
SSLError: ('The read operation timed out',)

Comment 16 James Slagle 2018-10-26 15:09:49 UTC
based on the above traceback, it's timeout trying to read from the websocket, so I think DFG:Workflows/CloudApps needs to look at this.

The timeout in tripleoclient in base.wait_for_messages() is 600 seconds.

The longest gap between message sends (found by grep'ing for zaqar.queue_post from openstack action execution list) is ~7.5 minutes:

| af195b60-7939-4169-8125-c6c260ed3fca | zaqar.queue_post                           | tripleo.messaging.v1.send                                  |                    | send_message                             | a0e347cf-a6e0-49dd-885f-4b1ed8f5b65f | SUCCESS | True     | 2018-10-25 20:35:47 | 2018-10-25 20:35:49 |
| fdff0f8e-2a04-449d-a763-d0c399b1cd8c | zaqar.queue_post                           | tripleo.messaging.v1.send                                  |                    | send_message                             | 26bbe053-5d2b-42ad-a26b-2099e16592fc | SUCCESS | True     | 2018-10-25 20:43:03 | 2018-10-25 20:43:05 |

That's within the 600s/10m timeout on the client side, but perhaps there is some unaccounted for overhead?

Or perhaps haproxy is misconfigured?

Comment 17 James Slagle 2018-10-26 15:44:31 UTC
alright, this might be the needed patch:

https://review.openstack.org/#/c/609993/

I was looking at the upstream code that has that patch and the timeout is set to 600s. However in the actual environment, it is still set to 360s. Going to try with the patch applied and see if it still reproduces.

Comment 18 James Slagle 2018-10-26 16:24:25 UTC
i'll push this patch downstream

Comment 26 Artem Hrechanychenko 2018-11-14 21:06:12 UTC
VERIFIED
python-tripleoclient-10.6.1-0.20181010222404.8c8f259.el7ost.noarch


2018-11-14 19:54:30Z [overcloud.ComputeServiceChain.PostUpgradeTasks]: CREATE_COMPLETE  state changed
^CTraceback (most recent call last):
  File "/usr/bin/openstack", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 210, in main
    return OpenStackShell().run(argv)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 281, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 175, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 402, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 887, in take_action
    self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 359, in _deploy_tripleo_heat_templates_tmpdir
    new_tht_root, tht_root)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 452, in _deploy_tripleo_heat_templates
    parsed_args.plan_environment_file)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 467, in _try_overcloud_deploy_with_compat_yaml
    plan_env_file)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 234, in _heat_deploy
    skip_deploy_identifier=skip_deploy_identifier)
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 101, in deploy_and_wait
    orchestration_client, plan_name, marker, action, verbose_events)
  File "/usr/lib/python2.7/site-packages/tripleoclient/utils.py", line 276, in wait_for_stack_ready
    poll_period=5, marker=marker, out=out, nested_depth=2)
  File "/usr/lib/python2.7/site-packages/heatclient/common/event_utils.py", line 240, in poll_for_events
    time.sleep(poll_period)
KeyboardInterrupt
(undercloud) [stack@undercloud-0 ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
/usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
/usr/lib/python2.7/site-packages/urllib3/connection.py:344: SubjectAltNameWarning: Certificate for 192.168.24.2 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for details.)
  SubjectAltNameWarning
+--------------------------------------+------------+--------------------+----------------------+--------------+----------------------------------+
| id                                   | stack_name | stack_status       | creation_time        | updated_time | project                          |
+--------------------------------------+------------+--------------------+----------------------+--------------+----------------------------------+
| 11249161-68e9-4a47-8fd4-db2a62cb4447 | overcloud  | CREATE_IN_PROGRESS | 2018-11-14T19:51:54Z | None         | aa6fde9247dc4da7a5d2a366a2166fe7 |



(undercloud) [stack@undercloud-0 ~]$ bash overcloud_deploy.sh 
Removing the current plan files
Uploading new plan files
Plan updated.
Processing templates in the directory /tmp/tripleoclient-cSTH1p/tripleo-heat-templates
(undercloud) [stack@undercloud-0 ~]$ bash overcloud_deploy.sh 
Removing the current plan files
Uploading new plan files
Plan updated.
Processing templates in the directory /tmp/tripleoclient-cSTH1p/tripleo-heat-templates
WARNING: Following parameter(s) are deprecated and still defined. Deprecated parameters will be removed soon!
  OvercloudControlFlavor
WARNING: Following parameter(s) are defined but not used in plan. Could be possible that parameter is valid but currently not used.
  CinderBackupBackend
Deploying templates in the directory /tmp/tripleoclient-cSTH1p/tripleo-heat-templates
Initializing overcloud plan deployment
Creating overcloud Heat stack
2018-11-14 20:09:24Z [Networks]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:24Z [overcloud-Networks-cqf6exrqw4b4]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:25Z [DefaultPasswords]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:26Z [overcloud-Networks-cqf6exrqw4b4.InternalApiNetwork]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:26Z [ServiceNetMap]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:26Z [overcloud-Networks-cqf6exrqw4b4.TenantNetwork]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:26Z [overcloud-ServiceNetMap-auvbhlaezzeu]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:27Z [DefaultPasswords]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:27Z [overcloud-ServiceNetMap-auvbhlaezzeu]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:27Z [overcloud-Networks-cqf6exrqw4b4.ExternalNetwork]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:27Z [overcloud-Networks-cqf6exrqw4b4-InternalApiNetwork-wo5sbj77bzez]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:27Z [overcloud-Networks-cqf6exrqw4b4-TenantNetwork-jnz4hhljbiaf]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:27Z [ServiceNetMap]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:29Z [overcloud-Networks-cqf6exrqw4b4-ExternalNetwork-u3dqa67yxnte]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:29Z [overcloud-Networks-cqf6exrqw4b4.StorageMgmtNetwork]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:29Z [overcloud-Networks-cqf6exrqw4b4-InternalApiNetwork-wo5sbj77bzez]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:30Z [overcloud-Networks-cqf6exrqw4b4-TenantNetwork-jnz4hhljbiaf]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:30Z [overcloud-Networks-cqf6exrqw4b4.ManagementNetwork]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:30Z [overcloud-Networks-cqf6exrqw4b4.InternalApiNetwork]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:30Z [overcloud-Networks-cqf6exrqw4b4-StorageMgmtNetwork-fhirddls7hla]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:30Z [overcloud-Networks-cqf6exrqw4b4.TenantNetwork]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:31Z [overcloud-Networks-cqf6exrqw4b4-ExternalNetwork-u3dqa67yxnte]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:31Z [overcloud-Networks-cqf6exrqw4b4.StorageNetwork]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:31Z [overcloud-Networks-cqf6exrqw4b4-ManagementNetwork-2twroz6aofsf]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:32Z [overcloud-Networks-cqf6exrqw4b4.ExternalNetwork]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:32Z [overcloud-Networks-cqf6exrqw4b4-StorageMgmtNetwork-fhirddls7hla]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:32Z [overcloud-Networks-cqf6exrqw4b4.StorageMgmtNetwork]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:32Z [overcloud-Networks-cqf6exrqw4b4-StorageNetwork-jama65js6y34]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:33Z [overcloud-Networks-cqf6exrqw4b4-ManagementNetwork-2twroz6aofsf]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:33Z [overcloud-Networks-cqf6exrqw4b4.ManagementNetwork]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:34Z [overcloud-Networks-cqf6exrqw4b4-StorageNetwork-jama65js6y34]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:34Z [overcloud-Networks-cqf6exrqw4b4.StorageNetwork]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:35Z [overcloud-Networks-cqf6exrqw4b4]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:36Z [Networks]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:44Z [InternalApiVirtualIP]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:44Z [StorageVirtualIP]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:44Z [PublicVirtualIP]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:44Z [RedisVirtualIP]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:44Z [StorageMgmtVirtualIP]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:46Z [overcloud-StorageVirtualIP-o62tgyevgy6p]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:46Z [overcloud-RedisVirtualIP-fwvwhkabavb7]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:47Z [overcloud-PublicVirtualIP-xmrsrngu24tk]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:47Z [overcloud-InternalApiVirtualIP-nj5p3g52pi5i]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:47Z [overcloud-StorageMgmtVirtualIP-edr34ijf5nme]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:49Z [overcloud-RedisVirtualIP-fwvwhkabavb7]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:49Z [overcloud-StorageVirtualIP-o62tgyevgy6p]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:49Z [overcloud-PublicVirtualIP-xmrsrngu24tk]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:49Z [overcloud-InternalApiVirtualIP-nj5p3g52pi5i]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:49Z [overcloud-StorageMgmtVirtualIP-edr34ijf5nme]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:49Z [StorageVirtualIP]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:50Z [RedisVirtualIP]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:50Z [PublicVirtualIP]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:50Z [InternalApiVirtualIP]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:50Z [StorageMgmtVirtualIP]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:52Z [VipMap]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:53Z [overcloud-VipMap-btlbzopfvbhw]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:53Z [overcloud-VipMap-btlbzopfvbhw]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:09:54Z [VipMap]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:54Z [EndpointMap]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:56Z [EndpointMap]: UPDATE_COMPLETE  state changed
2018-11-14 20:09:57Z [CephStorageServiceChain]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:57Z [ObjectStorageServiceChain]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:57Z [ComputeServiceChain]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:57Z [ControllerServiceChain]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:58Z [BlockStorageServiceChain]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:59Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:59Z [overcloud-ComputeServiceChain-dlan2xtjsndy]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:59Z [overcloud-ControllerServiceChain-m34umnbdos4s]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:59Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:09:59Z [overcloud-ComputeServiceChain-dlan2xtjsndy.LoggingConfiguration]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:09:59Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:10:00Z [overcloud-ControllerServiceChain-m34umnbdos4s.LoggingConfiguration]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:00Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp.ServiceChain]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:00Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy.LoggingConfiguration]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:00Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2.ServiceChain]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:01Z [overcloud-ComputeServiceChain-dlan2xtjsndy.ServiceChain]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:01Z [overcloud-ControllerServiceChain-m34umnbdos4s.LoggingConfiguration]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:01Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2.LoggingConfiguration]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:01Z [overcloud-ControllerServiceChain-m34umnbdos4s.ServiceChain]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:01Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp.LoggingConfiguration]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:01Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy.ServiceChain]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:02Z [overcloud-ComputeServiceChain-dlan2xtjsndy.LoggingConfiguration]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:03Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:10:03Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.2]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:04Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:10:04Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp.LoggingConfiguration]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:04Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2.LoggingConfiguration]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:04Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.3]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:05Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.3]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:05Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy.LoggingConfiguration]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:05Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:10:06Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.2]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:07Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.4]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:08Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.2]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:08Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.3]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:08Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.2]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:08Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.8]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:09Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.5]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:09Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.3]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:09Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.21]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:09Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.9]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:09Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.4]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:10Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:10:10Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.2]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:11Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.4]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:11Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.12]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:11Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.5]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:11Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.11]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:12Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.6]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:13Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.4]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:13Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.9]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:13Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.12]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:13Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.13]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:14Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.13]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:16Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.13]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:17Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.21]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:19Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.21]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:22Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.3]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:23Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.6]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:24Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.8]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:24Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.21]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:25Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.12]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:25Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.7]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:25Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.20]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:25Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.18]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:26Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.16]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:26Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.11]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:26Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.14]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:26Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.25]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:27Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.12]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:28Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.3]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:29Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.18]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:29Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.7]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:29Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.2]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:29Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.19]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:29Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.3]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:30Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.20]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:30Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.20]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:30Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.11]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:30Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.16]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:31Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.24]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:32Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.22]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:32Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.24]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:34Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.5]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:34Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.8]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:34Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.25]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:35Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.22]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:35Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.24]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:35Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.2]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:35Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.13]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:35Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv]: UPDATE_IN_PROGRESS  Stack UPDATE started
2018-11-14 20:10:35Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.20]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:35Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.11]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:35Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.19]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:36Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.2]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:36Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.24]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:37Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.14]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:37Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.24]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:37Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.19]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:37Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.5]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:38Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.15]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:40Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.23]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:41Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.24]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:42Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.23]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:42Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.21]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:43Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.22]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:43Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.15]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:44Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.3]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:44Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.10]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:45Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.25]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:46Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.19]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:46Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.23]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:46Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.22]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:46Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.8]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:46Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.23]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:47Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.26]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:47Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.7]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:48Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.23]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:48Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.12]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:49Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o.23]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:51Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.3]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:53Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.1]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:54Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.25]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:54Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.12]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:56Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.10]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:56Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.26]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:57Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.2]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:57Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.21]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:58Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.37]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:10:59Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.37]: UPDATE_COMPLETE  state changed
2018-11-14 20:10:59Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.3]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:02Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.4]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:02Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.14]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:02Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.16]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:04Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2-ServiceChain-bk37lz3zcr2o]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:11:05Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.38]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:06Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2.ServiceChain]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:07Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.25]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:08Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.16]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:08Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.14]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:09Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.2]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:09Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.38]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:09Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.11]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:10Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph.25]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:11Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.22]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:12Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.24]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:12Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.11]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:15Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp-ServiceChain-izbgyvwgdlph]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:11:15Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.7]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:15Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2.ExternalDeployTasks]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:16Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp.ServiceChain]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:16Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2.ExternalDeployTasks]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:22Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.1]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:23Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.11]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:29Z [overcloud-CephStorageServiceChain-jaa3jfavy3b2]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:11:29Z [CephStorageServiceChain]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:32Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.24]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:33Z [overcloud-ObjectStorageServiceChain-z6vazvpc2wtp]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:11:33Z [CephStorageServiceChainRoleData]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:33Z [ObjectStorageServiceChain]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:34Z [CephStorageServiceChainRoleData]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:34Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.4]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:35Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.22]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:42Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.12]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:42Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.35]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:42Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.26]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:44Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.13]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:44Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov.26]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:45Z [ObjectStorageServiceChainRoleData]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:45Z [ObjectStorageServiceChainRoleData]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:45Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.28]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:47Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.34]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:47Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.11]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:49Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy-ServiceChain-rwfgezvfosov]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:11:49Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.15]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:50Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy.ServiceChain]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:52Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.12]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:53Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.13]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:54Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.28]: UPDATE_COMPLETE  state changed
2018-11-14 20:11:57Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.16]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:57Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.36]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:11:58Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.34]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:00Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.17]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:00Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.35]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:01Z [overcloud-BlockStorageServiceChain-wnm3paudzxdy]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:12:01Z [BlockStorageServiceChain]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:02Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.15]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:06Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.16]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:07Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.36]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:08Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.40]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:09Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.39]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:10Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.19]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:10Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.40]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:10Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.17]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:12Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.46]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:12Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.20]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:14Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.46]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:15Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.21]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:16Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.39]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:16Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.19]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:18Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.20]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:21Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.23]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:22Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.54]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:25Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.51]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:33Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.54]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:35Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.36]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:36Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.23]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:38Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.58]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:38Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.51]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:39Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.21]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:40Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.27]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:40Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.58]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:42Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.27]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:42Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.55]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:44Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.35]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:46Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.35]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:47Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.39]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:48Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.36]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:51Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde.39]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:51Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.70]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:12:55Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.70]: UPDATE_COMPLETE  state changed
2018-11-14 20:12:58Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.55]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:04Z [overcloud-ComputeServiceChain-dlan2xtjsndy-ServiceChain-7rqfsyclhlde]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:13:04Z [overcloud-ComputeServiceChain-dlan2xtjsndy.ServiceChain]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:05Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.52]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:08Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.82]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:08Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.52]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:11Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.59]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:12Z [overcloud-ComputeServiceChain-dlan2xtjsndy.ExternalDeployTasks]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:12Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.56]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:13Z [overcloud-ComputeServiceChain-dlan2xtjsndy.ExternalDeployTasks]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:15Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.53]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:17Z [overcloud-ComputeServiceChain-dlan2xtjsndy]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:13:19Z [ComputeServiceChain]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:19Z [ComputeServiceChainRoleData]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:20Z [ComputeServiceChainRoleData]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:20Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.82]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:24Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.59]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:24Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.53]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:27Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.90]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:27Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.56]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:30Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.90]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:30Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.57]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:33Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.94]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:39Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.60]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:41Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.71]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:42Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.94]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:46Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.57]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:46Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.98]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:49Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.61]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:13:51Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.60]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:51Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.71]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:55Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.61]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:58Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.98]: UPDATE_COMPLETE  state changed
2018-11-14 20:13:59Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.68]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:14:06Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.102]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:14:07Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.91]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:14:14Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.68]: UPDATE_COMPLETE  state changed
2018-11-14 20:14:18Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.91]: UPDATE_COMPLETE  state changed
2018-11-14 20:14:19Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.89]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:14:19Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.102]: UPDATE_COMPLETE  state changed
2018-11-14 20:14:27Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.95]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:14:30Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.114]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:14:34Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.89]: UPDATE_COMPLETE  state changed
2018-11-14 20:14:35Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.88]: UPDATE_IN_PROGRESS  state changed
2018-11-14 20:14:36Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.95]: UPDATE_COMPLETE  state changed
2018-11-14 20:14:38Z [overcloud-ControllerServiceChain-m34umnbdos4s-ServiceChain-zqvgvqjhhrsv.88]: UPDATE_COMPLETE  state changed
...
018-11-14 20:20:26Z [overcloud-AllNodesDeploySteps-x7kqytsrgfi2.ObjectStorageExtraConfigPost]: UPDATE_COMPLETE  state changed
2018-11-14 20:20:26Z [overcloud-AllNodesDeploySteps-x7kqytsrgfi2.CephStorageExtraConfigPost]: UPDATE_COMPLETE  state changed
2018-11-14 20:20:26Z [overcloud-AllNodesDeploySteps-x7kqytsrgfi2.ComputeExtraConfigPost]: UPDATE_COMPLETE  state changed
2018-11-14 20:20:27Z [overcloud-AllNodesDeploySteps-x7kqytsrgfi2.BlockStorageExtraConfigPost]: UPDATE_COMPLETE  state changed
2018-11-14 20:20:27Z [overcloud-AllNodesDeploySteps-x7kqytsrgfi2.ControllerExtraConfigPost]: UPDATE_COMPLETE  state changed
2018-11-14 20:20:33Z [overcloud-AllNodesDeploySteps-x7kqytsrgfi2]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-11-14 20:20:34Z [AllNodesDeploySteps]: UPDATE_COMPLETE  state changed
2018-11-14 20:20:39Z [overcloud]: UPDATE_COMPLETE  Stack UPDATE completed successfully

 Stack overcloud/11249161-68e9-4a47-8fd4-db2a62cb4447 UPDATE_COMPLETE 

Deploying overcloud configuration
Enabling ssh admin (tripleo-admin) for hosts:
192.168.24.9 192.168.24.15 192.168.24.6 192.168.24.8 192.168.24.13 192.168.24.11 192.168.24.10 192.168.24.14
Using ssh user heat-admin for initial connection.
Using ssh key at /home/stack/.ssh/id_rsa for initial connection.
Generating public/private rsa key pair.
Your identification has been saved in /tmp/tmpg2WNch/id_rsa.
Your public key has been saved in /tmp/tmpg2WNch/id_rsa.pub.
The key fingerprint is:
SHA256:l1Q0QX6vT0CF+FJAew2+8ov+EuXQr/cuPeESuNPL4Rg TripleO split stack short term key
The key's randomart image is:
+---[RSA 4096]----+
|           +O+...|
|           o.+o+ |
...



PLAY [Server Post Deployments] *************************************************

TASK [include_tasks] ***********************************************************
Wednesday 14 November 2018  16:00:06 -0500 (0:00:00.318)       0:37:08.051 **** 

PLAY [External deployment Post Deploy tasks] ***********************************

PLAY RECAP *********************************************************************
ceph-0                     : ok=147  changed=55   unreachable=0    failed=0   
ceph-1                     : ok=146  changed=55   unreachable=0    failed=0   
ceph-2                     : ok=146  changed=55   unreachable=0    failed=0   
compute-0                  : ok=167  changed=69   unreachable=0    failed=0   
compute-1                  : ok=168  changed=69   unreachable=0    failed=0   
controller-0               : ok=221  changed=93   unreachable=0    failed=0   
controller-1               : ok=215  changed=92   unreachable=0    failed=0   
controller-2               : ok=215  changed=92   unreachable=0    failed=0   
undercloud                 : ok=40   changed=17   unreachable=0    failed=0   

Wednesday 14 November 2018  16:00:07 -0500 (0:00:00.359)       0:37:08.411 **** 
=============================================================================== 

Ansible passed.
Overcloud configuration completed.
Waiting for messages on queue 'tripleo' with no timeout.
Overcloud Endpoint: https://10.0.0.101:13000
Overcloud Horizon Dashboard URL: https://10.0.0.101:443/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed

Comment 28 errata-xmlrpc 2019-01-11 11:50:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045

Comment 29 Red Hat Bugzilla 2023-09-14 04:31:56 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.