Attachment 1452949 Details for Bug 1575912 – Amphora shutoff -> loadbalancer ONLINE -> Amphora not restarted

Amphora shutoff -> loadbalancer ONLINE -> Amphora not restarted

Amphora VM shutoff does not update loadbalancer and octavia service status.txt (text/plain), 25.60 KB, created by Noam Manos on 2018-06-19 13:48:54 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Noam Manos

Created: 2018-06-19 13:48:54 UTC

Size: 25.60 KB

patch

obsolete

>
>
>(overcloud) [stack@undercloud-0 ~]$ . stackrc
>(undercloud) [stack@undercloud-0 ~]$ openstack server list --all
>+--------------------------------------+--------------+--------+------------------------+----------------+------------+
>| ID                                   | Name         | Status | Networks               | Image          | Flavor     |
>+--------------------------------------+--------------+--------+------------------------+----------------+------------+
>| 885d2c04-e2da-4504-bc38-a1c7d0465bc8 | controller-0 | ACTIVE | ctlplane=192.168.24.17 | overcloud-full | controller |
>| fadab01e-6888-49cd-bf55-0faf0672d5f7 | compute-1    | ACTIVE | ctlplane=192.168.24.12 | overcloud-full | compute    |
>| 1ef0778f-1e78-4f75-a04a-a381050324ad | compute-0    | ACTIVE | ctlplane=192.168.24.6  | overcloud-full | compute    |
>| 417343a8-b9cb-41f0-985b-0ecc8253187c | controller-1 | ACTIVE | ctlplane=192.168.24.16 | overcloud-full | controller |
>| f90ce67b-d009-4c8e-8525-c5b8dedc97c9 | controller-2 | ACTIVE | ctlplane=192.168.24.9  | overcloud-full | controller |
>+--------------------------------------+--------------+--------+------------------------+----------------+------------+
>
>
>(undercloud) [stack@undercloud-0 ~]$ . overcloudrc
>(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list
>
>
>[08:06:08.598] (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer create --name LB-0 --vip-subnet-id $int_subnet_id
>+---------------------+--------------------------------------+
>| Field               | Value                                |
>+---------------------+--------------------------------------+
>| admin_state_up      | True                                 |
>| created_at          | 2018-06-19T12:06:34                  |
>| description         |                                      |
>| flavor              |                                      |
>| id                  | ccbc588d-ccc0-4f4a-8b82-55de8a941450 |
>| listeners           |                                      |
>| name                | LB-0                                 |
>| operating_status    | OFFLINE                              |
>| pools               |                                      |
>| project_id          | 8ea5e0946ce54d448df01a77d058b2df     |
>| provider            | octavia                              |
>| provisioning_status | PENDING_CREATE                       |
>| updated_at          | None                                 |
>| vip_address         | 192.168.2.33                         |
>| vip_network_id      | fa6b2bdc-8f64-49b2-9f60-5215deb3ca2c |
>| vip_port_id         | 000d2651-aafa-4b5b-96a2-97d59c2e49ae |
>| vip_qos_policy_id   | None                                 |
>| vip_subnet_id       | aa1f6061-a47e-4158-b055-ac239442d02e |
>+---------------------+--------------------------------------+
>[08:06:40.339] (overcloud) [stack@undercloud-0 ~]$ watch -e "! openstack loadbalancer amphora list | grep -E 'STANDALONE|BACKUP|MASTER' -B 3"
>[08:07:37.564] (overcloud) [stack@undercloud-0 ~]$ ssh heat-admin@controller-0
>Last login: Tue Jun 19 08:38:44 2018 from 192.168.24.1
>[heat-admin@controller-0 ~]$ sudo -i
>[root@controller-0 ~]# tail /var/log/containers/octavia/worker.log
>2018-06-19 08:42:46.750 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:42:47.759 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 12 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:42:59.776 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:00.786 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:01.800 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 14 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:15.822 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:16.831 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:17.842 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 16 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:33.870 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:34.915 23 INFO oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] Reconnected to AMQP server on controller-0.internalapi.localdomain:5672 via [amqp] client with port 37582.
>[root@controller-0 ~]#
>[root@controller-0 ~]#
>[root@controller-0 ~]# logout
>[heat-admin@controller-0 ~]$ logout
>Connection to controller-0 closed.
>[08:09:11.411] (overcloud) [stack@undercloud-0 ~]$ ssh heat-admin@controller-1 "sudo tail /var/log/containers/octavia/worker.log"
>2018-06-18 10:45:04.896 23 INFO octavia.controller.queue.endpoint [-] Deleting member '0ec041e7-57b8-4eac-9d32-9b294351f2ea'...
>2018-06-18 10:45:21.433 23 INFO octavia.controller.queue.endpoint [req-3378ef59-3dba-415f-9f66-8e380052e99e - 2e83123c71d6430dbdf9228b6733cb89 - - -] Deleting pool '2bc9fbce-518a-4131-9346-601ae57b0051'...
>2018-06-18 12:36:26.228 23 INFO octavia.controller.queue.endpoint [-] Deleting load balancer '9a9d3369-b7cc-45b3-948f-bcc174f55b2d'...
>2018-06-19 08:43:15.200 1 INFO octavia.common.config [-] Logging enabled!
>2018-06-19 08:43:15.203 1 INFO octavia.common.config [-] /usr/bin/octavia-worker version 2.0.1
>2018-06-19 08:43:15.237 23 INFO octavia.controller.queue.consumer [-] Starting consumer...
>2018-06-19 08:43:21.463 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [4f2ce345-94f7-49a0-b74a-e5d879884b15] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:22.484 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [4f2ce345-94f7-49a0-b74a-e5d879884b15] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:23.517 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [4f2ce345-94f7-49a0-b74a-e5d879884b15] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:24.602 23 INFO oslo.messaging._drivers.impl_rabbit [-] [4f2ce345-94f7-49a0-b74a-e5d879884b15] Reconnected to AMQP server on controller-0.internalapi.localdomain:5672 via [amqp] client with port 44094.
>[08:10:17.042] (overcloud) [stack@undercloud-0 ~]$
>[08:10:21.012] (overcloud) [stack@undercloud-0 ~]$
>[08:10:21.204] (overcloud) [stack@undercloud-0 ~]$
>[08:10:21.404] (overcloud) [stack@undercloud-0 ~]$
>[08:10:21.581] (overcloud) [stack@undercloud-0 ~]$
>[08:10:21.732] (overcloud) [stack@undercloud-0 ~]$ ssh heat-admin@controller-2 "sudo tail /var/log/containers/octavia/worker.log"
>2018-06-19 12:06:39.673 23 INFO octavia.certificates.generator.local [-] Signing a certificate request using OpenSSL locally.
>2018-06-19 12:06:39.673 23 INFO octavia.certificates.generator.local [-] Using CA Certificate from config.
>2018-06-19 12:06:39.674 23 INFO octavia.certificates.generator.local [-] Using CA Private Key from config.
>2018-06-19 12:06:39.674 23 INFO octavia.certificates.generator.local [-] Using CA Private Key Passphrase from config.
>2018-06-19 12:06:42.392 23 WARNING octavia.compute.drivers.nova_driver [-] A single Glance image should be tagged with amphora-image tag, but at least two were found. Using 44125cc1-a0b1-4103-9b1b-3b2b31d30e1a.
>2018-06-19 12:07:26.276 23 INFO octavia.controller.worker.tasks.database_tasks [-] Mark ALLOCATED in DB for amphora: 1b1958c8-d52d-460c-929a-cd03878a3a77 with compute id d5d89c58-9c1d-4d4d-99f4-93b62fa6a5b6 for load balancer: ccbc588d-ccc0-4f4a-8b82-55de8a941450
>2018-06-19 12:07:28.112 23 INFO octavia.network.drivers.neutron.allowed_address_pairs [-] Port 000d2651-aafa-4b5b-96a2-97d59c2e49ae already exists. Nothing to be done.
>2018-06-19 12:07:46.187 23 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: ConnectionError: HTTPSConnectionPool(host='172.24.0.15', port=9443): Max retries exceeded with url: /0.5/plug/vip/192.168.2.33 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f5294548290>: Failed to establish a new connection: [Errno 113] No route to host',))
>2018-06-19 12:07:51.192 23 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: ConnectionError: HTTPSConnectionPool(host='172.24.0.15', port=9443): Max retries exceeded with url: /0.5/plug/vip/192.168.2.33 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f5294548350>: Failed to establish a new connection: [Errno 111] Connection refused',))
>2018-06-19 12:08:03.452 23 INFO octavia.controller.worker.tasks.database_tasks [-] Mark ACTIVE in DB for load balancer id: ccbc588d-ccc0-4f4a-8b82-55de8a941450
>[08:10:27.437] (overcloud) [stack@undercloud-0 ~]$
>[08:12:27.065] (overcloud) [stack@undercloud-0 ~]$
>[08:12:27.414] (overcloud) [stack@undercloud-0 ~]$
>[08:12:27.631] (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list
>+--------------------------------------+--------------------------------------+-----------+------------+---------------+--------------+
>| id                                   | loadbalancer_id                      | status    | role       | lb_network_ip | ha_ip        |
>+--------------------------------------+--------------------------------------+-----------+------------+---------------+--------------+
>| 1b1958c8-d52d-460c-929a-cd03878a3a77 | ccbc588d-ccc0-4f4a-8b82-55de8a941450 | ALLOCATED | STANDALONE | 172.24.0.15   | 192.168.2.33 |
>+--------------------------------------+--------------------------------------+-----------+------------+---------------+--------------+
>
>[08:12:41.427] (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list
>+--------------------------------------+------+----------------------------------+--------------+---------------------+----------+
>| id                                   | name | project_id                       | vip_address  | provisioning_status | provider |
>+--------------------------------------+------+----------------------------------+--------------+---------------------+----------+
>| ccbc588d-ccc0-4f4a-8b82-55de8a941450 | LB-0 | 8ea5e0946ce54d448df01a77d058b2df | 192.168.2.33 | ACTIVE              | octavia  |
>+--------------------------------------+------+----------------------------------+--------------+---------------------+----------+
>
>[09:24:17.238] (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer show LB-0
>+---------------------+--------------------------------------+
>| Field               | Value                                |
>+---------------------+--------------------------------------+
>| admin_state_up      | True                                 |
>| created_at          | 2018-06-19T12:06:34                  |
>| description         |                                      |
>| flavor              |                                      |
>| id                  | ccbc588d-ccc0-4f4a-8b82-55de8a941450 |
>| listeners           |                                      |
>| name                | LB-0                                 |
>| operating_status    | ONLINE                               |
>| pools               |                                      |
>| project_id          | 8ea5e0946ce54d448df01a77d058b2df     |
>| provider            | octavia                              |
>| provisioning_status | ACTIVE                               |
>| updated_at          | 2018-06-19T12:08:03                  |
>| vip_address         | 192.168.2.33                         |
>| vip_network_id      | fa6b2bdc-8f64-49b2-9f60-5215deb3ca2c |
>| vip_port_id         | 000d2651-aafa-4b5b-96a2-97d59c2e49ae |
>| vip_qos_policy_id   | None                                 |
>| vip_subnet_id       | aa1f6061-a47e-4158-b055-ac239442d02e |
>+---------------------+--------------------------------------+
>
>[09:25:48.138] (overcloud) [stack@undercloud-0 ~]$ openstack server list --all
>+--------------------------------------+----------------------------------------------+--------+-----------------------------------------------+---------------------------------+--------+
>| ID                                   | Name                                         | Status | Networks                                      | Image                           | Flavor |
>+--------------------------------------+----------------------------------------------+--------+-----------------------------------------------+---------------------------------+--------+
>| d5d89c58-9c1d-4d4d-99f4-93b62fa6a5b6 | amphora-1b1958c8-d52d-460c-929a-cd03878a3a77 | ACTIVE | int_net=192.168.2.46; lb-mgmt-net=172.24.0.15 | octavia-amphora-13.0-20180608.1 |        |
>| 8d59874e-ef2f-4433-9c02-1a96ba1cd893 | vm-rht-2                                     | ACTIVE | int_net=192.168.2.43, 10.0.0.218              | rhel75                          | rhel7  |
>| 504d3190-f94e-4910-aac0-cf5f4efde0c8 | vm-rht-1                                     | ACTIVE | int_net=192.168.2.40, 10.0.0.210              | rhel75                          | rhel7  |
>+--------------------------------------+----------------------------------------------+--------+-----------------------------------------------+---------------------------------+--------+
>
>[09:27:20.781] (overcloud) [stack@undercloud-0 ~]$ openstack server stop d5d89c58-9c1d-4d4d-99f4-93b62fa6a5b6
>
>[09:27:33.409] (overcloud) [stack@undercloud-0 ~]$ openstack server list --all
>+--------------------------------------+----------------------------------------------+---------+-----------------------------------------------+---------------------------------+--------+
>| ID                                   | Name                                         | Status  | Networks                                      | Image                           | Flavor |
>+--------------------------------------+----------------------------------------------+---------+-----------------------------------------------+---------------------------------+--------+
>| d5d89c58-9c1d-4d4d-99f4-93b62fa6a5b6 | amphora-1b1958c8-d52d-460c-929a-cd03878a3a77 | SHUTOFF | int_net=192.168.2.46; lb-mgmt-net=172.24.0.15 | octavia-amphora-13.0-20180608.1 |        |
>| 8d59874e-ef2f-4433-9c02-1a96ba1cd893 | vm-rht-2                                     | ACTIVE  | int_net=192.168.2.43, 10.0.0.218              | rhel75                          | rhel7  |
>| 504d3190-f94e-4910-aac0-cf5f4efde0c8 | vm-rht-1                                     | ACTIVE  | int_net=192.168.2.40, 10.0.0.210              | rhel75                          | rhel7  |
>+--------------------------------------+----------------------------------------------+---------+-----------------------------------------------+---------------------------------+--------+
>
>[09:27:44.823] (overcloud) [stack@undercloud-0 ~]$ ssh heat-admin@controller-2 "sudo tail -f /var/log/containers/octavia/worker.log"
>2018-06-19 12:06:39.673 23 INFO octavia.certificates.generator.local [-] Signing a certificate request using OpenSSL locally.
>2018-06-19 12:06:39.673 23 INFO octavia.certificates.generator.local [-] Using CA Certificate from config.
>2018-06-19 12:06:39.674 23 INFO octavia.certificates.generator.local [-] Using CA Private Key from config.
>2018-06-19 12:06:39.674 23 INFO octavia.certificates.generator.local [-] Using CA Private Key Passphrase from config.
>2018-06-19 12:06:42.392 23 WARNING octavia.compute.drivers.nova_driver [-] A single Glance image should be tagged with amphora-image tag, but at least two were found. Using 44125cc1-a0b1-4103-9b1b-3b2b31d30e1a.
>2018-06-19 12:07:26.276 23 INFO octavia.controller.worker.tasks.database_tasks [-] Mark ALLOCATED in DB for amphora: 1b1958c8-d52d-460c-929a-cd03878a3a77 with compute id d5d89c58-9c1d-4d4d-99f4-93b62fa6a5b6 for load balancer: ccbc588d-ccc0-4f4a-8b82-55de8a941450
>2018-06-19 12:07:28.112 23 INFO octavia.network.drivers.neutron.allowed_address_pairs [-] Port 000d2651-aafa-4b5b-96a2-97d59c2e49ae already exists. Nothing to be done.
>2018-06-19 12:07:46.187 23 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: ConnectionError: HTTPSConnectionPool(host='172.24.0.15', port=9443): Max retries exceeded with url: /0.5/plug/vip/192.168.2.33 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f5294548290>: Failed to establish a new connection: [Errno 113] No route to host',))
>2018-06-19 12:07:51.192 23 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: ConnectionError: HTTPSConnectionPool(host='172.24.0.15', port=9443): Max retries exceeded with url: /0.5/plug/vip/192.168.2.33 (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f5294548350>: Failed to establish a new connection: [Errno 111] Connection refused',))
>2018-06-19 12:08:03.452 23 INFO octavia.controller.worker.tasks.database_tasks [-] Mark ACTIVE in DB for load balancer id: ccbc588d-ccc0-4f4a-8b82-55de8a941450
>
>
>[09:28:26.472] (overcloud) [stack@undercloud-0 ~]$ ssh heat-admin@controller-1 "sudo tail -f /var/log/containers/octavia/worker.log"
>2018-06-19 12:15:03.440 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/controller/worker/tasks/network_tasks.py", line 363, in execute
>2018-06-19 12:15:03.440 23 ERROR oslo_messaging.rpc.server     self.network_driver.deallocate_vip(vip)
>2018-06-19 12:15:03.440 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 298, in deallocate_vip
>2018-06-19 12:15:03.440 23 ERROR oslo_messaging.rpc.server     self._delete_security_group(vip, port)
>2018-06-19 12:15:03.440 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 274, in _delete_security_group
>2018-06-19 12:15:03.440 23 ERROR oslo_messaging.rpc.server     self._delete_vip_security_group(sec_grp)
>2018-06-19 12:15:03.440 23 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 244, in _delete_vip_security_group
>2018-06-19 12:15:03.440 23 ERROR oslo_messaging.rpc.server     raise base.DeallocateVIPException(message)
>2018-06-19 12:15:03.440 23 ERROR oslo_messaging.rpc.server DeallocateVIPException: All attempts to remove security group 49ce052a-b930-40c7-a865-db283dfd4b4e have failed.
>2018-06-19 12:15:03.440 23 ERROR oslo_messaging.rpc.server
>
>
>[09:28:53.016] (overcloud) [stack@undercloud-0 ~]$ ssh heat-admin@controller-0 "sudo tail -f /var/log/containers/octavia/worker.log"
>2018-06-19 08:42:46.750 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:42:47.759 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 12 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:42:59.776 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:00.786 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:01.800 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 14 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:15.822 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:16.831 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:17.842 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 16 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:33.870 23 ERROR oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused
>2018-06-19 08:43:34.915 23 INFO oslo.messaging._drivers.impl_rabbit [-] [596ef96e-7af1-428c-9a64-d5fdf763cb25] Reconnected to AMQP server on controller-0.internalapi.localdomain:5672 via [amqp] client with port 37582.
>
>
>[09:29:14.810] (overcloud) [stack@undercloud-0 ~]$
>[09:29:15.390] (overcloud) [stack@undercloud-0 ~]$
>[09:29:15.590] (overcloud) [stack@undercloud-0 ~]$
>[09:29:15.759] (overcloud) [stack@undercloud-0 ~]$
>[09:29:15.911] (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer amphora list
>+--------------------------------------+--------------------------------------+--------+------------+---------------+--------------+
>| id                                   | loadbalancer_id                      | status | role       | lb_network_ip | ha_ip        |
>+--------------------------------------+--------------------------------------+--------+------------+---------------+--------------+
>| 1b1958c8-d52d-460c-929a-cd03878a3a77 | ccbc588d-ccc0-4f4a-8b82-55de8a941450 | ERROR  | STANDALONE | 172.24.0.15   | 192.168.2.33 |
>+--------------------------------------+--------------------------------------+--------+------------+---------------+--------------+
>
>[09:29:39.498] (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer show LB-0
>+---------------------+--------------------------------------+
>| Field               | Value                                |
>+---------------------+--------------------------------------+
>| admin_state_up      | True                                 |
>| created_at          | 2018-06-19T12:06:34                  |
>| description         |                                      |
>| flavor              |                                      |
>| id                  | ccbc588d-ccc0-4f4a-8b82-55de8a941450 |
>| listeners           |                                      |
>| name                | LB-0                                 |
>| operating_status    | ONLINE                               |
>| pools               |                                      |
>| project_id          | 8ea5e0946ce54d448df01a77d058b2df     |
>| provider            | octavia                              |
>| provisioning_status | ERROR                                |
>| updated_at          | 2018-06-19T13:29:21                  |
>| vip_address         | 192.168.2.33                         |
>| vip_network_id      | fa6b2bdc-8f64-49b2-9f60-5215deb3ca2c |
>| vip_port_id         | 000d2651-aafa-4b5b-96a2-97d59c2e49ae |
>| vip_qos_policy_id   | None                                 |
>| vip_subnet_id       | aa1f6061-a47e-4158-b055-ac239442d02e |
>+---------------------+--------------------------------------+
>[09:30:16.554] (overcloud) [stack@undercloud-0 ~]$

Actions: View

Attachments on bug 1575912: 1452949