Bug 1607567

Summary:	Upgrading existing OSP13 installation to add Octavia provides "update_complete" but Octavia not installed
Product:	Red Hat OpenStack	Reporter:	rlopez
Component:	rhosp-director	Assignee:	RHOS Maint <rhos-maint>
Status:	CLOSED WORKSFORME	QA Contact:	Gurenko Alex <agurenko>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	13.0 (Queens)	CC:	cgoncalves, dbecker, dpeacock, jslagle, mburns, morazi, rlopez
Target Milestone:	---	Keywords:	ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-08-24 13:42:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description rlopez 2018-07-23 18:47:08 UTC

Description of problem:

When attempting to update a successful deployment of OSP13 with Octavia, the installation procedure states it is successful, however, Octavia is not enabled. 

Deleting the overcloud installation and doing a fresh install with including Octavia did produce a successful installation. 


Steps to reproduce:

Install your overcloud without Octavia.

Example:

openstack overcloud deploy --templates \
 -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible-external.yaml \
 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
 -e /home/stack/custom_templates/node-info.yaml \
 -e /home/stack/custom_templates/overcloud_images.yaml \
 -e /home/stack/custom_templates/network_environment.yaml \
 -n /home/stack/custom_templates/network_data.yaml \
 -e /home/stack/custom_templates/ceph_external_environment.yaml


The above assumes a successful OSP13 install without Octavia.


Next, ensure the following:

* yum install octavia-amphora-image

* This line is part of your network-environment.yaml
OctaviaAmphoraImageFilename: '/usr/share/rhosp-director-images/octavia-amphora.qcow2'

* Octavia is part of your local registry images (if using local registry) local_registry_images.yaml

Attempt to update your overcloud with Octavia:

openstack overcloud deploy --templates \
 -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible-external.yaml \
 -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
 -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
 -e /home/stack/custom_templates/node-info.yaml \
 -e /home/stack/custom_templates/overcloud_images.yaml \
 -e /home/stack/custom_templates/network_environment.yaml \
 -n /home/stack/custom_templates/network_data.yaml \
 -e /home/stack/custom_templates/ceph_external_environment.yaml


This will provide an update complete. However, the following error will be seen when attempting openstack loadbalancer list

$ openstack loadbalancer list
Unable to establish connection to http://10.19.114.43:9876/v2.0/lbaas/loadbalancers: HTTPConnectionPool(host='10.19.114.43', port=9876): Max retries exceeded with url: /v2.0/lbaas/loadbalancers (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f07dc5cfa90>: Failed to establish a new connection: [Errno 110] Connection timed out',))


Looking at the controllers, I did not see any ocatvia containers.

Workaround:

* Delete existing overcloud
* Redeploy overcloud with Ocatvia fresh install

Result: Successful overcloud with Ocatvia

Comment 1 Nir Magnezi 2018-08-01 13:36:55 UTC

Hi,

This was a known GA bug, that was fixed in Z1.
Could you please share the version that you were using?

Comment 2 rlopez 2018-08-01 14:44:31 UTC

Nir,

To my knowledge GA. Here is a print out of the versions from my director node

]# rpm -qa | grep openstack
openstack-swift-container-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-nova-common-17.0.3-0.20180420001141.el7ost.noarch
openstack-neutron-ml2-12.0.2-0.20180421011364.0ec54fd.el7ost.noarch
openstack-mistral-engine-6.0.2-1.el7ost.noarch
openstack-neutron-12.0.2-0.20180421011364.0ec54fd.el7ost.noarch
openstack-heat-api-cfn-10.0.1-0.20180411125640.el7ost.noarch
openstack-ironic-staging-drivers-0.9.0-4.el7ost.noarch
openstack-nova-conductor-17.0.3-0.20180420001141.el7ost.noarch
openstack-tripleo-puppet-elements-8.0.0-2.el7ost.noarch
openstack-nova-compute-17.0.3-0.20180420001141.el7ost.noarch
python2-openstackclient-3.14.1-1.el7ost.noarch
openstack-neutron-common-12.0.2-0.20180421011364.0ec54fd.el7ost.noarch
openstack-ironic-api-10.1.2-4.el7ost.noarch
openstack-ironic-inspector-7.2.1-0.20180409163360.el7ost.noarch
openstack-tripleo-ui-8.3.1-3.el7ost.noarch
openstack-zaqar-6.0.1-1.el7ost.noarch
openstack-tripleo-image-elements-8.0.1-1.el7ost.noarch
openstack-selinux-0.8.14-12.el7ost.noarch
python-openstackclient-lang-3.14.1-1.el7ost.noarch
python2-openstacksdk-0.11.3-1.el7ost.noarch
puppet-openstack_extras-12.4.1-0.20180413042250.2634296.el7ost.noarch
puppet-openstacklib-12.4.0-0.20180329042555.4b30e6f.el7ost.noarch
openstack-tripleo-heat-templates-8.0.2-43.el7ost.noarch
openstack-swift-object-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-neutron-openvswitch-12.0.2-0.20180421011364.0ec54fd.el7ost.noarch
openstack-swift-proxy-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-heat-common-10.0.1-0.20180411125640.el7ost.noarch
openstack-ironic-conductor-10.1.2-4.el7ost.noarch
openstack-mistral-api-6.0.2-1.el7ost.noarch
openstack-tripleo-validations-8.4.1-5.el7ost.noarch
openstack-keystone-13.0.1-0.20180420194847.7bd6454.el7ost.noarch
openstack-heat-api-10.0.1-0.20180411125640.el7ost.noarch
openstack-tripleo-common-8.6.1-23.el7ost.noarch
openstack-nova-placement-api-17.0.3-0.20180420001141.el7ost.noarch
openstack-swift-account-2.17.1-0.20180314165245.caeeb54.el7ost.noarch
openstack-glance-16.0.1-2.el7ost.noarch
openstack-ironic-common-10.1.2-4.el7ost.noarch
openstack-mistral-executor-6.0.2-1.el7ost.noarch
openstack-tripleo-common-containers-8.6.1-23.el7ost.noarch
openstack-heat-engine-10.0.1-0.20180411125640.el7ost.noarch
openstack-tempest-18.0.0-2.el7ost.noarch
openstack-mistral-common-6.0.2-1.el7ost.noarch
openstack-nova-api-17.0.3-0.20180420001141.el7ost.noarch
openstack-nova-scheduler-17.0.3-0.20180420001141.el7ost.noarch

Comment 3 Carlos Goncalves 2018-08-15 14:11:47 UTC

Have you prepared the container images for Octavia before deploying overcloud?

For documentation on how to do it, please refer to https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/transitioning_to_containerized_services/

Make sure the local registry lists the Octavia docker images:

[stack@undercloud-0 ~]$ curl 192.168.24.1:8787/v2/_catalog | grep octavia
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1292  100  1292    0     0   175k      0 --:--:-- --:--:-- --:--:--  210k
{"repositories":["rhosp13/openstack-cinder-api","rhosp13/openstack-cinder-scheduler","rhosp13/openstack-cinder-volume","rhosp13/openstack-cron","rhosp13/openstack-glance-api","rhosp13/openstack-haproxy","rhosp13/openstack-heat-api","rhosp13/openstack-heat-api-cfn","rhosp13/openstack-heat-engine","rhosp13/openstack-horizon","rhosp13/openstack-iscsid","rhosp13/openstack-keystone","rhosp13/openstack-mariadb","rhosp13/openstack-memcached","rhosp13/openstack-neutron-metadata-agent-ovn","rhosp13/openstack-neutron-server-ovn","rhosp13/openstack-nova-api","rhosp13/openstack-nova-compute","rhosp13/openstack-nova-conductor","rhosp13/openstack-nova-consoleauth","rhosp13/openstack-nova-libvirt","rhosp13/openstack-nova-novncproxy","rhosp13/openstack-nova-placement-api","rhosp13/openstack-nova-scheduler","rhosp13/openstack-octavia-api","rhosp13/openstack-octavia-health-manager","rhosp13/openstack-octavia-housekeeping","rhosp13/openstack-octavia-worker","rhosp13/openstack-ovn-controller","rhosp13/openstack-ovn-nb-db-server","rhosp13/openstack-ovn-northd","rhosp13/openstack-ovn-sb-db-server","rhosp13/openstack-rabbitmq","rhosp13/openstack-redis","rhosp13/openstack-swift-account","rhosp13/openstack-swift-container","rhosp13/openstack-swift-object","rhosp13/openstack-swift-proxy-server"]}

Comment 4 rlopez 2018-08-15 14:18:43 UTC

Hi Carlos,

Yes I updated the container images for Octavia before deploying overcloud. I never got the change to confirm the curl you mentioned here. 


What I did:

1) I had already prepared the container image before deploying overcloud, I just didn't call it out on my overcloud deploy as at the time I was attempting just to get a working OSP13 with the basics.

2) After overcloud comeplete, went ahead and made the changes in my network-environment.yaml as I listed above and then added the  -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml to my deploy command.

3) Re-ran the deploy command which said successful but no Octavia was deployed.

4) deleted my overcloud 

5) Re-ran the same exact command and this time Octavia was successfully installed.

Comment 5 Carlos Goncalves 2018-08-16 13:24:33 UTC

The steps you performed seem exactly like the steps I also did some weeks ago when I tried to enable Octavia on an existing overcloud with the same result: deployment succeeded but Octavia containers were not running; deleting and deploying again worked. My problem was that the octavia container images had not been prepared *and* uploaded to the local registry. After uploading them, I could update an existing overcloud. So, I'd ask you to kindly double-check if you uploaded them to the local registry with the curl command I mentioned before.

BTW, you don't need to manually install the octavia-amphora-image RPM and set OctaviaAmphoraImageFilename. Director can do that automatically.

Comment 6 rlopez 2018-08-16 13:55:47 UTC

Hi Carlos,

At this point, I can't confirm it because my environment is currently working with Octavia. What I can tell you is that my prepare command did have Octavia in it and the local_registry_images.yaml had Octavia in it. However, there is no way for me to confirm the curl command as I'm no longer at that step. 

This was the command: 

#   openstack overcloud container image prepare -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/ironic.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/manila.yaml --namespace=registry.access.redhat.com/rhosp13 --push-destination=192.3.2.1:8787 --prefix=openstack- --tag-from-label {version}-{release} --output-env-file=/home/stack/templates/overcloud_images.yaml --output-images-file /home/stack/local_registry_images.yaml
#

Good to know about the RPM and setting the image file name.

Comment 7 Carlos Goncalves 2018-08-24 13:42:00 UTC

It is likely that while you prepared the container images, you did not upload them to the local registry (openstack overcloud container image upload).

I tried to reproduce on a fresh deployment without Octavia enabled. I could not reproduce. See below the step-by-step on how I did it, following closely to our official documentation (https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/transitioning_to_containerized_services/).

Please feel free to reopen if you happen to run again into this issue.

----------------------------------------
Fresh deployment without Octavia enabled
----------------------------------------

(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list
public endpoint for load-balancer service not found

(undercloud) [stack@undercloud-0 ~]$ curl 192.168.24.1:8787/v2/_catalog
{"repositories":["rhosp13/openstack-cinder-api","rhosp13/openstack-cinder-scheduler","rhosp13/openstack-cinder-volume","rhosp13/openstack-cron","rhosp13/openstack-glance-api","rhosp13/openstack-haproxy","rhosp13/openstack-heat-api","rhosp13/openstack-heat-api-cfn","rhosp13/openstack-heat-engine","rhosp13/openstack-horizon","rhosp13/openstack-iscsid","rhosp13/openstack-keystone","rhosp13/openstack-mariadb","rhosp13/openstack-memcached","rhosp13/openstack-neutron-dhcp-agent","rhosp13/openstack-neutron-l3-agent","rhosp13/openstack-neutron-metadata-agent","rhosp13/openstack-neutron-openvswitch-agent","rhosp13/openstack-neutron-server","rhosp13/openstack-nova-api","rhosp13/openstack-nova-compute","rhosp13/openstack-nova-conductor","rhosp13/openstack-nova-consoleauth","rhosp13/openstack-nova-libvirt","rhosp13/openstack-nova-novncproxy","rhosp13/openstack-nova-placement-api","rhosp13/openstack-nova-scheduler","rhosp13/openstack-rabbitmq","rhosp13/openstack-redis","rhosp13/openstack-swift-account","rhosp13/openstack-swift-container","rhosp13/openstack-swift-object","rhosp13/openstack-swift-proxy-server"]


----------------
Enabling Octavia
----------------

Append -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml to the "openstack overcloud container image prepare" command.

(undercloud) [stack@undercloud-0 ~]$ openstack overcloud container image prepare --namespace=docker-registry.engineering.redhat.com/rhosp13 --output-images-file /home/stack/local_registry_images.yaml --output-env-file=/home/stack/cgoncalves-docker-images.yaml -e /home/stack/virt/config_lvm.yaml -e /home/stack/virt/network/network-environment.yaml -e /home/stack/virt/inject-trust-anchor.yaml -e /home/stack/virt/hostnames.yml -e /home/stack/virt/debug.yaml -e /home/stack/virt/nodes_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml --prefix=openstack- --tag=2018-08-20.1 --set ceph_namespace=docker-registry.engineering.redhat.com --set ceph_image=rhceph --set ceph_tag=3-11 --push-destination=192.168.24.1:8787

Confirm Octavia bits in the container image environment file for the overcloud: 

(undercloud) [stack@undercloud-0 ~]$ grep Octavia cgoncalves-docker-images.yaml 
  DockerOctaviaApiImage: 192.168.24.1:8787/rhosp13/openstack-octavia-api:2018-08-20.1
  DockerOctaviaConfigImage: 192.168.24.1:8787/rhosp13/openstack-octavia-api:2018-08-20.1
  DockerOctaviaHealthManagerImage: 192.168.24.1:8787/rhosp13/openstack-octavia-health-manager:2018-08-20.1
  DockerOctaviaHousekeepingImage: 192.168.24.1:8787/rhosp13/openstack-octavia-housekeeping:2018-08-20.1
  DockerOctaviaWorkerImage: 192.168.24.1:8787/rhosp13/openstack-octavia-worker:2018-08-20.1

(undercloud) [stack@undercloud-0 ~]$ grep octavia local_registry_images.yaml
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-octavia-api:2018-08-20.1
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-octavia-health-manager:2018-08-20.1
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-octavia-housekeeping:2018-08-20.1
- imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-octavia-worker:2018-08-20.1


Pull the container images to the undercloud:

(undercloud) [stack@undercloud-0 ~]$ sudo openstack overcloud container image upload --config-file /home/stack/local_registry_images.yaml


Confirm Octavia container images are available now in the local registry

(undercloud) [stack@undercloud-0 ~]$ curl 192.168.24.1:8787/v2/_catalog | grep octavia
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1266  100  1266    0     0   173k      0 --:--:-- --:--:-- --:--:--  206k
{"repositories":["rhosp13/openstack-cinder-api","rhosp13/openstack-cinder-scheduler","rhosp13/openstack-cinder-volume","rhosp13/openstack-cron","rhosp13/openstack-glance-api","rhosp13/openstack-haproxy","rhosp13/openstack-heat-api","rhosp13/openstack-heat-api-cfn","rhosp13/openstack-heat-engine","rhosp13/openstack-horizon","rhosp13/openstack-iscsid","rhosp13/openstack-keystone","rhosp13/openstack-mariadb","rhosp13/openstack-memcached","rhosp13/openstack-neutron-dhcp-agent","rhosp13/openstack-neutron-l3-agent","rhosp13/openstack-neutron-metadata-agent","rhosp13/openstack-neutron-openvswitch-agent","rhosp13/openstack-neutron-server","rhosp13/openstack-nova-api","rhosp13/openstack-nova-compute","rhosp13/openstack-nova-conductor","rhosp13/openstack-nova-consoleauth","rhosp13/openstack-nova-libvirt","rhosp13/openstack-nova-novncproxy","rhosp13/openstack-nova-placement-api","rhosp13/openstack-nova-scheduler","rhosp13/openstack-octavia-api","rhosp13/openstack-octavia-health-manager","rhosp13/openstack-octavia-housekeeping","rhosp13/openstack-octavia-worker","rhosp13/openstack-rabbitmq","rhosp13/openstack-redis","rhosp13/openstack-swift-account","rhosp13/openstack-swift-container","rhosp13/openstack-swift-object","rhosp13/openstack-swift-proxy-server"]}


Append octavia environment to the "openstack overcloud deploy" and run it to enable the service:

openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
-e /home/stack/virt/docker-images.yaml \
--log-file overcloud_deployment_93.log


2018-08-24 12:42:55Z [overcloud-AllNodesDeploySteps-z54u4a465ljy]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2018-08-24 12:42:55Z [AllNodesDeploySteps]: UPDATE_COMPLETE  state changed
2018-08-24 12:43:01Z [overcloud]: UPDATE_COMPLETE  Stack UPDATE completed successfully

 Stack overcloud UPDATE_COMPLETE 

Started Mistral Workflow tripleo.deployment.v1.get_horizon_url. Execution ID: f67b0d05-984a-4efa-9d00-41e96658a6c7
Overcloud Endpoint: http://10.0.0.103:5000/
Overcloud Horizon Dashboard URL: http://10.0.0.103:80/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed
(undercloud) [stack@undercloud-0 ~]$ . overcloudrc 
(overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list

(overcloud) [stack@undercloud-0 ~]$

Comment 8 rlopez 2018-08-24 14:27:47 UTC

Thanks Carlos! Appreciate your do diligence on this :) If I do run into it again I will def let ya know.