Description of problem: When attempting to update a successful deployment of OSP13 with Octavia, the installation procedure states it is successful, however, Octavia is not enabled. Deleting the overcloud installation and doing a fresh install with including Octavia did produce a successful installation. Steps to reproduce: Install your overcloud without Octavia. Example: openstack overcloud deploy --templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible-external.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/custom_templates/node-info.yaml \ -e /home/stack/custom_templates/overcloud_images.yaml \ -e /home/stack/custom_templates/network_environment.yaml \ -n /home/stack/custom_templates/network_data.yaml \ -e /home/stack/custom_templates/ceph_external_environment.yaml The above assumes a successful OSP13 install without Octavia. Next, ensure the following: * yum install octavia-amphora-image * This line is part of your network-environment.yaml OctaviaAmphoraImageFilename: '/usr/share/rhosp-director-images/octavia-amphora.qcow2' * Octavia is part of your local registry images (if using local registry) local_registry_images.yaml Attempt to update your overcloud with Octavia: openstack overcloud deploy --templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible-external.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \ -e /home/stack/custom_templates/node-info.yaml \ -e /home/stack/custom_templates/overcloud_images.yaml \ -e /home/stack/custom_templates/network_environment.yaml \ -n /home/stack/custom_templates/network_data.yaml \ -e /home/stack/custom_templates/ceph_external_environment.yaml This will provide an update complete. However, the following error will be seen when attempting openstack loadbalancer list $ openstack loadbalancer list Unable to establish connection to http://10.19.114.43:9876/v2.0/lbaas/loadbalancers: HTTPConnectionPool(host='10.19.114.43', port=9876): Max retries exceeded with url: /v2.0/lbaas/loadbalancers (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f07dc5cfa90>: Failed to establish a new connection: [Errno 110] Connection timed out',)) Looking at the controllers, I did not see any ocatvia containers. Workaround: * Delete existing overcloud * Redeploy overcloud with Ocatvia fresh install Result: Successful overcloud with Ocatvia
Hi, This was a known GA bug, that was fixed in Z1. Could you please share the version that you were using?
Nir, To my knowledge GA. Here is a print out of the versions from my director node ]# rpm -qa | grep openstack openstack-swift-container-2.17.1-0.20180314165245.caeeb54.el7ost.noarch openstack-nova-common-17.0.3-0.20180420001141.el7ost.noarch openstack-neutron-ml2-12.0.2-0.20180421011364.0ec54fd.el7ost.noarch openstack-mistral-engine-6.0.2-1.el7ost.noarch openstack-neutron-12.0.2-0.20180421011364.0ec54fd.el7ost.noarch openstack-heat-api-cfn-10.0.1-0.20180411125640.el7ost.noarch openstack-ironic-staging-drivers-0.9.0-4.el7ost.noarch openstack-nova-conductor-17.0.3-0.20180420001141.el7ost.noarch openstack-tripleo-puppet-elements-8.0.0-2.el7ost.noarch openstack-nova-compute-17.0.3-0.20180420001141.el7ost.noarch python2-openstackclient-3.14.1-1.el7ost.noarch openstack-neutron-common-12.0.2-0.20180421011364.0ec54fd.el7ost.noarch openstack-ironic-api-10.1.2-4.el7ost.noarch openstack-ironic-inspector-7.2.1-0.20180409163360.el7ost.noarch openstack-tripleo-ui-8.3.1-3.el7ost.noarch openstack-zaqar-6.0.1-1.el7ost.noarch openstack-tripleo-image-elements-8.0.1-1.el7ost.noarch openstack-selinux-0.8.14-12.el7ost.noarch python-openstackclient-lang-3.14.1-1.el7ost.noarch python2-openstacksdk-0.11.3-1.el7ost.noarch puppet-openstack_extras-12.4.1-0.20180413042250.2634296.el7ost.noarch puppet-openstacklib-12.4.0-0.20180329042555.4b30e6f.el7ost.noarch openstack-tripleo-heat-templates-8.0.2-43.el7ost.noarch openstack-swift-object-2.17.1-0.20180314165245.caeeb54.el7ost.noarch openstack-neutron-openvswitch-12.0.2-0.20180421011364.0ec54fd.el7ost.noarch openstack-swift-proxy-2.17.1-0.20180314165245.caeeb54.el7ost.noarch openstack-heat-common-10.0.1-0.20180411125640.el7ost.noarch openstack-ironic-conductor-10.1.2-4.el7ost.noarch openstack-mistral-api-6.0.2-1.el7ost.noarch openstack-tripleo-validations-8.4.1-5.el7ost.noarch openstack-keystone-13.0.1-0.20180420194847.7bd6454.el7ost.noarch openstack-heat-api-10.0.1-0.20180411125640.el7ost.noarch openstack-tripleo-common-8.6.1-23.el7ost.noarch openstack-nova-placement-api-17.0.3-0.20180420001141.el7ost.noarch openstack-swift-account-2.17.1-0.20180314165245.caeeb54.el7ost.noarch openstack-glance-16.0.1-2.el7ost.noarch openstack-ironic-common-10.1.2-4.el7ost.noarch openstack-mistral-executor-6.0.2-1.el7ost.noarch openstack-tripleo-common-containers-8.6.1-23.el7ost.noarch openstack-heat-engine-10.0.1-0.20180411125640.el7ost.noarch openstack-tempest-18.0.0-2.el7ost.noarch openstack-mistral-common-6.0.2-1.el7ost.noarch openstack-nova-api-17.0.3-0.20180420001141.el7ost.noarch openstack-nova-scheduler-17.0.3-0.20180420001141.el7ost.noarch
Have you prepared the container images for Octavia before deploying overcloud? For documentation on how to do it, please refer to https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/transitioning_to_containerized_services/ Make sure the local registry lists the Octavia docker images: [stack@undercloud-0 ~]$ curl 192.168.24.1:8787/v2/_catalog | grep octavia % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1292 100 1292 0 0 175k 0 --:--:-- --:--:-- --:--:-- 210k {"repositories":["rhosp13/openstack-cinder-api","rhosp13/openstack-cinder-scheduler","rhosp13/openstack-cinder-volume","rhosp13/openstack-cron","rhosp13/openstack-glance-api","rhosp13/openstack-haproxy","rhosp13/openstack-heat-api","rhosp13/openstack-heat-api-cfn","rhosp13/openstack-heat-engine","rhosp13/openstack-horizon","rhosp13/openstack-iscsid","rhosp13/openstack-keystone","rhosp13/openstack-mariadb","rhosp13/openstack-memcached","rhosp13/openstack-neutron-metadata-agent-ovn","rhosp13/openstack-neutron-server-ovn","rhosp13/openstack-nova-api","rhosp13/openstack-nova-compute","rhosp13/openstack-nova-conductor","rhosp13/openstack-nova-consoleauth","rhosp13/openstack-nova-libvirt","rhosp13/openstack-nova-novncproxy","rhosp13/openstack-nova-placement-api","rhosp13/openstack-nova-scheduler","rhosp13/openstack-octavia-api","rhosp13/openstack-octavia-health-manager","rhosp13/openstack-octavia-housekeeping","rhosp13/openstack-octavia-worker","rhosp13/openstack-ovn-controller","rhosp13/openstack-ovn-nb-db-server","rhosp13/openstack-ovn-northd","rhosp13/openstack-ovn-sb-db-server","rhosp13/openstack-rabbitmq","rhosp13/openstack-redis","rhosp13/openstack-swift-account","rhosp13/openstack-swift-container","rhosp13/openstack-swift-object","rhosp13/openstack-swift-proxy-server"]}
Hi Carlos, Yes I updated the container images for Octavia before deploying overcloud. I never got the change to confirm the curl you mentioned here. What I did: 1) I had already prepared the container image before deploying overcloud, I just didn't call it out on my overcloud deploy as at the time I was attempting just to get a working OSP13 with the basics. 2) After overcloud comeplete, went ahead and made the changes in my network-environment.yaml as I listed above and then added the -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml to my deploy command. 3) Re-ran the deploy command which said successful but no Octavia was deployed. 4) deleted my overcloud 5) Re-ran the same exact command and this time Octavia was successfully installed.
The steps you performed seem exactly like the steps I also did some weeks ago when I tried to enable Octavia on an existing overcloud with the same result: deployment succeeded but Octavia containers were not running; deleting and deploying again worked. My problem was that the octavia container images had not been prepared *and* uploaded to the local registry. After uploading them, I could update an existing overcloud. So, I'd ask you to kindly double-check if you uploaded them to the local registry with the curl command I mentioned before. BTW, you don't need to manually install the octavia-amphora-image RPM and set OctaviaAmphoraImageFilename. Director can do that automatically.
Hi Carlos, At this point, I can't confirm it because my environment is currently working with Octavia. What I can tell you is that my prepare command did have Octavia in it and the local_registry_images.yaml had Octavia in it. However, there is no way for me to confirm the curl command as I'm no longer at that step. This was the command: # openstack overcloud container image prepare -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/ironic.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/manila.yaml --namespace=registry.access.redhat.com/rhosp13 --push-destination=192.3.2.1:8787 --prefix=openstack- --tag-from-label {version}-{release} --output-env-file=/home/stack/templates/overcloud_images.yaml --output-images-file /home/stack/local_registry_images.yaml # Good to know about the RPM and setting the image file name.
It is likely that while you prepared the container images, you did not upload them to the local registry (openstack overcloud container image upload). I tried to reproduce on a fresh deployment without Octavia enabled. I could not reproduce. See below the step-by-step on how I did it, following closely to our official documentation (https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/transitioning_to_containerized_services/). Please feel free to reopen if you happen to run again into this issue. ---------------------------------------- Fresh deployment without Octavia enabled ---------------------------------------- (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list public endpoint for load-balancer service not found (undercloud) [stack@undercloud-0 ~]$ curl 192.168.24.1:8787/v2/_catalog {"repositories":["rhosp13/openstack-cinder-api","rhosp13/openstack-cinder-scheduler","rhosp13/openstack-cinder-volume","rhosp13/openstack-cron","rhosp13/openstack-glance-api","rhosp13/openstack-haproxy","rhosp13/openstack-heat-api","rhosp13/openstack-heat-api-cfn","rhosp13/openstack-heat-engine","rhosp13/openstack-horizon","rhosp13/openstack-iscsid","rhosp13/openstack-keystone","rhosp13/openstack-mariadb","rhosp13/openstack-memcached","rhosp13/openstack-neutron-dhcp-agent","rhosp13/openstack-neutron-l3-agent","rhosp13/openstack-neutron-metadata-agent","rhosp13/openstack-neutron-openvswitch-agent","rhosp13/openstack-neutron-server","rhosp13/openstack-nova-api","rhosp13/openstack-nova-compute","rhosp13/openstack-nova-conductor","rhosp13/openstack-nova-consoleauth","rhosp13/openstack-nova-libvirt","rhosp13/openstack-nova-novncproxy","rhosp13/openstack-nova-placement-api","rhosp13/openstack-nova-scheduler","rhosp13/openstack-rabbitmq","rhosp13/openstack-redis","rhosp13/openstack-swift-account","rhosp13/openstack-swift-container","rhosp13/openstack-swift-object","rhosp13/openstack-swift-proxy-server"] ---------------- Enabling Octavia ---------------- Append -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml to the "openstack overcloud container image prepare" command. (undercloud) [stack@undercloud-0 ~]$ openstack overcloud container image prepare --namespace=docker-registry.engineering.redhat.com/rhosp13 --output-images-file /home/stack/local_registry_images.yaml --output-env-file=/home/stack/cgoncalves-docker-images.yaml -e /home/stack/virt/config_lvm.yaml -e /home/stack/virt/network/network-environment.yaml -e /home/stack/virt/inject-trust-anchor.yaml -e /home/stack/virt/hostnames.yml -e /home/stack/virt/debug.yaml -e /home/stack/virt/nodes_data.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml --prefix=openstack- --tag=2018-08-20.1 --set ceph_namespace=docker-registry.engineering.redhat.com --set ceph_image=rhceph --set ceph_tag=3-11 --push-destination=192.168.24.1:8787 Confirm Octavia bits in the container image environment file for the overcloud: (undercloud) [stack@undercloud-0 ~]$ grep Octavia cgoncalves-docker-images.yaml DockerOctaviaApiImage: 192.168.24.1:8787/rhosp13/openstack-octavia-api:2018-08-20.1 DockerOctaviaConfigImage: 192.168.24.1:8787/rhosp13/openstack-octavia-api:2018-08-20.1 DockerOctaviaHealthManagerImage: 192.168.24.1:8787/rhosp13/openstack-octavia-health-manager:2018-08-20.1 DockerOctaviaHousekeepingImage: 192.168.24.1:8787/rhosp13/openstack-octavia-housekeeping:2018-08-20.1 DockerOctaviaWorkerImage: 192.168.24.1:8787/rhosp13/openstack-octavia-worker:2018-08-20.1 (undercloud) [stack@undercloud-0 ~]$ grep octavia local_registry_images.yaml - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-octavia-api:2018-08-20.1 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-octavia-health-manager:2018-08-20.1 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-octavia-housekeeping:2018-08-20.1 - imagename: docker-registry.engineering.redhat.com/rhosp13/openstack-octavia-worker:2018-08-20.1 Pull the container images to the undercloud: (undercloud) [stack@undercloud-0 ~]$ sudo openstack overcloud container image upload --config-file /home/stack/local_registry_images.yaml Confirm Octavia container images are available now in the local registry (undercloud) [stack@undercloud-0 ~]$ curl 192.168.24.1:8787/v2/_catalog | grep octavia % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1266 100 1266 0 0 173k 0 --:--:-- --:--:-- --:--:-- 206k {"repositories":["rhosp13/openstack-cinder-api","rhosp13/openstack-cinder-scheduler","rhosp13/openstack-cinder-volume","rhosp13/openstack-cron","rhosp13/openstack-glance-api","rhosp13/openstack-haproxy","rhosp13/openstack-heat-api","rhosp13/openstack-heat-api-cfn","rhosp13/openstack-heat-engine","rhosp13/openstack-horizon","rhosp13/openstack-iscsid","rhosp13/openstack-keystone","rhosp13/openstack-mariadb","rhosp13/openstack-memcached","rhosp13/openstack-neutron-dhcp-agent","rhosp13/openstack-neutron-l3-agent","rhosp13/openstack-neutron-metadata-agent","rhosp13/openstack-neutron-openvswitch-agent","rhosp13/openstack-neutron-server","rhosp13/openstack-nova-api","rhosp13/openstack-nova-compute","rhosp13/openstack-nova-conductor","rhosp13/openstack-nova-consoleauth","rhosp13/openstack-nova-libvirt","rhosp13/openstack-nova-novncproxy","rhosp13/openstack-nova-placement-api","rhosp13/openstack-nova-scheduler","rhosp13/openstack-octavia-api","rhosp13/openstack-octavia-health-manager","rhosp13/openstack-octavia-housekeeping","rhosp13/openstack-octavia-worker","rhosp13/openstack-rabbitmq","rhosp13/openstack-redis","rhosp13/openstack-swift-account","rhosp13/openstack-swift-container","rhosp13/openstack-swift-object","rhosp13/openstack-swift-proxy-server"]} Append octavia environment to the "openstack overcloud deploy" and run it to enable the service: openstack overcloud deploy \ --timeout 100 \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack overcloud \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /home/stack/virt/config_lvm.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/hostnames.yml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \ -e /home/stack/virt/docker-images.yaml \ --log-file overcloud_deployment_93.log 2018-08-24 12:42:55Z [overcloud-AllNodesDeploySteps-z54u4a465ljy]: UPDATE_COMPLETE Stack UPDATE completed successfully 2018-08-24 12:42:55Z [AllNodesDeploySteps]: UPDATE_COMPLETE state changed 2018-08-24 12:43:01Z [overcloud]: UPDATE_COMPLETE Stack UPDATE completed successfully Stack overcloud UPDATE_COMPLETE Started Mistral Workflow tripleo.deployment.v1.get_horizon_url. Execution ID: f67b0d05-984a-4efa-9d00-41e96658a6c7 Overcloud Endpoint: http://10.0.0.103:5000/ Overcloud Horizon Dashboard URL: http://10.0.0.103:80/dashboard Overcloud rc file: /home/stack/overcloudrc Overcloud Deployed (undercloud) [stack@undercloud-0 ~]$ . overcloudrc (overcloud) [stack@undercloud-0 ~]$ openstack loadbalancer list (overcloud) [stack@undercloud-0 ~]$
Thanks Carlos! Appreciate your do diligence on this :) If I do run into it again I will def let ya know.