Bug 1564188 - Can't use Heat with OS::Neutron::LBaaS resources with Octavia due to Neutron not listing lbaasv2 extension
Summary: Can't use Heat with OS::Neutron::LBaaS resources with Octavia due to Neutron ...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Emilien Macchi
QA Contact: Gurenko Alex
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-05 15:29 UTC by Tzu-Mainn Chen
Modified: 2019-09-10 14:09 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-10 12:44:40 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 559120 0 None None None 2018-04-05 16:20:21 UTC

Description Tzu-Mainn Chen 2018-04-05 15:29:56 UTC
Description of problem:

An OSP13 overcloud installs Octavia but doesn't add the lbaasv2 extension to Neutron. That means that Heat does not recognize OS::Neutron::LBaaS::* resources, so Heat templates that could work with either Octavia or LBaasV2 can only work with one or the other.

Version-Release number of selected component (if applicable):

OSP13

How reproducible:

Steps to Reproduce:
1. Install an OSP13 overcloud
2. Try to deploy a Heat stack that uses a  OS::Neutron::LBaaS::* resource
3.

Actual results:

Stack deployment will fail.

Expected results:

Stack deployment succeeds.

Additional info:

Comment 1 Antoni Segura Puimedon 2018-04-05 15:36:49 UTC
If we want to enable users to keep using he OS::Neutron::LBaaS:: resources I see three possible ways:

a) Tripleo installs neutron-lbaas as well so the extension will be listed. The calls to octavia would go via neutron-lbaas IIRC.
b) Heat drops the required_service_extension in openstack/neutron/lbaas/*.py resources.
c) Neutron adds a check on the /extensions.json that uses keystone to find out if there is a registered endpoint for lbaasv2 and if there is, it puts lbaasv2 in the extension list.

Comment 2 Bogdan Dobrelya 2018-04-05 15:46:07 UTC
I think the option (a) is the way to go. Would it work if lbaasv2 extension is added for octavia heat params, omitting neutron-lbaas installation for the environments/services/octavia.yaml?..

Comment 3 Bogdan Dobrelya 2018-04-05 16:20:21 UTC
Let's give https://review.openstack.org/#/c/559120 a try folks!

Comment 4 Carlos Goncalves 2018-04-05 22:57:53 UTC
https://review.openstack.org/#/c/559120/1 will not do what you'd like it to do because:

a) OSP13 overcloud is containerized. Patch touches only the baremetal octavia service. You want to look at THT/environments/services-docker/octavia.yaml
b) adding lbaasv2 to the neutron service plugin list is not enough. Check https://review.openstack.org/#/c/526093/ which removed what you're now trying to add.

Also, OSP13 will not support Octavia as provider of neutron-lbaas. I believe that, that alone, invalidates the use case you mentioned.

Changing component to where a potential fix would be needed in.

Comment 5 Carlos Goncalves 2018-04-05 23:03:02 UTC
FYI, starting in stable/queens Heat supports Octavia resources: https://review.openstack.org/#/q/status:merged+project:openstack/heat+branch:master+topic:bug/1737567

Migration from OS::Neutron::LBaaS to OS::Octavia:: is relatively low effort.

Comment 6 Antoni Segura Puimedon 2018-04-06 07:53:59 UTC
(In reply to Carlos Goncalves from comment #5)
> FYI, starting in stable/queens Heat supports Octavia resources:
> https://review.openstack.org/#/q/status:merged+project:openstack/heat+branch:
> master+topic:bug/1737567
> 
> Migration from OS::Neutron::LBaaS to OS::Octavia:: is relatively low effort.

We are aware of the possibility to use OS::Octavia::* . It is however a bit artificial to have the heat stack change just because of a check when the functionality would otherwise work. Until neutron-lbaasv2 is dropped for good it means having two sets of templates. If there is no other way to address this, openshift-ansible will have to add some code to choose the templates depending on who is providing the service in the keystone endpoints.

Comment 7 Tomas Sedovic 2018-04-06 09:05:13 UTC
For the record, the upstream Octavia docs recommend that we use the LBAAS v2 API:

https://docs.openstack.org/octavia/latest/reference/introduction.html

"For this reason, we recommend that tenants configure load balancing services with Octavia through the Neutron LBaaS version 2 CLI and API."

So we did that.

As far as our openshift-on-openstack project goes, this is more than just fixing the Heat templates. We also call the LBaaSv2 API directly and getting the same error.

We are able to work around this, as far as we're aware, the API and endpoints are not supposed to go away and using Octavia in this manner should work.

Comment 9 Carlos Goncalves 2018-04-06 15:39:03 UTC
(In reply to Tomas Sedovic from comment #7)
> For the record, the upstream Octavia docs recommend that we use the LBAAS v2
> API:
> 
> https://docs.openstack.org/octavia/latest/reference/introduction.html
> 
> "For this reason, we recommend that tenants configure load balancing
> services with Octavia through the Neutron LBaaS version 2 CLI and API."
> 
> So we did that.

Thanks for pointing that out! I hope https://review.openstack.org/#/c/559352/ will improve the documentation to reflect the present state of affairs.

Comment 11 Carlos Goncalves 2018-04-06 17:36:28 UTC
One other possible workaround maybe worth investigating is, if well received downstream (aka officially supported), to enable the octavia proxy plugin in neutron-lbaas [1]. All API calls to neutron-lbaas will be proxied to Octavia. It is still buggy at the moment [2].

Do you want to try it out by adding a gate job in upstream?

[1] https://review.openstack.org/#/c/418530/
[2] https://review.openstack.org/#/c/554004/

Comment 12 Nir Yechiel 2018-04-08 11:27:18 UTC
(In reply to Tomas Sedovic from comment #7)
> For the record, the upstream Octavia docs recommend that we use the LBAAS v2
> API:
> 
> https://docs.openstack.org/octavia/latest/reference/introduction.html
> 
> "For this reason, we recommend that tenants configure load balancing
> services with Octavia through the Neutron LBaaS version 2 CLI and API."
> 
> So we did that.
> 
> As far as our openshift-on-openstack project goes, this is more than just
> fixing the Heat templates. We also call the LBaaSv2 API directly and getting
> the same error.
> 
> We are able to work around this, as far as we're aware, the API and
> endpoints are not supposed to go away and using Octavia in this manner
> should work.

The recommendation from the Network team at this time is to not use LBaaS v2 but to call the Octavia API directly. Technically, this is a better solution as the requests will go directly into Octavia and not via the Neutron API. 

We, however, will take a closer look at how we can make this LBaaS v2 and Octavia situation more transparent to the user as the APIs are compatible.

Comment 13 Nir Magnezi 2018-04-10 09:28:36 UTC
As of Upstream Pike[1],  Octavia is an OpenStack top-level project and the python-octaviaclient got in a good shape in Queens.

For those reasons, the way we incorporate Octavia into OSP13 is by registering it as a Keystone endpoint and not as a Neutron LBaaS provider and thus, the interaction with Octavia should be handled by the python-octaviaclient (and both dedicated horizon and tempest plugins).

The Neutron Client is not smart enough to distinguish which command should be directed to which API endpoint (both Octavia-API and Neutron-Server) and quite frankly I don't think this logic belong there.

(In reply to Antoni Segura Puimedon from comment #1)

The way I see it, we should look at option (d):
* We should not search for an LBaaS extension to Neutron, but rather an Octavia endpoint in Keystone.
* We should convert the heat templates to use Octavia (as Carlos pointed out in Comment#5)
* We need to implement an Octavia driver in Kuryr (I can imagine it will take a significant amount of work, but obviously you know Kuryr better than me)

(In reply to Tomas Sedovic from comment #7)
Sadly, the docs were not up to date. Carlos was right to submit a patch (thanks Carlos!) in comment#9

(In reply to Carlos Goncalves from comment #11)
IMHO, introducing a the proxy driver now brings risks:
1. We don't have any TripleO support for it.
2. No one in our squad tested it.
3. This introduces more complexities: we will essentially base our reference LBaaS solution in OSP13 (an LTS version..) with a deprecated neutron-lbaas code.
4. No migration path for eventually migrating to Octavia without neutron-lbaas.
 

(In reply to Nir Yechiel from comment #12)

> The recommendation from the Network team at this time is to not use LBaaS v2
> but to call the Octavia API directly. Technically, this is a better solution
> as the requests will go directly into Octavia and not via the Neutron API. 
> 

+1

> We, however, will take a closer look at how we can make this LBaaS v2 and
> Octavia situation more transparent to the user as the APIs are compatible.

I'll be happy to help with that.

@Toni, we will do our best to help and provide any information you need, but IMHO the bug should be addressed in Kuryr+Heat templates.

Comment 14 Nir Magnezi 2018-04-10 09:29:29 UTC
Forgot to attach this link:
[1] https://docs.openstack.org/releasenotes/octavia/pike.html#prelude

Comment 15 Carlos Goncalves 2018-04-10 09:54:35 UTC
Thanks for sharing your view, Nir!

Yesterday I tested the experimental lbaasv2-proxy on a devstack environment. It is not in a good shape to serve as workaround here. I tried to reproduce a use case similar to yours: I ran a Heat stack using OS::Neutron::LBaaS::* resources where it requests creation of various resources: load balancer, listener, pool, 2 members and 2 HTTP VMs. Stack failed on member creation:

stack_status_reason   | Resource CREATE failed: Conflict: resources.pool_member1: {"debuginfo": null, "faultcode": "Client", "faultstring": "Load Balancer 4fbeda8b-528d-4173-9434-6e58de044628 is immutable and cannot be updated."}

Thus, the lbaasv2-proxy is out of question.

One other option would be to setup L7 rules to forward API requests from http://192.168.1.13:9696/v2.0/lbaas/ to http://192.168.1.13/load-balancer (Octavia's Keystone endpoint). It will not help either as one would run into the same issues.

Comment 16 Tomas Sedovic 2018-04-10 10:17:53 UTC
Thanks for your replies.

We'll update our code to use the Octavia API. We did consider that from the get-go, but were led astray by the docs, thinking this is going to be a long-term supported solution.

If that's not how the project is supposed to be used, we'll have to fix it on our side.

Comment 17 Carlos Goncalves 2018-04-10 12:44:40 UTC
Tomas, thanks for your understanding and apologies for the outdated doc.
As per your comment, I'm closing this BZ. Please reopen it if necessary.


Note You need to log in before you can comment on or make changes to this bug.