Bug 1392235

This sounds like the same issue from https://bugzilla.redhat.com/show_bug.cgi?id=1311165  and can be fixed by making the same changesto the Keystone config file.  Please confirm.

Hello sir,

   We tried that and it didn't solve the issue.   Do we need to backport 40c3942c12d1dd2c826d836987616838a73a64a1 to RHOSP 6?

Thank you very much,

David Hill

that is not a trivial backport.  Is it possible to work around it in your setup by explicitly setting the public or private URL config options?

The case got closed but I reopened it and I'm asking the customer about it.  Let me get back at you with this.  thx

Well, this defeat the purpose of having internal/public/admin APIs doesn't it?   How hard is it to backport?  If it's not possible at all, the customer will have to update in order to get this fix but if it's possible to backport, it'd solve this issue for the customer using the openstack CLI.  We already tried the keystone solution but it doesn't fix the issue.

There is a suggestion that bug "Keystone API GET 5000/v3 returns wrong
endpoint URL in response body"
(https://bugs.launchpad.net/keystone/+bug/1381961)

Which is fixed by this commit:
40c3942c12d1dd2c826d836987616838a73a64a1
in Gerrit review https://review.openstack.org/#/c/226464/

is a likely cause of this problem, however I find no evidence of
that. First of all note the the patch replaces the 'host_url' value
from webob with a call to wsgref.application_uri on the belief it was
not utilzing the correct URL reconstruction logic based on the HTTP
request received. This logic requires the HOST header in the request
to be preferred over the host the web server is running on. It is a
requirement clients include a HOST header. However if the HOST header
is absent (which it shouldn't be) then the next correct step is to
utilize the ServerName which should be set in the VirtualHost section
of Apache. This value SHOULD include both the scheme and port, not
just the hostname. Finally barring that the hostname the server is
running on can be used, but this should never happen.

However note that the code that was replaced (webob.host_url)
implments the same logic as the code that replaced it
(wsgiref.application_uri). These implementations are what are installed
in RHEL 7.x with our Juno release.

wsgiref/util.py:

def application_uri(environ):
    """Return the application's base URI (no PATH_INFO or QUERY_STRING)"""
    url = environ['wsgi.url_scheme']+'://'
    from urllib import quote

    if environ.get('HTTP_HOST'):
        url += environ['HTTP_HOST']
    else:
        url += environ['SERVER_NAME']

        if environ['wsgi.url_scheme'] == 'https':
            if environ['SERVER_PORT'] != '443':
                url += ':' + environ['SERVER_PORT']
        else:
            if environ['SERVER_PORT'] != '80':
                url += ':' + environ['SERVER_PORT']

    url += quote(environ.get('SCRIPT_NAME') or '/')
    return url

webob/request.py:

    @property
    def host_url(self):
        """
        The URL through the host (no path)
        """
        e = self.environ
        scheme = e.get('wsgi.url_scheme')
        url = scheme + '://'
        host = e.get('HTTP_HOST')
        if host is not None:
            if ':' in host:
                host, port = host.split(':', 1)
            else:
                port = None
        else:
            host = e.get('SERVER_NAME')
            port = e.get('SERVER_PORT')
        if scheme == 'https':
            if port == '443':
                port = None
        elif scheme == 'http':
            if port == '80':
                port = None
        url += host
        if port:
            url += ':%s' % port
        return url

Therefore I cannot see how the aforementioned commit would actually
fix the observed problem.

Furthermore I performed extensive testing in an OSP Juno deployment
and I was unable to get the endpoints in the version response to be
incorrect.

However, it is important to note the observed problem is likely to be
the result of incorrect catalog endpoints being returned. I surmise
this because it appears other services are trying to contact keystone
with an invalid endpoint.

It is also important to note that version endpoints and catalog
endpoints have no DIRECT CORRELATION. When Keystone reports version
endpoints it does so by examining the request URI as seen above. But
when catalog information is requested the endpoints are looked up in a
database. The catalog endpoints are set by calls the endpoint API.

One of the difficulties of evaluating this bug report is version
information has been conflated with catalog endpoints and there is
insufficient information concerning the state of the catalog endpoints
or how they were set.

Another speculation and potential fix occurs in this Gerrit review:
https://review.openstack.org/#/c/118522/1 with this commit:
af0cec12e750061e78a5c4fdcdd998cd3f506574

The observation is that in Juno Keystone creates two identical API
mapping routes, one for the admin and one for main. The admin routes
are created first. The way the route mapping works is that the first
route that matches wins. That means the admin routes will always be
invoked. However the admin and main routes map to identical handlers
and nowhere does the admin port appear as part of the admin route
handler (at least that I could find). The superflous admin route was
removed by the the above patch (there wasn't even a bug associated
with it). Although what this commit fixes looks suspiciously like it
could contribute to admin endpoints being returned instead of the main
endpoints I can find no evidence to support that, rather it just seems
to be a needless duplication that yields the same values.

If there are problems with the catalog endpoints then I suspect it is
the result of an installer and/or configuration management tool. It is
possible that this bug

https://bugzilla.redhat.com/show_bug.cgi?id=1311165

with this Gerrit review and patch:

https://review.openstack.org/#/c/368969/

which removes the keystone public endpoint from Puppet control is a
potential problem.

One the problems with this bug report and the bug reports it
references is the enormous amount of non-relevant speculation. It is
very difficult to work with bug reports like this because they don't
articulate a single problem nor do they use rigerous methods to
isolate the problem all of which is compouned by a lack of relevant
information.

At this junction I have been unable to reproduce the problem in a
fresh Juno install. It does not seem to be a problem inherent in
Keystone itself, rather it appears it may have more to do with
configuration management than with a Keystone defect.

My recommendation is to do a condtional NAK on this bug until such
time as a SIMPLE CLEAN REPRODUCER can be provided. Without a
reproducer we're just spinning our wheels and contributing to the mass
of speculation.

We are closing this bug because we have not received sufficient information to reproduce the bug and make progress. Please feel free to open this bug again when you are able to provide the required information requested.