Bug 1122536

Summary: HEAD and GET inconsistencies in Keystone
Product: Red Hat OpenStack Reporter: Jaroslav Henner <jhenner>
Component: openstack-keystoneAssignee: Nathan Kinder <nkinder>
Status: CLOSED ERRATA QA Contact: Udi Kalifon <ukalifon>
Severity: medium Docs Contact:
Priority: low    
Version: 5.0 (RHEL 6)CC: aberezin, ajeain, ayoung, dnavale, jhenner, mkollaro, nkinder, sclewis, sgordon, yeylon
Target Milestone: z1Keywords: Rebase, ZStream
Target Release: 5.0 (RHEL 7)   
Hardware: x86_64   
OS: All   
Whiteboard:
Fixed In Version: openstack-keystone-2014.1.2.1-1.el7ost Doc Type: Bug Fix
Doc Text:
Identity service returned different response codes for GET and HEAD requests, even though the HTTP specification stated that the same response codes should be returned. As a result, when Identity service was deployed in Apache httpd with mod_wsgi, clients received a different response as compared to running 'keystone'all' command, since mod_wsgi translated HEAD requests into GET requests. This led to client interoperability issues when Identity service was deployed in different web servers. With this update, Identity service returns consistent responses for HEAD and GET requests. And client interoperability is improved by consistent API responses, regardless of the web server that Identity service is deployed on.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-30 17:51:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jaroslav Henner 2014-07-23 12:58:01 UTC
Description of problem:
please include the LP#1334368. It causes some failures in our icehouse tempest runs:

tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_trust_expire[
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_trust_impersonate
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_trust_noimpersonate


Version-Release number of selected component (if applicable):
puddle 2014-07-18.1

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Nathan Kinder 2014-07-23 17:10:18 UTC
This will be picked up as a part of the rebase of the upstream 2014.1.2 release.  That upstream release is currently planned for Aug 7 2014.  Our A2 update will be based on this upstream release.

I don't think that this is worthy of an earlier one-off update since it's purely an inconsistency in the response codes between keystone-all and httpd deployments.  This inconsistency is not a new behavior, and it should only be affecting the tests.  I'm targeting OSP 5.0 A2.

Comment 3 Nathan Kinder 2014-09-20 02:18:53 UTC
We rebased to the upstream keystone 2014.1.2.release for 5.0.z A1.  Setting to MODIFIED since this rebase has been completed.

Comment 5 Udi Kalifon 2014-09-24 09:48:15 UTC
The problem is that all HEAD calls return 404 for me... For example even the simplest call to list all domains:

# curl --head -H "x-auth-token:$TOKEN" http://localhost:35357/v3/domains
HTTP/1.1 404 Not Found
Vary: X-Auth-Token
Content-Type: application/json
Content-Length: 93
Date: Wed, 24 Sep 2014 09:46:25 GMT

When doing a regular GET on the same url there is no problem.

Comment 6 Nathan Kinder 2014-09-25 04:57:45 UTC
(In reply to Udi from comment #5)
> The problem is that all HEAD calls return 404 for me... For example even the
> simplest call to list all domains:

You can't simply issue a HEAD request for any resource, as not all resources support a HEAD request.  You need to test against a resource that supports HEAD according to the Identity API spec (http://docs.openstack.org/api/openstack-identity-service/3/content/core-api.html).  An example of an API call that supports head is the request to check/validate tokens described here:

  http://docs.openstack.org/api/openstack-identity-service/3/content/check-token-head-authtokens.html

You can use curl like this to make the correct HEAD request:

  curl -v -H "X-Auth-Token: <admin token>" 
       -H "X-Subject-Token: <token to validate>" \
       -H "Content-Type: application/json" \
       --head http://localhost:5000/v3/auth/tokens/

Note that this will return a 404 is the token in "X-Subject-Token" is not found, so you will want to validate a valid token id (not the admin service token) if you want to get a 20x response.  Prior to the fix, you will see a difference in the response codes for this call between GET and HEAD.  With the fix, the response code should be the same for both GET and HEAD requests.

Comment 7 Martina Kollarova 2014-09-25 13:31:45 UTC
Failed QE:

I reproduced the error again on openstack-keystone-2014.1.2.1-2.el7ost.noarch, where it's supposed to be fixed already. I also reproduced it on RDO Icehouse. It caused these 3 tests to fail in Tempest [1]:

tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_trust_expire[gate,smoke]
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_trust_impersonate[gate,smoke]
tempest.api.identity.admin.v3.test_trusts.TrustsV3TestJSON.test_trust_noimpersonate[gate,smoke]

The failure was always something like this:
Traceback (most recent call last):
  File "/root/tempest/tempest/api/identity/admin/v3/test_trusts.py", line 221, in test_trust_noimpersonate
    self.check_trust_roles()
  File "/root/tempest/tempest/api/identity/admin/v3/test_trusts.py", line 169, in check_trust_roles
    self.assertEqual('204', resp['status'])
  File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 321, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/usr/lib/python2.7/site-packages/testtools/testcase.py", line 406, in assertThat
    raise mismatch_error
MismatchError: '204' != '200'



[1] I tried it on a 6 month old version of Tempest from the icehouse branch of https://github.com/redhat-openstack/tempest

Comment 8 Martina Kollarova 2014-09-25 13:38:42 UTC
I tried it now on the latest Tempest and there it's not failing anymore. Is this enough to consider it verified?

Comment 10 Nathan Kinder 2014-09-25 18:44:33 UTC
(In reply to Martina Kollarova from comment #8)
> I tried it now on the latest Tempest and there it's not failing anymore. Is
> this enough to consider it verified?

Yes, there were associated changes made to the Tempest tests.  Details are here:

  https://bugs.launchpad.net/keystone/+bug/1334368

  https://git.openstack.org/cgit/openstack/tempest/commit/?id=59a9d7a8d4f9620cdcf783b9ef94a99acbe20873

  https://git.openstack.org/cgit/openstack/tempest/commit/?id=883311d64fd749315ee3639bff6c730e86026ac5

If the latest tempest is passing, then it should be enough to consider this VERIFIED.

Comment 15 errata-xmlrpc 2014-09-30 17:51:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1347.html