Bug 919526

Summary: Temporary network outage results in connection refused and invalid token
Product: Red Hat OpenStack Reporter: Alan Pevec <apevec>
Component: openstack-keystoneAssignee: Alan Pevec <apevec>
Status: CLOSED ERRATA QA Contact: Pavel Sedlák <psedlak>
Severity: high Docs Contact:
Priority: high    
Version: 2.0 (Folsom)CC: ajeain, ayoung
Target Milestone: asyncKeywords: Triaged
Target Release: 2.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-keystone-2012.2.4-1.el6ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-05-09 18:16:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 950132    
Bug Blocks:    
Attachments:
Description Flags
Script used for verification - sends requests during keystone restart or while traffic is rejected none

Description Alan Pevec 2013-03-08 17:53:09 UTC 
Description of problem:
The auth_token middleware does not retry if it gets an http error. This means a temporary network outage or keystone restart will cause the token to be listed as invalid. The middleware should retry a few times before failing.
Comment 4 Pavel Sedlák 2013-05-02 19:29:14 UTC 
Created attachment 742897 [details]
Script used for verification - sends requests during keystone restart or while traffic is rejected

This script obtains token from keystone, then launches requests for nova servers list.

First, third and fifth are alignment tests and should always succeed.
When this bug is not fixed:
- second request should fail because keystone is restarting
- fourth request should fail because keystone traffic is rejected by iptables

Output for new version - openstack-keystone-2012.2.4-1.el6ost:
> $ ./919526-network-outage-unauth.py 
> OK
> Stopping keystone:                                         [  OK  ]
> Starting keystone:                                         [  OK  ]
> OLD SHOULD FAIL ::  OK
> OK
> OLD SHOULD FAIL ::  OK
> OK

Output for old version - openstack-keystone-2012.2.3-8.el6ost:
> OK
> Stopping keystone: OLD SHOULD FAIL ::  Failed:
> 401 Unauthorized
> 
> This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.
> 
>  Authentication required  
>                                                            [  OK  ]
> Starting keystone:                                         [  OK  ]
> OK
> OLD SHOULD FAIL ::  Failed:
> 401 Unauthorized
> 
> This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required.
> 
>  Authentication required  
> OK
Comment 5 Pavel Sedlák 2013-05-02 19:30:13 UTC 
Verified - details provided with previous comment for example test script.
Comment 7 errata-xmlrpc 2013-05-09 18:16:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0806.html