Bug 1415570

Summary: Identity storage location in etcd changed from 3.3 to 3.4, upgrading causes all identities to be seen as missing
Product: OpenShift Container Platform Reporter: Jordan Liggitt <jliggitt>
Component: apiserver-authAssignee: Jordan Liggitt <jliggitt>
Status: CLOSED ERRATA QA Contact: ge liu <geliu>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.4.0CC: abhgupta, adellape, aos-bugs, bleanhar, ccoleman, dakini, dmace, jliggitt, mbarrett, xtian, zhezli
Target Milestone: ---Keywords: TestCaseProvided
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously, upgrading from OpenShift Container Platform 3.3 to 3.4 caused all user identities to disappear, though they were still present in etcd, and OAuth-based users could no longer log in. New 3.4 installations were also affected. This was caused by an unintentional change in the etcd prefix for user identities; egressnetworkpolicies were similarly affected. This bug fix restores the previous etcd prefix for user identities and egressnetworkpolicies, and as a result users can log in again successfully. Administrators who have already installed or upgraded to v3.4.0.39 must upgrade to v3.4.0.40, then perform a data migration using a data migration tool.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-24 21:09:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1303130    
Attachments:
Description Flags
testcase none

Description Jordan Liggitt 2017-01-23 05:35:14 UTC 
Description of problem:

The etcd prefix for user identities was unintentionally changed during 3.4 development.

This means that upgrading from 3.3 to 3.4 makes all user identities disappear (though they are still present in etcd).



Version-Release number of selected component (if applicable):


How reproducible:
Always


Steps to Reproduce:
1. Start 3.3
2. Log in or create user identities
3. Upgrade to 3.4

Actual results:

User identities are not visible, logins no longer work


Expected results:


Additional info:
Comment 9 ge liu 2017-01-24 08:36:57 UTC 
Verified with: v3.4.0.40,

Verify Steps:

1). Install openshift 3.3.1.11 
     - oc login with user1
     - oc create -f old-objects.json
     - oc get identity:

                ...........
                 name: geliu
                 ..............
                  source: original
                  name: x:a
                ............
                 source: original
                 name: x:b

      - oc get egressnetworkpolicy :
                 kind: EgressNetworkPolicy
                 source: original
                   name: a
                 ................
                 source: original
                   name: b
2). upgrade to openshift 3.4.0.39
     - oc login with user1 successfully
     - # oc get identity
           No resources found.
       # oc get egressnetworkpolicy
           No resources found.
     - oc create -f new-objects.json
     - # oc create -f new-object.json 
              egressnetworkpolicy "b" created
              egressnetworkpolicy "c" created
              identity "x:b" created
              identity "x:c" created
     - # oc get identity
           NAME      IDP NAME   IDP USER NAME   USER NAME   USER UID
             x:b       x          b                            
             x:c       x          c                           
     - # oc get egressnetworkpolicy
           NAME
             b
             c

3). upgrade to 3.4.0.40

     - oc login with user1 successfully
      
     - # oc get identity
       NAME              IDP NAME    IDP USER NAME   USER NAME   USER UID

        allow_all:geliu   allow_all   geliu           geliu      xxxxxx
        x:a               x           a                           
        x:b               x           b                          
     - # oc get egressnetworkpolicy 
          NAME
           a
           b
     both identity a,b and egressnetworkpolicy a,b are meet: "source: original"


latest upgrade openshift version:

openshift v3.4.0.40
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0
Comment 10 Stefanie Forrester 2017-01-24 16:33:28 UTC 
*** Bug 1413496 has been marked as a duplicate of this bug. ***
Comment 12 errata-xmlrpc 2017-01-24 21:09:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0186
Comment 13 ge liu 2017-02-07 08:28:35 UTC 
@jliggitt, could u help to confirm with customer which  "mappingMethod" mode used in master-config.yaml file, it related with the 
customer's 'login fail‘ error, we need this info to improve our upgrad testcase, thank in advance:


#################customer situation#######################
Actual results:

User identities are not visible, logins no longer work
############################################################

master-config.yaml：

####################################
oauthConfig:
..............
    mappingMethod: claim  => default is claim, there are some other mode, we want to know which mode the customer used
####################################
Comment 14 Jordan Liggitt 2017-02-08 05:20:23 UTC 
mappingMethod "claim" and "lookup" would both encounter login problems if the mapping between a user and identity was broken.
Comment 15 ge liu 2017-02-17 06:29:06 UTC 
Created attachment 1251792 [details]
testcase

testcase url: https://url.corp.redhat.com/01421f4