Bug 1415570 - Identity storage location in etcd changed from 3.3 to 3.4, upgrading causes all identities to be seen as missing
Summary: Identity storage location in etcd changed from 3.3 to 3.4, upgrading causes a...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: apiserver-auth
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: ---
Assignee: Jordan Liggitt
QA Contact: ge liu
URL:
Whiteboard:
: 1413496 (view as bug list)
Depends On:
Blocks: OSOPS_V3
TreeView+ depends on / blocked
 
Reported: 2017-01-23 05:35 UTC by Jordan Liggitt
Modified: 2017-03-08 18:43 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, upgrading from OpenShift Container Platform 3.3 to 3.4 caused all user identities to disappear, though they were still present in etcd, and OAuth-based users could no longer log in. New 3.4 installations were also affected. This was caused by an unintentional change in the etcd prefix for user identities; egressnetworkpolicies were similarly affected. This bug fix restores the previous etcd prefix for user identities and egressnetworkpolicies, and as a result users can log in again successfully. Administrators who have already installed or upgraded to v3.4.0.39 must upgrade to v3.4.0.40, then perform a data migration using a data migration tool.
Clone Of:
Environment:
Last Closed: 2017-01-24 21:09:57 UTC
Target Upstream Version:


Attachments (Terms of Use)
testcase (3.58 KB, text/plain)
2017-02-17 06:29 UTC, ge liu
no flags Details


Links
System ID Priority Status Summary Last Updated
Origin (Github) 12598 None None None 2017-01-23 05:36:52 UTC
Red Hat Product Errata RHBA-2017:0186 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.4.0.40 bug fix update 2017-01-25 02:09:07 UTC

Description Jordan Liggitt 2017-01-23 05:35:14 UTC
Description of problem:

The etcd prefix for user identities was unintentionally changed during 3.4 development.

This means that upgrading from 3.3 to 3.4 makes all user identities disappear (though they are still present in etcd).



Version-Release number of selected component (if applicable):


How reproducible:
Always


Steps to Reproduce:
1. Start 3.3
2. Log in or create user identities
3. Upgrade to 3.4

Actual results:

User identities are not visible, logins no longer work


Expected results:


Additional info:

Comment 9 ge liu 2017-01-24 08:36:57 UTC
Verified with: v3.4.0.40,

Verify Steps:

1). Install openshift 3.3.1.11 
     - oc login with user1
     - oc create -f old-objects.json
     - oc get identity:

                ...........
                 name: geliu
                 ..............
                  source: original
                  name: x:a
                ............
                 source: original
                 name: x:b

      - oc get egressnetworkpolicy :
                 kind: EgressNetworkPolicy
                 source: original
                   name: a
                 ................
                 source: original
                   name: b
2). upgrade to openshift 3.4.0.39
     - oc login with user1 successfully
     - # oc get identity
           No resources found.
       # oc get egressnetworkpolicy
           No resources found.
     - oc create -f new-objects.json
     - # oc create -f new-object.json 
              egressnetworkpolicy "b" created
              egressnetworkpolicy "c" created
              identity "x:b" created
              identity "x:c" created
     - # oc get identity
           NAME      IDP NAME   IDP USER NAME   USER NAME   USER UID
             x:b       x          b                            
             x:c       x          c                           
     - # oc get egressnetworkpolicy
           NAME
             b
             c

3). upgrade to 3.4.0.40

     - oc login with user1 successfully
      
     - # oc get identity
       NAME              IDP NAME    IDP USER NAME   USER NAME   USER UID

        allow_all:geliu   allow_all   geliu           geliu      xxxxxx
        x:a               x           a                           
        x:b               x           b                          
     - # oc get egressnetworkpolicy 
          NAME
           a
           b
     both identity a,b and egressnetworkpolicy a,b are meet: "source: original"


latest upgrade openshift version:

openshift v3.4.0.40
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0

Comment 10 Stefanie Forrester 2017-01-24 16:33:28 UTC
*** Bug 1413496 has been marked as a duplicate of this bug. ***

Comment 12 errata-xmlrpc 2017-01-24 21:09:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0186

Comment 13 ge liu 2017-02-07 08:28:35 UTC
@jliggitt@redhat.com, could u help to confirm with customer which  "mappingMethod" mode used in master-config.yaml file, it related with the 
customer's 'login fail‘ error, we need this info to improve our upgrad testcase, thank in advance:


#################customer situation#######################
Actual results:

User identities are not visible, logins no longer work
############################################################

master-config.yaml:

####################################
oauthConfig:
..............
    mappingMethod: claim  => default is claim, there are some other mode, we want to know which mode the customer used
####################################

Comment 14 Jordan Liggitt 2017-02-08 05:20:23 UTC
mappingMethod "claim" and "lookup" would both encounter login problems if the mapping between a user and identity was broken.

Comment 15 ge liu 2017-02-17 06:29:06 UTC
Created attachment 1251792 [details]
testcase

testcase url: https://url.corp.redhat.com/01421f4


Note You need to log in before you can comment on or make changes to this bug.