Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1628834

Summary: [3.10] master static pod failed to start when kerberos auth is set
Product: OpenShift Container Platform Reporter: Johnny Liu <jialiu>
Component: DocumentationAssignee: Vikram Goyal <vigoyal>
Status: CLOSED DEFERRED QA Contact: scheng
Severity: medium Docs Contact: Vikram Goyal <vigoyal>
Priority: high    
Version: 3.10.0CC: aos-bugs, jialiu, jokerman, mgugino, mmccomas, vrutkovs, wsun
Target Milestone: ---Keywords: Regression
Target Release: 3.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1628837 (view as bug list) Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1628837    
Attachments:
Description Flags
installation log with inventory file embedded none

Description Johnny Liu 2018-09-14 07:16:04 UTC 
Created attachment 1483203 [details]
installation log with inventory file embedded

Description of problem:

Version-Release number of the following components:
openshift-ansible-3.10.47-1.git.0.95bc2d2.el7_5.noarch

How reproducible:
Always

Steps to Reproduce:
1. Enable kerberos auth for a fresh install
openshift_master_identity_providers=[{'name': 'kerberos_auth', 'login': 'true', 'challenge': 'true', 'mappingMethod': 'claim', 'kind': 'RequestHeaderIdentityProvider', 'headers': ['X-Remote-User'], 'challengeURL': 'https://host-8-252-241.host.centralci.eng.rdu2.redhat.com/challenging-proxy/oauth/authorize?${query}', 'loginURL': 'https://xxx/login-proxy/oauth/authorize?${query}', 'clientCA': '/etc/origin/master/ca.crt'}]
2.
3.

Actual results:
master api failed to start.
api log:
<--snip-->
I0914 06:39:00.315873       1 plugins.go:84] Registered admission plugin "StorageObjectInUseProtection"
Invalid MasterConfig /etc/origin/master/master-config.yaml
  oauthConfig.identityProvider[0].provider.clientCA: Invalid value: "/etc/origin/master/kerberos_auth_request_header_ca.crt": could not read file: stat /etc/origin/master/kerberos_auth_request_header_ca.crt: no such file or directory

In QE's testing, we do not intend to specify a local customized ca file, just want to utilize the master ca.crt generated when master bootstrap. The was working well for several years, now this becomes broken.

Expected results:
No breaking for install

Additional info:
This is similar to BZ#1627764, I dig into code a little:
    def __init__(self, api_version, idp):
        super(RequestHeaderIdentityProvider, self).__init__(api_version, idp)
        self._allow_additional = False
        self._required += [['headers']]
        self._optional += [['challengeURL', 'challenge_url'],
                           ['loginURL', 'login_url'],
                           ['clientCA', 'client_ca'],
                           ['clientCommonNames', 'client_common_names'],
                           ['emailHeaders', 'email_headers'],
                           ['nameHeaders', 'name_headers'],
                           ['preferredUsernameHeaders', 'preferred_username_headers']]
        self._idp['clientCA'] = \
            '/etc/origin/master/{}_request_header_ca.crt'.format(idp['name'])

I think _idp['clientCA' should be set *ONLY* when clientCA is not specified by user in openshift_master_identity_providers.
Comment 1 Vadim Rutkovsky 2018-09-17 13:08:41 UTC 
Right, `clientCA` is now ignored on 3.10+, as its not clear where this file should be present.

`openshift_master_request_header_ca_file` or `openshift_master_request_header_ca` should be set to have this file copied as `/etc/origin/master/<name>_request_header_ca.crt` - does it work when these variables set?
Comment 2 Johnny Liu 2018-09-18 01:19:26 UTC 
(In reply to Vadim Rutkovsky from comment #1)
> Right, `clientCA` is now ignored on 3.10+, as its not clear where this file
> should be present.
> 
> `openshift_master_request_header_ca_file` or
> `openshift_master_request_header_ca` should be set to have this file copied
> as `/etc/origin/master/<name>_request_header_ca.crt` - does it work when
> these variables set?

According to doc:
openshift_master_request_header_ca_file=<path to local ca file to use>

The ca should be some local ca file in my ansible controller host, but in my test case, I want to use '/etc/origin/master/ca.crt' which is generated by openshift_ca and located in master host. So openshift_master_request_header_ca_file and openshift_master_request_header_ca do not help my testing.
Comment 3 Vadim Rutkovsky 2018-09-18 08:29:11 UTC 
(In reply to Johnny Liu from comment #2)
> The ca should be some local ca file in my ansible controller host, but in my
> test case, I want to use '/etc/origin/master/ca.crt' which is generated by
> openshift_ca and located in master host.


In this case it needs to be renamed and put in the specified location. This has been introduced in https://github.com/openshift/openshift-ansible/pull/9731 and docs have not been updated yet.
Comment 4 Johnny Liu 2018-09-18 09:32:53 UTC 
(In reply to Vadim Rutkovsky from comment #3)
> (In reply to Johnny Liu from comment #2)
> > The ca should be some local ca file in my ansible controller host, but in my
> > test case, I want to use '/etc/origin/master/ca.crt' which is generated by
> > openshift_ca and located in master host.
> 
> 
> In this case it needs to be renamed and put in the specified location. This
> has been introduced in
> https://github.com/openshift/openshift-ansible/pull/9731 and docs have not
> been updated yet.

You mean I have to prepare the client CA file in my local ansible host before install, and set openshift_master_request_header_ca_file to my local path, have no way to utilize /etc/origin/master/ca.crt, right?

And other critical issue, even I did not set clientCA in openshift_master_identity_providers, installer would set _idp['clientCA'] automatically, and set it to some non-existing file.
Comment 5 Vadim Rutkovsky 2018-09-18 10:02:52 UTC 
(In reply to Johnny Liu from comment #4)
> (In reply to Vadim Rutkovsky from comment #3)
> > (In reply to Johnny Liu from comment #2)
> > > The ca should be some local ca file in my ansible controller host, but in my
> > > test case, I want to use '/etc/origin/master/ca.crt' which is generated by
> > > openshift_ca and located in master host.
> > 
> > 
> > In this case it needs to be renamed and put in the specified location. This
> > has been introduced in
> > https://github.com/openshift/openshift-ansible/pull/9731 and docs have not
> > been updated yet.
> 
> You mean I have to prepare the client CA file in my local ansible host
> before install, and set openshift_master_request_header_ca_file to my local
> path, have no way to utilize /etc/origin/master/ca.crt, right?

There are three options here:
1) Set openshift_master_request_header_ca with contents
2) Set openshift_master_request_header_ca_file pointing to a local path
3) Prepare a host with /etc/origin/master/<name>_request_header_ca.crt already being present.

There is no way to specify the path and point it to /ca.crt - this caused to errors for customers, especially with static pods - i.e. two CA for different identification providers, paths which were not mounted in the static pods etc.

> 
> And other critical issue, even I did not set clientCA in
> openshift_master_identity_providers, installer would set _idp['clientCA']
> automatically, and set it to some non-existing file.

Correct, since openshift_master_request_header_ca/openshift_master_request_header_ca_file were not set the installer assumes the user is responsible for this file to be present there.
Comment 6 Michael Gugino 2018-11-29 19:17:36 UTC 
Moving to documentation.
Comment 7 Stephen Cuppett 2019-11-20 18:57:55 UTC 
OCP 3.6-3.10 is no longer on full support [1]. Marking CLOSED DEFERRED. If you have a customer case with a support exception or have reproduced on 3.11+, please reopen and include those details. When reopening, please set the Target Release to the appropriate version where needed.

[1]: https://access.redhat.com/support/policy/updates/openshift
Comment 8 Stephen Cuppett 2019-11-20 18:58:49 UTC 
OCP 3.6-3.10 is no longer on full support [1]. Marking CLOSED DEFERRED. If you have a customer case with a support exception or have reproduced on 3.11+, please reopen and include those details. When reopening, please set the Target Release to the appropriate version where needed.

[1]: https://access.redhat.com/support/policy/updates/openshift