Bug 1570935

Summary: Upgrade should terminate early if it detects files that will be inaccessible after upgrade
Product: OpenShift Container Platform Reporter: Justin Pierce <jupierce>
Component: InstallerAssignee: Michael Gugino <mgugino>
Status: CLOSED ERRATA QA Contact: liujia <jiajliu>
Severity: high Docs Contact:
Priority: high    
Version: 3.10.0CC: aos-bugs, bleanhar, geliu, jokerman, kdube, mmccomas
Target Milestone: ---   
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-30 19:13:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Description Flags
Problematic files on free-int none

Description Justin Pierce 2018-04-23 18:50:29 UTC
Created attachment 1425719 [details]
Problematic files on free-int

Description of problem:
On the starter clusters, there are at least two files referenced outside of /etc/origin/master which would have caused the control-plane to fail to start after a successful upgrade (see attachment).

Version-Release number of the following components:

How reproducible:
By design in v3.10

Actual results:
The upgrade proceeded without report this potential problem. 

Expected results:
Prior to upgrading v3.9 to v3.10, the upgrade should check for files which could cause the control-plane to fail. If any problematic paths are detected, the upgrade should abort and inform the user.

** I've asked the online team to move these particular files, but others might be lurking on non-starter clusters.

Comment 1 Scott Dodson 2018-04-23 19:35:30 UTC
I think we need to scan /etc/origin/master/master-config.yaml for any path that's not /etc/origin/master and /etc/origin/cloudprovider and error.

Comment 4 Michael Gugino 2018-05-01 19:29:30 UTC
PR merged: https://github.com/openshift/openshift-ansible/pull/8189

I need to file a follow-on patch to check to ensure no cert paths have been modified for auth providers (other than htpasswd).

Comment 5 Scott Dodson 2018-05-02 14:16:44 UTC
In openshift-ansible-3.10.0-0.32.0

Comment 6 Scott Dodson 2018-05-02 15:54:23 UTC
*** Bug 1565555 has been marked as a duplicate of this bug. ***

Comment 7 liujia 2018-05-03 09:41:05 UTC

Scenario1: Abort upgrade if needed files(not in oauthConfig.identityProviders) were not placed in master pod's mounted path.

1. rpm install ocp
2. update scheduler.json path in master-config.yaml
schedulerConfigFile: /etc/origin/master/scheduler.json
schedulerConfigFile: /etc/origin/scheduler.json
3. move scheduler.json from /etc/origin/master/ to /etc/origin/
4. restart master service
5. upgrade above ocp

Upgrade will abort without any changes in pre check.

Failure summary:

  1. Hosts:    x.x.x.x
     Play:     Retrieve existing master configs and validate
     Task:     Check for file paths outside of /etc/origin/master in master's config
     Message:  A string value that appears to be a file path located outside of
               /etc/origin/master/, /var/lib/origin, /etc/origin/cloudprovider has been found in /etc/origin/master/master-config.yaml.
               In 3.10 and newer, all files needed by the master must reside inside of
               those directories or a subdirectory or it will not be readable by the
               master process. Please migrate all files needed by the master into
               one of /etc/origin/master/, /var/lib/origin, /etc/origin/cloudprovider or a subdirectory and update your master configs before
               proceeding. The string found was: /etc/origin/scheduler.json
               NOTE: the following items do not need to be migrated, they will be migrated
               for you: oauthConfig.identityProviders

Scenario2: upgrade succeed with migrating existed files to mounted path when needed files(oauthConfig.identityProviders) were not placed in master pod's mounted path.(refer to https://bugzilla.redhat.com/show_bug.cgi?id=1570539)

1. rpm install with htpasswd auth.
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/htpasswd'}]

# ls -la /etc/origin/|grep ht
-rw-r--r--.  1 root root   45 May  3 03:18 htpasswd

# cat /etc/origin/master/master-config.yaml|grep htpasswd
    name: htpasswd_auth
      file: /etc/origin/htpasswd

2. upgrade above ocp

Upgrade succeed with htpasswd file moved to mounted path(/etc/origin/master/)
# cat /etc/origin/master/master-config.yaml|grep htpasswd
    name: htpasswd_auth
      file: /etc/origin/master/htpasswd

# ls -la /etc/origin/master/ |grep ht
-rw-------. 1 root root     45 May  3 04:08 htpasswd

Comment 8 liujia 2018-05-03 09:41:41 UTC
Added test cases about above two scenarios.

Comment 10 errata-xmlrpc 2018-07-30 19:13:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.