Bug 1591632

Summary: master controllers edits the secret after upgrade to 3.9
Product: OpenShift Container Platform Reporter: Vladislav Walek <vwalek>
Component: MasterAssignee: Michal Fojtik <mfojtik>
Status: CLOSED ERRATA QA Contact: ge liu <geliu>
Severity: high Docs Contact:
Priority: urgent    
Version: 3.9.0CC: aos-bugs, bleanhar, bmchugh, geliu, haowang, jchevret, jgoulding, jokerman, jrosenta, mfojtik, mmccomas, rhowe, smunilla, stobin, tbielawa, vwalek
Target Milestone: ---Keywords: OpsBlocker
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-18 09:18:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vladislav Walek 2018-06-15 07:20:28 UTC
Description of problem:

On a 3.9.30 cluster, when restarting the "atomic-openshift-master-controllers" service, all secrets are overwritten. This is a big issue, since this makes the cluster practically not usable. in attachment what we see, first "oc get secret", than restart the services, again "oc get secret", you can see the value changed. In the logs of the controller I see this:

Jun 14 18:17:11 master-1 atomic-openshift-master-controllers[72597]: I0614 18:17:11.439085   72597 start_master.go:652] Starting "openshift.io/serviceaccount-pull-secrets"
Jun 14 18:17:11 master-1 atomic-openshift-master-controllers[72597]: I0614 18:17:11.455637   72597 start_master.go:662] Started "openshift.io/serviceaccount-pull-secrets"

all secrets containing docker credentials for private docker repo's where changed. instead of the url to the private docker repo, they now contained the IP for the integrated registry. This means no images couldn't be pulled from the registries anymore.

and with the loglevel set to 6, and 1 secret modified to its correct value again, you see the secret gets overwritten.
Jun 14 18:37:20 master-1 atomic-openshift-master-controllers[79076]: I0614 18:37:18.237210   79076 round_trippers.go:436] PUT https://master-1:8443/api/v1/namespaces/<namespace>/secrets/registrysecret-k8s 200 OK in 22 milliseconds

Version-Release number of selected component (if applicable):
OpenShift Container Platform 3.9.30



Additional info:
will attach all the logs

Comment 4 Michal Fojtik 2018-06-15 11:47:53 UTC
The fix was merged and should be available in next 3.9.z update.

Comment 6 ge liu 2018-06-19 09:28:51 UTC
@vwalek, is there any key difference between our recreate steps? thanks in advance!

Comment 11 Ryan Howe 2018-06-28 18:10:51 UTC
*** Bug 1596333 has been marked as a duplicate of this bug. ***

Comment 26 errata-xmlrpc 2018-07-18 09:18:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2213