Bug 2019710

Summary: When global pull secret is updated, openshift-apiserver does not reflect it.
Product: OpenShift Container Platform Reporter: Seunghwan Jung <jseunghw>
Component: Image RegistryAssignee: Oleg Bulatov <obulatov>
Status: CLOSED DUPLICATE QA Contact: XiuJuan Wang <xiuwang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.8CC: aos-bugs, dramseur, mfojtik, ngutu
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-23 11:04:31 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Seunghwan Jung 2021-11-03 08:17:03 UTC
Description of problem:

When global pull secret is updated, openshift-apiserver pod do not reflect it,
resulting in 'oc import-image' command to fail.


How reproducible:
always according to my test on OCP4.7 and 4.8. 

Steps to Reproduce:
1. Your pull secret should have been changed. 
2. Update the global pull secret on your cluster with current one according to doc[1].
3. Once applied to the cluster, check the global pull secret and one that is mounted by openshift-apiserver pods.

$ oc get secret pull-secret -n openshift-config --template='{{index .data ".dockerconfigjson" | base64decode}}' | jq '.auths."registry.redhat.io".auth' > global-pullsecret

$ oc -n openshift-apiserver rsh apiserver-XXXXnnnn-xxxx cat /var/lib/kubelet/config.json | jq '.auths."registry.redhat.io".auth'  > apipod-pullsecret

$ diff global-pullsecret apipod-pullsecret

Actual results:
openshift-apiserver pods still have old pull secret. 

Expected results:
openshift-apiserver pods should have the updated pull secret mounted. 


Additional info:

To workaround it, delete apiserver deployment to recreate the pods.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ oc -n openshift-apiserver delete deployment apiserver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


[1]https://docs.openshift.com/container-platform/4.8/openshift_images/managing_images/using-image-pull-secrets.html#images-update-global-pull-secret_using-image-pull-secrets

Comment 1 Stefan Schimanski 2021-11-03 09:23:17 UTC
Moving to image registry. They own the configuration of the OpenShift components to use the pull secret.

Comment 2 Oleg Bulatov 2021-11-23 11:04:31 UTC

*** This bug has been marked as a duplicate of bug 1984592 ***