Created attachment 1440749 [details] Pod in ImagePullBackOff Description of problem: Pods began ImagePullBackoff after upgrading to v3.10.0-0.50.0 . Pod was attempting to pull IP based image URL and getting a certificate error. Docker registry hostname was not present in master-config : https://docs.openshift.org/latest/install_config/registry/extended_registry_configuration.html#setting-the-registry-hostname Version-Release number of selected component (if applicable): v3.10.0-0.50.0 Additional info: - See attachment for pod listing with error condition. - Adding OPENSHIFT_DEFAULT_REGISTRY=docker-registry.default.svc:5000 to master.env allowed pod to start pulling again, but we are speculating this setting should in the master-config & openshift-ansible.
Ben, Where's the most appropriate place to set this? Previously we've been setting the environment variable in /etc/sysconfig/atomic-openshift-master-{api,controllers} but if a YAML based configuration variable is appropriate it seems easier to maintain it there. Is this a candidate for devex taking ownership?
The only reason that this wouldn't have been set is because we didn't sign the certificate with the hostname 'docker-registry.default.svc' prior to 3.7. However in 3.7 and 3.9 upgrades we've been updating the certificate so effectively by the time that they get to 3.10 we should have 100% assurance that the certificate has been signed with hostname so there's no need for conditional logic any longer. We should default this in the product unless there's a reason not to do so. And if there's a reason not to do so, like `oc cluster up` needs to reconfigure it or something, then we should force it via openshift-ansible.
yeah i think it should be in the master-config and i can't think of a reason not to default it... if we're setting the registry url on the registry DC, we should be setting it in the master configuration also. As for us taking ownership, sure, but we'll need pointers :)
Ended up taking care of it, this forces the value during upgrade via master-config.yaml and it makes sure if they've specified openshift_master_image_policy dictionary that we merge in the value we care about. https://github.com/openshift/openshift-ansible/pull/8521
Fix is in openshift-ansible-3.10.0-0.52.0
*** Bug 1571608 has been marked as a duplicate of this bug. ***
Verified on openshift-ansible-3.10.0-0.53.0.git.0.53fe016.el7.noarch After upgrade, sti-build still push image with docker registry's DNS address. And docker registry was added into master-config. # oc get is NAME DOCKER REPO TAGS UPDATED nodejs-mongodb-example docker-registry.default.svc:5000/mytest/nodejs-mongodb-example # cat /etc/origin/master/master-config.yaml|grep -A 1 imagePolicyConfig imagePolicyConfig: internalRegistryHostname: docker-registry.default.svc:5000
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1816