Bug 1581876

Summary: [free-int] docker registry hostname not set in master-config
Product: OpenShift Container Platform Reporter: Justin Pierce <jupierce>
Component: InstallerAssignee: Scott Dodson <sdodson>
Status: CLOSED ERRATA QA Contact: liujia <jiajliu>
Severity: high Docs Contact:
Priority: medium    
Version: 3.10.0CC: aos-bugs, bparees, jiajliu, jokerman, mmccomas, vrutkovs
Target Milestone: ---Keywords: DeliveryBlocker
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-30 19:16:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Pod in ImagePullBackOff none

Description Justin Pierce 2018-05-23 20:11:59 UTC
Created attachment 1440749 [details]
Pod in ImagePullBackOff

Description of problem:
Pods began ImagePullBackoff after upgrading to v3.10.0-0.50.0 . Pod was attempting to pull IP based image URL and getting a certificate error. Docker registry hostname was not present in master-config : https://docs.openshift.org/latest/install_config/registry/extended_registry_configuration.html#setting-the-registry-hostname

Version-Release number of selected component (if applicable):
v3.10.0-0.50.0

Additional info:
- See attachment for pod listing with error condition. 
- Adding OPENSHIFT_DEFAULT_REGISTRY=docker-registry.default.svc:5000 to master.env allowed pod to start pulling again, but we are speculating this setting should in the master-config & openshift-ansible.

Comment 1 Scott Dodson 2018-05-24 12:46:29 UTC
Ben,

Where's the most appropriate place to set this?

Previously we've been setting the environment variable in /etc/sysconfig/atomic-openshift-master-{api,controllers} but if a YAML based configuration variable is appropriate it seems easier to maintain it there.

Is this a candidate for devex taking ownership?

Comment 2 Scott Dodson 2018-05-24 13:39:27 UTC
The only reason that this wouldn't have been set is because we didn't sign the certificate with the hostname 'docker-registry.default.svc' prior to 3.7. However in 3.7 and 3.9 upgrades we've been updating the certificate so effectively by the time that they get to 3.10 we should have 100% assurance that the certificate has been signed with hostname so there's no need for conditional logic any longer.

We should default this in the product unless there's a reason not to do so. And if there's a reason not to do so, like `oc cluster up` needs to reconfigure it or something, then we should force it via openshift-ansible.

Comment 4 Ben Parees 2018-05-24 15:51:17 UTC
yeah i think it should be in the master-config and i can't think of a reason not to default it... if we're setting the registry url on the registry DC, we should be setting it in the master configuration also.

As for us taking ownership, sure, but we'll need pointers :)

Comment 5 Scott Dodson 2018-05-24 18:57:27 UTC
Ended up taking care of it, this forces the value during upgrade via master-config.yaml and it makes sure if they've specified openshift_master_image_policy dictionary that we merge in the value we care about.

https://github.com/openshift/openshift-ansible/pull/8521

Comment 6 Vadim Rutkovsky 2018-05-25 09:34:57 UTC
Fix is in openshift-ansible-3.10.0-0.52.0

Comment 7 Scott Dodson 2018-05-25 12:41:32 UTC
*** Bug 1571608 has been marked as a duplicate of this bug. ***

Comment 8 liujia 2018-05-28 11:02:35 UTC
Verified on openshift-ansible-3.10.0-0.53.0.git.0.53fe016.el7.noarch

After upgrade, sti-build still push image with docker registry's DNS address. And docker registry was added into master-config.

# oc get is
NAME                     DOCKER REPO                                                      TAGS      UPDATED
nodejs-mongodb-example   docker-registry.default.svc:5000/mytest/nodejs-mongodb-example             

# cat /etc/origin/master/master-config.yaml|grep -A 1 imagePolicyConfig
imagePolicyConfig:
  internalRegistryHostname: docker-registry.default.svc:5000

Comment 10 errata-xmlrpc 2018-07-30 19:16:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816