Bug 1418191

Summary: Getting 'Failed to pull image .... x509: certificate signed by unknown authority', after redeployed certificates
Product: OpenShift Container Platform Reporter: Takayuki Konishi <tkonishi>
Component: InstallerAssignee: Andrew Butcher <abutcher>
Status: CLOSED ERRATA QA Contact: Gaoyun Pei <gpei>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.3.1CC: abutcher, aos-bugs, jokerman, mmccomas, nhashimo, rromerom
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Previously the registry certificate was not properly updated when running the certificate re-deploy playbooks which may have prevented pushing or pulling images. The playbooks have been updated to ensure that the registry certificate is correctly updated.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-12 18:49:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Takayuki Konishi 2017-02-01 08:58:07 UTC 
Description of problem:
After this bug [1][2] fixed, docker-registry is secured by default.
However, redeploy-certificates.yml does not handle the server certificate.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1373106
[2] https://access.redhat.com/errata/RHBA-2016:2122

Version-Release number of selected component (if applicable):
v3.3.1.11

How reproducible: 100%

Steps to Reproduce:
1. Install OSCP 3.3 (In my env, 1 master and 1 node)
2. ansible-playbook -i <inventory> playbooks/byo/openshift-cluster/redeploy-certificates.yml --extra-vars "openshift_certificates_redeploy_ca=true"
3. Create a project, add an application then build it.

Actual results:
Fail to push the image because of unknown authority

Pushing image 172.30.252.228:5000/test/test:latest ...

E0201 07:55:23.092616       1 dockerutil.go:92] push for image 172.30.252.228:5000/test/test:latest failed, will retry in 5s ...
Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount
Registry server Password: <<non-empty>>
error: build error: Failed to push image: unable to ping registry endpoint https://172.30.252.228:5000/v0/
v2 ping attempt failed with error: Get https://172.30.252.228:5000/v2/: x509: certificate signed by unknown authority
 v1 ping attempt failed with error: Get https://172.30.252.228:5000/v1/_ping: x509: certificate signed by unknown authority

Expected results:

Succeed to push the image

Additional info:

Workaround: renew the server certificate by hand.

REGISTRY_IP=$(oc get service docker-registry --template='{{.spec.clusterIP}}')
oadm ca create-server-cert \
    --signer-cert=/etc/origin/master/ca.crt \
    --signer-key=/etc/origin/master/ca.key \
    --signer-serial=/etc/origin/master/ca.serial.txt \
    --hostnames="docker-registry.default.svc.cluster.local,$REGISTRY_IP" \
    --cert=/etc/origin/master/registry.crt \
    --key=/etc/origin/master/registry.key
oc delete secrets registry-certificates
oc secrets new registry-certificates \
    /etc/origin/master/registry.crt \
    /etc/origin/master/registry.key
oc deploy docker-registry --latest
sleep 15 # wait for deploy

-------
These bugs may be related to this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1397958
https://bugzilla.redhat.com/show_bug.cgi?id=1388691
Comment 3 Gaoyun Pei 2017-02-10 07:28:39 UTC 
Test this with openshift-ansible-3.5.6-1.git.0.5e6099d.el7.noarch.

For an ocp-3.5 cluster, redeploy the openshift CA certificate files with running command:
ansible-playbook -i inventory playbooks/byo/openshift-cluster/redeploy-openshift-ca.yml

After playbook finished, create an app, sti-build could be done successfully, built image could be pushed into docker-registry.


Redeploy all openshift certificates(etcd/master/node/router/registry certificates) with running command:
ansible-playbook -i inventory playbooks/byo/openshift-cluster/redeploy-certificates.yml

After playbook finished, create an app, sti-build could be done, image could be pushed into docker-registry.
Comment 5 errata-xmlrpc 2017-04-12 18:49:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0903