Bug 1418191 - Getting 'Failed to pull image .... x509: certificate signed by unknown authority', after redeployed certificates
Summary: Getting 'Failed to pull image .... x509: certificate signed by unknown author...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.3.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Andrew Butcher
QA Contact: Gaoyun Pei
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-01 08:58 UTC by Takayuki Konishi
Modified: 2017-07-24 14:11 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously the registry certificate was not properly updated when running the certificate re-deploy playbooks which may have prevented pushing or pulling images. The playbooks have been updated to ensure that the registry certificate is correctly updated.
Clone Of:
Environment:
Last Closed: 2017-04-12 18:49:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0903 0 normal SHIPPED_LIVE OpenShift Container Platform atomic-openshift-utils bug fix and enhancement 2017-04-12 22:45:42 UTC

Description Takayuki Konishi 2017-02-01 08:58:07 UTC
Description of problem:
After this bug [1][2] fixed, docker-registry is secured by default.
However, redeploy-certificates.yml does not handle the server certificate.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1373106
[2] https://access.redhat.com/errata/RHBA-2016:2122

Version-Release number of selected component (if applicable):
v3.3.1.11

How reproducible: 100%

Steps to Reproduce:
1. Install OSCP 3.3 (In my env, 1 master and 1 node)
2. ansible-playbook -i <inventory> playbooks/byo/openshift-cluster/redeploy-certificates.yml --extra-vars "openshift_certificates_redeploy_ca=true"
3. Create a project, add an application then build it.

Actual results:
Fail to push the image because of unknown authority

Pushing image 172.30.252.228:5000/test/test:latest ...

E0201 07:55:23.092616       1 dockerutil.go:92] push for image 172.30.252.228:5000/test/test:latest failed, will retry in 5s ...
Registry server Address: 
Registry server User Name: serviceaccount
Registry server Email: serviceaccount
Registry server Password: <<non-empty>>
error: build error: Failed to push image: unable to ping registry endpoint https://172.30.252.228:5000/v0/
v2 ping attempt failed with error: Get https://172.30.252.228:5000/v2/: x509: certificate signed by unknown authority
 v1 ping attempt failed with error: Get https://172.30.252.228:5000/v1/_ping: x509: certificate signed by unknown authority

Expected results:

Succeed to push the image

Additional info:

Workaround: renew the server certificate by hand.

REGISTRY_IP=$(oc get service docker-registry --template='{{.spec.clusterIP}}')
oadm ca create-server-cert \
    --signer-cert=/etc/origin/master/ca.crt \
    --signer-key=/etc/origin/master/ca.key \
    --signer-serial=/etc/origin/master/ca.serial.txt \
    --hostnames="docker-registry.default.svc.cluster.local,$REGISTRY_IP" \
    --cert=/etc/origin/master/registry.crt \
    --key=/etc/origin/master/registry.key
oc delete secrets registry-certificates
oc secrets new registry-certificates \
    /etc/origin/master/registry.crt \
    /etc/origin/master/registry.key
oc deploy docker-registry --latest
sleep 15 # wait for deploy

-------
These bugs may be related to this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1397958
https://bugzilla.redhat.com/show_bug.cgi?id=1388691

Comment 3 Gaoyun Pei 2017-02-10 07:28:39 UTC
Test this with openshift-ansible-3.5.6-1.git.0.5e6099d.el7.noarch.

For an ocp-3.5 cluster, redeploy the openshift CA certificate files with running command:
ansible-playbook -i inventory playbooks/byo/openshift-cluster/redeploy-openshift-ca.yml

After playbook finished, create an app, sti-build could be done successfully, built image could be pushed into docker-registry.


Redeploy all openshift certificates(etcd/master/node/router/registry certificates) with running command:
ansible-playbook -i inventory playbooks/byo/openshift-cluster/redeploy-certificates.yml

After playbook finished, create an app, sti-build could be done, image could be pushed into docker-registry.

Comment 5 errata-xmlrpc 2017-04-12 18:49:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0903


Note You need to log in before you can comment on or make changes to this bug.