Document URL: https://docs.openshift.com/container-platform/3.3/install_config/redeploying_certificates.html Section Number and Name: Describe the issue: Documentation does not explain how to update the router (and registry - however the registry is less of an issue), certificates after this process is run. Because of these updates (and restarts of pods caused by: https://github.com/openshift/openshift-ansible/blob/master/playbooks/common/openshift-cluster/redeploy-certificates.yml#L207-L246) An outage to the "data" plan is possible because the routers are restarted, as part of "node evacuations", which result in the routers not being able to talk to the masters, because they have old/bad certificates. This causes you to see issues described by: https://bugzilla.redhat.com/1387714 Suggestions for improvement: At the very least we need to explain how to update the certificates for the router / registry (quickly) to minimize the outage. > Update the registry certificates manually. Edit the deployment config for the registry and update the OPENSHIFT_CA_DATA, OPENSHIFT_CERT_DATA and OPENSHIFT_KEY_DATA environment variables. >> If you are using the OpenShift CA for the registry certificate you can copy the contents of /etc/origin/master/ca.crt /etc/origin/master/openshift-registry.crt /etc/origin/master/openshift-registry.key in to those variables. >> Otherwise insert your own certificates. Additional information:
@Ashley - we have talked about this issue previously and am not sure if the current docs now explain this better. Please check in with Alex as he might have better guidance.
@abutcher What is the latest recommendation for this? Thanks!
Note that starting with 3.2 the router/registry don't need these env variables that contain cert-based authentication credentials anymore and can operate using token-based authentication via an associated serviceAccount: https://docs.openshift.com/enterprise/3.2/release_notes/ose_3_2_release_notes.html#ose-32-administrator-cli Therefore, an option here (for 3.2+ environments) would be to remove these variables instead of updating them (as long as proper service accounts are in place - probably worth adding some note about double checking that): # oc env dc/router OPENSHIFT_CA_DATA- OPENSHIFT_CERT_DATA- OPENSHIFT_KEY_DATA- # oc env dc/docker-registry OPENSHIFT_CA_DATA- OPENSHIFT_CERT_DATA- OPENSHIFT_KEY_DATA- An advantage of this is that the tokens (secrets/SA) should be automatically updated if the API certs ever change again, so this would be a one-off change.
The installer currently uses this process to automatically secure the registry during install: https://docs.openshift.com/container-platform/3.4/install_config/registry/securing_and_exposing_registry.html#securing-the-registry Registry certificates are added to a secret named "registry-certificates". In order to update the certificate a new cert must be created, updated within the "registry-certificates" secret and then the registry must be redeployed. REGISTRY_IP=`oc get service docker-registry -o jsonpath='{.spec.clusterIP}'` REGISTRY_HOSTNAME=`oc get route/docker-registry -o jsonpath='{.spec.host}'` oc adm ca create-server-cert \ --signer-cert=/etc/origin/master/ca.crt \ --signer-key=/etc/origin/master/ca.key \ --hostnames=$REGISTRY_IP,docker-registry.default.svc.cluster.local,$REGISTRY_HOSTNAME \ --cert=/etc/origin/master/registry.crt \ --key=/etc/origin/master/registry.key \ --signer-serial=/etc/origin/master/ca.serial.txt oc secret new registry-certificates \ /etc/origin/master/registry.crt \ /etc/origin/master/registry.key \ -o json | oc replace -f - oc deploy dc/docker-registry --latest The router is secured using a service serving certificate secret which was automatically created after adding an annotation to the router service. We can trigger that service serving certificate to be recreated by deleting the service and then clearing/re-adding annotations to the router service. This process may change / get simplified if the service serving certificate is automatically recreated after being deleted. Ref: https://github.com/openshift/origin/issues/12834. oc delete secret router-certs oc annotate service router \ service.alpha.openshift.io/serving-cert-secret-name- \ service.alpha.openshift.io/serving-cert-signed-by- oc annotate service router \ service.alpha.openshift.io/serving-cert-secret-name=router-certs oc deploy dc/router --latest NOTE: We're adding two new playbooks in 1.5 (part of cert redeploy) which can be executed separately to redeploy router/registry certificates. Ref: https://github.com/openshift/openshift-ansible/pull/2671. QE is still validating these new playbooks. ansible-playbook -i inventory playbooks/byo/openshift-cluster/redeploy-router-certificates.yml ansible-playbook -i inventory playbooks/byo/openshift-cluster/redeploy-registry-certificates.yml
Work in progress for 3.3 and 3.4: https://github.com/openshift/openshift-docs/pull/3712
Original PR closed and work is continued in https://github.com/openshift/openshift-docs/pull/3851.
Johnny, can you QE the new "Redeploying Registry Certificates Manually" and "Redeploying Router Certificates Manually" sections here? http://file.rdu.redhat.com/~adellape/022817/router-registry-manual/install_config/redeploying_certificates.html#redeploying-registry-and-router-certificates Planning to include these new sections in 3.4, 3.3, and 3.2 docs. Thanks!
Verified the steps are correct on ocp 3.4, 3.3 and 3.2 env. Registry and router certificates could be updated, docker-registry and router are also working well after cert redeployed.
Commit pushed to master at https://github.com/openshift/openshift-docs https://github.com/openshift/openshift-docs/commit/778a438f4c20745a6df8db6fec5fd90dc3ab5f46 Merge pull request #3851 from adellape/router-registry-manual Bug 1388691: Manual router/registry cert redeploy
Updates are published: https://access.redhat.com/documentation/en-us/openshift_container_platform/3.4/html/installation_and_configuration/install-config-redeploying-certificates