Bug 1701410

Summary: [OCP4 Beta] ingress-operator is not recognizing if the certificate Secret is changed
Product: OpenShift Container Platform Reporter: Stuart Auchterlonie <sauchter>
Component: NetworkingAssignee: Miciah Dashiel Butler Masters <mmasters>
Networking sub component: router QA Contact: Hongan Li <hongli>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: aos-bugs, bbennett, florin-alexandru.peter, jeder, jupierce, nagrawal, nmalik, weliang
Version: 4.1.0Keywords: DeliveryBlocker
Target Milestone: ---   
Target Release: 4.2.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-16 06:28:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Stuart Auchterlonie 2019-04-18 21:35:54 UTC
Description of problem:

Cu uses SSL certificates provided by letsencrypt.

The ingress-operator that is reconciling the router is not watching the 
certificate for changes.

Version-Release number of selected component (if applicable):

OCP HTB 3

How reproducible: 100%


Steps to Reproduce:
1. Provide custom SSL certificates for the router pod
2. Cause certificate renewal to occur
3.

Actual results:

The certificates are renewed, but the operator does not
detect this and take appropriate action

Expected results:

The operator should detect the certificate has been updated
and take appropriate action

Additional info:

Comment 3 Ben Bennett 2019-04-30 13:09:39 UTC
This is not really a release blocker because it is equivalent behavior to 3.11.  We should document the work-around in 4.1.

Miciah, if you happen to land this in 4.1 timeframe, please update the target release and let docs know.

Comment 4 Dan Mace 2019-05-01 23:39:30 UTC
*** Bug 1705297 has been marked as a duplicate of this bug. ***

Comment 6 Miciah Dashiel Butler Masters 2019-05-02 17:04:27 UTC
The workaround is to create a new secret with the updated certificate and modify the ingress controller's .spec.defaultCertificate field to reference the new secret.

Comment 7 Miciah Dashiel Butler Masters 2019-05-07 22:23:49 UTC
PR: https://github.com/openshift/router/pull/25

Comment 8 Miciah Dashiel Butler Masters 2019-05-07 22:31:41 UTC
In case they are helpful, here are the steps I used to test:

1. openssl s_client -connect xyz.apps.example.com:443 </dev/null 2>/dev/null | openssl x509 -noout -subject

2. oc -n openshift-ingress-operator get secrets/router-ca -o go-template --template='{{index .data "tls.key"}}' | base64 -d > router-ca.key

3. openssl genrsa -out router.key 2048 && openssl req -new -key router.key -out router.csr -subj '/C=US/ST=NC/L=Chocowinity/O=OS3/OU=Eng/CN=*.apps.example.com'

4. openssl x509 -req -days 1 -in router.csr -signkey router.key -CAkey router-ca.key -out router.crt

5. oc create secret tls router-certs-default --cert=router.crt --key=router.key --namespace=openshift-ingress --dry-run -o yaml > new-secret.yaml

6. oc replace -f new-secret.yaml

7. oc -n openshift-ingress rsh deploy/router-default ls -l /etc/pki/tls/private/..data

8. openssl s_client -connect xyz.apps.example.com:443 </dev/null 2>/dev/null | openssl x509 -noout -subject

Step 1 should show the old certificate's CN.  Step 8 should show the new certificate's CN.  Step 7 is useful to see if the secret has actually been updated yet—in my testing, it could take a couple of minutes from the `oc replace` command before the secret actually changed in the pod.

Comment 10 Weibin Liang 2019-06-25 21:59:53 UTC
Verified in 4.2.0-0.nightly-2019-06-21-041727.

# The output from step1
[root@dhcp-41-193 FILE]# openssl s_client -connect xyz.apps.qe-weliang-6255.qe.devcluster.openshift.com:443 </dev/null 2>/dev/null | openssl x509 -noout -subject
subject= /CN=*.apps.qe-weliang-6255.qe.devcluster.openshift.com

# The output from step8
[root@dhcp-41-193 FILE]# openssl s_client -connect xyz.apps.qe-weliang-6255.qe.devcluster.openshift.com:443 </dev/null 2>/dev/null | openssl x509 -noout -subject
subject= /C=US/ST=NC/L=Chocowinity/O=OS3/OU=Eng/CN=*.apps.qe-weliang-6255.qe.devcluster.openshift.com

Comment 11 errata-xmlrpc 2019-10-16 06:28:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922