Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1820400

Summary: Set up a watch on the default certificate secret even if it does not contain a valid certificate when openshift-router starts.
Product: OpenShift Container Platform Reporter: Miheer Salunke <misalunk>
Component: NetworkingAssignee: Miheer Salunke <misalunk>
Networking sub component: router QA Contact: Arvind iyengar <aiyengar>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: aos-bugs, hongli
Version: 4.2.z   
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Release Note
Doc Text:
Cause: https://github.com/openshift/router/pull/25/commits/b4a6b0823a2e71080eec39c021811cd603fc0be8 added logic to openshift-router to watch the default certificate secret contents and reload the default certificate when the file changed. However, the logic in the change does not set up the watch if openshift-router fails to read the secret when it starts. Consequence: So if the user (1) creates a secret (which must exist for the router pod to start) but the secret contains an invalid secret, then the user will need to (2) update the secret and then (3) delete the current router pods. Fix: If openshift-router established the watch even if it could not read the secret, then deleting the router pods would be unnecessary. So with this fix, the router watches for any changes in the default certificate secret. So if the secret is invalid due to any reason(Eg- invalid cert) then default router certificate will be used and served by the router. If the secret you mentioned is valid then the router will serve default certificate from that secret. Also using this fix you no longer need to delete the router as changes are made on the fly as the router is continuously watching for any changes done to the default router certificate secret. Result: If you make any changes to the default router secrets which are valid then they will be served by the router without the need for deleting the routers. If the default router secret is invalid default router cert will be served on the fly without deleting the router pods.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-07-13 17:25:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Miheer Salunke 2020-04-03 01:10:31 UTC 
Description of problem:

https://github.com/openshift/router/pull/25/commits/b4a6b0823a2e71080eec39c021811cd603fc0be8 added logic to openshift-router to watch the default certificate secret contents and reload the default certificate when the file changed.  However, the logic in the change does not set up the watch if openshift-router fails to read the secret when it starts.  This means that if the user (1) creates a secret (which must exist for the router pod to start) but the secret contains an invalid secret, then the user will need to (2) update the secret and then (3) delete the current router pods.  If openshift-router established the watch even if it could not read the secret, then (3) would be unnecessary.

Version-Release number of selected component (if applicable):
OCP 4.2


How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

https://issues.redhat.com/browse/NE-228

PR - https://github.com/openshift/router/pull/52
Comment 3 Miheer Salunke 2020-04-07 09:40:05 UTC 
test1 ->
[miheer@miheer router]$ cat tls.crt
nonsense
[miheer@miheer router]$ cat tls.key
nonsense
[miheer@miheer router]$ oc -n openshift-ingress create secret generic custom-default-cert --from-file=tls.crt=tls.crt
secret/custom-default-cert created
[miheer@miheer router]$ oc patch -n openshift-ingress-operator ingresscontroller/test --patch '{"spec":{"replicas": 0}}' --type=merge
ingresscontroller.operator.openshift.io/test patched
[miheer@miheer router]$ oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/test  --patch '{"spec":{"defaultCertificate":{"name":"custom-default-cert"}}}'
ingresscontroller.operator.openshift.io/test patched (no change)
[miheer@miheer router]$ oc patch -n openshift-ingress-operator ingresscontroller/test --patch '{"spec":{"replicas": 2}}' --type=merge
ingresscontroller.operator.openshift.io/test patched



[miheer@miheer router]$ oc get pods
NAME                              READY   STATUS       RESTARTS   AGE
ingress-router-10-build           0/1     Completed    0          50m
ingress-router-11-build           0/1     Completed    0          36m
ingress-router-12-build           0/1     Completed    0          32m
ingress-router-13-build           0/1     Completed    0          16m
ingress-router-3-build            0/1     Init:Error   0          9h
ingress-router-4-build            0/1     Init:Error   0          8h
ingress-router-5-build            0/1     Init:Error   0          7h49m
ingress-router-6-build            0/1     Error        0          7h49m
ingress-router-7-build            0/1     Error        0          7h32m
ingress-router-9-build            0/1     Completed    0          93m
router-default-7f7f47b998-dj6vv   1/1     Running      0          9m27s
router-default-7f7f47b998-kcflg   1/1     Running      0          9m3s
router-test-6db6b49d74-cq9xs      1/1     Running      0          104s
router-test-6db6b49d74-p66t4      1/1     Running      0          104s
[miheer@miheer router]$ oc logs router-test-6db6b49d74-cq9xs
2019-10-11T22:31:54.998Z    INFO    router.router   router/template.go:293  starting router {"version": "v0.0.0-master+$Format:%h$"}
2019-10-11T22:31:55.002Z    INFO    router.metrics  metrics/metrics.go:153  router health and metrics port listening on HTTP and HTTPS  {"address": "0.0.0.0:1936"}
2019-10-11T22:31:55.014Z    INFO    router.template template/router.go:357  router default cert from router container
2019-10-11T22:31:55.014Z    INFO    router.template template/router.go:294  watching for changes    {"path": "/etc/pki/tls/private"}
E1011 22:31:55.020681       1 haproxy.go:395] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
2019-10-11T22:31:55.044Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T22:31:55.044Z    INFO    router.router   router/router.go:257    router is including routes in all namespaces
2019-10-11T22:31:55.250Z    INFO    router.template template/router.go:1093 a edge terminated route with host downloads-openshift-console.apps.misalunk-ocp-42.paasology.com does not have the required certificates.  The route will still be created but no certificates will be written
2019-10-11T22:31:55.283Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T22:32:00.281Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T22:32:05.972Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T22:32:10.903Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
[miheer@miheer router]$ 


From the following it looks good ->

2019-10-11T22:31:55.014Z    INFO    router.template template/router.go:357  router default cert from router container
2019-10-11T22:31:55.014Z    INFO    router.template template/router.go:294  watching for changes    {"path": "/etc/pki/tls/private"}



test2 ->

oc -n openshift-ingress create secret generic custom-default-cert --from-file=tls.crt=example.pem --from-file=tls.key=example.key

019-10-11T22:58:11.265Z    INFO    router.template    template/router.go:320    got watch event for update    {"event": "\"/etc/pki/tls/private/..2019_10_11_22_58_11.189229392\": CREATE", "name": "/var/lib/haproxy/router/certs/default.pem"}
E1011 22:58:11.288367       1 limiter.go:140] error reloading router: exit status 1
[ALERT] 283/225811 (126) : parsing [/var/lib/haproxy/conf/haproxy.config:117] : 'bind 127.0.0.1:10444' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/225811 (126) : parsing [/var/lib/haproxy/conf/haproxy.config:154] : 'bind 127.0.0.1:10443' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/225811 (126) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 283/225811 (126) : Fatal errors found in configuration.
[miheer@miheer certs]$


Watch seems to be working from the above logs.
Comment 4 Miheer Salunke 2020-04-07 09:52:43 UTC 
Adding correct secret no inconsistencies issues seen. Following is the snip from the logs. Check [0] for log details. 

2019-10-11T23:04:12.363Z    INFO    router.template template/router.go:320  got watch event for update  {"event": "\"/etc/pki/tls/private/..data_tmp\": RENAME", "name": "/var/lib/haproxy/router/certs/default.pem"}
2019-10-11T23:04:12.396Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}

miheer@miheer certs]$ oc -n openshift-ingress create secret generic custom-default-cert --from-file=tls.crt=example.crt --from-file=tls.key=example.key
secret/custom-default-cert created
[miheer@miheer certs]$

Following looks good ->

2019-10-12T00:12:51.153Z    INFO    router.template    template/router.go:320    got watch event for update    {"event": "\"/etc/pki/tls/private/..data_tmp\": CREATE", "name": "/var/lib/haproxy/router/certs/default.pem"}


[0]

[miheer@miheer certs]$ oc logs router-test-6db6b49d74-cq9xs
2019-10-11T22:31:54.998Z    INFO    router.router   router/template.go:293  starting router {"version": "v0.0.0-master+$Format:%h$"}
2019-10-11T22:31:55.002Z    INFO    router.metrics  metrics/metrics.go:153  router health and metrics port listening on HTTP and HTTPS  {"address": "0.0.0.0:1936"}
2019-10-11T22:31:55.014Z    INFO    router.template template/router.go:357  router default cert from router container
2019-10-11T22:31:55.014Z    INFO    router.template template/router.go:294  watching for changes    {"path": "/etc/pki/tls/private"}
E1011 22:31:55.020681       1 haproxy.go:395] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
2019-10-11T22:31:55.044Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T22:31:55.044Z    INFO    router.router   router/router.go:257    router is including routes in all namespaces
2019-10-11T22:31:55.250Z    INFO    router.template template/router.go:1093 a edge terminated route with host downloads-openshift-console.apps.misalunk-ocp-42.paasology.com does not have the required certificates.  The route will still be created but no certificates will be written
2019-10-11T22:31:55.283Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T22:32:00.281Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T22:32:05.972Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T22:32:10.903Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T22:44:55.689Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T22:45:00.675Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T22:45:35.626Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T22:45:40.660Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T22:58:11.265Z    INFO    router.template template/router.go:320  got watch event for update  {"event": "\"/etc/pki/tls/private/..2019_10_11_22_58_11.189229392\": CREATE", "name": "/var/lib/haproxy/router/certs/default.pem"}
E1011 22:58:11.288367       1 limiter.go:140] error reloading router: exit status 1
[ALERT] 283/225811 (126) : parsing [/var/lib/haproxy/conf/haproxy.config:117] : 'bind 127.0.0.1:10444' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/225811 (126) : parsing [/var/lib/haproxy/conf/haproxy.config:154] : 'bind 127.0.0.1:10443' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/225811 (126) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 283/225811 (126) : Fatal errors found in configuration.
E1011 23:02:02.961397       1 limiter.go:140] error reloading router: exit status 1
[ALERT] 283/230202 (129) : parsing [/var/lib/haproxy/conf/haproxy.config:117] : 'bind 127.0.0.1:10444' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230202 (129) : parsing [/var/lib/haproxy/conf/haproxy.config:154] : 'bind 127.0.0.1:10443' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230202 (129) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 283/230202 (129) : Fatal errors found in configuration.
E1011 23:02:07.965464       1 limiter.go:140] error reloading router: exit status 1
[ALERT] 283/230207 (132) : parsing [/var/lib/haproxy/conf/haproxy.config:117] : 'bind 127.0.0.1:10444' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230207 (132) : parsing [/var/lib/haproxy/conf/haproxy.config:154] : 'bind 127.0.0.1:10443' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230207 (132) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 283/230207 (132) : Fatal errors found in configuration.
E1011 23:02:18.559465       1 limiter.go:140] error reloading router: exit status 1
[ALERT] 283/230218 (135) : parsing [/var/lib/haproxy/conf/haproxy.config:117] : 'bind 127.0.0.1:10444' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230218 (135) : parsing [/var/lib/haproxy/conf/haproxy.config:154] : 'bind 127.0.0.1:10443' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230218 (135) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 283/230218 (135) : Fatal errors found in configuration.
E1011 23:02:23.552547       1 limiter.go:140] error reloading router: exit status 1
[ALERT] 283/230223 (138) : parsing [/var/lib/haproxy/conf/haproxy.config:117] : 'bind 127.0.0.1:10444' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230223 (138) : parsing [/var/lib/haproxy/conf/haproxy.config:154] : 'bind 127.0.0.1:10443' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230223 (138) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 283/230223 (138) : Fatal errors found in configuration.
E1011 23:03:25.513377       1 limiter.go:140] error reloading router: exit status 1
[ALERT] 283/230325 (141) : parsing [/var/lib/haproxy/conf/haproxy.config:117] : 'bind 127.0.0.1:10444' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230325 (141) : parsing [/var/lib/haproxy/conf/haproxy.config:154] : 'bind 127.0.0.1:10443' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230325 (141) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 283/230325 (141) : Fatal errors found in configuration.
E1011 23:03:30.510597       1 limiter.go:140] error reloading router: exit status 1
[ALERT] 283/230330 (144) : parsing [/var/lib/haproxy/conf/haproxy.config:117] : 'bind 127.0.0.1:10444' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230330 (144) : parsing [/var/lib/haproxy/conf/haproxy.config:154] : 'bind 127.0.0.1:10443' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230330 (144) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 283/230330 (144) : Fatal errors found in configuration.
E1011 23:03:39.806915       1 limiter.go:140] error reloading router: exit status 1
[ALERT] 283/230339 (147) : parsing [/var/lib/haproxy/conf/haproxy.config:117] : 'bind 127.0.0.1:10444' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230339 (147) : parsing [/var/lib/haproxy/conf/haproxy.config:154] : 'bind 127.0.0.1:10443' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230339 (147) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 283/230339 (147) : Fatal errors found in configuration.
E1011 23:03:44.781257       1 limiter.go:140] error reloading router: exit status 1
[ALERT] 283/230344 (150) : parsing [/var/lib/haproxy/conf/haproxy.config:117] : 'bind 127.0.0.1:10444' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230344 (150) : parsing [/var/lib/haproxy/conf/haproxy.config:154] : 'bind 127.0.0.1:10443' : inconsistencies between private key and certificate loaded from PEM file '/var/lib/haproxy/router/certs/default.pem'.
[ALERT] 283/230344 (150) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 283/230344 (150) : Fatal errors found in configuration.
2019-10-11T23:04:12.363Z    INFO    router.template template/router.go:320  got watch event for update  {"event": "\"/etc/pki/tls/private/..data_tmp\": RENAME", "name": "/var/lib/haproxy/router/certs/default.pem"}
2019-10-11T23:04:12.396Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T23:05:02.805Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T23:05:07.786Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T23:05:25.880Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T23:05:30.840Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T23:05:38.919Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T23:05:43.699Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T23:08:59.367Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T23:09:04.344Z    INFO    router.template template/router.go:550  router reloaded {"output": "IC0gUHJveHkgcHJvdG9jb2wgb24sIGNoZWNraW5nIGh0dHA6Ly9sb2NhbGhvc3Q6ODAgLi4uCiAtIEhlYWx0aCBjaGVjayBvayA6IDAgcmV0cnkgYXR0ZW1wdChzKS4K"}
2019-10-11T23:09:30.267Z    INFO    router.template template/router.go:320  got watch event for update  {"event": "\"/etc/pki/tls/private/..20...
Comment 5 Arvind iyengar 2020-04-09 09:32:59 UTC 
Verified the functioning of the patch from PR#52 in "4.5.0-0.nightly-2020-04-03-084339" nightly release. The result of testing is as provided below: 

OCP version:
----
Server Version: 4.5.0-0.nightly-2020-04-03-084339
Kubernetes Version: v1.17.1
----

* Create test router which uses the default cert secret:
------
$ oc -n openshift-ingress get all                                                 
NAME                                  READY   STATUS    RESTARTS   AGE
pod/router-default-6f4cb8f674-g7tbh   1/1     Running   0          20h
pod/router-default-6f4cb8f674-pcsq6   1/1     Running   0          20h
pod/router-test-98cd745cc-z8plj       1/1     Running   0          6m8s --
pod/router-test-98cd745cc-c9qtm       1/1     Running   0          6m8s --
NAME                              TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                      AGE
service/router-default            LoadBalancer   172.30.104.172   a523eaf57425a4fe4b92981773773568-2079015431.us-east-2.elb.amazonaws.com   80:31509/TCP,443:31023/TCP   20h
service/router-internal-default   ClusterIP      172.30.198.119   <none>                                                                    80/TCP,443/TCP,1936/TCP      20h
service/router-internal-test      ClusterIP      172.30.50.217    <none>                                                                    80/TCP,443/TCP,1936/TCP      6m9s
service/router-test               LoadBalancer   172.30.245.210   a1183e43f87fd408dbd217e1549beb14-1395471993.us-east-2.elb.amazonaws.com   80:30451/TCP,443:32482/TCP   6m9s
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/router-default   2/2     2            2           20h
deployment.apps/router-test      2/2     2            2           6m10s
NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/router-default-6f4cb8f674   2         2         2       20h
replicaset.apps/router-test-98cd745cc       2         2         2       6m10s

certs inside the router pod are te default ones:

/var/lib/haproxy/router/certs/default.pem
------

* Added and patched the routed with an invalid certificate: 
------
$ cat tls.crt 
invalid-cert-test-PR52

$ oc -n openshift-ingress create secret generic custom-default-cert --from-file=tls.crt=tls.crt
secret/custom-default-cert created

$ oc -n openshift-ingress describe secret/custom-default-cert                                 
Name:         custom-default-cert
Namespace:    openshift-ingress
Labels:       <none>
Annotations:  <none>
Type:  Opaque
Data
====
tls.crt:  23 bytes

$ oc patch -n openshift-ingress-operator ingresscontroller/test --patch '{"spec":{"replicas": 0}}' --type=merge
ingresscontroller.operator.openshift.io/test patched

$ oc patch --type=merge --namespace openshift-ingress-operator ingresscontrollers/test  --patch '{"spec":{"defaultCertificate":{"name":"custom-default-cert"}}}'
ingresscontroller.operator.openshift.io/test patched

$ oc patch -n openshift-ingress-operator ingresscontroller/test --patch '{"spec":{"replicas": 2}}' --type=merge
ingresscontroller.operator.openshift.io/test patched
------

* This resulted in the certificate inside the pod to pick the wrong one:
-----
$ cat   /var/lib/haproxy/router/certs/default.pem 
invalid-cert-test-PR52
invalid-cert-test-PR52
-----

* Added a valid certificate and the key. This results in the changes being applied without any reload of the router pods:
-----
$ oc -n openshift-ingress delete secret custom-default-cert
secret "custom-default-cert" deleted

$ oc -n openshift-ingress create secret generic custom-default-cert --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key 
secret/custom-default-cert created

$ oc -n openshift-ingress rsh  pod/router-test-68f5465664-kgxds       

sh-4.2$ cat /var/lib/haproxy/router/certs/default.pem
-----BEGIN CERTIFICATE-----
MIIEFzCCAv+gAwIBAgIJALK1iUpF2VQLMA0GCSqGSIb3DQEBBQUAMIGhMQswCQYD
VQQGEwJVUzELMAkGA1UECAwCU0MxFTATBgNVBAcMDERlZmF1bHQgQ2l0eTEcMBoG
A1UECgwTRGVmYXVsdCBDb21wYW55IEx0ZDEQMA4GA1UECwwHVGVzdCBDQTEaMBgG
A1UEAwwRd3d3LmV4YW1wbGVjYS5jb20xIjAgBgkqhkiG9w0BCQEWE2V4YW1wbGVA
ZXhhbXBsZS5jb20wHhcNMTUwMTEyMTQxNTAxWhcNMjUwMTA5MTQxNTAxWjCBoTEL
.....
....
..
-----

In non-patched versions, the certificate inside the router pod will not be updated at all. the "default.pem" becomes missing in the pod if the secret certificated in updated on the fly. It is hence concluded that the patch works as expected.
Comment 7 errata-xmlrpc 2020-07-13 17:25:20 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409