Bug 1989005

Summary: router pod is CrashLoopBackOff if configure spec.clientTLS.allowedSubjectPatterns to "*.openshift.com"
Product: OpenShift Container Platform Reporter: Hongan Li <hongli>
Component: NetworkingAssignee: Miciah Dashiel Butler Masters <mmasters>
Networking sub component: router QA Contact: jechen <jechen>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: aos-bugs, jechen, mmasters
Version: 4.9   
Target Milestone: ---   
Target Release: 4.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-10-18 17:43:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Hongan Li 2021-08-02 09:24:11 UTC
Description of problem:
router pod is CrashLoopBackOff if configure spec.clientTLS.allowedSubjectPatterns to "*.openshift.com"

OpenShift release version:
4.9.0-0.nightly-2021-08-01-132055

Cluster Platform:
AWS

How reproducible:
100%

Steps to Reproduce (in detail):
1. edit ingresscontroller/defalt as below
spec:
  clientTLS:
    allowedSubjectPatterns:
    - '*.openshift.com'
    clientCA:
      name: test-client-ca
    clientCertificatePolicy: Optional

2.



Actual results:
$ oc -n openshift-ingress logs router-default-75b446bff6-bb6z6
I0802 09:19:58.529496       1 template.go:437] router "msg"="starting router"  "version"="majorFromGit: \nminorFromGit: \ncommitFromGit: dca0c64df1dcc00042218714ed3326fea1d9221e\nversionFromGit: 4.0.0-343-gdca0c64d\ngitTreeState: clean\nbuildDate: 2021-07-30T13:54:05Z\n"
I0802 09:19:58.531999       1 metrics.go:155] metrics "msg"="router health and metrics port listening on HTTP and HTTPS"  "address"="0.0.0.0:1936"
I0802 09:19:58.547116       1 router.go:191] template "msg"="creating a new template router"  "writeDir"="/var/lib/haproxy"
I0802 09:19:58.547223       1 router.go:273] template "msg"="router will coalesce reloads within an interval of each other"  "interval"="5s"
I0802 09:19:58.547588       1 router.go:337] template "msg"="watching for changes"  "path"="/etc/pki/tls/private"
I0802 09:19:58.547694       1 router.go:337] template "msg"="watching for changes"  "path"="/etc/pki/tls/client-ca"
I0802 09:19:58.547776       1 router.go:337] template "msg"="watching for changes"  "path"="/etc/pki/tls/client-ca-crl"
I0802 09:19:58.547836       1 router.go:262] router "msg"="router is including routes in all namespaces"  
E0802 09:19:58.672567       1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
E0802 09:19:58.720471       1 limiter.go:165] error reloading router: exit status 1
[NOTICE] 213/091958 (18) : haproxy version is 2.2.15-5e8f49d
[NOTICE] 213/091958 (18) : path to executable is /usr/sbin/haproxy
[ALERT] 213/091958 (18) : parsing [/var/lib/haproxy/conf/haproxy.config:142] : error detected while parsing ACL 'cert_cn_matches' : regex '(?:*.openshift.com)' is invalid (error=nothing to repeat, erroffset=3).
[ALERT] 213/091958 (18) : parsing [/var/lib/haproxy/conf/haproxy.config:143] : error detected while parsing an 'http-request deny' condition : no such ACL : 'cert_cn_matches'.
[ALERT] 213/091958 (18) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
E0802 09:20:03.659815       1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
E0802 09:20:03.676073       1 limiter.go:165] error reloading router: exit status 1
[NOTICE] 213/092003 (22) : haproxy version is 2.2.15-5e8f49d
[NOTICE] 213/092003 (22) : path to executable is /usr/sbin/haproxy
[ALERT] 213/092003 (22) : parsing [/var/lib/haproxy/conf/haproxy.config:142] : error detected while parsing ACL 'cert_cn_matches' : regex '(?:*.openshift.com)' is invalid (error=nothing to repeat, erroffset=3).
[ALERT] 213/092003 (22) : parsing [/var/lib/haproxy/conf/haproxy.config:143] : error detected while parsing an 'http-request deny' condition : no such ACL : 'cert_cn_matches'.
[ALERT] 213/092003 (22) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config



Expected results:
router pod should works well

Impact of the problem:


Additional info:



** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report.  You may also mark the bug private if you wish.

Comment 2 jechen 2021-08-25 23:47:10 UTC
Verified in 4.9.0-0.nightly-2021-08-25-185404

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-08-25-185404   True        False         10m     Cluster version is 4.9.0-0.nightly-2021-08-25-185404

$ oc -n openshift-ingress get pod
NAME                              READY   STATUS    RESTARTS   AGE
router-default-795d87f7c5-bttnh   1/1     Running   0          109s
router-default-795d87f7c5-w8df7   1/1     Running   0          109s


$ oc -n openshift-ingress-operator edit ingresscontroller/default
ingresscontroller.operator.openshift.io/default edited
<--snip-->
spec:
  clientTLS:
    allowedSubjectPatterns:
    - '*.openshift.com'
    clientCA:
      name: test-client-ca
    clientCertificatePolicy: Optional
<--snip-->

$ oc -n openshift-ingress get pod
NAME                              READY   STATUS    RESTARTS   AGE
router-default-795d87f7c5-bttnh   1/1     Running   0          5m29s
router-default-795d87f7c5-w8df7   1/1     Running   0          5m29s


router pods did not crash with spec.clientTLS.allowedSubjectPatterns set to "*.openshift.com"

Comment 5 errata-xmlrpc 2021-10-18 17:43:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759