HAproxy pod logs showing error "another server named 'pod:httpd-7c7ccfffdc-wdkvk:httpd:8080-tcp:10.128.x.x:8080' was already defined at line 326, please use distinct names"
Cause:
Remove selector from a service exposed via a route.
Consequence:
Duplicate endpointslices would be created for the service's pods, triggering HAProxy reload errors due to duplicate server entries.
Fix:
Filter out accidental duplicate server lines when writing out the HAProxy config file.
Result:
Deleting the selector from a service does not brick the router.
Verified in "4.6.0-0.nightly-2021-06-10-234022" payload. With this release, there are no more router reload errors when the selector is removed for a service mapped to a route:
-----
oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.6.0-0.nightly-2021-06-10-234022 True False 35m Cluster version is 4.6.0-0.nightly-2021-06-10-234022
oc get all
NAME READY STATUS RESTARTS AGE
pod/hello-openshift 1/1 Running 0 20s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hello-openshift ClusterIP 172.30.126.67 <none> 8080/TCP 13s
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
route.route.openshift.io/hello-openshift hello-openshift-test1.apps.aiyengar4611.qe.devcluster.openshift.com hello-openshift 8080 None
oc get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
hello-openshift ClusterIP 172.30.231.25 <none> 8080/TCP 4s name=hello-openshift
oc patch service hello-openshift --patch '{"spec":{"selector":null}}'
service/hello-openshift patched
oc get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
hello-openshift ClusterIP 172.30.126.67 <none> 8080/TCP 7m44s <none>
oc -n openshift-ingress logs router-default-9dfbdf579-m9gbc
I0611 07:24:29.221823 1 template.go:403] router "msg"="starting router" "version"="majorFromGit: \nminorFromGit: \ncommitFromGit: 09690e0f35d28e99587c261cf00d4ccb4ecc3e97\nversionFromGit: 4.0.0-237-g09690e0f\ngitTreeState: clean\nbuildDate: 2021-06-10T23:01:44Z\n"
I0611 07:24:29.228594 1 metrics.go:154] metrics "msg"="router health and metrics port listening on HTTP and HTTPS" "address"="0.0.0.0:1936"
I0611 07:24:29.238817 1 router.go:185] template "msg"="creating a new template router" "writeDir"="/var/lib/haproxy"
I0611 07:24:29.238927 1 router.go:263] template "msg"="router will coalesce reloads within an interval of each other" "interval"="5s"
I0611 07:24:29.239509 1 router.go:325] template "msg"="watching for changes" "path"="/etc/pki/tls/private"
I0611 07:24:29.239669 1 router.go:262] router "msg"="router is including routes in all namespaces"
E0611 07:24:29.346724 1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
I0611 07:24:29.426153 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I0611 07:24:34.383965 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I0611 07:24:48.331575 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I0611 07:24:56.343915 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I0611 07:25:02.958224 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I0611 07:25:07.946739 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I0611 07:25:12.956753 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I0611 07:25:17.939111 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
I0611 07:25:22.922245 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n"
------
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (OpenShift Container Platform 4.6.35 bug fix update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2021:2410
Verified in "4.6.0-0.nightly-2021-06-10-234022" payload. With this release, there are no more router reload errors when the selector is removed for a service mapped to a route: ----- oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.6.0-0.nightly-2021-06-10-234022 True False 35m Cluster version is 4.6.0-0.nightly-2021-06-10-234022 oc get all NAME READY STATUS RESTARTS AGE pod/hello-openshift 1/1 Running 0 20s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/hello-openshift ClusterIP 172.30.126.67 <none> 8080/TCP 13s NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD route.route.openshift.io/hello-openshift hello-openshift-test1.apps.aiyengar4611.qe.devcluster.openshift.com hello-openshift 8080 None oc get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR hello-openshift ClusterIP 172.30.231.25 <none> 8080/TCP 4s name=hello-openshift oc patch service hello-openshift --patch '{"spec":{"selector":null}}' service/hello-openshift patched oc get svc -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR hello-openshift ClusterIP 172.30.126.67 <none> 8080/TCP 7m44s <none> oc -n openshift-ingress logs router-default-9dfbdf579-m9gbc I0611 07:24:29.221823 1 template.go:403] router "msg"="starting router" "version"="majorFromGit: \nminorFromGit: \ncommitFromGit: 09690e0f35d28e99587c261cf00d4ccb4ecc3e97\nversionFromGit: 4.0.0-237-g09690e0f\ngitTreeState: clean\nbuildDate: 2021-06-10T23:01:44Z\n" I0611 07:24:29.228594 1 metrics.go:154] metrics "msg"="router health and metrics port listening on HTTP and HTTPS" "address"="0.0.0.0:1936" I0611 07:24:29.238817 1 router.go:185] template "msg"="creating a new template router" "writeDir"="/var/lib/haproxy" I0611 07:24:29.238927 1 router.go:263] template "msg"="router will coalesce reloads within an interval of each other" "interval"="5s" I0611 07:24:29.239509 1 router.go:325] template "msg"="watching for changes" "path"="/etc/pki/tls/private" I0611 07:24:29.239669 1 router.go:262] router "msg"="router is including routes in all namespaces" E0611 07:24:29.346724 1 haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory I0611 07:24:29.426153 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I0611 07:24:34.383965 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I0611 07:24:48.331575 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I0611 07:24:56.343915 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I0611 07:25:02.958224 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I0611 07:25:07.946739 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I0611 07:25:12.956753 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I0611 07:25:17.939111 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" I0611 07:25:22.922245 1 router.go:570] template "msg"="router reloaded" "output"=" - Proxy protocol on, checking http://localhost:80 ...\n - Health check ok : 0 retry attempt(s).\n" ------