Bug 1868521

Summary:	Defunct router doesn't trigger alerts
Product:	OpenShift Container Platform	Reporter:	OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component:	Networking	Assignee:	Stephen Greene <sgreene>
Networking sub component:	router	QA Contact:	Arvind iyengar <aiyengar>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	urgent
Priority:	urgent	CC:	aiyengar, amcdermo, aos-bugs, hongli, jeder, rrackow, sgreene, skuznets, wking
Version:	4.4	Keywords:	ServiceDeliveryImpact
Target Milestone:	---
Target Release:	4.4.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-08-25 06:20:31 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1866454
Bug Blocks:

Comment 3 Arvind iyengar 2020-08-17 05:47:16 UTC

The PR was merged and made into "4.4.0-0.nightly-2020-08-14-172657" release payload. With the patched version, it is noted that the ingress operator now changes the state to "degraded" in situation if the router gets crashed or stuck in restarts due to incorrect haproxy configuration such as improper route annotation options:
-----
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-08-14-172657   True        False         69m     Cluster version is 4.4.0-0.nightly-2020-08-14-172657

reload of router with improper route annotations:

$ oc -n openshift-ingress get pods -o wide
NAME                                  READY   STATUS             RESTARTS   AGE   IP            NODE                                         NOMINATED NODE   READINESS GATES
router-default-7c799755fd-7qrpc       1/1     Running            0          98m   10.128.2.6    ip-10-0-179-0.us-east-2.compute.internal     <none>           <none>
router-default-7c799755fd-xrnm5       1/1     Running            0          98m   10.131.0.10   ip-10-0-195-158.us-east-2.compute.internal   <none>           <none>
router-internalapps-7b546df4d-wk82n   0/1     CrashLoopBackOff   7          14m   10.128.2.20   ip-10-0-179-0.us-east-2.compute.internal     <none>           <none>

$ oc -n openshift-ingress logs router-internalapps-7b546df4d-wk82n --tail 5
[ALERT] 229/053547 (17) : parsing [/var/lib/haproxy/conf/haproxy.config:319] : timer overflow in argument '999d' to 'timeout server' (maximum value is 2147483647 ms or ~24.8 days)
[ALERT] 229/053547 (17) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 229/053547 (17) : Fatal errors found in configuration.
E0817 05:36:01.960006       1 haproxy.go:416] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
E0817 05:36:05.389062       1 haproxy.go:416] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory


$ oc get co ingress
NAME      VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
ingress   4.4.0-0.nightly-2020-08-14-172657   False       True          True       15m
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-08-14-172657   True        False         77m     Error while reconciling 4.4.0-0.nightly-2020-08-14-172657: the cluster operator ingress is degraded
-----

Comment 5 errata-xmlrpc 2020-08-25 06:20:31 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.4.18 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3440