Bug 1868521 - Defunct router doesn't trigger alerts
Summary: Defunct router doesn't trigger alerts
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Routing
Version: 4.4
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.4.z
Assignee: Stephen Greene
QA Contact: Arvind iyengar
URL:
Whiteboard:
Depends On: 1866454
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-12 23:22 UTC by OpenShift BugZilla Robot
Modified: 2020-08-25 06:20 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-25 06:20:31 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift router pull 171 None closed [release-4.4] Bug 1868521: Back port Remove initial haproxy template commitAndReload 2020-09-08 09:00:33 UTC
Red Hat Product Errata RHBA-2020:3440 None None None 2020-08-25 06:20:38 UTC

Comment 3 Arvind iyengar 2020-08-17 05:47:16 UTC
The PR was merged and made into "4.4.0-0.nightly-2020-08-14-172657" release payload. With the patched version, it is noted that the ingress operator now changes the state to "degraded" in situation if the router gets crashed or stuck in restarts due to incorrect haproxy configuration such as improper route annotation options:
-----
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-08-14-172657   True        False         69m     Cluster version is 4.4.0-0.nightly-2020-08-14-172657

reload of router with improper route annotations:

$ oc -n openshift-ingress get pods -o wide
NAME                                  READY   STATUS             RESTARTS   AGE   IP            NODE                                         NOMINATED NODE   READINESS GATES
router-default-7c799755fd-7qrpc       1/1     Running            0          98m   10.128.2.6    ip-10-0-179-0.us-east-2.compute.internal     <none>           <none>
router-default-7c799755fd-xrnm5       1/1     Running            0          98m   10.131.0.10   ip-10-0-195-158.us-east-2.compute.internal   <none>           <none>
router-internalapps-7b546df4d-wk82n   0/1     CrashLoopBackOff   7          14m   10.128.2.20   ip-10-0-179-0.us-east-2.compute.internal     <none>           <none>

$ oc -n openshift-ingress logs router-internalapps-7b546df4d-wk82n --tail 5
[ALERT] 229/053547 (17) : parsing [/var/lib/haproxy/conf/haproxy.config:319] : timer overflow in argument '999d' to 'timeout server' (maximum value is 2147483647 ms or ~24.8 days)
[ALERT] 229/053547 (17) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 229/053547 (17) : Fatal errors found in configuration.
E0817 05:36:01.960006       1 haproxy.go:416] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
E0817 05:36:05.389062       1 haproxy.go:416] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory


$ oc get co ingress
NAME      VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
ingress   4.4.0-0.nightly-2020-08-14-172657   False       True          True       15m
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-08-14-172657   True        False         77m     Error while reconciling 4.4.0-0.nightly-2020-08-14-172657: the cluster operator ingress is degraded
-----

Comment 5 errata-xmlrpc 2020-08-25 06:20:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.4.18 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3440


Note You need to log in before you can comment on or make changes to this bug.