Bug 1442860 - [3.5] Router hangs on deadlock
Summary: [3.5] Router hangs on deadlock
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 3.4.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 3.5.z
Assignee: Ben Bennett
QA Contact: zhaozhanqi
URL:
Whiteboard:
Depends On: 1440977
Blocks: 1442859 1442863
TreeView+ depends on / blocked
 
Reported: 2017-04-17 19:25 UTC by Ben Bennett
Modified: 2022-08-04 22:20 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1440977
Environment:
Last Closed: 2017-04-26 05:38:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 13717 0 None closed Prevent the router from deadlocking itself when calling Commit() 2020-08-24 08:53:11 UTC
Red Hat Knowledge Base (Solution) 2995641 0 None None None 2017-04-17 19:25:14 UTC
Red Hat Product Errata RHBA-2017:1129 0 normal SHIPPED_LIVE OpenShift Container Platform 3.5, 3.4, 3.3, and 3.2 bug fix update 2017-04-26 09:35:35 UTC

Comment 1 Eric Paris 2017-04-18 01:41:45 UTC
https://github.com/openshift/ose/pull/707

Comment 2 Troy Dawson 2017-04-18 19:38:23 UTC
This has been merged into ocp and is in OCP v3.5.5.7 or newer.

Comment 4 zhaozhanqi 2017-04-20 10:09:32 UTC
verified this bug on OCP v3.5.5.7

steps:


echo "begin test" > fail1.route;
oc create -f https://raw.githubusercontent.com/openshift-qe/v3-testfiles/master/routing/edge/route_edge.json;

while true;
  do
  oc patch route secured-edge-route -p '{"spec":{"path":"/test"}}';
  oc patch route secured-edge-route -p '{"spec":{"path":"/hello"}}';

done

When running this on OCP v3.5.5.7 about 3 hours, No errors found..

When I running this on old env. found error logs in router pod:
E0420 09:53:13.419135       1 ratelimiter.go:52] error reloading router: exit status 1
[ALERT] 109/095313 (36177) : parsing [/var/lib/haproxy/conf/haproxy.config:118] : 'bind 127.0.0.1:10444' : 'crt-list' : error processing line 1 in file '/var/lib/haproxy/conf/cert_config.map' : unable to load SSL private key from PEM file '/var/lib/haproxy/router/certs/z1:secured-edge-route.pem'.
[ALERT] 109/095313 (36177) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config

@Ben Bennett 
 Could you help check if this is the correct steps to verify this bug?

Comment 5 Ben Bennett 2017-04-21 13:52:39 UTC
That is a reasonable test.  But Phil Cameron had something that would lock it up far sooner, Phil can you tell us what you were doing that hit this bug?

Comment 6 Phil Cameron 2017-04-24 16:57:24 UTC
I just ran Webin's test. It creates and deletes routes in a tight loop. I run several concurrent scripts.

Comment 8 errata-xmlrpc 2017-04-26 05:38:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1129


Note You need to log in before you can comment on or make changes to this bug.