Web console is not usable due to numerous 429 response codes from the api server on starter-us-east-1. Simply trying to load the web console. GET /apis/policy/v1beta1 HTTP/1.1 Host: api.starter-us-east-1.openshift.com Connection: keep-alive Pragma: no-cache Cache-Control: no-cache Accept: */* Origin: https://console.starter-us-east-1.openshift.com User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36 DNT: 1 Referer: https://console.starter-us-east-1.openshift.com/console/error?error_description=Unable%20to%20load%20details%20about%20the%20server.%20If%20the%20problem%20continues,%20please%20contact%20your%20system%20administrator.&error=API_DISCOVERY Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.8 HTTP/1.1 429 Too Many Requests Content-Type: text/plain; charset=utf-8 Retry-After: 1 X-Content-Type-Options: nosniff Date: Wed, 19 Jul 2017 01:39:42 GMT Content-Length: 43 OpenShift Master: v3.6.126.1 (online version 3.5.0.20) Kubernetes Master: v1.6.1+5115d708d7 See dakini or eparis for cluster logs
resolved with multiple changes: - increased max connections (config change) - upgraded nodes to 3.6.170 to send fewer event requests - resolved issue with long-hanging requests from nodes (https://github.com/openshift/origin/pull/15515)
Verified with v3.6.170 (online version 3.5.0.20), no such issue found.
the max connections are set in the master config: servingInfo: maxRequestsInFlight: ... ... That was increased to 1200 by default
We updated these changes in 3 places. We went to 9000. assetConfig: servingInfo: maxRequestsInFlight: 9000 ... servingInfo: maxRequestsInFlight: 9000 .... kubernetesMasterConfig: apiServerArguments: max-requests-inflight: - '9000'
*** Bug 1479582 has been marked as a duplicate of this bug. ***