Bug 1663501

Summary:	Setting DEBUG_LOGLEVEL higher than 5 in /etc/origin/master/master.env breaks HPA and oc adm top
Product:	OpenShift Container Platform	Reporter:	Brendan Mchugh <bmchugh>
Component:	Master	Assignee:	Michal Fojtik <mfojtik>
Status:	CLOSED DUPLICATE	QA Contact:	Xingxing Xia <xxia>
Severity:	urgent	Docs Contact:
Priority:	unspecified
Version:	3.11.0	CC:	aos-bugs, decarr, jesusr, jokerman, mfojtik, mirollin, mmccomas, msheermo, oarribas, palonsor, suchaudh, vhernand
Target Milestone:	---
Target Release:	3.11.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-07-10 16:02:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Brendan Mchugh 2019-01-04 15:40:14 UTC

Description of problem:
Setting DEBUG_LOGLEVEL higher than 5 in /etc/origin/master/master.env breaks HPA and "oc adm top"

Version-Release number of selected component (if applicable):
v3.11.51

How reproducible:
Always

Steps to Reproduce:
1. Set DEBUG_LOGLEVEL=6 in /etc/origin/master/master.env
2. Restart master services
3.

Actual results:
$ oc adm top node
Error from server (InternalError): an error on the server ("unable to set dialer for openshift-metrics-server/metrics-server as rest transport is of type *transport.debuggingRoundTripper") has prevented the request from succeeding (get nodes.metrics.k8s.io)

$ oc get hpa
NAME      REFERENCE                  TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
someapp   DeploymentConfig/someapp   <unknown>/9%   2         5         2          13h

Expected results:
"oc adm top" and HPA should work even when DEBUG_LOGLEVEL increased.

Additional info:
Seems fixed in upstream 1.12
https://github.com/kubernetes/kubernetes/issues/65848
https://github.com/kubernetes/kubernetes/blob/release-1.12/staging/src/k8s.io/kube-aggregator/pkg/apiserver/handler_proxy.go

Comment 2 Mitchell Rollinson 2019-03-05 21:43:23 UTC

Please advise on any updates that we might be able to pass on to the cu.

Comment 3 Michal Fojtik 2019-03-07 12:01:03 UTC

Backport PR: https://github.com/openshift/ose/pull/1503

Comment 6 Jesus M. Rodriguez 2019-07-10 16:02:41 UTC

After further investigation, I believe this bug is a duplicate of BZ #1689366 which was fixed in version 3.11.98 and released by an errata RHBA-2019:0636.

The version in this bug is 3.11.51 which is also older than 3.11.98.
The version in BZ #1695461 bug as seen in comment #3 shows 3.11.43 which is older than 3.11.98.

Please apply the following errata.
https://access.redhat.com/errata/RHBA-2019:0636

*** This bug has been marked as a duplicate of bug 1689366 ***

Comment 7 Pablo Alonso Rodriguez 2019-07-10 16:10:28 UTC

It seems to be no longer reproducible in a more modern errata, as per my tests on a test cluster. Thanks for letting us know