Bug 1695326

Summary:	Unit test flake post rebase, TestLegacyScaleUpUnreadyLessScale
Product:	OpenShift Container Platform	Reporter:	Clayton Coleman <ccoleman>
Component:	Node	Assignee:	Joel Smith <joelsmith>
Status:	CLOSED ERRATA	QA Contact:	Jianwei Hou <jhou>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	4.1.0	CC:	anpicker, aos-bugs, erooth, fbranczy, jokerman, mloibl, mmccomas, pkrupa, schoudha, sjenning, surbania
Target Milestone:	---
Target Release:	4.1.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-04 10:47:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Clayton Coleman 2019-04-02 20:13:24 UTC

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/22455/pull-ci-openshift-origin-master-unit/4554#githubcomopenshiftoriginvendork8siokubernetespkgcontrollerpodautoscaler-testlegacyscaleupunreadylessscale

== RUN   TestLegacyScaleUpUnreadyLessScale
E0402 19:32:53.847496    2111 runtime.go:69] Observed a panic: &runtime.TypeAssertionError{interfaceString:"", concreteString:"testing.PatchActionImpl", assertedString:"testing.CreateAction", missingMethod:"GetObject"} (interface conversion: testing.PatchActionImpl is not testing.CreateAction: missing method GetObject)
/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:573
/usr/local/go/src/runtime/panic.go:502
/usr/local/go/src/runtime/iface.go:85
/usr/local/go/src/runtime/iface.go:562
/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/podautoscaler/legacy_horizontal_test.go:475
/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/testing/fixture.go:504
/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/testing/fake.go:140
/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/kubernetes/typed/core/v1/fake/fake_event_expansion.go:64
/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/kubernetes/typed/core/v1/event_expansion.go:163
/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/record/event.go:178
/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/record/event.go:142
/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/record/event.go:124
/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/record/event.go:238
/usr/local/go/src/runtime/asm_amd64.s:2361
--- PASS: TestLegacyScaleUpUnreadyLessScale (0.13s)
panic: interface conversion: testing.PatchActionImpl is not testing.CreateAction: missing method GetObject [recovered]
	panic: interface conversion: testing.PatchActionImpl is not testing.CreateAction: missing method GetObject

goroutine 1859 [running]:
github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x16b
panic(0x1a2a1c0, 0xc4200ae840)
	/usr/local/go/src/runtime/panic.go:502 +0x24a
github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/podautoscaler.(*legacyTestCase).runTest.func1(0x1d51b80, 0xc4203cac80, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/podautoscaler/legacy_horizontal_test.go:475 +0x1da
github.com/openshift/origin/vendor/k8s.io/client-go/testing.(*SimpleReactor).React(0xc420e26d80, 0x1d51b80, 0xc4203cac80, 0x1, 0x2, 0xc4204f6240, 0x1, 0x2)
	/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/testing/fixture.go:504 +0x65
github.com/openshift/origin/vendor/k8s.io/client-go/testing.(*Fake).Invokes(0xc4205526e0, 0x1d51b80, 0xc4203ca500, 0x1d31240, 0xc4204e8280, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/testing/fake.go:140 +0x276
github.com/openshift/origin/vendor/k8s.io/client-go/kubernetes/typed/core/v1/fake.(*FakeEvents).PatchWithEventNamespace(0xc42085aae0, 0xc4204e8280, 0xc420c3e000, 0x91, 0xd2, 0x0, 0xc4205fe7b0, 0x30)
	/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/kubern

Failing almost every time

Comment 1 Frederic Branczyk 2019-04-08 14:21:02 UTC

Joel from the Pod team is taking care of pod autoscaler things. Re-assigning.

Comment 2 Seth Jennings 2019-04-10 14:50:10 UTC

This has stopped happening.

Most recent occurrence was 5d ago.  Seems there was a 2d window 5d ago where this was happening.

https://search.svc.ci.openshift.org/?search=TestLegacyScaleUpUnreadyLessScale&maxAge=336h&context=2&type=all

Comment 3 Joel Smith 2019-04-17 14:54:14 UTC

I'm not sure if this flake is fixed by the deadlock in the same file, but it's certainly possible. Since the flake isn't happening now, I'm going to move forward under the assumption that the fix will address this flake. If it doesn't, we can always re-open this bug.

Here's the PR: https://github.com/openshift/origin/pull/22591

And here's how to trigger the deadlock flake. From the Origin source directory, run:

GOMAXPROCS=1 go test ./vendor/k8s.io/kubernetes/pkg/controller/podautoscaler/... -timeout 60s -count 1

Comment 5 Sunil Choudhary 2019-04-30 14:45:24 UTC

$ GOMAXPROCS=1 go test ./vendor/k8s.io/kubernetes/pkg/controller/podautoscaler/.. -timeout 60s -count 1
ok  	origin/vendor/k8s.io/kubernetes/pkg/controller	0.028s

And no longer see the panic in https://search.svc.ci.openshift.org/?search=TestLegacyScaleUpUnreadyLessScale&maxAge=336h&context=2&type=all

Comment 7 errata-xmlrpc 2019-06-04 10:47:00 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758