Bug 1711563

Summary:	github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue TestHighProirotyFlushUnschedulableQLeftover
Product:	OpenShift Container Platform	Reporter:	Maciej Szulik <maszulik>
Component:	Master	Assignee:	ravig <rgudimet>
Status:	CLOSED ERRATA	QA Contact:	Sunil Choudhary <schoudha>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.2.0	CC:	aos-bugs, jokerman, mfojtik, mmccomas
Target Milestone:	---	Keywords:	Rebase
Target Release:	4.2.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-10-16 06:29:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Maciej Szulik 2019-05-18 16:59:28 UTC

In https://github.com/openshift/origin/pull/22833 I've seen this happening a few times:

=== RUN   TestHighProirotyFlushUnschedulableQLeftover
==================
WARNING: DATA RACE
Read at 0x00c00000c408 by goroutine 90:
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.(*PriorityQueue).flushUnschedulableQLeftover()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:497 +0x24a
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.(*PriorityQueue).flushUnschedulableQLeftover-fm()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:315 +0x41
  github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x61
  github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xce
  github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x5a

Previous write at 0x00c00000c408 by goroutine 88:
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.TestHighProirotyFlushUnschedulableQLeftover()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue_test.go:979 +0x60d
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162

Goroutine 90 (running) created at:
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.(*PriorityQueue).run()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:315 +0x18a
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.NewPriorityQueueWithClock()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:307 +0x875
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.NewPriorityQueue()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:290 +0x68
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.TestHighProirotyFlushUnschedulableQLeftover()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue_test.go:933 +0x4f
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162

Goroutine 88 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:878 +0x659
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1119 +0xa8
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1117 +0x4ee
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1034 +0x2ee
  main.main()
      _testmain.go:126 +0x332
==================
==================
WARNING: DATA RACE
Read at 0x00c00000c428 by goroutine 90:
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.(*PriorityQueue).flushUnschedulableQLeftover()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:497 +0x24a
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.(*PriorityQueue).flushUnschedulableQLeftover-fm()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:315 +0x41
  github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x61
  github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xce
  github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x5a

Previous write at 0x00c00000c428 by goroutine 88:
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.TestHighProirotyFlushUnschedulableQLeftover()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue_test.go:980 +0x7d7
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162

Goroutine 90 (running) created at:
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.(*PriorityQueue).run()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:315 +0x18a
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.NewPriorityQueueWithClock()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:307 +0x875
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.NewPriorityQueue()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:290 +0x68
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.TestHighProirotyFlushUnschedulableQLeftover()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue_test.go:933 +0x4f
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162

Goroutine 88 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:878 +0x659
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1119 +0xa8
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1117 +0x4ee
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1034 +0x2ee
  main.main()
      _testmain.go:126 +0x332
==================
==================
WARNING: DATA RACE
Write at 0x00c000412420 by goroutine 90:
  runtime.mapdelete_faststr()
      /usr/local/go/src/runtime/map_faststr.go:281 +0x0
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.(*UnschedulablePodsMap).delete()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:802 +0xc4
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.(*PriorityQueue).movePodsToActiveQueue()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:674 +0xaf
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.(*PriorityQueue).flushUnschedulableQLeftover()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:504 +0x44c
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.(*PriorityQueue).flushUnschedulableQLeftover-fm()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:315 +0x41
  github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x61
  github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xce
  github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x5a

Previous read at 0x00c000412420 by goroutine 88:
  runtime.mapaccess1_faststr()
      /usr/local/go/src/runtime/map_faststr.go:12 +0x0
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.TestHighProirotyFlushUnschedulableQLeftover()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue_test.go:979 +0x58a
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162

Goroutine 90 (running) created at:
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.(*PriorityQueue).run()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:315 +0x18a
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.NewPriorityQueueWithClock()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:307 +0x875
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.NewPriorityQueue()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue.go:290 +0x68
  github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue.TestHighProirotyFlushUnschedulableQLeftover()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/scheduler/internal/queue/scheduling_queue_test.go:933 +0x4f
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162

Goroutine 88 (running) created at:
  testing.(*T).Run()
      /usr/local/go/src/testing/testing.go:878 +0x659
  testing.runTests.func1()
      /usr/local/go/src/testing/testing.go:1119 +0xa8
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:827 +0x162
  testing.runTests()
      /usr/local/go/src/testing/testing.go:1117 +0x4ee
  testing.(*M).Run()
      /usr/local/go/src/testing/testing.go:1034 +0x2ee
  main.main()
      _testmain.go:126 +0x332
==================
--- FAIL: TestHighProirotyFlushUnschedulableQLeftover (0.00s)
    testing.go:771: race detected during execution of test

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/22833/pull-ci-openshift-origin-master-unit/6570/

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/22833/pull-ci-openshift-origin-master-unit/6567/

Comment 1 Michal Fojtik 2019-06-10 13:36:57 UTC

Potential fix: https://github.com/openshift/origin/pull/23094

Comment 2 Maciej Szulik 2019-06-19 14:40:58 UTC

This looks like it should be fixed by now. I'm gonna close it, it should be reopened if re-appears.

Comment 4 errata-xmlrpc 2019-10-16 06:29:06 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922