Bug 1880283

Summary: oc get events fails when sorting by lastTimestamp
Product: OpenShift Container Platform Reporter: Jan Chaloupka <jchaloup>
Component: ocAssignee: Maciej Szulik <maszulik>
Status: CLOSED ERRATA QA Contact: RamaKasturi <knarra>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.4CC: aos-bugs, cpassare, jchaloup, jokerman, knarra, maszulik, mfojtik, oarribas, plarsen, xxia, yinzhou
Target Milestone: ---Keywords: Reopened
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: Some events might have empty/nil lastTimestamp. Consequence: When sorting events by lastTimestamp that nil was causing a crash in the algorithm. Fix: Make sorting resistant to nil elements. Result: Sorting works properly.
Story Points: ---
Clone Of: 1857268 Environment:
Last Closed: 2020-10-27 16:42:20 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1857268    

Comment 1 RamaKasturi 2020-09-21 07:46:28 UTC
Moving the bug back as i still see the same error when run the command below with the payload as given.

[ramakasturinarra@dhcp35-60 openshift-client-linux-4.6.0-0.nightly-2020-09-21-030155]$ ./oc version
Client Version: 4.6.0-0.nightly-2020-09-21-030155
Server Version: 4.6.0-0.nightly-2020-09-20-184226
Kubernetes Version: v1.19.0+f5121a6

[ramakasturinarra@dhcp35-60 openshift-client-linux-4.6.0-0.nightly-2020-09-21-030155]$ ./oc get events -n openshift-sdn --sort-by='.lastTimestamp'
F0921 13:14:36.555979   12596 sorter.go:353] Field {.lastTimestamp} in [][][]reflect.Value is an unsortable type: interface, err: unsortable type: <nil>
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc000010001, 0xc0012241e0, 0x99, 0xeb)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:996 +0xb8
k8s.io/klog/v2.(*loggingT).output(0x5223080, 0xc000000003, 0x0, 0x0, 0xc00085e150, 0x4d9ef58, 0x9, 0x161, 0x0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:945 +0x19d
k8s.io/klog/v2.(*loggingT).printf(0x5223080, 0x3, 0x0, 0x0, 0x2f3341b, 0x31, 0xc001d0f5f0, 0x4, 0x4)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:733 +0x17b
k8s.io/klog/v2.Fatalf(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1463
k8s.io/kubectl/pkg/cmd/get.(*TableSorter).Less(0xc0005fc600, 0x0, 0x1a, 0xc001d0f788)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/sorter.go:353 +0x2d7
sort.medianOfThree(0x3582e80, 0xc0005fc600, 0x0, 0x1a, 0x34)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/sort/sort.go:76 +0x49
sort.doPivot(0x3582e80, 0xc0005fc600, 0x0, 0xd7, 0xc0005eb4c0, 0xc000eebc70)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/sort/sort.go:101 +0x5c5
sort.quickSort(0x3582e80, 0xc0005fc600, 0x0, 0xd7, 0x10)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/sort/sort.go:190 +0x9a
sort.Sort(0x3582e80, 0xc0005fc600)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/sort/sort.go:218 +0x79
k8s.io/kubectl/pkg/cmd/get.(*TableSorter).Sort(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/sorter.go:359
k8s.io/kubectl/pkg/cmd/get.(*SortingPrinter).PrintObj(0xc001a2d6b0, 0x3543f40, 0xc001a42090, 0x3517b40, 0xc0014b8b40, 0x0, 0xc0014b8b40)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/sorter.go:56 +0x208
k8s.io/kubectl/pkg/cmd/get.(*TablePrinter).PrintObj(0xc001a32810, 0x3544080, 0xc000404310, 0x3517b40, 0xc0014b8b40, 0xc0000209c0, 0xc0009f47a8)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/table_printer.go:41 +0x1ba
k8s.io/cli-runtime/pkg/printers.ResourcePrinterFunc.PrintObj(0xc0004f0500, 0x3544080, 0xc000404310, 0x3517b40, 0xc0014b8b40, 0x0, 0x20)
	/go/src/github.com/openshift/oc/vendor/k8s.io/cli-runtime/pkg/printers/interface.go:31 +0x4e
k8s.io/kubectl/pkg/cmd/get.(*GetOptions).Run(0xc001264690, 0x35cb980, 0xc001367d10, 0xc0014bf8c0, 0xc0016da700, 0x1, 0x4, 0x0, 0x0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/get.go:579 +0x83a
k8s.io/kubectl/pkg/cmd/get.NewCmdGet.func1(0xc0014bf8c0, 0xc0016da700, 0x1, 0x4)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/get.go:167 +0x12d
github.com/spf13/cobra.(*Command).execute(0xc0014bf8c0, 0xc0016da6c0, 0x4, 0x4, 0xc0014bf8c0, 0xc0016da6c0)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:846 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0xc0009df8c0, 0x2, 0xc0009df8c0, 0x2)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:887
main.main()
	/go/src/github.com/openshift/oc/cmd/oc/oc.go:110 +0x81a

goroutine 6 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x5223080)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1131 +0x8b
created by k8s.io/klog/v2.init.0
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:416 +0xd6

goroutine 9 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x5222fa0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:1010 +0x8b
created by k8s.io/klog.init.0
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:411 +0xd6

goroutine 66 [select]:
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x30bdf40, 0x351a560, 0xc0002d4000, 0x1, 0xc00005a4e0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x13f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x30bdf40, 0x12a05f200, 0x0, 0xc0011a2401, 0xc00005a4e0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/apimachinery/pkg/util/wait.Forever(0x30bdf40, 0x12a05f200)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f
created by k8s.io/component-base/logs.InitLogs
	/go/src/github.com/openshift/oc/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a

goroutine 67 [select]:
io.(*pipe).Read(0xc001568780, 0xc0002fc000, 0x1000, 0x1000, 0x2988460, 0x1, 0xc0002fc000)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/io/pipe.go:57 +0xe7
io.(*PipeReader).Read(0xc000405860, 0xc0002fc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/io/pipe.go:134 +0x4c
bufio.(*Scanner).Scan(0xc001580b00, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/bufio/scan.go:213 +0xa4
github.com/openshift/oc/pkg/cli/admin/mustgather.newPrefixWriter.func1(0xc001580b00, 0x351bf80, 0xc000010018, 0x2ed85ce, 0x17)
	/go/src/github.com/openshift/oc/pkg/cli/admin/mustgather/mustgather.go:413 +0x14d
created by github.com/openshift/oc/pkg/cli/admin/mustgather.newPrefixWriter
	/go/src/github.com/openshift/oc/pkg/cli/admin/mustgather/mustgather.go:412 +0x1c1

goroutine 54 [IO wait]:
internal/poll.runtime_pollWait(0x7f39d57f2100, 0x72, 0xffffffffffffffff)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000e83798, 0x72, 0x3500, 0x3571, 0xffffffffffffffff)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000e83780, 0xc001324000, 0x3571, 0x3571, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc000e83780, 0xc001324000, 0x3571, 0x3571, 0x203000, 0x768050, 0xc000b76138)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000114418, 0xc001324000, 0x3571, 0x3571, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc000028320, 0xc001324000, 0x3571, 0x3571, 0xfcd, 0x254d, 0xc0005b79a8)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc000b76258, 0x3516540, 0xc000028320, 0x41c735, 0x2acd0c0, 0x2dc97a0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc000b76000, 0x351baa0, 0xc000114418, 0x5, 0xc000114418, 0xfbc)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc000b76000, 0x0, 0x0, 0xc0005b7d18)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc000b76000, 0xc0012a0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc00083b740, 0xc0001969d8, 0x9, 0x9, 0xc0005b7d18, 0x30c1900, 0x9892b5)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x35162e0, 0xc00083b740, 0xc0001969d8, 0x9, 0x9, 0x9, 0xc000092060, 0x0, 0x35167e0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/io/io.go:310 +0x87
io.ReadFull(...)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0001969d8, 0x9, 0x9, 0x35162e0, 0xc00083b740, 0x0, 0x0, 0xc0011c1440, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001969a0, 0xc0011c1440, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0005b7fa8, 0x0, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1794 +0xd8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc001180780)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1716 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:695 +0x64a
[ramakasturinarra@dhcp35-60 openshift-client-linux-4.6.0-0.nightly-2020-09-21-030155]$ 
[ramakasturinarra@dhcp35-60 openshift-client-linux-4.6.0-0.nightly-2020-09-21-030155]$ ./oc get events -n openshift-sdn --sort-by='.lastTimestamp'
F0921 13:14:43.784088   12611 sorter.go:353] Field {.lastTimestamp} in [][][]reflect.Value is an unsortable type: interface, err: unsortable type: <nil>
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc000010001, 0xc000db62d0, 0x99, 0xeb)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:996 +0xb8
k8s.io/klog/v2.(*loggingT).output(0x5223080, 0xc000000003, 0x0, 0x0, 0xc000f8c000, 0x4d9ef58, 0x9, 0x161, 0x0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:945 +0x19d
k8s.io/klog/v2.(*loggingT).printf(0x5223080, 0x3, 0x0, 0x0, 0x2f3341b, 0x31, 0xc001c0f5f0, 0x4, 0x4)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:733 +0x17b
k8s.io/klog/v2.Fatalf(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1463
k8s.io/kubectl/pkg/cmd/get.(*TableSorter).Less(0xc000ff1a40, 0x0, 0x1a, 0xc001c0f788)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/sorter.go:353 +0x2d7
sort.medianOfThree(0x3582e80, 0xc000ff1a40, 0x0, 0x1a, 0x34)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/sort/sort.go:76 +0x49
sort.doPivot(0x3582e80, 0xc000ff1a40, 0x0, 0xd7, 0xc000feb9a0, 0xc0005825b0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/sort/sort.go:101 +0x5c5
sort.quickSort(0x3582e80, 0xc000ff1a40, 0x0, 0xd7, 0x10)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/sort/sort.go:190 +0x9a
sort.Sort(0x3582e80, 0xc000ff1a40)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/sort/sort.go:218 +0x79
k8s.io/kubectl/pkg/cmd/get.(*TableSorter).Sort(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/sorter.go:359
k8s.io/kubectl/pkg/cmd/get.(*SortingPrinter).PrintObj(0xc001b3b710, 0x3543f40, 0xc00178c240, 0x3517b40, 0xc0017a80c0, 0x0, 0xc0017a80c0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/sorter.go:56 +0x208
k8s.io/kubectl/pkg/cmd/get.(*TablePrinter).PrintObj(0xc001b427e0, 0x3544080, 0xc000010fb8, 0x3517b40, 0xc0017a80c0, 0xc000216760, 0xc0014e4108)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/table_printer.go:41 +0x1ba
k8s.io/cli-runtime/pkg/printers.ResourcePrinterFunc.PrintObj(0xc001aeb3a0, 0x3544080, 0xc000010fb8, 0x3517b40, 0xc0017a80c0, 0x0, 0x20)
	/go/src/github.com/openshift/oc/vendor/k8s.io/cli-runtime/pkg/printers/interface.go:31 +0x4e
k8s.io/kubectl/pkg/cmd/get.(*GetOptions).Run(0xc000daeb40, 0x35cb980, 0xc00144c2d0, 0xc001515340, 0xc000418700, 0x1, 0x4, 0x0, 0x0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/get.go:579 +0x83a
k8s.io/kubectl/pkg/cmd/get.NewCmdGet.func1(0xc001515340, 0xc000418700, 0x1, 0x4)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/get.go:167 +0x12d
github.com/spf13/cobra.(*Command).execute(0xc001515340, 0xc000418680, 0x4, 0x4, 0xc001515340, 0xc000418680)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:846 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0xc001243340, 0x2, 0xc001243340, 0x2)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:887
main.main()
	/go/src/github.com/openshift/oc/cmd/oc/oc.go:110 +0x81a

goroutine 6 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x5223080)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1131 +0x8b
created by k8s.io/klog/v2.init.0
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:416 +0xd6

goroutine 10 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x5222fa0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:1010 +0x8b
created by k8s.io/klog.init.0
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:411 +0xd6

goroutine 67 [select]:
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x30bdf40, 0x351a560, 0xc00044e030, 0x1, 0xc00005a4e0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x13f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x30bdf40, 0x12a05f200, 0x0, 0xc001219c01, 0xc00005a4e0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/apimachinery/pkg/util/wait.Forever(0x30bdf40, 0x12a05f200)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f
created by k8s.io/component-base/logs.InitLogs
	/go/src/github.com/openshift/oc/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a

goroutine 36 [select]:
io.(*pipe).Read(0xc000a66f00, 0xc001568000, 0x1000, 0x1000, 0x2988460, 0x1, 0xc001568000)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/io/pipe.go:57 +0xe7
io.(*PipeReader).Read(0xc000308330, 0xc001568000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/io/pipe.go:134 +0x4c
bufio.(*Scanner).Scan(0xc000490300, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/bufio/scan.go:213 +0xa4
github.com/openshift/oc/pkg/cli/admin/mustgather.newPrefixWriter.func1(0xc000490300, 0x351bf80, 0xc000010018, 0x2ed85ce, 0x17)
	/go/src/github.com/openshift/oc/pkg/cli/admin/mustgather/mustgather.go:413 +0x14d
created by github.com/openshift/oc/pkg/cli/admin/mustgather.newPrefixWriter
	/go/src/github.com/openshift/oc/pkg/cli/admin/mustgather/mustgather.go:412 +0x1c1

goroutine 277 [IO wait]:
internal/poll.runtime_pollWait(0x7fd124753f70, 0x72, 0xffffffffffffffff)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc00132c898, 0x72, 0x3500, 0x350e, 0xffffffffffffffff)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00132c880, 0xc001790000, 0x350e, 0x350e, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc00132c880, 0xc001790000, 0x350e, 0x350e, 0x203000, 0x768050, 0xc0016c5d38)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000487608, 0xc001790000, 0x350e, 0x350e, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc00024bfa0, 0xc001790000, 0x350e, 0x350e, 0xfd5, 0x24ea, 0xc000b279a8)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc0016c5e58, 0x3516540, 0xc00024bfa0, 0x41c735, 0x2acd0c0, 0x2dc97a0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc0016c5c00, 0x351baa0, 0xc000487608, 0x5, 0xc000487608, 0x8)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc0016c5c00, 0x0, 0x0, 0xc000b27d18)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc0016c5c00, 0xc00172c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc000a45800, 0xc0000228f8, 0x9, 0x9, 0xc000b27d18, 0x30c1900, 0x9892b5)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x35162e0, 0xc000a45800, 0xc0000228f8, 0x9, 0x9, 0x9, 0xc000092060, 0x0, 0x35167e0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/io/io.go:310 +0x87
io.ReadFull(...)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc0000228f8, 0x9, 0x9, 0x35162e0, 0xc000a45800, 0x0, 0x0, 0xc0010139e0, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000228c0, 0xc0010139e0, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000b27fa8, 0x0, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1794 +0xd8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000ec5200)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1716 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:695 +0x64a
[ramakasturinarra@dhcp35-60 openshift-client-linux-4.6.0-0.nightly-2020-09-21-030155]$ 
[ramakasturinarra@dhcp35-60 openshift-client-linux-4.6.0-0.nightly-2020-09-21-030155]$ ./oc get events -n openshift-sdn --sort-by='.lastTimestamp'
F0921 13:15:15.281566   12647 sorter.go:353] Field {.lastTimestamp} in [][][]reflect.Value is an unsortable type: interface, err: unsortable type: <nil>
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc000130001, 0xc0004923c0, 0x99, 0xeb)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:996 +0xb8
k8s.io/klog/v2.(*loggingT).output(0x5223080, 0xc000000003, 0x0, 0x0, 0xc000f66000, 0x4d9ef58, 0x9, 0x161, 0x0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:945 +0x19d
k8s.io/klog/v2.(*loggingT).printf(0x5223080, 0x3, 0x0, 0x0, 0x2f3341b, 0x31, 0xc0011975f0, 0x4, 0x4)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:733 +0x17b
k8s.io/klog/v2.Fatalf(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1463
k8s.io/kubectl/pkg/cmd/get.(*TableSorter).Less(0xc000e505a0, 0x0, 0x1a, 0xc001197788)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/sorter.go:353 +0x2d7
sort.medianOfThree(0x3582e80, 0xc000e505a0, 0x0, 0x1a, 0x34)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/sort/sort.go:76 +0x49
sort.doPivot(0x3582e80, 0xc000e505a0, 0x0, 0xd7, 0xc000c7b880, 0xc0004380d0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/sort/sort.go:101 +0x5c5
sort.quickSort(0x3582e80, 0xc000e505a0, 0x0, 0xd7, 0x10)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/sort/sort.go:190 +0x9a
sort.Sort(0x3582e80, 0xc000e505a0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/sort/sort.go:218 +0x79
k8s.io/kubectl/pkg/cmd/get.(*TableSorter).Sort(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/sorter.go:359
k8s.io/kubectl/pkg/cmd/get.(*SortingPrinter).PrintObj(0xc001ad41e0, 0x3543f40, 0xc000996990, 0x3517b40, 0xc000203a40, 0x0, 0xc000203a40)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/sorter.go:56 +0x208
k8s.io/kubectl/pkg/cmd/get.(*TablePrinter).PrintObj(0xc001ac4f90, 0x3544080, 0xc000011a98, 0x3517b40, 0xc000203a40, 0xc000214f40, 0xc00120e7a8)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/table_printer.go:41 +0x1ba
k8s.io/cli-runtime/pkg/printers.ResourcePrinterFunc.PrintObj(0xc0002b1fc0, 0x3544080, 0xc000011a98, 0x3517b40, 0xc000203a40, 0x0, 0x20)
	/go/src/github.com/openshift/oc/vendor/k8s.io/cli-runtime/pkg/printers/interface.go:31 +0x4e
k8s.io/kubectl/pkg/cmd/get.(*GetOptions).Run(0xc001282690, 0x35cb980, 0xc0013bf560, 0xc0014d98c0, 0xc0009162c0, 0x1, 0x4, 0x0, 0x0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/get.go:579 +0x83a
k8s.io/kubectl/pkg/cmd/get.NewCmdGet.func1(0xc0014d98c0, 0xc0009162c0, 0x1, 0x4)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/get.go:167 +0x12d
github.com/spf13/cobra.(*Command).execute(0xc0014d98c0, 0xc000916200, 0x4, 0x4, 0xc0014d98c0, 0xc000916200)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:846 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0xc000bfd8c0, 0x2, 0xc000bfd8c0, 0x2)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:887
main.main()
	/go/src/github.com/openshift/oc/cmd/oc/oc.go:110 +0x81a

goroutine 19 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x5223080)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1131 +0x8b
created by k8s.io/klog/v2.init.0
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:416 +0xd6

goroutine 22 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x5222fa0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:1010 +0x8b
created by k8s.io/klog.init.0
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:411 +0xd6

goroutine 36 [select]:
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x30bdf40, 0x351a560, 0xc00097c000, 0x1, 0xc00010e3c0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x13f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x30bdf40, 0x12a05f200, 0x0, 0xc000a43601, 0xc00010e3c0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/apimachinery/pkg/util/wait.Forever(0x30bdf40, 0x12a05f200)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f
created by k8s.io/component-base/logs.InitLogs
	/go/src/github.com/openshift/oc/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a

goroutine 37 [select]:
io.(*pipe).Read(0xc0014b39e0, 0xc001644000, 0x1000, 0x1000, 0x2988460, 0x2, 0x2)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/io/pipe.go:57 +0xe7
io.(*PipeReader).Read(0xc000311f20, 0xc001644000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/io/pipe.go:134 +0x4c
bufio.(*Scanner).Scan(0xc00157dd00, 0x6884b3)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/bufio/scan.go:213 +0xa4
github.com/openshift/oc/pkg/cli/admin/mustgather.newPrefixWriter.func1(0xc00157dd00, 0x351bf80, 0xc000130008, 0x2ed85ce, 0x17)
	/go/src/github.com/openshift/oc/pkg/cli/admin/mustgather/mustgather.go:413 +0x14d
created by github.com/openshift/oc/pkg/cli/admin/mustgather.newPrefixWriter
	/go/src/github.com/openshift/oc/pkg/cli/admin/mustgather/mustgather.go:412 +0x1c1

goroutine 288 [IO wait]:
internal/poll.runtime_pollWait(0x7f505c0c4c48, 0x72, 0xffffffffffffffff)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc001114218, 0x72, 0x3500, 0x350e, 0xffffffffffffffff)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc001114200, 0xc001780000, 0x350e, 0x350e, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc001114200, 0xc001780000, 0x350e, 0x350e, 0x203000, 0x768050, 0xc00106fd38)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc001712e70, 0xc001780000, 0x350e, 0x350e, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/net/net.go:184 +0x8e
crypto/tls.(*atLeastReader).Read(0xc0000dc8a0, 0xc001780000, 0x350e, 0x350e, 0xff8, 0x24ea, 0xc0008259a8)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:760 +0x60
bytes.(*Buffer).ReadFrom(0xc00106fe58, 0x3516540, 0xc0000dc8a0, 0x41c735, 0x2acd0c0, 0x2dc97a0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc00106fc00, 0x351baa0, 0xc001712e70, 0x5, 0xc001712e70, 0x8)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:782 +0xec
crypto/tls.(*Conn).readRecordOrCCS(0xc00106fc00, 0x0, 0x0, 0xc000825d18)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:589 +0x115
crypto/tls.(*Conn).readRecord(...)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:557
crypto/tls.(*Conn).Read(0xc00106fc00, 0xc000fdc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/crypto/tls/conn.go:1233 +0x15b
bufio.(*Reader).Read(0xc000a284e0, 0xc000568658, 0x9, 0x9, 0xc000825d18, 0x30c1900, 0x9892b5)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/bufio/bufio.go:226 +0x24f
io.ReadAtLeast(0x35162e0, 0xc000a284e0, 0xc000568658, 0x9, 0x9, 0x9, 0xc000118050, 0x0, 0x35167e0)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/io/io.go:310 +0x87
io.ReadFull(...)
	/opt/rh/go-toolset-1.14/root/usr/lib/go-toolset-1.14-golang/src/io/io.go:329
golang.org/x/net/http2.readFrameHeader(0xc000568658, 0x9, 0x9, 0x35162e0, 0xc000a284e0, 0x0, 0x0, 0xc001763260, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:237 +0x87
golang.org/x/net/http2.(*Framer).ReadFrame(0xc000568620, 0xc001763260, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:492 +0xa1
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000825fa8, 0x0, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1794 +0xd8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000fa1e00)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1716 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:695 +0x64a


Based on the above info, moving the bug back to assigned state.

Comment 2 Maciej Szulik 2020-09-21 10:05:33 UTC
We need https://github.com/kubernetes/kubernetes/pull/94666, I'll check what other upstream PRs we might want and will pick them into oc.

Comment 3 Jan Chaloupka 2020-09-21 14:16:08 UTC
Rama, can you upload the list of all events in openshift-sdn namespace? Just to see if both firstTimestamp and lastTimestamp fields are both empty.

Comment 4 Maciej Szulik 2020-09-22 14:40:57 UTC
This will be fixed in https://github.com/openshift/oc/pull/589

Comment 6 RamaKasturi 2020-09-23 12:26:34 UTC
still failing, will check tomorrow.

Comment 7 RamaKasturi 2020-09-24 10:11:57 UTC
Verified with the payload below and i do not see the error as reported here, also events are sorted as well

[ramakasturinarra@dhcp35-60 openshift-client-linux-4.6.0-0.nightly-2020-09-24-015627]$ ./oc version -o yaml
clientVersion:
  buildDate: "2020-09-24T00:23:34Z"
  compiler: gc
  gitCommit: 403c1a835a57e7c3713008aa14cc1f4bcf3c4e46
  gitTreeState: clean
  gitVersion: openshift-clients-4.6.0-202006250705.p0-148-g403c1a835
  goVersion: go1.14.4
  major: ""
  minor: ""
  platform: linux/amd64
openshiftVersion: 4.6.0-0.nightly-2020-09-24-015627
releaseClientVersion: 4.6.0-0.nightly-2020-09-24-015627

[ramakasturinarra@dhcp35-60 openshift-client-linux-4.6.0-0.nightly-2020-09-24-015627]$ ./oc version
Client Version: 4.6.0-0.nightly-2020-09-24-015627
Server Version: 4.6.0-0.nightly-2020-09-24-015627
Kubernetes Version: v1.19.0+fff8183


[ramakasturinarra@dhcp35-60 openshift-client-linux-4.6.0-0.nightly-2020-09-24-015627]$ ./oc get events -n openshift-dns --sort-by='.lastTimestamp'
LAST SEEN   TYPE      REASON             OBJECT                  MESSAGE
3h4m        Normal    Scheduled          pod/dns-default-kqc2z   Successfully assigned openshift-dns/dns-default-kqc2z to ip-10-0-189-37.ap-northeast-1.compute.internal
173m        Normal    Scheduled          pod/dns-default-4mcbz   Successfully assigned openshift-dns/dns-default-4mcbz to ip-10-0-186-86.ap-northeast-1.compute.internal
172m        Normal    Scheduled          pod/dns-default-n2k8c   Successfully assigned openshift-dns/dns-default-n2k8c to ip-10-0-202-185.ap-northeast-1.compute.internal
3h4m        Normal    Scheduled          pod/dns-default-9c64t   Successfully assigned openshift-dns/dns-default-9c64t to ip-10-0-128-207.ap-northeast-1.compute.internal
3h4m        Normal    Scheduled          pod/dns-default-pv46p   Successfully assigned openshift-dns/dns-default-pv46p to ip-10-0-202-182.ap-northeast-1.compute.internal
173m        Normal    Scheduled          pod/dns-default-6kcnf   Successfully assigned openshift-dns/dns-default-6kcnf to ip-10-0-132-185.ap-northeast-1.compute.internal
3h4m        Normal    SuccessfulCreate   daemonset/dns-default   Created pod: dns-default-9c64t
3h4m        Warning   FailedMount        pod/dns-default-kqc2z   MountVolume.SetUp failed for volume "metrics-tls" : secret "dns-default-metrics-tls" not found
3h4m        Normal    SuccessfulCreate   daemonset/dns-default   Created pod: dns-default-pv46p
3h4m        Normal    SuccessfulCreate   daemonset/dns-default   Created pod: dns-default-kqc2z
3h4m        Warning   FailedMount        pod/dns-default-9c64t   MountVolume.SetUp failed for volume "metrics-tls" : secret "dns-default-metrics-tls" not found
3h4m        Normal    AddedInterface     pod/dns-default-pv46p   Add eth0 [10.129.0.16/23]
3h4m        Normal    AddedInterface     pod/dns-default-kqc2z   Add eth0 [10.128.0.8/23]
3h4m        Normal    AddedInterface     pod/dns-default-9c64t   Add eth0 [10.130.0.9/23]
3h4m        Normal    Pulling            pod/dns-default-pv46p   Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96e3290d56c2c8a2b3acb8c61583d1f2132770d401a68cdde631ee75c5659cdb"
3h4m        Normal    Pulling            pod/dns-default-kqc2z   Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96e3290d56c2c8a2b3acb8c61583d1f2132770d401a68cdde631ee75c5659cdb"
3h4m        Normal    Pulling            pod/dns-default-9c64t   Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96e3290d56c2c8a2b3acb8c61583d1f2132770d401a68cdde631ee75c5659cdb"
3h4m        Normal    Pulling            pod/dns-default-kqc2z   Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee52f47649bbabb7cc880e6996cef0aeb6c142d5310a4b5e2e1db216af255b4"
3h4m        Normal    Started            pod/dns-default-pv46p   Started container dns
3h4m        Normal    Started            pod/dns-default-kqc2z   Started container kube-rbac-proxy
3h4m        Normal    Pulled             pod/dns-default-kqc2z   Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96e3290d56c2c8a2b3acb8c61583d1f2132770d401a68cdde631ee75c5659cdb" in 11.799179588s
3h4m        Normal    Created            pod/dns-default-pv46p   Created container dns
3h4m        Normal    Pulled             pod/dns-default-pv46p   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a1de7c7e809a2f370193def15751db5aef971fab245c1b07ad8c5a5c00ce4d" already present on machine
3h4m        Normal    Created            pod/dns-default-pv46p   Created container kube-rbac-proxy
3h4m        Normal    Started            pod/dns-default-pv46p   Started container kube-rbac-proxy
3h4m        Normal    Pulled             pod/dns-default-pv46p   Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96e3290d56c2c8a2b3acb8c61583d1f2132770d401a68cdde631ee75c5659cdb" in 12.270313597s
3h4m        Normal    Pulling            pod/dns-default-pv46p   Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee52f47649bbabb7cc880e6996cef0aeb6c142d5310a4b5e2e1db216af255b4"
3h4m        Normal    Started            pod/dns-default-kqc2z   Started container dns
3h4m        Normal    Pulled             pod/dns-default-kqc2z   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a1de7c7e809a2f370193def15751db5aef971fab245c1b07ad8c5a5c00ce4d" already present on machine
3h4m        Normal    Created            pod/dns-default-kqc2z   Created container dns
3h4m        Normal    Created            pod/dns-default-kqc2z   Created container kube-rbac-proxy
3h3m        Normal    Pulled             pod/dns-default-9c64t   Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96e3290d56c2c8a2b3acb8c61583d1f2132770d401a68cdde631ee75c5659cdb" in 16.784827866s
3h3m        Normal    Created            pod/dns-default-9c64t   Created container dns
3h3m        Normal    Started            pod/dns-default-9c64t   Started container dns
3h3m        Normal    Pulled             pod/dns-default-9c64t   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a1de7c7e809a2f370193def15751db5aef971fab245c1b07ad8c5a5c00ce4d" already present on machine
3h3m        Normal    Created            pod/dns-default-9c64t   Created container kube-rbac-proxy
3h3m        Normal    Started            pod/dns-default-9c64t   Started container kube-rbac-proxy
3h3m        Normal    Pulling            pod/dns-default-9c64t   Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee52f47649bbabb7cc880e6996cef0aeb6c142d5310a4b5e2e1db216af255b4"
3h3m        Normal    Pulled             pod/dns-default-pv46p   Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee52f47649bbabb7cc880e6996cef0aeb6c142d5310a4b5e2e1db216af255b4" in 12.051423057s
3h3m        Normal    Pulled             pod/dns-default-kqc2z   Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee52f47649bbabb7cc880e6996cef0aeb6c142d5310a4b5e2e1db216af255b4" in 12.446705375s
3h3m        Normal    Started            pod/dns-default-pv46p   Started container dns-node-resolver
3h3m        Normal    Created            pod/dns-default-pv46p   Created container dns-node-resolver
3h3m        Normal    Created            pod/dns-default-kqc2z   Created container dns-node-resolver
3h3m        Normal    Started            pod/dns-default-kqc2z   Started container dns-node-resolver
3h3m        Normal    Started            pod/dns-default-9c64t   Started container dns-node-resolver
3h3m        Normal    Created            pod/dns-default-9c64t   Created container dns-node-resolver
3h3m        Normal    Pulled             pod/dns-default-9c64t   Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee52f47649bbabb7cc880e6996cef0aeb6c142d5310a4b5e2e1db216af255b4" in 12.124869449s
173m        Normal    SuccessfulCreate   daemonset/dns-default   Created pod: dns-default-4mcbz
173m        Normal    SuccessfulCreate   daemonset/dns-default   Created pod: dns-default-6kcnf
173m        Normal    SuccessfulCreate   daemonset/dns-default   Created pod: dns-default-n2k8c
172m        Warning   NetworkNotReady    pod/dns-default-4mcbz   network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
172m        Warning   NetworkNotReady    pod/dns-default-6kcnf   network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
172m        Warning   NetworkNotReady    pod/dns-default-n2k8c   network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?
171m        Normal    AddedInterface     pod/dns-default-4mcbz   Add eth0 [10.131.0.3/23]
171m        Normal    AddedInterface     pod/dns-default-6kcnf   Add eth0 [10.128.2.3/23]
171m        Normal    AddedInterface     pod/dns-default-n2k8c   Add eth0 [10.129.2.3/23]
154m        Warning   NodeNotReady       pod/dns-default-kqc2z   Node is not ready
154m        Warning   NodeNotReady       pod/dns-default-4mcbz   Node is not ready
153m        Normal    AddedInterface     pod/dns-default-4mcbz   Add eth0 [10.131.0.3/23]
153m        Normal    Pulled             pod/dns-default-4mcbz   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee52f47649bbabb7cc880e6996cef0aeb6c142d5310a4b5e2e1db216af255b4" already present on machine
153m        Normal    Started            pod/dns-default-4mcbz   Started container kube-rbac-proxy
153m        Normal    Pulled             pod/dns-default-4mcbz   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a1de7c7e809a2f370193def15751db5aef971fab245c1b07ad8c5a5c00ce4d" already present on machine
153m        Normal    Started            pod/dns-default-4mcbz   Started container dns
153m        Normal    Created            pod/dns-default-4mcbz   Created container dns
153m        Normal    Pulled             pod/dns-default-4mcbz   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96e3290d56c2c8a2b3acb8c61583d1f2132770d401a68cdde631ee75c5659cdb" already present on machine
153m        Normal    Created            pod/dns-default-4mcbz   Created container kube-rbac-proxy
153m        Normal    Created            pod/dns-default-4mcbz   Created container dns-node-resolver
153m        Normal    Started            pod/dns-default-4mcbz   Started container dns-node-resolver
153m        Normal    AddedInterface     pod/dns-default-kqc2z   Add eth0 [10.128.0.2/23]
153m        Normal    Pulled             pod/dns-default-kqc2z   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96e3290d56c2c8a2b3acb8c61583d1f2132770d401a68cdde631ee75c5659cdb" already present on machine
153m        Normal    Created            pod/dns-default-kqc2z   Created container dns
153m        Normal    Pulled             pod/dns-default-kqc2z   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a1de7c7e809a2f370193def15751db5aef971fab245c1b07ad8c5a5c00ce4d" already present on machine
153m        Normal    Created            pod/dns-default-kqc2z   Created container kube-rbac-proxy
153m        Normal    Started            pod/dns-default-kqc2z   Started container dns
153m        Normal    Started            pod/dns-default-kqc2z   Started container kube-rbac-proxy
153m        Normal    Pulled             pod/dns-default-kqc2z   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee52f47649bbabb7cc880e6996cef0aeb6c142d5310a4b5e2e1db216af255b4" already present on machine
153m        Normal    Started            pod/dns-default-kqc2z   Started container dns-node-resolver
153m        Normal    Created            pod/dns-default-kqc2z   Created container dns-node-resolver
151m        Warning   NodeNotReady       pod/dns-default-pv46p   Node is not ready
151m        Warning   NodeNotReady       pod/dns-default-n2k8c   Node is not ready
151m        Normal    Started            pod/dns-default-n2k8c   Started container dns
151m        Normal    Pulled             pod/dns-default-n2k8c   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a1de7c7e809a2f370193def15751db5aef971fab245c1b07ad8c5a5c00ce4d" already present on machine
151m        Normal    AddedInterface     pod/dns-default-n2k8c   Add eth0 [10.129.2.3/23]
151m        Normal    Created            pod/dns-default-n2k8c   Created container dns
151m        Normal    Pulled             pod/dns-default-n2k8c   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96e3290d56c2c8a2b3acb8c61583d1f2132770d401a68cdde631ee75c5659cdb" already present on machine
151m        Normal    Created            pod/dns-default-n2k8c   Created container dns-node-resolver
151m        Normal    Started            pod/dns-default-n2k8c   Started container dns-node-resolver
151m        Normal    Created            pod/dns-default-n2k8c   Created container kube-rbac-proxy
151m        Normal    Started            pod/dns-default-n2k8c   Started container kube-rbac-proxy
151m        Normal    Pulled             pod/dns-default-n2k8c   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee52f47649bbabb7cc880e6996cef0aeb6c142d5310a4b5e2e1db216af255b4" already present on machine
151m        Normal    AddedInterface     pod/dns-default-pv46p   Add eth0 [10.129.0.6/23]
151m        Normal    Pulled             pod/dns-default-pv46p   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96e3290d56c2c8a2b3acb8c61583d1f2132770d401a68cdde631ee75c5659cdb" already present on machine
151m        Normal    Started            pod/dns-default-pv46p   Started container kube-rbac-proxy
151m        Normal    Pulled             pod/dns-default-pv46p   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a1de7c7e809a2f370193def15751db5aef971fab245c1b07ad8c5a5c00ce4d" already present on machine
151m        Normal    Started            pod/dns-default-pv46p   Started container dns
151m        Normal    Pulled             pod/dns-default-pv46p   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee52f47649bbabb7cc880e6996cef0aeb6c142d5310a4b5e2e1db216af255b4" already present on machine
151m        Normal    Created            pod/dns-default-pv46p   Created container dns
151m        Normal    Created            pod/dns-default-pv46p   Created container kube-rbac-proxy
151m        Normal    Created            pod/dns-default-pv46p   Created container dns-node-resolver
151m        Normal    Started            pod/dns-default-pv46p   Started container dns-node-resolver
149m        Normal    Pulled             pod/dns-default-6kcnf   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96e3290d56c2c8a2b3acb8c61583d1f2132770d401a68cdde631ee75c5659cdb" already present on machine
149m        Normal    Created            pod/dns-default-6kcnf   Created container dns
149m        Normal    Started            pod/dns-default-6kcnf   Started container dns
149m        Normal    AddedInterface     pod/dns-default-6kcnf   Add eth0 [10.128.2.3/23]
149m        Normal    Pulled             pod/dns-default-6kcnf   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a1de7c7e809a2f370193def15751db5aef971fab245c1b07ad8c5a5c00ce4d" already present on machine
149m        Normal    Started            pod/dns-default-6kcnf   Started container kube-rbac-proxy
149m        Normal    Created            pod/dns-default-6kcnf   Created container kube-rbac-proxy
149m        Normal    Started            pod/dns-default-6kcnf   Started container dns-node-resolver
149m        Normal    Created            pod/dns-default-6kcnf   Created container dns-node-resolver
149m        Normal    Pulled             pod/dns-default-6kcnf   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee52f47649bbabb7cc880e6996cef0aeb6c142d5310a4b5e2e1db216af255b4" already present on machine
149m        Warning   NodeNotReady       pod/dns-default-9c64t   Node is not ready
148m        Normal    Pulled             pod/dns-default-9c64t   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96e3290d56c2c8a2b3acb8c61583d1f2132770d401a68cdde631ee75c5659cdb" already present on machine
148m        Normal    AddedInterface     pod/dns-default-9c64t   Add eth0 [10.130.0.2/23]
148m        Normal    Started            pod/dns-default-9c64t   Started container dns
148m        Normal    Pulled             pod/dns-default-9c64t   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a1de7c7e809a2f370193def15751db5aef971fab245c1b07ad8c5a5c00ce4d" already present on machine
148m        Normal    Created            pod/dns-default-9c64t   Created container dns
148m        Normal    Pulled             pod/dns-default-9c64t   Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee52f47649bbabb7cc880e6996cef0aeb6c142d5310a4b5e2e1db216af255b4" already present on machine
148m        Normal    Started            pod/dns-default-9c64t   Started container kube-rbac-proxy
148m        Normal    Created            pod/dns-default-9c64t   Created container kube-rbac-proxy
148m        Normal    Started            pod/dns-default-9c64t   Started container dns-node-resolver
148m        Normal    Created            pod/dns-default-9c64t   Created container dns-node-resolver

Based on the above moving bug to verified state.

Comment 10 errata-xmlrpc 2020-10-27 16:42:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196