Description of problem: Sorting events by lastTimestamp item return this error: $ oc get events -n openshift-dns --sort-by='.lastTimestamp' F0715 09:18:21.343259 22629 sorter.go:354] Field {.lastTimestamp} in [][][]reflect.Value is an unsortable type: interface, err: unsortable interface: interface If are present events with "null" value on the item. Version-Release number of selected component (if applicable): Client Version: 4.4.12 Also older versions are affected. How reproducible: When in the event list is present at least one event with "<unknown>" in the "LAST SEEN" column the command will fail. Actual results: "F0715 09:18:21.343259 22629 sorter.go:354] Field {.lastTimestamp} in [][][]reflect.Value is an unsortable type: interface, err: unsortable interface: interface" Expected results: Additional info:
There are two types of events: - the core type: https://github.com/kubernetes/kubernetes/blob/8e8b6a01cf6bf55dea5e2e4f554597a95c82988a/staging/src/k8s.io/api/core/v1/types.go#L5309 - beta/v1 type: https://github.com/kubernetes/kubernetes/blob/8e8b6a01cf6bf55dea5e2e4f554597a95c82988a/staging/src/k8s.io/api/events/v1/types.go#L90 The core type has LastTimestamp, beta/v1 type does not. It has DeprecatedLastTimestamp instead. Depending on which version of the event is used in each component producing the events, `LastTimestamp` field is either set or not. Also, as you can see in the customer case logs, all the events with LAST SEEN equal to "<unknown>" are produced by the default scheduler (see the Scheduled keyword in the REASON column). Related upstream issue: https://github.com/kubernetes/kubernetes/issues/89689
https://stupefied-goodall-e282f7.netlify.app/contributors/design-proposals/instrumentation/events-redesign/ for more on the changes. The new events have series field which contains `.lastObservedTime` field as an equivalent for the deprecated `.lastTimestamp`. There's no fix in the oc/kubectl code allowing to sort both types of events at the same time. Unless a custom wrapper around kubectl/oc is implemented that lists all the relevant pods in json and sorts them locally.
Revisiting the code again, it's the apiserver who provides a meta.Table response with all the rows. The table is already populated with lastTimestamp field set to unknown so there's nothing to be done on the oc/kubectl side. If fixed, the apiserver needs to properly populated the field from .series.lastObservedTime if available.
Conversion from v1/v1beta1 uses DeprecatedLastTimestamp to set LastTimestamp. https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/events/v1beta1/conversion.go#L41 https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/events/v1/conversion.go#L41 Meaning DeprecatedLastTimestamp field in v1/v1beta1 event has to be empty.
kube-scheduler uses event recorder from k8s.io/client-go/tools/events [1] which does not take deprecated fields into account. [1] https://github.com/kubernetes/kubernetes/blob/db28b0239a1d09e46507e0b320f0b4ffa1cbc8b6/staging/src/k8s.io/client-go/tools/events/event_recorder.go#L66-L88 The issue needs to be addressed in k8s.io/client-go/tools/events.
Observations: - 4.4 deployed with 1.19 scheduler does not leave lastTimestamp field empty - 4.4 deployed with 1.18 scheduler leaves lastTimestamp field empty - 4.4 deployed with 1.17 scheduler leaves lastTimestamp field empty
1.18 and 1.17 kube-scheduler are choosing the event type (either core of v1/v1beta1) based on what the apiserver publishes (through Discovery). 4.4 is publishing v1beta1 so the scheduler is posting v1beta1 events. 1.19 checks for v1 only so it falls to the core event type in 4.4 since there's no event v1 version yet.
Upstream PR for 1.18: https://github.com/kubernetes/kubernetes/pull/93663 Once discussed upstream and accepted, I will backport it to release-1.17 and to 4.5 (v1.18) and 4.4 (v1.17).
*** Bug 1857420 has been marked as a duplicate of this bug. ***
Putting UpcomingSprint just in case 4.5 backport PR does not get merged EOW
Adding UpcomingSprint just in case the fix does not get merged by EOW
Verified bug with the payload below and i see that the fix works. [knarra@knarra openshift-client-linux-4.5.0-0.nightly-2020-10-23-025638]$ ./oc version Client Version: 4.5.0-0.nightly-2020-10-23-025638 Server Version: 4.5.0-0.nightly-2020-10-23-025638 Kubernetes Version: v1.18.3+70b8a93 [knarra@knarra openshift-client-linux-4.5.0-0.nightly-2020-10-23-025638]$ ./oc get events -n openshift-sdn --sort-by='.lastTimestamp' LAST SEEN TYPE REASON OBJECT MESSAGE 46m Normal Scheduled pod/sdn-mm59r Successfully assigned openshift-sdn/sdn-mm59r to ip-10-0-161-172.us-east-2.compute.internal 46m Normal Scheduled pod/ovs-ggssz Successfully assigned openshift-sdn/ovs-ggssz to ip-10-0-129-75.us-east-2.compute.internal 59m Normal Scheduled pod/ovs-2j7wb Successfully assigned openshift-sdn/ovs-2j7wb to ip-10-0-131-243.us-east-2.compute.internal 59m Normal Scheduled pod/sdn-controller-qns8k Successfully assigned openshift-sdn/sdn-controller-qns8k to ip-10-0-211-110.us-east-2.compute.internal 59m Normal Scheduled pod/sdn-controller-722hg Successfully assigned openshift-sdn/sdn-controller-722hg to ip-10-0-179-228.us-east-2.compute.internal 59m Normal Scheduled pod/sdn-bqrmv Successfully assigned openshift-sdn/sdn-bqrmv to ip-10-0-131-243.us-east-2.compute.internal 59m Normal Scheduled pod/ovs-724w9 Successfully assigned openshift-sdn/ovs-724w9 to ip-10-0-211-110.us-east-2.compute.internal 46m Normal Scheduled pod/sdn-dlpvp Successfully assigned openshift-sdn/sdn-dlpvp to ip-10-0-129-75.us-east-2.compute.internal 59m Normal Scheduled pod/sdn-controller-vhcmh Successfully assigned openshift-sdn/sdn-controller-vhcmh to ip-10-0-131-243.us-east-2.compute.internal 46m Normal Scheduled pod/ovs-gkrkg Successfully assigned openshift-sdn/ovs-gkrkg to ip-10-0-161-172.us-east-2.compute.internal 59m Normal Scheduled pod/sdn-8s868 Successfully assigned openshift-sdn/sdn-8s868 to ip-10-0-179-228.us-east-2.compute.internal 59m Normal Scheduled pod/ovs-9qznw Successfully assigned openshift-sdn/ovs-9qznw to ip-10-0-179-228.us-east-2.compute.internal 46m Normal Scheduled pod/sdn-lm5nb Successfully assigned openshift-sdn/sdn-lm5nb to ip-10-0-196-212.us-east-2.compute.internal 59m Normal Scheduled pod/sdn-m6lrx Successfully assigned openshift-sdn/sdn-m6lrx to ip-10-0-211-110.us-east-2.compute.internal 46m Normal Scheduled pod/ovs-ncmrc Successfully assigned openshift-sdn/ovs-ncmrc to ip-10-0-196-212.us-east-2.compute.internal 59m Normal SuccessfulCreate daemonset/sdn-controller Created pod: sdn-controller-qns8k 59m Normal SuccessfulCreate daemonset/sdn-controller Created pod: sdn-controller-vhcmh 59m Normal SuccessfulCreate daemonset/sdn-controller Created pod: sdn-controller-722hg 59m Normal Pulling pod/sdn-controller-qns8k Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Pulling pod/sdn-controller-722hg Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal SuccessfulCreate daemonset/sdn Created pod: sdn-m6lrx 59m Normal SuccessfulCreate daemonset/sdn Created pod: sdn-8s868 59m Normal SuccessfulCreate daemonset/sdn Created pod: sdn-bqrmv 59m Normal SuccessfulCreate daemonset/ovs Created pod: ovs-724w9 59m Normal SuccessfulCreate daemonset/ovs Created pod: ovs-9qznw 59m Normal SuccessfulCreate daemonset/ovs Created pod: ovs-2j7wb 59m Normal Pulling pod/ovs-9qznw Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Pulling pod/sdn-8s868 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Warning FailedToUpdateEndpoint endpoints/sdn Failed to update endpoint openshift-sdn/sdn: Operation cannot be fulfilled on endpoints "sdn": the object has been modified; please apply your changes to the latest version and try again 59m Warning FailedMount pod/ovs-2j7wb MountVolume.SetUp failed for volume "sdn-token-c4bw9" : failed to sync secret cache: timed out waiting for the condition 59m Normal Pulling pod/sdn-m6lrx Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Pulling pod/ovs-724w9 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Pulling pod/ovs-2j7wb Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Started pod/ovs-724w9 Started container openvswitch 59m Normal Created pod/ovs-724w9 Created container openvswitch 59m Normal Pulled pod/ovs-724w9 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Created pod/sdn-m6lrx Created container sdn 59m Normal Started pod/sdn-m6lrx Started container sdn 59m Normal Pulled pod/ovs-9qznw Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Pulled pod/sdn-8s868 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Pulling pod/sdn-controller-vhcmh Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Pulled pod/sdn-m6lrx Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Pulled pod/sdn-controller-722hg Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Pulling pod/sdn-bqrmv Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Started pod/sdn-controller-qns8k Started container sdn-controller 59m Normal Created pod/sdn-controller-qns8k Created container sdn-controller 59m Normal Pulled pod/sdn-controller-qns8k Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Created pod/sdn-8s868 Created container sdn 59m Normal Started pod/sdn-8s868 Started container sdn 59m Normal Started pod/ovs-9qznw Started container openvswitch 59m Normal Created pod/sdn-controller-722hg Created container sdn-controller 59m Normal Started pod/sdn-controller-722hg Started container sdn-controller 59m Normal Created pod/ovs-9qznw Created container openvswitch 59m Normal Pulled pod/sdn-controller-vhcmh Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Pulled pod/ovs-2j7wb Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Pulled pod/sdn-bqrmv Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 59m Normal Created pod/sdn-bqrmv Created container sdn 59m Normal Started pod/ovs-2j7wb Started container openvswitch 59m Normal Started pod/sdn-controller-vhcmh Started container sdn-controller 59m Normal Created pod/ovs-2j7wb Created container openvswitch 59m Normal Created pod/sdn-controller-vhcmh Created container sdn-controller 59m Normal Started pod/sdn-bqrmv Started container sdn 46m Normal SuccessfulCreate daemonset/ovs Created pod: ovs-gkrkg 46m Normal SuccessfulCreate daemonset/sdn Created pod: sdn-mm59r 46m Normal SuccessfulCreate daemonset/sdn Created pod: sdn-lm5nb 46m Normal SuccessfulCreate daemonset/ovs Created pod: ovs-ncmrc 46m Normal SuccessfulCreate daemonset/ovs Created pod: ovs-ggssz 46m Normal SuccessfulCreate daemonset/sdn Created pod: sdn-dlpvp 45m Normal Pulling pod/sdn-dlpvp Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 45m Normal Pulling pod/ovs-ggssz Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 45m Normal Pulling pod/ovs-ncmrc Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 45m Normal Pulling pod/sdn-lm5nb Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 45m Normal Pulling pod/sdn-mm59r Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 45m Normal Pulling pod/ovs-gkrkg Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 45m Normal Pulled pod/ovs-ggssz Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 45m Normal Pulled pod/sdn-dlpvp Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 45m Normal Created pod/sdn-dlpvp Created container sdn 45m Normal Started pod/sdn-dlpvp Started container sdn 45m Normal Started pod/ovs-ggssz Started container openvswitch 45m Normal Created pod/ovs-ggssz Created container openvswitch 45m Normal Pulled pod/ovs-ncmrc Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 45m Normal Pulled pod/sdn-lm5nb Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 45m Normal Created pod/ovs-ncmrc Created container openvswitch 45m Normal Started pod/ovs-ncmrc Started container openvswitch 45m Normal Started pod/sdn-lm5nb Started container sdn 45m Normal Created pod/sdn-lm5nb Created container sdn 45m Normal Pulled pod/sdn-mm59r Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 45m Normal Pulled pod/ovs-gkrkg Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f" 45m Normal Started pod/sdn-mm59r Started container sdn 45m Normal Created pod/sdn-mm59r Created container sdn 45m Normal Created pod/ovs-gkrkg Created container openvswitch 45m Normal Started pod/ovs-gkrkg Started container openvswitch Based on the above moving bug to verified state.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.5.17 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4325