Bug 1857268 - oc get events fails when sorting by lastTimestamp
Summary: oc get events fails when sorting by lastTimestamp
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.z
Assignee: Jan Chaloupka
QA Contact: RamaKasturi
URL:
Whiteboard:
: 1857420 (view as bug list)
Depends On: 1880283
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-15 15:14 UTC by Christian Passarelli
Modified: 2020-11-05 12:47 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
: 1880283 (view as bug list)
Environment:
Last Closed: 2020-11-05 12:46:54 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubernetes kubernetes pull 93663 0 None closed Set DeprecatedLastTimestamp and DeprecatedFirstTimestamp so LastTimestamp and FirstTimestamp are still set 2020-11-17 13:18:14 UTC
Github kubernetes kubernetes pull 94252 0 None closed Automated cherry pick of #89999: If firstTimestamp is not set use eventTime when printing event 2020-11-17 13:18:14 UTC
Github kubernetes kubernetes pull 94945 0 None closed Automated cherry pick of #94666: Handle nil elements when sorting, instead of panicking 2020-11-17 13:18:13 UTC
Github openshift kubernetes-kubectl pull 33 0 None closed Oc 4.5 kubernetes 1.18.8 2020-11-17 13:18:35 UTC
Github openshift oc pull 551 0 None closed Bug 1860789: Rebase k8s to 1.18.8 to enable approving CSRs on a 4.6 server 2020-11-17 13:18:14 UTC
Github openshift origin pull 25461 0 None closed bug 1857268: UPSTREAM: 94252: If firstTimestamp is not set use eventTime when printing event 2020-11-17 13:18:36 UTC
Red Hat Bugzilla 1772891 1 None None None 2021-01-20 06:05:38 UTC
Red Hat Knowledge Base (Solution) 5224831 0 None None None 2020-10-20 15:38:49 UTC
Red Hat Product Errata RHBA-2020:4325 0 None None None 2020-11-05 12:47:18 UTC

Internal Links: 1772891

Description Christian Passarelli 2020-07-15 15:14:09 UTC
Description of problem:

Sorting events by lastTimestamp item return this error:

$ oc get events -n openshift-dns --sort-by='.lastTimestamp'
F0715 09:18:21.343259   22629 sorter.go:354] Field {.lastTimestamp} in [][][]reflect.Value is an unsortable type: interface, err: unsortable interface: interface

If are present events with "null" value on the item.


Version-Release number of selected component (if applicable):

Client Version: 4.4.12
Also older versions are affected.

How reproducible:

When in the event list is present at least one event with "<unknown>" in the "LAST SEEN" column the command will fail.


Actual results:

"F0715 09:18:21.343259   22629 sorter.go:354] Field {.lastTimestamp} in [][][]reflect.Value is an unsortable type: interface, err: unsortable interface: interface"

Expected results:


Additional info:

Comment 1 Jan Chaloupka 2020-07-29 14:13:49 UTC
There are two types of events:
- the core type: https://github.com/kubernetes/kubernetes/blob/8e8b6a01cf6bf55dea5e2e4f554597a95c82988a/staging/src/k8s.io/api/core/v1/types.go#L5309
- beta/v1 type: https://github.com/kubernetes/kubernetes/blob/8e8b6a01cf6bf55dea5e2e4f554597a95c82988a/staging/src/k8s.io/api/events/v1/types.go#L90

The core type has LastTimestamp, beta/v1 type does not. It has DeprecatedLastTimestamp instead. Depending on which version of the event is used in each component producing the events, `LastTimestamp` field is either set or not.

Also, as you can see in the customer case logs, all the events with LAST SEEN equal to "<unknown>" are produced by the default scheduler (see the Scheduled keyword in the REASON column).

Related upstream issue: https://github.com/kubernetes/kubernetes/issues/89689

Comment 2 Jan Chaloupka 2020-07-29 14:27:05 UTC
https://stupefied-goodall-e282f7.netlify.app/contributors/design-proposals/instrumentation/events-redesign/ for more on the changes.

The new events have series field which contains `.lastObservedTime` field as an equivalent for the deprecated `.lastTimestamp`.

There's no fix in the oc/kubectl code allowing to sort both types of events at the same time. Unless a custom wrapper around kubectl/oc is implemented that lists all the relevant pods in json and sorts them locally.

Comment 3 Jan Chaloupka 2020-07-30 10:23:09 UTC
Revisiting the code again, it's the apiserver who provides a meta.Table response with all the rows. The table is already populated with lastTimestamp field set to unknown so there's nothing to be done on the oc/kubectl side. If fixed, the apiserver needs to properly populated the field from .series.lastObservedTime if available.

Comment 4 Jan Chaloupka 2020-07-30 10:44:02 UTC
Conversion from v1/v1beta1 uses DeprecatedLastTimestamp to set LastTimestamp.

https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/events/v1beta1/conversion.go#L41
https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/events/v1/conversion.go#L41

Meaning DeprecatedLastTimestamp field in v1/v1beta1 event has to be empty.

Comment 5 Jan Chaloupka 2020-07-30 11:04:58 UTC
kube-scheduler uses event recorder from k8s.io/client-go/tools/events [1] which does not take deprecated fields into account.

[1] https://github.com/kubernetes/kubernetes/blob/db28b0239a1d09e46507e0b320f0b4ffa1cbc8b6/staging/src/k8s.io/client-go/tools/events/event_recorder.go#L66-L88

The issue needs to be addressed in k8s.io/client-go/tools/events.

Comment 6 Jan Chaloupka 2020-08-03 17:03:29 UTC
Observations:
- 4.4 deployed with 1.19 scheduler does not leave lastTimestamp field empty
- 4.4 deployed with 1.18 scheduler leaves lastTimestamp field empty
- 4.4 deployed with 1.17 scheduler leaves lastTimestamp field empty

Comment 7 Jan Chaloupka 2020-08-03 22:30:25 UTC
1.18 and 1.17 kube-scheduler are choosing the event type (either core of v1/v1beta1) based on what the apiserver publishes (through Discovery). 4.4 is publishing v1beta1 so the scheduler is posting v1beta1 events.

1.19 checks for v1 only so it falls to the core event type in 4.4 since there's no event v1 version yet.

Comment 8 Jan Chaloupka 2020-08-03 23:23:39 UTC
Upstream PR for 1.18: https://github.com/kubernetes/kubernetes/pull/93663

Once discussed upstream and accepted, I will backport it to release-1.17 and to 4.5 (v1.18) and 4.4 (v1.17).

Comment 9 Maciej Szulik 2020-08-11 11:01:56 UTC
*** Bug 1857420 has been marked as a duplicate of this bug. ***

Comment 12 Jan Chaloupka 2020-09-10 14:22:10 UTC
Putting UpcomingSprint just in case 4.5 backport PR does not get merged EOW

Comment 13 Jan Chaloupka 2020-10-01 15:10:49 UTC
Adding UpcomingSprint just in case the fix does not get merged by EOW

Comment 15 RamaKasturi 2020-10-23 10:09:40 UTC
Verified bug with the payload below and i see that the fix works.

[knarra@knarra openshift-client-linux-4.5.0-0.nightly-2020-10-23-025638]$ ./oc version
Client Version: 4.5.0-0.nightly-2020-10-23-025638
Server Version: 4.5.0-0.nightly-2020-10-23-025638
Kubernetes Version: v1.18.3+70b8a93

[knarra@knarra openshift-client-linux-4.5.0-0.nightly-2020-10-23-025638]$ ./oc get events -n openshift-sdn --sort-by='.lastTimestamp'
LAST SEEN   TYPE      REASON                   OBJECT                     MESSAGE
46m         Normal    Scheduled                pod/sdn-mm59r              Successfully assigned openshift-sdn/sdn-mm59r to ip-10-0-161-172.us-east-2.compute.internal
46m         Normal    Scheduled                pod/ovs-ggssz              Successfully assigned openshift-sdn/ovs-ggssz to ip-10-0-129-75.us-east-2.compute.internal
59m         Normal    Scheduled                pod/ovs-2j7wb              Successfully assigned openshift-sdn/ovs-2j7wb to ip-10-0-131-243.us-east-2.compute.internal
59m         Normal    Scheduled                pod/sdn-controller-qns8k   Successfully assigned openshift-sdn/sdn-controller-qns8k to ip-10-0-211-110.us-east-2.compute.internal
59m         Normal    Scheduled                pod/sdn-controller-722hg   Successfully assigned openshift-sdn/sdn-controller-722hg to ip-10-0-179-228.us-east-2.compute.internal
59m         Normal    Scheduled                pod/sdn-bqrmv              Successfully assigned openshift-sdn/sdn-bqrmv to ip-10-0-131-243.us-east-2.compute.internal
59m         Normal    Scheduled                pod/ovs-724w9              Successfully assigned openshift-sdn/ovs-724w9 to ip-10-0-211-110.us-east-2.compute.internal
46m         Normal    Scheduled                pod/sdn-dlpvp              Successfully assigned openshift-sdn/sdn-dlpvp to ip-10-0-129-75.us-east-2.compute.internal
59m         Normal    Scheduled                pod/sdn-controller-vhcmh   Successfully assigned openshift-sdn/sdn-controller-vhcmh to ip-10-0-131-243.us-east-2.compute.internal
46m         Normal    Scheduled                pod/ovs-gkrkg              Successfully assigned openshift-sdn/ovs-gkrkg to ip-10-0-161-172.us-east-2.compute.internal
59m         Normal    Scheduled                pod/sdn-8s868              Successfully assigned openshift-sdn/sdn-8s868 to ip-10-0-179-228.us-east-2.compute.internal
59m         Normal    Scheduled                pod/ovs-9qznw              Successfully assigned openshift-sdn/ovs-9qznw to ip-10-0-179-228.us-east-2.compute.internal
46m         Normal    Scheduled                pod/sdn-lm5nb              Successfully assigned openshift-sdn/sdn-lm5nb to ip-10-0-196-212.us-east-2.compute.internal
59m         Normal    Scheduled                pod/sdn-m6lrx              Successfully assigned openshift-sdn/sdn-m6lrx to ip-10-0-211-110.us-east-2.compute.internal
46m         Normal    Scheduled                pod/ovs-ncmrc              Successfully assigned openshift-sdn/ovs-ncmrc to ip-10-0-196-212.us-east-2.compute.internal
59m         Normal    SuccessfulCreate         daemonset/sdn-controller   Created pod: sdn-controller-qns8k
59m         Normal    SuccessfulCreate         daemonset/sdn-controller   Created pod: sdn-controller-vhcmh
59m         Normal    SuccessfulCreate         daemonset/sdn-controller   Created pod: sdn-controller-722hg
59m         Normal    Pulling                  pod/sdn-controller-qns8k   Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Pulling                  pod/sdn-controller-722hg   Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    SuccessfulCreate         daemonset/sdn              Created pod: sdn-m6lrx
59m         Normal    SuccessfulCreate         daemonset/sdn              Created pod: sdn-8s868
59m         Normal    SuccessfulCreate         daemonset/sdn              Created pod: sdn-bqrmv
59m         Normal    SuccessfulCreate         daemonset/ovs              Created pod: ovs-724w9
59m         Normal    SuccessfulCreate         daemonset/ovs              Created pod: ovs-9qznw
59m         Normal    SuccessfulCreate         daemonset/ovs              Created pod: ovs-2j7wb
59m         Normal    Pulling                  pod/ovs-9qznw              Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Pulling                  pod/sdn-8s868              Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Warning   FailedToUpdateEndpoint   endpoints/sdn              Failed to update endpoint openshift-sdn/sdn: Operation cannot be fulfilled on endpoints "sdn": the object has been modified; please apply your changes to the latest version and try again
59m         Warning   FailedMount              pod/ovs-2j7wb              MountVolume.SetUp failed for volume "sdn-token-c4bw9" : failed to sync secret cache: timed out waiting for the condition
59m         Normal    Pulling                  pod/sdn-m6lrx              Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Pulling                  pod/ovs-724w9              Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Pulling                  pod/ovs-2j7wb              Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Started                  pod/ovs-724w9              Started container openvswitch
59m         Normal    Created                  pod/ovs-724w9              Created container openvswitch
59m         Normal    Pulled                   pod/ovs-724w9              Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Created                  pod/sdn-m6lrx              Created container sdn
59m         Normal    Started                  pod/sdn-m6lrx              Started container sdn
59m         Normal    Pulled                   pod/ovs-9qznw              Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Pulled                   pod/sdn-8s868              Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Pulling                  pod/sdn-controller-vhcmh   Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Pulled                   pod/sdn-m6lrx              Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Pulled                   pod/sdn-controller-722hg   Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Pulling                  pod/sdn-bqrmv              Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Started                  pod/sdn-controller-qns8k   Started container sdn-controller
59m         Normal    Created                  pod/sdn-controller-qns8k   Created container sdn-controller
59m         Normal    Pulled                   pod/sdn-controller-qns8k   Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Created                  pod/sdn-8s868              Created container sdn
59m         Normal    Started                  pod/sdn-8s868              Started container sdn
59m         Normal    Started                  pod/ovs-9qznw              Started container openvswitch
59m         Normal    Created                  pod/sdn-controller-722hg   Created container sdn-controller
59m         Normal    Started                  pod/sdn-controller-722hg   Started container sdn-controller
59m         Normal    Created                  pod/ovs-9qznw              Created container openvswitch
59m         Normal    Pulled                   pod/sdn-controller-vhcmh   Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Pulled                   pod/ovs-2j7wb              Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Pulled                   pod/sdn-bqrmv              Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
59m         Normal    Created                  pod/sdn-bqrmv              Created container sdn
59m         Normal    Started                  pod/ovs-2j7wb              Started container openvswitch
59m         Normal    Started                  pod/sdn-controller-vhcmh   Started container sdn-controller
59m         Normal    Created                  pod/ovs-2j7wb              Created container openvswitch
59m         Normal    Created                  pod/sdn-controller-vhcmh   Created container sdn-controller
59m         Normal    Started                  pod/sdn-bqrmv              Started container sdn
46m         Normal    SuccessfulCreate         daemonset/ovs              Created pod: ovs-gkrkg
46m         Normal    SuccessfulCreate         daemonset/sdn              Created pod: sdn-mm59r
46m         Normal    SuccessfulCreate         daemonset/sdn              Created pod: sdn-lm5nb
46m         Normal    SuccessfulCreate         daemonset/ovs              Created pod: ovs-ncmrc
46m         Normal    SuccessfulCreate         daemonset/ovs              Created pod: ovs-ggssz
46m         Normal    SuccessfulCreate         daemonset/sdn              Created pod: sdn-dlpvp
45m         Normal    Pulling                  pod/sdn-dlpvp              Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
45m         Normal    Pulling                  pod/ovs-ggssz              Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
45m         Normal    Pulling                  pod/ovs-ncmrc              Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
45m         Normal    Pulling                  pod/sdn-lm5nb              Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
45m         Normal    Pulling                  pod/sdn-mm59r              Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
45m         Normal    Pulling                  pod/ovs-gkrkg              Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
45m         Normal    Pulled                   pod/ovs-ggssz              Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
45m         Normal    Pulled                   pod/sdn-dlpvp              Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
45m         Normal    Created                  pod/sdn-dlpvp              Created container sdn
45m         Normal    Started                  pod/sdn-dlpvp              Started container sdn
45m         Normal    Started                  pod/ovs-ggssz              Started container openvswitch
45m         Normal    Created                  pod/ovs-ggssz              Created container openvswitch
45m         Normal    Pulled                   pod/ovs-ncmrc              Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
45m         Normal    Pulled                   pod/sdn-lm5nb              Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
45m         Normal    Created                  pod/ovs-ncmrc              Created container openvswitch
45m         Normal    Started                  pod/ovs-ncmrc              Started container openvswitch
45m         Normal    Started                  pod/sdn-lm5nb              Started container sdn
45m         Normal    Created                  pod/sdn-lm5nb              Created container sdn
45m         Normal    Pulled                   pod/sdn-mm59r              Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
45m         Normal    Pulled                   pod/ovs-gkrkg              Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:717096a098ffc30131136c600941a866c3ee792f3adf6dc312088aee354ffb6f"
45m         Normal    Started                  pod/sdn-mm59r              Started container sdn
45m         Normal    Created                  pod/sdn-mm59r              Created container sdn
45m         Normal    Created                  pod/ovs-gkrkg              Created container openvswitch
45m         Normal    Started                  pod/ovs-gkrkg              Started container openvswitch

Based on the above moving bug to verified state.

Comment 18 errata-xmlrpc 2020-11-05 12:46:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5.17 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4325


Note You need to log in before you can comment on or make changes to this bug.