Bug 1785445 - mgr/k8sevents does not account for incomplete events passed in from kubernetes
Summary: mgr/k8sevents does not account for incomplete events passed in from kubernetes
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: Ceph-Mgr Plugins
Version: 4.0
Hardware: x86_64
OS: Linux
medium
low
Target Milestone: z1
: 4.1
Assignee: Paul Cuzner
QA Contact: Madhavi Kasturi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-19 21:44 UTC by Paul Cuzner
Modified: 2020-07-20 14:21 UTC (History)
8 users (show)

Fixed In Version: ceph-14.2.8-64.el8cp, ceph-14.2.8-68.el7cp
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-20 14:20:55 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2020:3003 0 None None None 2020-07-20 14:21:11 UTC

Description Paul Cuzner 2019-12-19 21:44:41 UTC
Description of problem:
The namespacewatcher in k8sevents pulls in events from kubernetes within the namespace the mgr is running under. If these events have an improper/non-existent  timestamp, the 'ls' command fails when it tries to sort the events.

The impact is low, since k8sevents carries on - but it means that from a ceph console you no longer have visibility of pod events within the namespace

Version-Release number of selected component (if applicable):
OCS 4.2.4

How reproducible:
Whenever the k8sevents module is loaded

Steps to Reproduce:
1. deploy OCS
2. launch the toolbox and enable the module
3. issue ceph k8sevents ls command

Actual results:
command fails referring to NoneType and datetime.datetime comparisoon

Expected results:
ls should show all events in the namespace, sorted - most recent first

Additional info:

Comment 3 Yaniv Kaul 2020-02-26 16:17:21 UTC
Jeff?

Comment 7 Yaniv Kaul 2020-04-22 15:54:40 UTC
This one has the blocker flag set for 4.1. Can we ensure this is worked on, moved or closed?

Comment 8 Jeff Brown 2020-04-22 16:33:46 UTC
I will review with Paul when he's online.

Comment 9 Jeff Brown 2020-04-22 16:34:06 UTC
I will review with Paul when he's online.

Comment 10 Federico Lucifredi 2020-04-22 21:12:49 UTC
This is not an issue for RHCS.  This is not a regression for Ceph (OCS). 

The bug does not meet blocker criteria. Moving to 4.1z1.

Comment 11 Paul Cuzner 2020-04-22 22:22:21 UTC
+1 This is not a blocker for 4.1

Some background;
- k8sevents is NOT currently enabled by default in either RHCS or OCS4.x
- k8sevents performs two tasks
  - it sends ceph events (pool created, osd added/removed, pool protection level changed, host added/removed) to the k8s events API(this is the main feature of the module and it's not affected by the bug)
  - it pulls events from the namespace that the mgr is running in, so at the ceph command line the admin can see k8sevents happening in the namespace. This was a secondary feature, and IS where the bug lies

To hit the bug in OCS the user would need to start the toolbox pod, rsh to it, and run the ceph k8sevents ls command. However, this workflow is not exactly normal.

Comment 15 Paul Cuzner 2020-06-11 02:09:42 UTC
upstream is merged (master) - upstream backport to do

downstream packport complete, commit pushed to ceph-4.1-rhel-patches

Comment 22 errata-xmlrpc 2020-07-20 14:20:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:3003


Note You need to log in before you can comment on or make changes to this bug.