Bug 1632430 - KubeletNotReady "pleg was last seen active.." string reports corrupt 'last seen active' timing
Summary: KubeletNotReady "pleg was last seen active.." string reports corrupt 'last se...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.1.0
Assignee: Seth Jennings
QA Contact: Weinan Liu
Depends On:
TreeView+ depends on / blocked
Reported: 2018-09-24 19:13 UTC by Justin Pierce
Modified: 2019-10-31 16:30 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Improves log message when PLEG has not been healthy since the kubelet started
Clone Of:
Last Closed: 2019-06-04 10:40:35 UTC
Target Upstream Version:

Attachments (Terms of Use)
"impossible" last seen active output (2.27 KB, text/plain)
2018-09-24 19:13 UTC, Justin Pierce
no flags Details

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 0 None None None 2019-06-04 10:40:45 UTC

Description Justin Pierce 2018-09-24 19:13:28 UTC
Created attachment 1486508 [details]
"impossible" last seen active output

Description of problem:
Observed the following string `oc describe` Conditions output: container runtime is down,PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s,network state unknown

2562047h equates to hundreds of years.

Version-Release number of selected component (if applicable):

How reproducible:
Consistent in this particular environment for the NotReady node in question.

Additional info:
The node is running cri-o-1.11.1-2.rhaos3.11.git1759204.el7.x86_64.

Comment 1 Seth Jennings 2018-09-24 19:25:39 UTC
Yes, that is a known log readability issue.

Basically is it a unit64 underflow due to the last relist time being 0. It indicates that PLEG has not succeeded, ever, since the kubelet process started.  

If you see this when just starting the kubelet, it is not a real error.

If you see this after the kubelet has been up for a while, typically indicates that the runtime is not running or unresponsive.

Comment 2 Seth Jennings 2018-09-24 19:31:33 UTC
Actually, it isn't an underflow.  It is just the current time - 0 as a duration:


Comment 3 Seth Jennings 2018-09-24 20:27:43 UTC
No time like the present to fix this.  Upstream PR:

Comment 5 Seth Jennings 2018-10-31 16:48:07 UTC
Origin PR:

Comment 9 Ryan Howe 2019-05-31 18:33:46 UTC
I just would like to note again (also see comment #1) this bug fix is purely cosmetic and does not point to an error with the node or container-runtime 

The following log is a result time being 0 since the relist time, when the node process is first started. 

PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago

Comment 11 errata-xmlrpc 2019-06-04 10:40:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.