Bug 1937709

This bug is very similar to https://bugzilla.redhat.com/show_bug.cgi?id=1917009. The reporter mentioned that after some time the error logs stopped appearing. Maybe there is a time drift happening at some point and the error appears until it catches up.

Hello Ashwini, could you perhaps try to follow this blog post which explains how to debug out-of-order samples https://www.robustperception.io/debugging-out-of-order-samples?

The most likely reason for this to be happening is that the time on your node jumped backward at some point. The most reasonable way to solve this is either to restart Prometheus to force a resync or wait for the system time to catch back up.

From the blog post:
```
A related potential cause is if the system time goes backwards on the machine running Prometheus. This would show up as widespread out-of-order for all targets that existed both before the time change and afterwards, affecting both scraped series and generated series like up. To check if this is the cause, evaluate timestamp(up) for some target the Prometheus is scraping and see if it is in the future. To fix this you can either wait for the system time to catch back up, or delete the "future" data from Prometheus.
```

Since the errors are reappearing over time there might me something else happening. In order to find what, I will need more information.

Could you please provide me the following data:

1. The logs of the Prometheus instance where we can see the errors with the log level set to `debug`. https://docs.openshift.com/container-platform/4.7/monitoring/configuring-the-monitoring-stack.html#setting-log-levels-for-monitoring-components_configuring-the-monitoring-stack

2. The sos-report of the node where the problematic instance of Prometheus is running.

3. The sos-report of one of the node for which metrics are reported to arrive out-of-order in the Prometheus log. The node name should be visible in the Prometheus logs once you run it with the debug log level.

Any details on the NTP configuration, like what servers is the client using, how they are synchronized, are there any virtual machines involved, etc?

The "Forward time jump detected" message indicates that something else is messing with the clock, or the machine was suspended/resumed/migrated.

Are there any "System clock wrong by" messages?

I am having the same problem, starting around OCP 4.6.x … it still drags on to OCP 4.7.x.

For me, I am running OCP inside of VMs, on top of RHEL 8.x. The hosts have a default configuration with Chrony, and the state looks good (to me):

~~~
Reference ID    : 81468422 (stratum2-1.NTP.TechFak.NET)
Stratum         : 3
Ref time (UTC)  : Wed May 26 07:41:30 2021
System time     : 0.000042926 seconds slow of NTP time
Last offset     : -0.000008695 seconds
RMS offset      : 0.000111053 seconds
Frequency       : 0.211 ppm fast
Residual freq   : +0.001 ppm
Skew            : 0.026 ppm
Root delay      : 0.031511676 seconds
Root dispersion : 0.002671151 seconds
Update interval : 1029.9 seconds
Leap status     : Normal
~~~

Doing a `dmesg | grep clock` turns up nothing like "clock wrong".

As this does not seem to be an issue in chrony, I'm closing the bug. Please reopen and/or reassign if you have new information.