Bug 836886 - systems have wrong system time and local watchdog is messing with it too
systems have wrong system time and local watchdog is messing with it too
Status: CLOSED INSUFFICIENT_DATA
Product: Beaker
Classification: Community
Component: beah (Show other bugs)
0.5
Unspecified Unspecified
urgent Severity urgent (vote)
: ---
: ---
Assigned To: Nick Coghlan
MC
: TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-02 02:21 EDT by Karel Srot
Modified: 2012-11-07 02:22 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-11-07 02:22:59 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Karel Srot 2012-07-02 02:21:38 EDT
Please resolve ASAP since the test review becomes very difficult and time consuming when there are AVC denials causing failures of tests executed at different time.

Description of problem:

Systems have wrong system time after the installation, probably because of bad ntpd configuration. But also local watchdog is somehow affecting the system time.

At the beginning there were tests with AVC denials that were not caused by the test itself but occurred in another test running different time. During the debugging of those failures I have observed that system time was changing during the test run. 

Take a look at https://beaker.engineering.redhat.com/jobs/255436 as the example of two systems with wrong time. Now I will speak about recipe T:6658458 but the same applies also for the other task in the job.

System installation has been finished at 12:08:14 +02:00 according to the Beaker time, this is
10:08 UTC. In TESTOUT.log we can see that the system time was 10:07:51 2012 which looks promising but later on we will find out that the timezone is missing here and the system is in EDT timezone. 

Task /examples/rhts_tutorial/ksrot/execute-remote-script is doing two things. It stops ntpd and it sets up system time using "ntpdate clock.redhat.com" command. In the TESTOUT.log we can see that the system time changed from 10:08:37 EDT to 06:08:56 EDT. Since EDT is UTC-4 h we can see that by the end of this task the Beaker time 12:09:04 +02:00 and system time 06:08:56 EDT are in sync (both have 10 hours in UTC).

Also, ntpd is stopped and therefore there should not be any system time changes till the end of the beaker job. /CoreOS/rsyslog/Sanity/omfile-module-test is executed for testing purposes and we can see that the job finished at 06:09:56 EDT and 12:10:03 +02:00 Beaker, thus the time is still OK.

Next task /examples/rhts_tutorial/ksrot/will-be-killed-by-local-watchdog is terminated by local watchdog. We can see in  ./LOCALWATCHDOG/avc that by the end of the task the time was 06:10:29 2012 (be aware that timezone info is missing here).

Right after the test task /CoreOS/rsyslog/Sanity/omfile-module-test is executed again, in TESTOUT.log we can see that system time has changed to 10:15:52 EDT, which is 4 hours and few minutes ahead!!! Since ntpd is stopped, there should not be any time updates.

Beside this 2 recipe job I can provide more recipes with time issues, e.g. 
https://beaker.engineering.redhat.com/recipes/542334
and 
https://beaker.engineering.redhat.com/recipes/542335
In T:542334 the time has changed after the 1st local watchdog intervention 4 hours ahead 
(/CoreOS/openldap/Regression/bz608112-connection-freeze-when-using-TLS)
and after the second intervention 8 hours ahead (/CoreOS/wireshark/fuzz-test).
Comment 1 Bill Peck 2012-07-05 10:32:26 EDT
(In reply to comment #0)
> Please resolve ASAP since the test review becomes very difficult and time
> consuming when there are AVC denials causing failures of tests executed at
> different time.
> 
> Description of problem:
> 
> Systems have wrong system time after the installation, probably because of
> bad ntpd configuration. But also local watchdog is somehow affecting the
> system time.
> 
> At the beginning there were tests with AVC denials that were not caused by
> the test itself but occurred in another test running different time. During
> the debugging of those failures I have observed that system time was
> changing during the test run. 
> 
> Take a look at https://beaker.engineering.redhat.com/jobs/255436 as the
> example of two systems with wrong time. Now I will speak about recipe
> T:6658458 but the same applies also for the other task in the job.
> 
> System installation has been finished at 12:08:14 +02:00 according to the
> Beaker time, this is
> 10:08 UTC. In TESTOUT.log we can see that the system time was 10:07:51 2012
> which looks promising but later on we will find out that the timezone is
> missing here and the system is in EDT timezone. 
> 
> Task /examples/rhts_tutorial/ksrot/execute-remote-script is doing two
> things. It stops ntpd and it sets up system time using "ntpdate
> clock.redhat.com" command. In the TESTOUT.log we can see that the system
> time changed from 10:08:37 EDT to 06:08:56 EDT. Since EDT is UTC-4 h we can
> see that by the end of this task the Beaker time 12:09:04 +02:00 and system
> time 06:08:56 EDT are in sync (both have 10 hours in UTC).
> 
> Also, ntpd is stopped and therefore there should not be any system time
> changes till the end of the beaker job.
> /CoreOS/rsyslog/Sanity/omfile-module-test is executed for testing purposes
> and we can see that the job finished at 06:09:56 EDT and 12:10:03 +02:00
> Beaker, thus the time is still OK.
> 
> Next task /examples/rhts_tutorial/ksrot/will-be-killed-by-local-watchdog is
> terminated by local watchdog. We can see in  ./LOCALWATCHDOG/avc that by the
> end of the task the time was 06:10:29 2012 (be aware that timezone info is
> missing here).

What is the purpose of the test?  I'm not sure if you realize this but after a localwatchdog is triggered we reboot the system to attempt to put it back into a sane state.

This reboot will start ntpd again unless you have disabled the service.

> 
> Right after the test task /CoreOS/rsyslog/Sanity/omfile-module-test is
> executed again, in TESTOUT.log we can see that system time has changed to
> 10:15:52 EDT, which is 4 hours and few minutes ahead!!! Since ntpd is
> stopped, there should not be any time updates.

See above.  ntpd may not actually be stopped anymore.

> 
> Beside this 2 recipe job I can provide more recipes with time issues, e.g. 
> https://beaker.engineering.redhat.com/recipes/542334
> and 
> https://beaker.engineering.redhat.com/recipes/542335
> In T:542334 the time has changed after the 1st local watchdog intervention 4
> hours ahead 
> (/CoreOS/openldap/Regression/bz608112-connection-freeze-when-using-TLS)
> and after the second intervention 8 hours ahead
> (/CoreOS/wireshark/fuzz-test).
Comment 2 Karel Srot 2012-07-09 01:46:34 EDT
I didn't know that local watchdog is rebooting the machine.. so we are getting back to the wrong system time after the installation.
Comment 3 Karel Srot 2012-07-10 10:03:16 EDT
Well, it might be a bug in RHEL7, I did not encounter time issues on RHEL6.
Comment 4 Nick Coghlan 2012-10-17 00:35:56 EDT
Bulk reassignment of issues as Bill has moved to another team.
Comment 5 Min Shin 2012-11-07 02:22:59 EST
This bugs is closed as it is either not in the current Beaker scope or we could not find sufficient data in the bug report for consideration.
Please feel free to reopen the bug with additional information and/or business cases behind it.

Note You need to log in before you can comment on or make changes to this bug.