Bug 1248860 - Guest time is affected by slew adjustments of host system time even start the guest with "clock=vm"
Guest time is affected by slew adjustments of host system time even start the...
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev (Show other bugs)
7.2
All Linux
low Severity unspecified
: rc
: 7.5
Assigned To: Marcelo Tosatti
Sitong Liu
:
Depends On:
Blocks: 1444027
  Show dependency treegraph
 
Reported: 2015-07-30 23:31 EDT by Gu Nini
Modified: 2018-02-16 13:44 EST (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-01-16 15:23:03 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
clock_gettime.c (290 bytes, text/plain)
2017-03-23 10:40 EDT, Marcelo Tosatti
no flags Details
clock_realtime.c (312 bytes, text/plain)
2018-01-16 14:41 EST, Marcelo Tosatti
no flags Details
clock_gettime.c (314 bytes, text/plain)
2018-01-16 14:42 EST, Marcelo Tosatti
no flags Details
gettimes.sh (1.13 KB, text/plain)
2018-01-16 14:43 EST, Marcelo Tosatti
no flags Details

  None (edit)
Description Gu Nini 2015-07-30 23:31:37 EDT
Description of problem:

Even start the guest with "-rtc base=utc,clock=vm" or "-rtc base=2006-06-06,clock=vm", it's found the guest hwclock is still affected by slew adjustments of the host system time although it's unaffected by sudden jumps in the host time, i.e. the guest hwclock would still tick with the same frenqency as the host system time.

In another way to say the problem, that's: If I changed the host system time one hour behind, it would tick quicker to chase up the time server(maybe), then the guest hwclock would ticks quicker too, I can check this from the time difference between guest system time and hwclock; if I changed the host system time one hour ahead, it would tick slower to wait for the time server, then the guest hwclock would ticks slower too.

Version-Release number of selected component (if applicable):
host kernel: 3.10.0-300.el7.ppc64le
Guest kernel: 3.10.0-295.el7.ppc64/3.10.0-295.el7.ppc64le
Qemu-kvm-rhev: qemu-kvm-rhev-2.3.0-13.el7.ppc64le


How reproducible:
100%

Steps to Reproduce:
1. Start a guest with guest clock "-rtc base=utc,clock=vm" or "-rtc base=2006-06-06,clock=vm"
2. After the guest boots up, check its system time and hwclock
3. Change the host system time with cmd 'date -s XX:XX:XX' one hour ahead
4. Inside the guest, track its system time and hwclock

Actual results:
In step2, the guest hwclock and system time are the same;
In step4, the guest hwclock is still affected by slew adjustments of the host system time, finally there will be one hour lag between the guest hwclock and system time

Expected results:
In step4, there is no time lag between the guest hwclock and system time

Additional info:
The bug is originally found in the verification of bz 1170132, please refer to https://bugzilla.redhat.com/show_bug.cgi?id=1170132#c31 and https://bugzilla.redhat.com/show_bug.cgi?id=1170132#c32 there for details
Comment 2 David Gibson 2015-07-31 00:26:49 EDT
As noted in the comments for bug 1170132, fixing this requires some substantial changes to qemu's timekeeping core.  Because it only appears in an odd edge case, it's probably not worth fixing, but it's useful to have this tracker.
Comment 3 Gu Nini 2015-08-04 01:20:31 EDT
A similar problem exists on x86_64 system, i.e. the actual result in step4 is: Both the guest hwclock and system time are affected by slew adjustments of host system time, the guest hwclock and system time are always the same.

The detailed sw versions are as follows:

host kernel: 3.10.0-300.el7.x86_64
Guest kernel: 3.10.0-295.el7.x86_64
Qemu-kvm-rhev: qemu-kvm-rhev-2.3.0-13.el7.x86_64

So change the bug summary and hardware field to be both ppc64le and x86_64 suitable.
Comment 5 David Gibson 2016-09-07 01:15:09 EDT
This is a subtle edge case and the right call here isn't totally obvious.  Due to the late stage in 7.3 release, deferring to 7.4.
Comment 6 David Gibson 2017-03-20 20:49:45 EDT
Since this is not Power specific, moving back to default owner.  I suspect this is actually intended behaviour, so I'm hoping this can find someone who can confirm or refute that.
Comment 7 Laurent Vivier 2017-03-21 07:27:44 EDT
Just a note about POWER case:
- hwclock time is provided by the hypervisor (RTAS call "get-time-of-day")
- date time is provided by the host hardware register "Time Base Register".

So "hwclock" gives you the date&time of the host, and "date" is computed according the number of ticks since the last boot.

According to that, it's "normal" only hwclock is affected by a time change on the host.
Comment 8 Marcelo Tosatti 2017-03-23 10:39:09 EDT
(In reply to Gu Nini from comment #0)
> Description of problem:
> 
> Even start the guest with "-rtc base=utc,clock=vm" or "-rtc
> base=2006-06-06,clock=vm", it's found the guest hwclock is still affected by
> slew adjustments of the host system time although it's unaffected by sudden
> jumps in the host time, i.e. the guest hwclock would still tick with the
> same frenqency as the host system time.

Yes, this is the documented behaviour, from the manpage:

       -rtc [base=utc|localtime|date][,clock=host|vm][,driftfix=none|slew]
           Specify base as "utc" or "localtime" to let the RTC start at the current UTC or local time,
           respectively. "localtime" is required for correct date in MS-DOS or Windows. To start at a
           specific point in time, provide date in the format "2006-06-17T16:01:21" or "2006-06-17". The
           default base is UTC.

           By default the RTC is driven by the host system time. This allows to use the RTC as accurate
           reference clock inside the guest, specifically if the host time is smoothly following an
           accurate external reference clock, e.g. via NTP.  If you want to isolate the guest time from
           the host, you can set clock to "rt" instead.  To even prevent it from progressing during
           suspension, you can set it to "vm".


> In another way to say the problem, that's: If I changed the host system time
> one hour behind, it would tick quicker to chase up the time server(maybe),

I don't think this will happen: with one hour behind NTP should step 
the host clock, which is not going to affect the RTC clock (which is CLOCK_MONOTONIC essentially).

> then the guest hwclock would ticks quicker too, I can check this from the
> time difference between guest system time and hwclock; if I changed the host
> system time one hour ahead, it would tick slower to wait for the time
> server, then the guest hwclock would ticks slower too.

Do you have a testcase to reproduce this? What you are saying is that 
changes to correct the host clock's CLOCK_REALTIME will be propagated to 
the guests RTC therefore turning the guests RTC incorrect.

This should not be true because the guests RTC is emulated with CLOCK_MONOTONIC, and only frequency difference between the local system clock and UTC are propagated to CLOCK_MONOTONIC.
 
> Version-Release number of selected component (if applicable):
> host kernel: 3.10.0-300.el7.ppc64le
> Guest kernel: 3.10.0-295.el7.ppc64/3.10.0-295.el7.ppc64le
> Qemu-kvm-rhev: qemu-kvm-rhev-2.3.0-13.el7.ppc64le
> 
> 
> How reproducible:
> 100%
> 
> Steps to Reproduce:
> 1. Start a guest with guest clock "-rtc base=utc,clock=vm" or "-rtc
> base=2006-06-06,clock=vm"
> 2. After the guest boots up, check its system time and hwclock
> 3. Change the host system time with cmd 'date -s XX:XX:XX' one hour ahead
> 4. Inside the guest, track its system time and hwclock
> 
> Actual results:
> In step2, the guest hwclock and system time are the same;
> In step4, the guest hwclock is still affected by slew adjustments of the
> host system time, finally there will be one hour lag between the guest
> hwclock and system time

This is then a generic problem related to assumptions about behaviour of CLOCK_MONOTONIC and not a QEMU problem: the clock_gettime man page says:

       CLOCK_MONOTONIC
              Clock  that  cannot be set and represents monotonic time
              since some unspecified starting point.   This  clock  is
              not  affected  by discontinuous jumps in the system time
              (e.g., if the system administrator manually changes  the
              clock),  but  is affected by the incremental adjustments
              performed by adjtime(3) and NTP.

And the assumption (at least in my mind) was: corrections to CLOCK_MONOTONIC 
are to correct the local system clock frequency difference to UTC frequency (that is how the clock drifts relative to UTC frequency). 

Now if Chrony uses sys_adjtimex's ADJ_FREQUENCY to correct for a 1 hour clock difference, and that is propagated to CLOCK_MONOTONIC, then things obviously fail as you described.

Now can you check, in the host, via an application that does clock_gettime(CLOCK_MONOTONIC) (attached), the following information. 

Collect every 10 minutes 3 values and save them in a table:

host date | host clock_gettime | hand clock

where host date is the output of "date", 
host clock_gettime is the output of "clock_gettime" (see clock_gettime.c), 
hand clock is any physical clock you have around
(or another computers clock).

So we can check whether Chrony corrections are affecting 
CLOCK_MONOTONIC of the host.


> 
> Expected results:
> In step4, there is no time lag between the guest hwclock and system time
> 
> Additional info:
> The bug is originally found in the verification of bz 1170132, please refer
> to https://bugzilla.redhat.com/show_bug.cgi?id=1170132#c31 and
> https://bugzilla.redhat.com/show_bug.cgi?id=1170132#c32 there for details
Comment 9 Marcelo Tosatti 2017-03-23 10:40 EDT
Created attachment 1265793 [details]
clock_gettime.c
Comment 13 FuXiangChun 2017-06-12 07:00:08 EDT
According to comment8. I did related test on host only. It is not relevant to guest.  If my test step are wrong. please correct me. 

On host.
1.#ntpdate clock.redhat.com
12 Jun 06:25:22 ntpdate[6322]: adjust time server 10.16.255.1 offset -0.000165 sec

2.run system date from a loop script.

# cat system-date.sh 
#/usr/bin/bash
while true
do
 date
 sleep 600
done

3.host clock_gettime

# cat clock_gettime.sh 
#/usr/bin/bash
while true
do
 ./clock_gettime
 sleep 600
done

4.physical clock
# cat physical-clock.sh 
#/usr/bin/bash
while true
do
 hwclock
 sleep 600
done

5.Change the host system time with cmd 'date -s XX:XX:XX' one hour ahead

# date
Mon Jun 12 06:27:33 EDT 2017

# date -s 07:27:33
Mon Jun 12 07:27:33 EDT 2017

# date
Mon Jun 12 07:27:35 EDT 2017

result:
---------------------------------------------------------
         system-date      clock_gettime          physical-clock
---------------------------------------------------------
0/mins   06:25:34     sec:25533 nsec:592796263   06:25:27
---------------------------------------------------------
10/mins  07:35:20     sec:26133 nsec:593658314   06:35:48
---------------------------------------------------------
20/mins  07:45:20     sec:26733 nsec:594777939   06:46:43
---------------------------------------------------------
30/mins  07:55:20     sec:27333 nsec:595872341   06:57:38
---------------------------------------------------------
Comment 18 Marcelo Tosatti 2018-01-16 14:41 EST
Created attachment 1382134 [details]
clock_realtime.c
Comment 19 Marcelo Tosatti 2018-01-16 14:42 EST
Created attachment 1382135 [details]
clock_gettime.c
Comment 20 Marcelo Tosatti 2018-01-16 14:43 EST
Created attachment 1382137 [details]
gettimes.sh
Comment 21 Marcelo Tosatti 2018-01-16 15:23:03 EST
(In reply to Laurent Vivier from comment #7)
> Just a note about POWER case:
> - hwclock time is provided by the hypervisor (RTAS call "get-time-of-day")
> - date time is provided by the host hardware register "Time Base Register".
> 
> So "hwclock" gives you the date&time of the host, and "date" is computed
> according the number of ticks since the last boot.
> 
> According to that, it's "normal" only hwclock is affected by a time change
> on the host.

Actually this means that this is expected behaviour on PPC, closing as NOTABUG.
Comment 22 Sitong Liu 2018-01-18 03:00:57 EST
Hi Marcelo,

According to #comment13, could we confirm that it is not a bug in x86_64?

Best regards,
Sitong
Comment 23 Marcelo Tosatti 2018-02-16 13:44:53 EST
(In reply to FuXiangChun from comment #13)
> According to comment8. I did related test on host only. It is not relevant
> to guest.  If my test step are wrong. please correct me. 
> 
> On host.
> 1.#ntpdate clock.redhat.com
> 12 Jun 06:25:22 ntpdate[6322]: adjust time server 10.16.255.1 offset
> -0.000165 sec
> 
> 2.run system date from a loop script.
> 
> # cat system-date.sh 
> #/usr/bin/bash
> while true
> do
>  date
>  sleep 600
> done
> 
> 3.host clock_gettime
> 
> # cat clock_gettime.sh 
> #/usr/bin/bash
> while true
> do
>  ./clock_gettime
>  sleep 600
> done
> 
> 4.physical clock
> # cat physical-clock.sh 
> #/usr/bin/bash
> while true
> do
>  hwclock
>  sleep 600
> done
> 
> 5.Change the host system time with cmd 'date -s XX:XX:XX' one hour ahead
> 
> # date
> Mon Jun 12 06:27:33 EDT 2017
> 
> # date -s 07:27:33
> Mon Jun 12 07:27:33 EDT 2017
> 
> # date
> Mon Jun 12 07:27:35 EDT 2017
> 
> result:
> ---------------------------------------------------------
>          system-date      clock_gettime          physical-clock
> ---------------------------------------------------------
> 0/mins   06:25:34     sec:25533 nsec:592796263   06:25:27
> ---------------------------------------------------------
> 10/mins  07:35:20     sec:26133 nsec:593658314   06:35:48
> ---------------------------------------------------------
> 20/mins  07:45:20     sec:26733 nsec:594777939   06:46:43
> ---------------------------------------------------------
> 30/mins  07:55:20     sec:27333 nsec:595872341   06:57:38
> ---------------------------------------------------------

So on x86 CLOCK_MONOTONIC ticks as expected.

Note You need to log in before you can comment on or make changes to this bug.