Bug 1139766

Summary:	need a non-event way to determine qemu's current offset from utc
Product:	Red Hat Enterprise Linux 7	Reporter:	Laine Stump <laine>
Component:	libvirt	Assignee:	Michal Privoznik <mprivozn>
Status:	CLOSED ERRATA	QA Contact:	Virtualization Bugs <virt-bugs>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	7.0	CC:	dyuan, fjin, mprivozn, mzhan, rbalakri
Target Milestone:	rc	Keywords:	Upstream
Target Release:	---
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	libvirt-1.3.5-1.el7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	1110429	Environment:
Last Closed:	2016-11-03 18:10:36 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1110429
Bug Blocks:

Description Laine Stump 2014-09-09 15:14:41 UTC

+++ This bug was initially created as a clone of Bug #1110429 +++

qemu now has a way to query the current RTC clock offset; libvirt needs to use it on all running domains each time libvirtd is restarted and reconnects to all of its guests.

See Bug 964177 for the discussion that prompted the request for qemu to implement this feature for libvirt:

--- Additional comment from Eric Blake on 2014-05-22 13:15:23 MDT ---

I think qemu has a bug.  Laine's proposed patches would have libvirt track all RTC change events from qemu, to keep a running sum.  But if we miss an event, such as due to a libvirtd restart, we've lost an accurate state.  I think the only sane thing is for qemu to have a QMP interface that can be polled to ask what qemu's current offset is.  Any time we receive the RTC change event (which tells us only how much the current offset was adjusted), we then followup with the query to learn the new offset.  We can't change the RTC change event semantics, but we CAN make qemu smarter about giving us the right information in the format we need, and in a way that works even with lost events.

--- Additional comment from Laine Stump on 2014-05-23 06:36:31 BRT ---

Note that the problem Eric is describing exists during the window between when the guest changes its RTC while libvirtd is not connected, and the next time that the guest changes its RTC - at that time libvirtd will again have the correct offset. Of course if the domain is migrated or saved/restored while in this state, then the proper offset is lost forever, so it certainly is a bad problem that needs a solution.

As a matter of fact, once the command he suggests is implemented, any RTC_CHANGE event, regardless of clock settings, could simply ignore the sent offset and poll qemu for an up-to-date "offset from UTC" (which is what libvirt wants to send in its own RTC_CHANGE event).

--- Additional comment from Eric Blake on 2014-05-23 14:42:06 BRT ---

(In reply to Laine Stump from comment #2)
> Note that the problem Eric is describing exists during the window between
> when the guest changes its RTC while libvirtd is not connected, and the next
> time that the guest changes its RTC - at that time libvirtd will again have
> the correct offset. Of course if the domain is migrated or saved/restored
> while in this state, then the proper offset is lost forever, so it certainly
> is a bad problem that needs a solution.

If I'm understanding how things work, the RTC offset is part of the migration stream. For save/restore and live migration, we want the destination to start with the same command line as the source (and NOT fold the RTC offset into that command line) - that's because the destination qemu will apply the same RTC offset that it read from the migration stream as what the source qemu had left things in, with both qemu's doing it relative to the same reference point.  The ONLY time where libvirt needs to care about the RTC offset is when the guest stops - at THAT instant, libvirt needs to know the current RTC offset, and rewrite the persistent domain XML, so that the next time a new qemu process boots the same guest, the new qemu uses a NEW command line offset based on the old qemu's command-line offset + the old qemu's RTC offset when the guest shut down (that is, every time a new qemu process boots fresh, rather than doing an incoming migration, the RTC offset resets to 0, so to avoid losing that RTC offset from before, the command line offset has to be adjusted).

So yes, if libvirt doesn't know the current RTC offset when a guest shuts down (because it missed an event and the guest shuts down prior to causing another RTC change event), then libvirt will write the wrong time into the domain xml, and the next boot of the guest will have the wrong time.

It also raises the point that for transient guests, the management app using libvirt needs a way to query what the current RTC offset is.  Just as libvirt cannot rely on qemu events always being delivered, VDSM cannot rely on libvirt events always being delivered.  So if a transient guest shuts down, VDSM still needs a way to query what the RTC offset of that guest was at the time it shut down, so that VDSM can then create a new transient guest with the correct offset for the next time that guest is to be booted.  But that raises the question - if a transient guest shutting down makes the guest disappear, then what recourse does VDSM have to get that information?

--- Additional comment from Paolo Bonzini on 2014-07-08 04:35:26 EDT ---

Now upstream (in qemu):

commit 654a36d857ff949e0d1989904b76f53fded9dc83
Author: Marcelo Tosatti <mtosatti>
Date:   Wed Jun 4 14:52:03 2014 -0300

    mc146818rtc: add "rtc-time" link to "/machine/rtc"
    
    Add a link to rtc under /machine providing a stable
    location for management apps to query the value of the
    time.  The link should be added by any object that sends
    RTC_TIME_CHANGE events.
    
    {"execute":"qom-get","arguments":{"path":"/machine","property":"rtc-time"} }
    
    Suggested by Paolo Bonzini and Andreas Faerber.
    
    Signed-off-by: Marcelo Tosatti <mtosatti>
    Signed-off-by: Paolo Bonzini <pbonzini>

Comment 5 Michal Privoznik 2016-04-29 16:47:13 UTC

Patches proposed upstream:

https://www.redhat.com/archives/libvir-list/2016-April/msg02001.html

Comment 6 Michal Privoznik 2016-05-03 09:49:43 UTC

I've just pushed the patches upstream:

commit 6ee78d334a9e8e841880d6931892a8ebaa73faeb
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Apr 29 18:01:39 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Tue May 3 11:44:13 2016 +0200

    qemu: Refresh RTC adjustment on qemuProcessReconnect
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1139766
    
    Thing is, for some reasons you can have your domain's RTC to be
    in something different than UTC. More weirdly, it's not only time
    zone what you can shift it of, but an arbitrary value. So, if
    domain is configured that way, libvirt will correctly put it onto
    qemu cmd line and moreover track it as this offset changes during
    domain's life time (e.g. because guest OS decides the best thing
    to do is set new time to RTC). Anyway, they way in which this
    tracking is implemented is events. But we've got a problem if
    change in guest's RTC occurs and the daemon is not running. The
    event is lost and we end up reporting invalid value in domain
    XML. Therefore, when the daemon is starting up again and it is
    reconnecting to all running domains, re-fetch their RTC so the
    correct offset value can be computed.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit b1e2f2d84d149e141f86f286a243536243862b55
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Apr 29 16:01:47 2016 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Tue May 3 11:44:13 2016 +0200

    qemu: Introduce qemuMonitorGetRTCTime
    
    Signed-off-by: Michal Privoznik <mprivozn>



v1.3.4-81-g6ee78d3

Comment 8 Fangge Jin 2016-06-29 09:00:11 UTC

Reproduce this BZ on build libvirt-1.3.4-1.el7.x86_64, qemu-kvm-rhev-2.6.0-9.el7.x86_64

Scenario 1: adjust guest hwclock when libvirtd.service is inactive, then restart libvirtd.service and check clock adjustment in guest live xml.

1. Add the following xml elements into domain xml:
# virsh edit rhel7.2-min
<clock offset='variable' adjustment='600' basis='utc'>
<timer name='rtc' tickpolicy='catchup' track='guest'>
<catchup threshold='123' slew='120' limit='10000'/>
</timer>
<timer name='pit' tickpolicy='delay'/>
</clock>

2. # virsh start rhel7.2-min

3. In guest, adjust hwclock forward by 10 mins via a script:
# hwclock;date
Wed 29 Jun 2016 04:23:42 PM CST -0.449830 seconds
Wed Jun 29 16:23:42 CST 2016

# ./addsecs 600

# hwclock;date
Wed 29 Jun 2016 04:35:01 PM CST -10.011560 seconds
Wed Jun 29 16:25:01 CST 2016

4. In host:
# virsh dumpxml rhel7|grep utc
<clock offset='variable' adjustment='1199' basis='utc'>

5. # systemctl stop libvirtd

6. In guest, adjust hwclock forward by 10 mins again:
# ./addsecs 600

# hwclock;date
Wed 29 Jun 2016 04:50:42 PM CST -10.011302 seconds
Wed Jun 29 16:30:42 CST 2016

7. # systemctl start libvirtd

8. In host:
# virsh dumpxml rhel7|grep utc
<clock offset='variable' adjustment='1199' basis='utc'> ====> the adjustment should be 1800 approximately.

9. In guest, adjust hwclock forward by 10 mins again:
# ./addsecs 600

10. In host:
# virsh dumpxml rhel7|grep utc
<clock offset='variable' adjustment='2399' basis='utc'> ====> the adjustment now is correct.



Scenario 2: check time inside guest after migration
11. # systemctl stop libvirtd

12. In guest, adjust hwclock forward by 10 mins again:
# ./addsecs 600

13. # systemctl start libvirtd

14. # virsh dumpxml rhel7|grep utc
<clock offset='variable' adjustment='2399' basis='utc'>

15.Do migration:
# virsh migrate rhel7 qemu+ssh://hp-dl385g7-06.lab.eng.pek2.redhat.com/system --live --verbose
Migration: [100 %]

16. In guest:
# hwclock;date
Wed 29 Jun 2016 05:30:10 PM CST -0.694085 seconds
Wed Jun 29 16:50:11 CST 2016

The time in guest is correct after migration.

17.On target host:
# virsh dumpxml rhel7|grep utc
<clock offset='variable' adjustment='2398' basis='utc'>

Comment 9 Fangge Jin 2016-06-30 08:02:51 UTC

Verify this BZ on buildlibvirt-1.3.5-1.el7.x86_64

Steps:
1. Add the following xml elements into domain xml:
# virsh edit rhel7.2-min
<clock offset='variable' adjustment='600' basis='utc'>
<timer name='rtc' tickpolicy='catchup' track='guest'>
<catchup threshold='123' slew='120' limit='10000'/>
</timer>
<timer name='pit' tickpolicy='delay'/>
</clock>

2. # virsh start rhel7.2-min

3. In guest, adjust hwclock forward by 10 mins via a script:
# hwclock;date
Mon 27 Jun 2016 04:22:06 PM CST -0.636120 seconds
Mon Jun 27 16:22:05 CST 2016

# ./addsecs 600

# hwclock;date
Mon 27 Jun 2016 04:32:21 PM CST -0.727867 seconds
Mon Jun 27 16:22:20 CST 2016

4. Stop libvirtd service
# systemctl stop libvirtd

5. In guest, adjust hwclock forward by 10 mins again:
# ./addsecs 600

# hwclock;date
Mon 27 Jun 2016 04:53:31 PM CST -0.937840 seconds
Mon Jun 27 16:33:30 CST 2016

6. Start libvirtd service:
# systemctl start libvirtd

7. Check the time offset in domain dumpxml:
# virsh dumpxml rhel7.2-min
<clock offset='variable' adjustment='1798' basis='utc'>
<timer name='rtc' tickpolicy='catchup' track='guest'>
<catchup threshold='123' slew='120' limit='10000'/>
</timer>
<timer name='pit' tickpolicy='delay'/>
</clock>

8. Do managedsave&start
# virsh managedsave rhel7.2-min
# virsh start rhel7.2-min

9.# virsh dumpxml rhel7.2-min
<clock offset='variable' adjustment='1798' basis='utc'>
<timer name='rtc' tickpolicy='catchup' track='guest'>
<catchup threshold='123' slew='120' limit='10000'/>
</timer>
<timer name='pit' tickpolicy='delay'/>
</clock>

10. In guest:
# hwclock;date
Mon 27 Jun 2016 04:56:54 PM CST -0.452379 seconds
Mon Jun 27 16:36:53 CST 2016

11. Do migration:
# virsh migrate rhel7.2-min qemu+ssh://hp-dl385g7-05.lab.eng.pek2.redhat.com/system --live --verbose
Migration: [100 %]

12. In guest:
# hwclock;date
Mon 27 Jun 2016 04:57:28 PM CST -0.661982 seconds
Mon Jun 27 16:37:27 CST 2016

13. Destroy guest:
# virsh destroy rhel7.2-min

# virsh dumpxml rhel7.2-min
<clock offset='variable' adjustment='600' basis='utc'> ==> The adjustment is 600 instead of 1798

Comment 10 Fangge Jin 2016-06-30 08:06:37 UTC

One question:
I'm not sure if libvirt should rewrite the persistent domain XML with the latest RTC adjustment value when guest shuts down, in my test, this is not done(see comment 9 step13)

Comment 11 Michal Privoznik 2016-07-01 09:40:34 UTC

(In reply to JinFangge from comment #10)
> One question:
> I'm not sure if libvirt should rewrite the persistent domain XML with the
> latest RTC adjustment value when guest shuts down, in my test, this is not
> done(see comment 9 step13)

No, I don't think we should change that. If users want to keep the adjustment as you described, they can edit the domain XML. Otherwise, changing persistent XML without letting users know is a big no-no.

Comment 13 errata-xmlrpc 2016-11-03 18:10:36 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html