Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1620573

Summary: [downstream clone - 4.2.7] [RFE] Time sync in VM after resuming from PAUSE state
Product: Red Hat Enterprise Virtualization Manager Reporter: RHV bug bot <rhv-bugzilla-bot>
Component: vdsmAssignee: Steven Rosenberg <srosenbe>
Status: CLOSED ERRATA QA Contact: Vitalii Yerys <vyerys>
Severity: high Docs Contact:
Priority: medium    
Version: 4.2.0CC: aperotti, fdelorey, gscott, lsurette, mavital, michal.skrivanek, mkalinin, mtessun, omachace, rmcswain, sgoodman, srevivo, srosenbe, ycui
Target Milestone: ovirt-4.2.7Keywords: FutureFeature, ZStream
Target Release: ---Flags: lsvaty: testing_plan_complete-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: v4.20.40 Doc Type: Enhancement
Doc Text:
Large snapshots can result in long pauses of a VM that can affect the accuracy of the System Time, upon which time stamps and other time related functions depend. Guest Time Synchronization enables synchronization of the VM’s System Time during the creation of snapshots when enabled. When this feature is enabled and the Guest Agent is running, the VDSM process on the Host attempts to synchronize the System Time of the VM with the Host’s System Time when snapshots are completed and the VM is un-paused. To turn on Guest Time Synchronization for snapshots, use the time_sync_snapshot_enable option. For synchronizing the VM’s System Time during abnormal scenarios that may cause the VM to pause, you can enable the time_sync_cont_enable option. By default, these features are disabled for backward compatibility.
Story Points: ---
Clone Of: 1510856 Environment:
Last Closed: 2018-11-05 15:02:07 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1510856    
Bug Blocks:    

Description RHV bug bot 2018-08-23 08:18:29 UTC
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1510856 +++
======================================================================

Description of problem:
Thanks to BZ#1156194, when a VM is suspended and later restored, the time is synced.

The request is to extend the automatic sync of the timing even when the VM restore from a Pause state.

Version-Release number of selected component (if applicable):
RHEV 4.1.x

How reproducible:
Always

Steps to Reproduce:
1. turn on a VM with guest agent enabled
2. create a transient storage issue or a high impact IO operation (like a snapshot with memory for a huge VM) happen.
2. VM went to pause
3. solve the issue or wait for the IO operation to happen

Actual results:
the time is mis-aligned.

Expected results:
RHV do force the time sync like when the VM go into suspend state.

Additional info:
This RFE is critical for those time sensitive workloads running on RHV (transational apps and so on).

(Originally by Andrea Perotti)

Comment 3 RHV bug bot 2018-08-23 08:18:45 UTC
possible, but not sure it needs to be configurable or not

(Originally by michal.skrivanek)

Comment 4 RHV bug bot 2018-08-23 08:18:51 UTC
can you add more details about the actual scenario? Even when VM pauses due to drive extension it shouldn't really take too much time, not more than few seconds which are better handled by NTP inside the guest rather than abrupt clock changes done externally.

(Originally by michal.skrivanek)

Comment 5 RHV bug bot 2018-08-23 08:18:56 UTC
The scenario we are talking about is very time sensitive app, an in-memory app,  like a jboss datagrid installation, running on VM with huge amount of RAM.

Dealing with a pause of that VM *can* be worked out with ntp, but require a constant aggressive configuration of the tool, while having the same behaviour for paused like for suspended VMs can be more practical for some users.

Eventually this can be make configurable, like, setting after how many seconds of pause state RHV should enforce the clock changes.

(Originally by Andrea Perotti)

Comment 6 RHV bug bot 2018-08-23 08:19:02 UTC
(In reply to Andrea Perotti from comment #4)
> The scenario we are talking about is very time sensitive app, an in-memory
> app,  like a jboss datagrid installation, running on VM with huge amount of
> RAM.
> 
> Dealing with a pause of that VM *can* be worked out with ntp, but require a
> constant aggressive configuration of the tool, while having the same
> behaviour for paused like for suspended VMs can be more practical for some
> users.

if it is a time sensitive app, wouldn't it be better to avoid ENOSPC paused states? Either bigger allocation chunks, lower watermark so it starts extending the drive sooner, bigger initial size of the thin provisioned disk, etc.

> Eventually this can be make configurable, like, setting after how many
> seconds of pause state RHV should enforce the clock changes.

Creating a config option to do a time sync after resume from pause is feasible. I would still leave it off by default though. Implementing a configurable interval when it should be set is more complicated and will delay this RFE, but if that's required it's doable too.

Still, before starting on this I believe we should check if we are really solving the right thing, making VMs not to pause in the first place might make more sense.

(Originally by michal.skrivanek)

Comment 7 RHV bug bot 2018-08-23 08:19:06 UTC
(In reply to Michal Skrivanek from comment #5)
> if it is a time sensitive app, wouldn't it be better to avoid ENOSPC paused
> states? Either bigger allocation chunks, lower watermark so it starts
> extending the drive sooner, bigger initial size of the thin provisioned
> disk, etc.

Customer is triggering this event when doing a full snapshot of VM included with memory, but also transient connectivity storage issues can lead to Pause.

> Creating a config option to do a time sync after resume from pause is
> feasible. I would still leave it off by default though. 

I think just having it would be good enough for my customer, and having it now is more important than having it perfectly configurable.

(Originally by Andrea Perotti)

Comment 9 RHV bug bot 2018-08-23 08:19:16 UTC
Overall I believe that we should address the reason for the pausing in the first place. If that happens for snapshots, we should probably also check if we can get rid of the pausing that does happen.

(Originally by Martin Tessun)

Comment 11 RHV bug bot 2018-08-23 08:19:26 UTC
perhaps after_vm_cont hook can be used? Hopefully it's not suffering from the same problem as the after_vm_pause hook in bug 1543103


Other than that, the solution could look similar to Openstack's https://review.openstack.org/#/c/316116/

(Originally by michal.skrivanek)

Comment 12 RHV bug bot 2018-08-23 08:19:31 UTC
I'll add another large application to this one too.  In this case, storage goes offline, the VM pauses, storage comes back, the VM resumes, but now its clock is way off, and that sets a whole bad cascade of events.  

Nothing we can do about the storage going offline, and pausing the VM is the correct action when that happens. If we can inject the correct time into that VM when it resumes, we can make lots of people happy.

- Greg

(Originally by Greg Scott)

Comment 26 Vitalii Yerys 2018-09-07 15:36:24 UTC
Verified upstream:

ovirt-engine-4.2.6.5-0.0.master.20180831090131.git1d64d4c.el7.noarch
vdsm-4.20.39-6.git00d5340.el7.x86_64

Comment 28 Raz Tamir 2018-09-19 12:30:38 UTC
QE verification bot: the bug was verified upstream

Comment 30 errata-xmlrpc 2018-11-05 15:02:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:3478