Bug 1052921 - novnc didn't connect because the clocks of websocket proxy and the host weren't in sync.
Summary: novnc didn't connect because the clocks of websocket proxy and the host weren...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-websocket-proxy
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.5.0
Assignee: Frantisek Kobzik
QA Contact: Ilanit Stein
URL:
Whiteboard: virt
Depends On:
Blocks: rhev3.5beta 1156165
TreeView+ depends on / blocked
 
Reported: 2014-01-14 12:05 UTC by Ilanit Stein
Modified: 2015-02-17 08:28 UTC (History)
8 users (show)

Fixed In Version: ovirt-3.5.0-alpha2
Doc Type: Bug Fix
Doc Text:
The fix adds add 5-second time tolerance to websocket proxy to prevent connection refusal caused by really small time drifts between clocks of the engine and the machine where the proxy is deployed.
Clone Of:
Environment:
Last Closed: 2015-02-17 08:28:54 UTC
oVirt Team: ---
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 27643 0 master MERGED core: add time tolerance to websocket proxy Never

Description Ilanit Stein 2014-01-14 12:05:02 UTC 
Description of problem:

novnc console failed to connect in case websocket proxy configured on host other than engine, due to clocks of engine host and websocket proxy server no synced.

fkobzik: "The machine with engine was 40 seconds ahead of the host. So engine issues a ticket that is not yet valid on the host..." 

Version-Release number of selected component (if applicable):
is30
Comment 1 Michal Skrivanek 2014-01-15 13:07:09 UTC 
IIRC there was an infra request to NTP  sync everything...is it still planned? 
Even just reporting host clocks would be great to see they are out of sync
I don't think we should fix this specific case as there are quite likely other races like this.
Comment 2 Frantisek Kobzik 2014-01-15 15:02:28 UTC 
But the problem is that we tolerate some drift (120 secs by default) but only "to the future". We have 0 tolerance to the past. And even if we have all host in sync using NTP, small drift still can be present which cause the ticket to be invalid.
Comment 3 Michal Skrivanek 2014-01-16 08:36:58 UTC 
yeah, but if you're NTP-synced it should never exceed the RTT you need for getting the ticket, acting on it in the backend(blazing fast!), and then send it
Comment 4 Alon Bar-Lev 2014-03-16 20:41:21 UTC 
We can have 5 seconds tolerance into the past, no need more than that in sane environment.

Clock synchronization is a must. Even in disconnected environment, ntpd can be installed at engine machine to sync the entire environment.
Comment 5 Michal Skrivanek 2014-03-24 10:42:32 UTC 
(In reply to Alon Bar-Lev from comment #4)
+1
there should be an infra feature to alert on nonsychronized hosts
Comment 6 Ilanit Stein 2014-08-12 14:15:50 UTC 
Verified on ovirt-engine 3.5 -rc1.

Set host, on which runs the web socket to 1 minute behind engine, and see that noVNC console fail to connect.

Then sync time back, and see that the noVNC console is working OK.
Comment 7 Michal Skrivanek 2014-09-01 12:49:47 UTC 
Barak, I'm actually still interested in the answer;-)
Are there any plans to manage time keeping between hosts&engine?
Comment 8 Barak 2014-09-02 18:29:23 UTC 
This is actually under discussion,
The original feature was to enable as a part of the engine setup (whether to install and configure ntpd) and than configure all hypervisors to sync with it.

There were various issues with such an implementetion:
- it looks like the ntp config should be per DC, as DC may  be remote that will influence sync
- post installation this can not be changed without redeploying each DC
and ...

Currently we explore the option of integration with foreman(through puppet) to configure the hypervisor's ntp.

The time sync influence mostly the migration ...
Comment 9 Omer Frenkel 2015-02-17 08:28:54 UTC 
RHEV-M 3.5.0 has been released
Note You need to log in before you can comment on or make changes to this bug.