Bug 1800901 - Make chrony use NTP settings from DHCP
Summary: Make chrony use NTP settings from DHCP
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RHCOS
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.6.0
Assignee: Sohan Kunkerkar
QA Contact: Michael Nguyen
URL:
Whiteboard:
: 1809063 (view as bug list)
Depends On:
Blocks: 1186913 1759923 1771572
TreeView+ depends on / blocked
 
Reported: 2020-02-08 23:23 UTC by William Caban
Modified: 2024-06-13 22:27 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 15:55:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 15:55:31 UTC

Internal Links: 1780165 2047415

Description William Caban 2020-02-08 23:23:43 UTC
Description of problem:

Affected platforms:
- BM deployments
- OSP deployments (when Instances distributed to different hardware with 1+ minute time difference)
- libvirt deployments (when VMs distributed to different hardware with 1+ minute time difference)

When nodes have time/date mismatch between the bootstrap and masters or among master nodes, the deployment fails because the nodes will either register the certificate as invalid (e.g. because of future date) or etcd cannot achieve quorum because of the time difference between master nodes.

In this situation injecting the chrony.conf as MachineConfig does not work because the corresponding CRDs are never created as the error is before etcd and KK8s API is ready.

The default chrony.conf configuration assumes RHCOS nodes can reach RHT (2.rhel.pool.ntp.org) and uses a stepped clock synchronization (makestep 1.0 3). 

1) Either in environments using Proxy or when doing air-gapped deployments, it is no possible to reach RHT NTP pool.
2) The current stepped synchronization does not work fast enough to allow a successful deployment of the platform.

Request:
1) The installer should provide a way to specify the organization NTP server and force an immediate time synchronization (makestep 1 -1)
2) When the DHCP NTP option is provided, the system should do an immediate time synchronization (makestep 1 -1)

Workaround:
The current workaround is to modify the initial Ignition files and inject the chrony.conf configuration with the NTP server to use and modify the stepped sync. This is not an IPI friendly workaround.

Comment 1 Abhinav Dahiya 2020-02-11 00:18:46 UTC
> Request:
> 1) The installer should provide a way to specify the organization NTP server and force an immediate time synchronization (makestep 1 -1)

that should be an RFE on JIRA https://issues.redhat.com/projects/RFE/issues/RFE-507?filter=allopenissues

> When the DHCP NTP option is provided, the system should do an immediate time synchronization (makestep 1 -1)

I think the RHCOS needs to maybe change the default. SO moving over to the RHCOS team to see what's best.

Comment 2 Abhinav Dahiya 2020-02-11 00:20:15 UTC
> The current workaround is to modify the initial Ignition files and inject the chrony.conf configuration with the NTP server to use and modify the stepped sync. This is not an IPI friendly workaround.

Can you explain why that's not IPI friendly? it should be easy enough using multiple invocations for advanced users..

Comment 3 William Caban 2020-02-11 12:56:29 UTC
Since the time sync needs to happen before the MachineConfig CRDs are in place, currently, the only way to achieve this is by modifying the actual bootstrap, master & worker Ignition files and add a .storage.files.path and add a section to inject the "/etc/chrony.conf". All that is manually editing a JSON file and, even when possible, the Ignition files are not intended for direct editing. The situation is more complex when the customer has hundreds of clusters to deploy.


Having an array variable in install-config.yaml like "ntpServer" to address this allows the user to maintain a single file in an organization git repo to redeploy those clusters at any time without having to edit the Ignition files for every attempt.

Comment 4 Micah Abbott 2020-02-12 18:16:36 UTC
(In reply to Abhinav Dahiya from comment #1)

> that should be an RFE on JIRA
> https://issues.redhat.com/projects/RFE/issues/RFE-507?filter=allopenissues

https://issues.redhat.com/browse/RFE-582

> I think the RHCOS needs to maybe change the default. SO moving over to the
> RHCOS team to see what's best.

RHCOS generally does not like to change default values that come from RHEL, as those default values have typically been selected for good reason.

However, we can investigate the possibility of changing the default settings with regards to the DHCP NTP option and synchronization settings (makestep)

Comment 5 Jonathan Lebon 2020-02-13 14:27:00 UTC
Re.

> Since the time sync needs to happen before the MachineConfig CRDs are in place, currently, the only way to achieve this is by modifying the actual bootstrap, master & worker Ignition files and add a .storage.files.path and add a section to inject the "/etc/chrony.conf". All that is manually editing a JSON file and, even when possible, the Ignition files are not intended for direct editing. The situation is more complex when the customer has hundreds of clusters to deploy.

Note you don't have to edit the Ignition files directly. There is a workflow for install-time MCs: https://github.com/openshift/installer/blob/master/docs/user/customization.md#install-time-customization-for-machine-configuration. Does that help?


So there are two parts here:
1. Provide a more streamlined path for install-time configuration of NTP servers. Micah filed a Jira RFE for this already, though it'll probably depend on the resolution of the upstream discussions about this (https://github.com/openshift/machine-config-operator/issues/629).
2. Make sure chrony picks up NTP settings from DHCP if available (and not overridden).

Let's make this ticket strictly about that second one.

Comment 6 William Caban 2020-02-13 15:16:58 UTC
Jonathan,

Please note the customization for machine configuration you are pointing to DOES use MachineConfigs and those are interpreted AFTER the CRDs created which require etcd and k8s API fully operational. This is precisely what does not happens because of the lack of time synchronization among masters.

WRT the two parts: Sounds good about #2 but it should also force the sync (the "makestep" settings) so it properly brings the node in time sync so the etcd deployment can complete successfully.

Comment 7 Jonathan Lebon 2020-02-13 16:01:50 UTC
> Please note the customization for machine configuration you are pointing to DOES use MachineConfigs and those are interpreted AFTER the CRDs created which require etcd and k8s API fully operational. This is precisely what does not happens because of the lack of time synchronization among masters.

Hmm my understanding is that those MCs are part of the initial Ignition config, which means that Ignition on firstboot should apply those modifications. See e.g. this part of the text:

> However, there may be some cases where these workloads need to be executed prior to the node joining the Kubernetes cluster. For example, a compliance mandate like "the user must run auditing tools as soon as the operating system comes up" might require a custom systemd unit for an auditing container in the Ignition config for some or all nodes.

If that's not happening then I think there is a bug in the installer.

Ahh but yes, they're not applied to the boostrap machine's Ignition config itself, which Colin pointed out in https://github.com/openshift/machine-config-operator/issues/629#issuecomment-557645447. So yeah, leveraging this as part of #1 would require fixing that too.

> WRT the two parts: Sounds good about #2 but it should also force the sync (the "makestep" settings) so it properly brings the node in time sync so the etcd deployment can complete successfully.

Yeah, that makes sense to me.

Comment 8 William Caban 2020-02-13 17:43:31 UTC
(In reply to Jonathan Lebon from comment #7)
> > Please note the customization for machine configuration you are pointing to DOES use MachineConfigs and those are interpreted AFTER the CRDs created which require etcd and k8s API fully operational. This is precisely what does not happens because of the lack of time synchronization among masters.
> 
> Hmm my understanding is that those MCs are part of the initial Ignition
> config, which means that Ignition on firstboot should apply those
> modifications. See e.g. this part of the text:
> 
> > However, there may be some cases where these workloads need to be executed prior to the node joining the Kubernetes cluster. For example, a compliance mandate like "the user must run auditing tools as soon as the operating system comes up" might require a custom systemd unit for an auditing container in the Ignition config for some or all nodes.
> 
> If that's not happening then I think there is a bug in the installer.

Yes, those MC's become part of the Ignition and it is NOT an installer bug. During the installation, if you take a look at the logs of bootkube service in the bootstrap node, you will see it will fail to render the MC's until after the base control plane is up (meaning etcd quorum + K8s API) and the CRDs for MC's are defined so it can apply all the MC's required for the deployment. It just happens so behind the scenes that unless you look at the logs the different steps are looks like one.

> 
> Ahh but yes, they're not applied to the boostrap machine's Ignition config
> itself, which Colin pointed out in
> https://github.com/openshift/machine-config-operator/issues/629#issuecomment-
> 557645447. So yeah, leveraging this as part of #1 would require fixing that
> too.
> 
> > WRT the two parts: Sounds good about #2 but it should also force the sync (the "makestep" settings) so it properly brings the node in time sync so the etcd deployment can complete successfully.
> 
> Yeah, that makes sense to me.

Comment 9 Colin Walters 2020-02-27 16:05:48 UTC
Related to having things apply to the bootstrap node, one thing that would probably help here is to have the MCO support an "every machine" selector for MachineConfig, then we could define that as applying to the bootstrap as well.

We're effectively doing this manually with the `fips` installconfig flag for example.

Comment 12 Colin Walters 2020-05-12 17:07:44 UTC
First, a lot of related discussion in https://github.com/openshift/machine-config-operator/issues/629

> 2) When the DHCP NTP option is provided, the system should do an immediate time synchronization (makestep 1 -1)

This makes sense; I don't actually know why it's not done that way by default.  It might simply be a missing integration between NetworkManager and chrony.

See e.g. this bit from the systemd NEWS:

        * systemd-timesyncd has been updated to automatically acquire
          NTP server information from systemd-networkd, which might
          have been discovered via DHCP.


But if we need to support explicitly specified configs that also apply to the bootstrap...that gets much messier per above.

Comment 13 Robert Fairley 2020-05-13 14:21:58 UTC
(In reply to Colin Walters from comment #12)
> First, a lot of related discussion in
> https://github.com/openshift/machine-config-operator/issues/629
> 


Had a read through - makes sense from the upstream discussion here that long term, being able to configure the Ignition config of the bootstrap machine through an install-time MachineConfig would help by providing an easy way for a sysadmin to reconfigure with different settings, e.g. to set `makestep 1 -1` in `/etc/chrony.conf` in the bootstrap machine in this case.

> > 2) When the DHCP NTP option is provided, the system should do an immediate time synchronization (makestep 1 -1)
> 
> This makes sense; I don't actually know why it's not done that way by
> default.  It might simply be a missing integration between NetworkManager
> and chrony.
> 
> See e.g. this bit from the systemd NEWS:
> 
>         * systemd-timesyncd has been updated to automatically acquire
>           NTP server information from systemd-networkd, which might
>           have been discovered via DHCP.
> 
> 
> But if we need to support explicitly specified configs that also apply to
> the bootstrap...that gets much messier per above.

Thanks for the pointer here. Seems that default arguments for NTP servers configured by DHCP are given in `/etc/dhcp/dhclient.d/chrony.sh` in the line:

`echo "$server ${NTPSERVERARGS:-iburst}" >> "$SERVERFILE"`

Changing this to:

`echo "$server ${NTPSERVERARGS:-iburst makestep 1 -1}" >> "$SERVERFILE"`

may be what's needed.

Still reading up on what the exact interactions between NetworkManager and chrony are - not completely sure how/when the `/etc/dhcp/dhclient.d/chrony.sh` script is invoked.

We may also require checking for DHCP events (`dhcp4-change`, `dhcp6-change`) in `/etc/NetworkManager/dispatcher.d/20-chrony` to invoke `/etc/dhcp/dhclient.d/chrony.sh`, to apply the `makestep 1 -1` setting to the new NTP servers configured from DHCP.

Will investigate further.

Comment 14 Colin Walters 2020-05-13 17:28:00 UTC
> Seems that default arguments for NTP servers configured by DHCP are given in `/etc/dhcp/dhclient.d/chrony.sh` 

Hah except...we're not shipping `dhclient` by default, we're using the dhcp=internal backend to NetworkManager, see also https://bugzilla.redhat.com/show_bug.cgi?id=1204226

So I doubt that code does anything today for us.

IMO...this "glue" should live either in NM or chrony directly.  Among other reasons, this is a totally generic issue; e.g. Fedora Workstation (using Anaconda/yum, nothing CoreOS) uses NM and chrony too.

> We may also require checking for DHCP events (`dhcp4-change`, `dhcp6-change`) in `/etc/NetworkManager/dispatcher.d/20-chrony` to invoke `/etc/dhcp/dhclient.d/chrony.sh`,

Hmm I wouldn't invoke that script, it's meant for "dhclient".  If we're going to implement this outside of NM/chrony (e.g. in fedora-coreos-config) as a short term thing I think it's simplest to follow an example similar to
https://github.com/coreos/fedora-coreos-config/pull/393

And also related to that PR...we need to be sure the DHCP handling code doesn't conflict with the more specific platform code there.

Comment 15 Robert Fairley 2020-05-14 20:46:15 UTC
(In reply to Colin Walters from comment #14)
> > Seems that default arguments for NTP servers configured by DHCP are given in `/etc/dhcp/dhclient.d/chrony.sh` 
> 
> Hah except...we're not shipping `dhclient` by default, we're using the
> dhcp=internal backend to NetworkManager, see also
> https://bugzilla.redhat.com/show_bug.cgi?id=1204226
> 
> So I doubt that code does anything today for us.
> 
> IMO...this "glue" should live either in NM or chrony directly.  Among other
> reasons, this is a totally generic issue; e.g. Fedora Workstation (using
> Anaconda/yum, nothing CoreOS) uses NM and chrony too.
> 

Ack on NM using the internal DHCP client by default - I did not realize this before. After some more research/thinking, another way may be to instead directly invoke `/usr/libexec/chrony-helper` in response to NM `dhcp4-change` through a dispatch script. Something similar has been done for `dhclient` in [1] which first goes through `/etc/dhcp/dhclient/chrony.sh`. So we'd effectively be generalizing the chrony integration existing in the `11-dhclient` script to write snippets containing NTP servers and chrony config out (similar to what `/etc/dhcp/dhclient/chrony.sh` does [2]), and invoke `chrony-helper update-daemon` directly from the NM dispatch script. This logic could live in the chrony RPM package (either in a new `11-chrony-dhcp` script or make it an addition to the existing `20-chrony` script).

Reading [3], it seems `chrony-helper` is written in a way it can be used generically, as long as the program invoking it writes new NTP configuration to a snippet matching the path `/var/lib/dhclient/chrony.servers.*`. It would be better to read the snippets from a general location - e.g. `/var/lib/chrony-dhcp`, if we intend the snippets from any DHCP client (through NM) to be picked up by chrony. To implement this, the `chrony-helper` script in the chrony RPM could be modified to use the general location. For compatibility with dhclient, `chrony-helper` could still look in the original `/var/lib/dhclient` location in addition to the new common location.

With the sketch above, the remaining issue is conflicts with platform-specific NTP settings. The chrony.d fix in https://bugzilla.redhat.com/show_bug.cgi?id=1828434 would help here I think. Possibly instead of the proposed NM dispatch writing to a directory like `/var/lib/chrony-dhcp`, the dispatch script would write out a snippet to `/run/chrony.d/20-dhcp-...`. The config written by DHCP could be overridden by a `/usr/lib/chrony.d/90-platform-...`, if there was a conflict with a platform-specific NTP server.

Without the `chrony.d` dropin/override functionality packaged, we'd probably need a way to check that `/etc/chrony.conf` doesn't already contain config for any of the NTP servers being added through DHCP; if config for the NTP server already exists, do not write a NTP config file to `/var/lib/chrony-dhcp`. I think this reasoning generally makes sense - if a static config in `/etc/chrony.conf` was already provided, a chrony config that DHCP wants to use for an NTP server should not override the static config?

Apologies for a lengthy comment - this summarizes possibilities for a long term fix I've got in mind so far. Let me know if there is anything I can clarify - also happy to discuss out-of-band.

[1] http://pkgs.devel.redhat.com/cgit/rpms/dhcp/tree/11-dhclient?h=rhel-8.2.0#n28
[2] http://pkgs.devel.redhat.com/cgit/rpms/chrony/tree/chrony.dhclient?h=rhel-8.2.0#n9
[3] http://pkgs.devel.redhat.com/cgit/rpms/chrony/tree/chrony.helper?h=rhel-8.2.0#n2

Comment 16 Robert Fairley 2020-05-14 20:49:41 UTC
> If we're
> going to implement this outside of NM/chrony (e.g. in fedora-coreos-config)
> as a short term thing I think it's simplest to follow an example similar to
> https://github.com/coreos/fedora-coreos-config/pull/393
> 
> And also related to that PR...we need to be sure the DHCP handling code
> doesn't conflict with the more specific platform code there.

For a short-term fix without modifying chrony upstream, maybe we could carry an NM dispatch script (e.g. `11-rhcos-chrony-dhcp`) in the FCOS overlay, which writes out NTP servers and chrony config snippets from DHCP to a separate temporary location in `/run` (e.g. `/run/rhcos-chrony-dhcp`)? The written DHCP-configured NTP snippets will include the required `makestep 1 -1` config. To talk to chrony, `11-rhcos-chrony-dhcp` would execute code similar to `update_daemon()` from the `chrony-helper` script. To avoid conflict with platform-specific NTP config, without having `chrony.d` available, a check in `/etc/chrony.conf` like described in https://bugzilla.redhat.com/show_bug.cgi?id=1800901#c15 would be done before writing the DHCP-configured NTP snippets (probably grepping `/etc/chrony.conf` for the NTP hostname that DHCP is providing).

Comment 17 Miroslav Lichvar 2020-05-19 14:06:27 UTC
Few comments wrt the chrony package.

Servers obtained from DHCP should be configured via chronyc and they should be specific to the interface as the chrony-helper script currently does. Adding/modifying configuration files and restarting the service should be avoided.

Forcing a clock update using the chronyc burst and makestep command might be ok in some cases, but should not be done by default.

Extending the chrony NM dispatcher script to add the servers when the dhclient package is not installed would make sense to me.

Comment 18 Robert Fairley 2020-05-20 17:20:57 UTC
Opened https://github.com/coreos/fedora-coreos-config/pull/412 which would be short-term fix for CoreOS specifically and meant to be temporary. This illustrates the changes proposed in https://bugzilla.redhat.com/show_bug.cgi?id=1800901#c16. 

> Forcing a clock update using the chronyc burst and makestep command might be ok in some cases, but should not be done by default.

Just to check that we are in consensus - by "default" are you referring to the default configuration in `/etc/chrony.conf`, or the default chrony config given to NTP servers obtained via DHCP? As I understand, here we'd like to make `iburst makestep 1 -1` the default for NTP servers obtained by DHCP, but leave the default unchanged for all other cases (i.e. leave `/etc/chrony.conf` unchanged). I.e. NTP servers from DHCP would by default be written with `iburst makestep 1 -1`, by editing this line https://src.fedoraproject.org/rpms/chrony/blob/cafc2b7a759c623057666ae578e847c78b6e7811/f/chrony.dhclient#_9 (not planning to edit this file, but in the patch to the chrony NM dispatcher script I'm planning to add `makestep 1 -1`).

> Extending the chrony NM dispatcher script to add the servers when the dhclient package is not installed would make sense to me.

Makes sense - working on a patch to https://git.tuxfamily.org/chrony/chrony.git/tree/examples/chrony.nm-dispatcher for this.

Comment 19 Robert Fairley 2020-05-20 17:31:35 UTC
> It might simply be a missing integration between NetworkManager and chrony.

> As I understand, here we'd like to make `iburst makestep 1 -1` the default for NTP servers obtained by DHCP

Reading this again - makes sense that it may just be missing integration between NM internal client and chrony, NTP servers obtained via DHCP are not getting passsed to chrony currently, resulting in the time sync issues. So extending https://git.tuxfamily.org/chrony/chrony.git/tree/examples/chrony.nm-dispatcher *without* changing default NTP parameters for servers obtained by DHCP may be sufficient for fixing this (i.e. keeping just `iburst`). Will first try with `iburst` only to see if it fixes the time sync issues in this bug.

Comment 20 Miroslav Lichvar 2020-05-21 06:49:54 UTC
(In reply to Robert Fairley from comment #18) 
> Just to check that we are in consensus - by "default" are you referring to
> the default configuration in `/etc/chrony.conf`, or the default chrony
> config given to NTP servers obtained via DHCP?

The latter. By default, makestep should not be changed for NTP servers from DHCP. When a suspended system is moved to an untrusted network (e.g. a laptop moving between different wireless networks), we don't want MITM attackers to be able to shift the client's clock to the far future or past there.

As for a quick fix, maybe you could just install the dhcp-client package to get its NM dispatcher working? It won't change the NM behavior, right?

Comment 21 Robert Fairley 2020-05-21 14:34:11 UTC
> The latter. By default, makestep should not be changed for NTP servers from DHCP. When a suspended system is moved to an untrusted network (e.g. a laptop moving between different wireless networks), we don't want MITM attackers to be able to shift the client's clock to the far future or past there.

Right, that makes sense.

> As for a quick fix, maybe you could just install the dhcp-client package to get its NM dispatcher working? It won't change the NM behavior, right?

Yes, that'd work too - I don't see a problem having the dhcp-client package present as NM will still be configured to use the internal client. Will try this out.

I have also tested out a patch for the `20-chrony` dispatcher script and posted the patch to the chrony-dev mailing list. An accompanying patch in the chrony RPM package is https://src.fedoraproject.org/rpms/chrony/pull-request/3 (also links to the commit for the upstream patch, that I posted to the mailing list).

Comment 22 Robert Fairley 2020-05-26 19:58:46 UTC
Proposed the fix upstream and in the Fedora chrony RPM: https://src.fedoraproject.org/rpms/chrony/pull-request/3 (upstream patch posted to chrony-dev mailing list).

Did initial testing in FCOS stable and next, and the latest two RHCOS versions, and the proposed fixes above worked (with and without dhclient installed).

Also proposed a short-term fix which includes the same scripts, slightly modified from the upstream patches to avoid collision: https://github.com/coreos/fedora-coreos-config/pull/412

Now testing the short-term fix. For the testing method used, see https://github.com/coreos/fedora-coreos-config/pull/412#issuecomment-634233469. Will write up more detailed verification steps here also.

Comment 26 Colin Walters 2020-05-29 14:41:35 UTC
OK right - the core tension is basically around "trusted networks".

Anyone who wants to set `makestep 1 -1` can inject `/etc/chrony.conf` via Ignition/MachineConfig.  That's probably our baseline fix.

But that overwrites the whole config file; it'd be even more ideal if chrony supported drop-in files (per the discussions above).
It also means the "source of truth" isn't DHCP anymore which could be annoying for systems that want to support both CoreOS and other systems.

Comment 27 Robert Fairley 2020-05-29 20:53:54 UTC
Planning to continue to work on this through the next sprint.

Comment 29 Robert Fairley 2020-06-02 18:08:32 UTC
Note the pool is still enabled by default in RHCOS - `pool 2.rhel.pool.ntp.org iburst` in `/etc/chrony.conf`. Could machines have been trying to sync to the pool previously and reached the default 3 steps limit before the NTP servers are added from DHCP? Assuming the default `makestep 1 3`, and the pool NTP servers are reachable from the deployment. Or possibly there is too great a difference between the pool and the DHCP-provided NTP servers and some nodes sync with either one, causing cert validations to fail? (though this would probably be more a problem with the DHCP-provided NTP servers).

We do disable the pool for platform-specific config in https://github.com/coreos/fedora-coreos-config/blob/e9448fe108fd6dcca279fee3281f0eebd3076ca9/overlay.d/20platform-chrony/usr/lib/systemd/system-generators/coreos-platform-chrony#L35.

Possibly the default pool can be disabled as a fix if the deployment is not intending to use it, and/or add `makestep 1 -1` for some cases (e.g. a drifting hardware clock keeps needing resyncing). These measures shouldn't apply as a default, but should be easily configurable and mentioned in docs (as Colin mentioned, chrony.conf can be overridden but would be made nicer using config snippets, which are already in upstream, instead).

Will try to verify whether the pool being disabled and `makestep 1 -1` can help with timesync issues in the situations above. Overall though, the current defaults seem sane, and the fix would be making the defaults configurable through chrony.d snippets (to users, and for things like overlaying platform-specific config).

So I'm thinking this BZ would be fixed by:

1. consider splitting the pool and makestep config from `/etc/chrony.conf` into chrony.d snippets, so they can be overritten without touching the rest of `/etc/chrony.conf`
2. mention the potential time sync problem in install/deployment docs, and point to the section on reconfiguring chrony (and suggest using `makestep 1 -1` and/or commenting out the pool)
3. update the example in the docs to reconfigure chrony using chrony.d snippets instead of overwriting chrony.conf (https://docs.openshift.com/container-platform/4.4/installing/install_config/installing-customizing.html#installation-special-config-crony_installing-customizing)

Comment 32 Miroslav Lichvar 2020-06-03 07:07:24 UTC
(In reply to Robert Fairley from comment #29)
> Note the pool is still enabled by default in RHCOS - `pool
> 2.rhel.pool.ntp.org iburst` in `/etc/chrony.conf`. Could machines have been
> trying to sync to the pool previously and reached the default 3 steps limit
> before the NTP servers are added from DHCP?

If the pool servers are not reachable, the first clock update will be from a server added from DHCP.

If the pool servers are reachable, the first clock update will be from a pool server or local server added from DHCP (assuming a majority is found among them).

If the local servers from DHCP are reachable later than the pool servers (e.g. when there are multiple interfaces?), the first clock will be from a pool server. An additional step might be needed in a later update with a DHCP-provided server, but I think that would happen only if the pool servers were consistently wrong, or someone was messing with the packets in network. In this case an unlimited makestep would make a difference.

Comment 35 Colin Walters 2020-06-10 19:04:12 UTC
*** Bug 1809063 has been marked as a duplicate of this bug. ***

Comment 38 Micah Abbott 2020-07-20 15:27:58 UTC
Additional internal discussion has indicated that we need to pursue the resolution of this BZ for 4.6 as we have changes related to how networking is configured in RHCOS that will affect the chrony/NTP/NM interaction.

Comment 39 Sohan Kunkerkar 2020-07-30 17:45:09 UTC
Continue working on this in the next sprint.

Comment 47 Sohan Kunkerkar 2020-09-28 15:27:05 UTC
This BZ can be verified by simply using https://github.com/coreos/fedora-coreos-config/blob/testing-devel/tests/kola/chrony/dhcp-propagation against the latest RHCOS/FCOS builds.

Something like this:

[root@localhost ~]# rpm-ostree status
State: idle
Deployments:
* ostree://d8380af85c6e4a5be0bf960aa56238fc887ba1ff780b6e83496d94f10729b2d1
                   Version: 46.82.202008231640-0 (2020-08-23T16:44:56Z)
[root@localhost ~]# ./dhcp-propagation 
+ main
+ test_setup
+ setsebool container_manage_cgroup 1
+ ip netns add container
+ ip link add veth-host type veth peer name veth-container
+ ip link set veth-container netns container
+ ip netns exec container ip address add 172.16.0.1/24 dev veth-container
+ ip netns exec container ip link set veth-container up
+ nmcli dev set veth-host managed yes
+ ip link set veth-host up
++ mktemp -d
+ pushd /tmp/tmp.IuCbg7UYU0
/tmp/tmp.IuCbg7UYU0 ~
++ cut -d ' ' -f 1
++ getent hosts time-c-g.nist.gov
+ NTPHOSTIP=129.6.15.30
+ cat
+ podman build -t dnsmasq .
STEP 1: FROM registry.fedoraproject.org/fedora:32
Getting image source signatures
Copying blob dd9f43919ba0 done  
Copying config 00ff39a8bf done  
Writing manifest to image destination
Storing signatures
STEP 2: RUN dnf -y install systemd dnsmasq iproute iputils && dnf clean all && systemctl enable dnsmasq
Fedora 32 openh264 (From Cisco) - x86_64                                                                                                                                             2.1 kB/s | 2.5 kB     00:01    
Fedora Modular 32 - x86_64                                                                                                                                                           5.6 MB/s | 4.9 MB     00:00    
Fedora Modular 32 - x86_64 - Updates                                                                                                                                                 1.8 MB/s | 3.7 MB     00:02    
Fedora 32 - x86_64 - Updates                                                                                                                                                          12 MB/s |  23 MB     00:01    
Fedora 32 - x86_64                                                                                                                                                                    15 MB/s |  70 MB     00:04    
Dependencies resolved.
=====================================================================================================================================================================================================================
 Package                                                     Architecture                                Version                                                  Repository                                    Size
=====================================================================================================================================================================================================================
Installing:
 dnsmasq                                                     x86_64                                      2.81-4.fc32                                              updates                                      321 k
 iproute                                                     x86_64                                      5.7.0-1.fc32                                             updates                                      662 k
 iputils                                                     x86_64                                      20190515-7.fc32                                          updates                                      141 k
 systemd                                                     x86_64                                      245.8-2.fc32                                             updates                                      4.2 M
Upgrading:
 systemd-libs                                                x86_64                                      245.8-2.fc32                                             updates                                      559 k
Installing dependencies:
 acl                                                         x86_64                                      2.2.53-5.fc32                                            fedora                                        72 k
 cryptsetup-libs                                             x86_64                                      2.3.4-1.fc32                                             updates                                      475 k
 dbus                                                        x86_64                                      1:1.12.20-1.fc32                                         updates                                      9.7 k
 dbus-broker                                                 x86_64                                      24-1.fc32                                                updates                                      174 k
 dbus-common                                                 noarch                                      1:1.12.20-1.fc32                                         updates                                       16 k
 dbus-libs                                                   x86_64                                      1:1.12.20-1.fc32                                         updates                                      163 k
 device-mapper                                               x86_64                                      1.02.171-1.fc32                                          fedora                                       143 k
 device-mapper-libs                                          x86_64                                      1.02.171-1.fc32                                          fedora                                       180 k
 iptables-libs                                               x86_64                                      1.8.4-9.fc32                                             updates                                      368 k
 kmod-libs                                                   x86_64                                      27-1.fc32                                                fedora                                        64 k
 libargon2                                                   x86_64                                      20171227-4.fc32                                          fedora                                        30 k
 libmnl                                                      x86_64                                      1.0.4-11.fc32                                            fedora                                        28 k
 libnetfilter_conntrack                                      x86_64                                      1.0.7-4.fc32                                             fedora                                        62 k
 libnfnetlink                                                x86_64                                      1.0.1-17.fc32                                            fedora                                        29 k
 libpcap                                                     x86_64                                      14:1.9.1-3.fc32                                          fedora                                       172 k
 libseccomp                                                  x86_64                                      2.5.0-3.fc32                                             updates                                       73 k
 linux-atm-libs                                              x86_64                                      2.5.1-26.fc32                                            fedora                                        38 k
 psmisc                                                      x86_64                                      23.3-3.fc32                                              fedora                                       159 k
 qrencode-libs                                               x86_64                                      4.0.2-5.fc32                                             fedora                                        60 k
 systemd-pam                                                 x86_64                                      245.8-2.fc32                                             updates                                      300 k
 systemd-rpm-macros                                          noarch                                      245.8-2.fc32                                             updates                                       25 k
 xkeyboard-config                                            noarch                                      2.29-1.fc32                                              fedora                                       772 k
Installing weak dependencies:
 diffutils                                                   x86_64                                      3.7-4.fc32                                               fedora                                       397 k
 iproute-tc                                                  x86_64                                      5.7.0-1.fc32                                             updates                                      421 k
 libxkbcommon                                                x86_64                                      0.10.0-2.fc32                                            fedora                                       121 k

Transaction Summary
=====================================================================================================================================================================================================================
Install  29 Packages
Upgrade   1 Package

Total download size: 10 M
Downloading Packages:
(1/30): dbus-1.12.20-1.fc32.x86_64.rpm                                                                                                                                                35 kB/s | 9.7 kB     00:00    
(2/30): dbus-common-1.12.20-1.fc32.noarch.rpm                                                                                                                                        776 kB/s |  16 kB     00:00    
(3/30): dbus-broker-24-1.fc32.x86_64.rpm                                                                                                                                             494 kB/s | 174 kB     00:00    
(4/30): dbus-libs-1.12.20-1.fc32.x86_64.rpm                                                                                                                                          2.2 MB/s | 163 kB     00:00    
(5/30): dnsmasq-2.81-4.fc32.x86_64.rpm                                                                                                                                               3.6 MB/s | 321 kB     00:00    
(6/30): iproute-5.7.0-1.fc32.x86_64.rpm                                                                                                                                              5.4 MB/s | 662 kB     00:00    
(7/30): iproute-tc-5.7.0-1.fc32.x86_64.rpm                                                                                                                                           5.1 MB/s | 421 kB     00:00    
(8/30): iptables-libs-1.8.4-9.fc32.x86_64.rpm                                                                                                                                        8.9 MB/s | 368 kB     00:00    
(9/30): iputils-20190515-7.fc32.x86_64.rpm                                                                                                                                           4.0 MB/s | 141 kB     00:00    
(10/30): libseccomp-2.5.0-3.fc32.x86_64.rpm                                                                                                                                          1.7 MB/s |  73 kB     00:00    
(11/30): systemd-pam-245.8-2.fc32.x86_64.rpm                                                                                                                                         6.6 MB/s | 300 kB     00:00    
(12/30): systemd-rpm-macros-245.8-2.fc32.noarch.rpm                                                                                                                                  457 kB/s |  25 kB     00:00    
(13/30): acl-2.2.53-5.fc32.x86_64.rpm                                                                                                                                                144 kB/s |  72 kB     00:00    
(14/30): device-mapper-1.02.171-1.fc32.x86_64.rpm                                                                                                                                    1.2 MB/s | 143 kB     00:00    
(15/30): device-mapper-libs-1.02.171-1.fc32.x86_64.rpm                                                                                                                               1.7 MB/s | 180 kB     00:00    
(16/30): cryptsetup-libs-2.3.4-1.fc32.x86_64.rpm                                                                                                                                     336 kB/s | 475 kB     00:01    
(17/30): diffutils-3.7-4.fc32.x86_64.rpm                                                                                                                                             3.1 MB/s | 397 kB     00:00    
(18/30): libargon2-20171227-4.fc32.x86_64.rpm                                                                                                                                        609 kB/s |  30 kB     00:00    
(19/30): libmnl-1.0.4-11.fc32.x86_64.rpm                                                                                                                                             442 kB/s |  28 kB     00:00    
(20/30): libnetfilter_conntrack-1.0.7-4.fc32.x86_64.rpm                                                                                                                              1.1 MB/s |  62 kB     00:00    
(21/30): kmod-libs-27-1.fc32.x86_64.rpm                                                                                                                                              207 kB/s |  64 kB     00:00    
(22/30): libnfnetlink-1.0.1-17.fc32.x86_64.rpm                                                                                                                                       701 kB/s |  29 kB     00:00    
(23/30): libxkbcommon-0.10.0-2.fc32.x86_64.rpm                                                                                                                                       1.9 MB/s | 121 kB     00:00    
(24/30): linux-atm-libs-2.5.1-26.fc32.x86_64.rpm                                                                                                                                     697 kB/s |  38 kB     00:00    
(25/30): libpcap-1.9.1-3.fc32.x86_64.rpm                                                                                                                                             1.2 MB/s | 172 kB     00:00    
(26/30): psmisc-23.3-3.fc32.x86_64.rpm                                                                                                                                               2.5 MB/s | 159 kB     00:00    
(27/30): qrencode-libs-4.0.2-5.fc32.x86_64.rpm                                                                                                                                       963 kB/s |  60 kB     00:00    
(28/30): xkeyboard-config-2.29-1.fc32.noarch.rpm                                                                                                                                     5.9 MB/s | 772 kB     00:00    
(29/30): systemd-libs-245.8-2.fc32.x86_64.rpm                                                                                                                                        627 kB/s | 559 kB     00:00    
(30/30): systemd-245.8-2.fc32.x86_64.rpm                                                                                                                                             787 kB/s | 4.2 MB     00:05    
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                1.5 MB/s |  10 MB     00:06     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                             1/1 
  Upgrading        : systemd-libs-245.8-2.fc32.x86_64                                                                                                                                                           1/31 
  Running scriptlet: systemd-libs-245.8-2.fc32.x86_64                                                                                                                                                           1/31 
  Installing       : libmnl-1.0.4-11.fc32.x86_64                                                                                                                                                                2/31 
  Installing       : dbus-libs-1:1.12.20-1.fc32.x86_64                                                                                                                                                          3/31 
  Installing       : xkeyboard-config-2.29-1.fc32.noarch                                                                                                                                                        4/31 
  Installing       : libxkbcommon-0.10.0-2.fc32.x86_64                                                                                                                                                          5/31 
  Installing       : qrencode-libs-4.0.2-5.fc32.x86_64                                                                                                                                                          6/31 
  Installing       : psmisc-23.3-3.fc32.x86_64                                                                                                                                                                  7/31 
  Installing       : linux-atm-libs-2.5.1-26.fc32.x86_64                                                                                                                                                        8/31 
  Installing       : libpcap-14:1.9.1-3.fc32.x86_64                                                                                                                                                             9/31 
  Installing       : libnfnetlink-1.0.1-17.fc32.x86_64                                                                                                                                                         10/31 
  Installing       : libnetfilter_conntrack-1.0.7-4.fc32.x86_64                                                                                                                                                11/31 
  Installing       : iptables-libs-1.8.4-9.fc32.x86_64                                                                                                                                                         12/31 
  Installing       : iproute-tc-5.7.0-1.fc32.x86_64                                                                                                                                                            13/31 
  Installing       : iproute-5.7.0-1.fc32.x86_64                                                                                                                                                               14/31 
  Installing       : libargon2-20171227-4.fc32.x86_64                                                                                                                                                          15/31 
  Installing       : kmod-libs-27-1.fc32.x86_64                                                                                                                                                                16/31 
  Installing       : diffutils-3.7-4.fc32.x86_64                                                                                                                                                               17/31 
  Installing       : acl-2.2.53-5.fc32.x86_64                                                                                                                                                                  18/31 
  Installing       : systemd-rpm-macros-245.8-2.fc32.noarch                                                                                                                                                    19/31 
  Installing       : libseccomp-2.5.0-3.fc32.x86_64                                                                                                                                                            20/31 
  Installing       : device-mapper-1.02.171-1.fc32.x86_64                                                                                                                                                      21/31 
  Installing       : device-mapper-libs-1.02.171-1.fc32.x86_64                                                                                                                                                 22/31 
  Installing       : cryptsetup-libs-2.3.4-1.fc32.x86_64                                                                                                                                                       23/31 
  Installing       : dbus-1:1.12.20-1.fc32.x86_64                                                                                                                                                              24/31 
  Installing       : systemd-pam-245.8-2.fc32.x86_64                                                                                                                                                           25/31 
  Running scriptlet: systemd-245.8-2.fc32.x86_64                                                                                                                                                               26/31 
  Installing       : systemd-245.8-2.fc32.x86_64                                                                                                                                                               26/31 
  Running scriptlet: systemd-245.8-2.fc32.x86_64                                                                                                                                                               26/31 
  Installing       : dbus-common-1:1.12.20-1.fc32.noarch                                                                                                                                                       27/31 
  Running scriptlet: dbus-common-1:1.12.20-1.fc32.noarch                                                                                                                                                       27/31 
Created symlink /etc/systemd/system/sockets.target.wants/dbus.socket → /usr/lib/systemd/system/dbus.socket.
Created symlink /etc/systemd/user/sockets.target.wants/dbus.socket → /usr/lib/systemd/user/dbus.socket.

  Running scriptlet: dbus-broker-24-1.fc32.x86_64                                                                                                                                                              28/31 
  Installing       : dbus-broker-24-1.fc32.x86_64                                                                                                                                                              28/31 
  Running scriptlet: dbus-broker-24-1.fc32.x86_64                                                                                                                                                              28/31 
Created symlink /etc/systemd/system/dbus.service → /usr/lib/systemd/system/dbus-broker.service.
Created symlink /etc/systemd/user/dbus.service → /usr/lib/systemd/user/dbus-broker.service.

  Running scriptlet: dnsmasq-2.81-4.fc32.x86_64                                                                                                                                                                29/31 
  Installing       : dnsmasq-2.81-4.fc32.x86_64                                                                                                                                                                29/31 
  Running scriptlet: dnsmasq-2.81-4.fc32.x86_64                                                                                                                                                                29/31 
  Installing       : iputils-20190515-7.fc32.x86_64                                                                                                                                                            30/31 
  Running scriptlet: iputils-20190515-7.fc32.x86_64                                                                                                                                                            30/31 
  Cleanup          : systemd-libs-245.6-2.fc32.x86_64                                                                                                                                                          31/31 
  Running scriptlet: systemd-libs-245.6-2.fc32.x86_64                                                                                                                                                          31/31 
  Verifying        : cryptsetup-libs-2.3.4-1.fc32.x86_64                                                                                                                                                        1/31 
  Verifying        : dbus-1:1.12.20-1.fc32.x86_64                                                                                                                                                               2/31 
  Verifying        : dbus-broker-24-1.fc32.x86_64                                                                                                                                                               3/31 
  Verifying        : dbus-common-1:1.12.20-1.fc32.noarch                                                                                                                                                        4/31 
  Verifying        : dbus-libs-1:1.12.20-1.fc32.x86_64                                                                                                                                                          5/31 
  Verifying        : dnsmasq-2.81-4.fc32.x86_64                                                                                                                                                                 6/31 
  Verifying        : iproute-5.7.0-1.fc32.x86_64                                                                                                                                                                7/31 
  Verifying        : iproute-tc-5.7.0-1.fc32.x86_64                                                                                                                                                             8/31 
  Verifying        : iptables-libs-1.8.4-9.fc32.x86_64                                                                                                                                                          9/31 
  Verifying        : iputils-20190515-7.fc32.x86_64                                                                                                                                                            10/31 
  Verifying        : libseccomp-2.5.0-3.fc32.x86_64                                                                                                                                                            11/31 
  Verifying        : systemd-245.8-2.fc32.x86_64                                                                                                                                                               12/31 
  Verifying        : systemd-pam-245.8-2.fc32.x86_64                                                                                                                                                           13/31 
  Verifying        : systemd-rpm-macros-245.8-2.fc32.noarch                                                                                                                                                    14/31 
  Verifying        : acl-2.2.53-5.fc32.x86_64                                                                                                                                                                  15/31 
  Verifying        : device-mapper-1.02.171-1.fc32.x86_64                                                                                                                                                      16/31 
  Verifying        : device-mapper-libs-1.02.171-1.fc32.x86_64                                                                                                                                                 17/31 
  Verifying        : diffutils-3.7-4.fc32.x86_64                                                                                                                                                               18/31 
  Verifying        : kmod-libs-27-1.fc32.x86_64                                                                                                                                                                19/31 
  Verifying        : libargon2-20171227-4.fc32.x86_64                                                                                                                                                          20/31 
  Verifying        : libmnl-1.0.4-11.fc32.x86_64                                                                                                                                                               21/31 
  Verifying        : libnetfilter_conntrack-1.0.7-4.fc32.x86_64                                                                                                                                                22/31 
  Verifying        : libnfnetlink-1.0.1-17.fc32.x86_64                                                                                                                                                         23/31 
  Verifying        : libpcap-14:1.9.1-3.fc32.x86_64                                                                                                                                                            24/31 
  Verifying        : libxkbcommon-0.10.0-2.fc32.x86_64                                                                                                                                                         25/31 
  Verifying        : linux-atm-libs-2.5.1-26.fc32.x86_64                                                                                                                                                       26/31 
  Verifying        : psmisc-23.3-3.fc32.x86_64                                                                                                                                                                 27/31 
  Verifying        : qrencode-libs-4.0.2-5.fc32.x86_64                                                                                                                                                         28/31 
  Verifying        : xkeyboard-config-2.29-1.fc32.noarch                                                                                                                                                       29/31 
  Verifying        : systemd-libs-245.8-2.fc32.x86_64                                                                                                                                                          30/31 
  Verifying        : systemd-libs-245.6-2.fc32.x86_64                                                                                                                                                          31/31 

Upgraded:
  systemd-libs-245.8-2.fc32.x86_64                                                                                                                                                                                   

Installed:
  acl-2.2.53-5.fc32.x86_64               cryptsetup-libs-2.3.4-1.fc32.x86_64       dbus-1:1.12.20-1.fc32.x86_64                    dbus-broker-24-1.fc32.x86_64             dbus-common-1:1.12.20-1.fc32.noarch     
  dbus-libs-1:1.12.20-1.fc32.x86_64      device-mapper-1.02.171-1.fc32.x86_64      device-mapper-libs-1.02.171-1.fc32.x86_64       diffutils-3.7-4.fc32.x86_64              dnsmasq-2.81-4.fc32.x86_64              
  iproute-5.7.0-1.fc32.x86_64            iproute-tc-5.7.0-1.fc32.x86_64            iptables-libs-1.8.4-9.fc32.x86_64               iputils-20190515-7.fc32.x86_64           kmod-libs-27-1.fc32.x86_64              
  libargon2-20171227-4.fc32.x86_64       libmnl-1.0.4-11.fc32.x86_64               libnetfilter_conntrack-1.0.7-4.fc32.x86_64      libnfnetlink-1.0.1-17.fc32.x86_64        libpcap-14:1.9.1-3.fc32.x86_64          
  libseccomp-2.5.0-3.fc32.x86_64         libxkbcommon-0.10.0-2.fc32.x86_64         linux-atm-libs-2.5.1-26.fc32.x86_64             psmisc-23.3-3.fc32.x86_64                qrencode-libs-4.0.2-5.fc32.x86_64       
  systemd-245.8-2.fc32.x86_64            systemd-pam-245.8-2.fc32.x86_64           systemd-rpm-macros-245.8-2.fc32.noarch          xkeyboard-config-2.29-1.fc32.noarch     

Complete!
42 files removed
Created symlink /etc/systemd/system/multi-user.target.wants/dnsmasq.service → /usr/lib/systemd/system/dnsmasq.service.
--> 2eb7372dca3
STEP 3: RUN echo -e 'dhcp-range=172.16.0.10,172.16.0.20,12h\nbind-interfaces\ninterface=veth-container\ndhcp-option=option:ntp-server,129.6.15.30' > /etc/dnsmasq.d/dhcp
--> ca4e7ddca01
STEP 4: CMD [ "/sbin/init" ]
STEP 5: COMMIT dnsmasq
--> d78f87da60f
d78f87da60f4e48bf09348691ebe53a9ebbd1d247bcc64aad90d3a4265c4d56b
+ popd
~
+ podman run -d --rm --name dnsmasq --privileged --network ns:/var/run/netns/container dnsmasq
105548a24dfcec86657588559465544f038a020acafaa4221236012f15b15b69
+ NTPSERVER=
+ retries=300
+ [[ 300 -gt 0 ]]
+ [[ ! '' =~ time-c-g\.nist\.gov ]]
++ chronyc sources
+ NTPSERVER='210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    39  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    38  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   167  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    37  -2030us[-2030us] +/-   69ms'
+ [[ 210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    39  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    38  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   167  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    37  -2030us[-2030us] +/-   69ms != *\t\i\m\e\-\c\-\g\.\n\i\s\t\.\g\o\v* ]]
+ echo 'waiting for ntp server to appear'
waiting for ntp server to appear
+ sleep 1
+ retries=299
+ [[ 299 -gt 0 ]]
+ [[ ! 210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    39  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    38  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   167  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    37  -2030us[-2030us] +/-   69ms =~ time-c-g\.nist\.gov ]]
++ chronyc sources
+ NTPSERVER='210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    41  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    40  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   169  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    39  -2030us[-2030us] +/-   69ms'
+ [[ 210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    41  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    40  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   169  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    39  -2030us[-2030us] +/-   69ms != *\t\i\m\e\-\c\-\g\.\n\i\s\t\.\g\o\v* ]]
+ echo 'waiting for ntp server to appear'
waiting for ntp server to appear
+ sleep 1
+ retries=298
+ [[ 298 -gt 0 ]]
+ [[ ! 210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    41  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    40  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   169  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    39  -2030us[-2030us] +/-   69ms =~ time-c-g\.nist\.gov ]]
++ chronyc sources
+ NTPSERVER='210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    44  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    43  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   172  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    42  -2030us[-2030us] +/-   69ms'
+ [[ 210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    44  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    43  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   172  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    42  -2030us[-2030us] +/-   69ms != *\t\i\m\e\-\c\-\g\.\n\i\s\t\.\g\o\v* ]]
+ echo 'waiting for ntp server to appear'
waiting for ntp server to appear
+ sleep 1
+ retries=297
+ [[ 297 -gt 0 ]]
+ [[ ! 210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    44  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    43  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   172  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    42  -2030us[-2030us] +/-   69ms =~ time-c-g\.nist\.gov ]]
++ chronyc sources
+ NTPSERVER='210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    46  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    45  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   174  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    44  -2030us[-2030us] +/-   69ms'
+ [[ 210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    46  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    45  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   174  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    44  -2030us[-2030us] +/-   69ms != *\t\i\m\e\-\c\-\g\.\n\i\s\t\.\g\o\v* ]]
+ echo 'waiting for ntp server to appear'
waiting for ntp server to appear
+ sleep 1
+ retries=296
+ [[ 296 -gt 0 ]]
+ [[ ! 210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    46  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    45  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   174  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    44  -2030us[-2030us] +/-   69ms =~ time-c-g\.nist\.gov ]]
++ chronyc sources
+ NTPSERVER='210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    48  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    47  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   176  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    46  -2030us[-2030us] +/-   69ms'
+ [[ 210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    48  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    47  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   176  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    46  -2030us[-2030us] +/-   69ms != *\t\i\m\e\-\c\-\g\.\n\i\s\t\.\g\o\v* ]]
+ echo 'waiting for ntp server to appear'
waiting for ntp server to appear
+ sleep 1
+ retries=295
+ [[ 295 -gt 0 ]]
+ [[ ! 210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    48  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    47  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   176  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    46  -2030us[-2030us] +/-   69ms =~ time-c-g\.nist\.gov ]]
++ chronyc sources
+ NTPSERVER='210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    50  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    49  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   178  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    48  -2030us[-2030us] +/-   69ms'
+ [[ 210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    50  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    49  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   178  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    48  -2030us[-2030us] +/-   69ms != *\t\i\m\e\-\c\-\g\.\n\i\s\t\.\g\o\v* ]]
+ echo 'waiting for ntp server to appear'
waiting for ntp server to appear
+ sleep 1
+ retries=294
+ [[ 294 -gt 0 ]]
+ [[ ! 210 Number of sources = 4
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    50  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    49  +2590us[+2590us] +/-  110ms
^- triton.ellipse.net            2   6    36   178  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    48  -2030us[-2030us] +/-   69ms =~ time-c-g\.nist\.gov ]]
++ chronyc sources
+ NTPSERVER='210 Number of sources = 5
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    52  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    51  +2590us[+2590us] +/-  110ms
^? triton.ellipse.net            2   6    36   180  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    50  -2030us[-2030us] +/-   69ms
^? time-c-g.nist.gov             1   6     1     0    -15ms[  -15ms] +/-   35ms'
+ [[ 210 Number of sources = 5
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    52  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    51  +2590us[+2590us] +/-  110ms
^? triton.ellipse.net            2   6    36   180  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    50  -2030us[-2030us] +/-   69ms
^? time-c-g.nist.gov             1   6     1     0    -15ms[  -15ms] +/-   35ms != *\t\i\m\e\-\c\-\g\.\n\i\s\t\.\g\o\v* ]]
+ [[ 294 -gt 0 ]]
+ [[ ! 210 Number of sources = 5
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 108.61.73.243                 2   6    77    52  -5267us[-8466us] +/-   41ms
^- 108.61.73.244                 2   6    77    51  +2590us[+2590us] +/-  110ms
^? triton.ellipse.net            2   6    36   180  -1864us[-3124us] +/-   75ms
^+ 208.67.72.50                  2   6    77    50  -2030us[-2030us] +/-   69ms
^? time-c-g.nist.gov             1   6     1     0    -15ms[  -15ms] +/-   35ms =~ time-c-g\.nist\.gov ]]
+ '[' 294 -eq 0 ']'

Comment 48 Colin Walters 2020-09-28 18:13:59 UTC
Since we have an automated test for this and it's passing, I think we should just mark it VERIFIED.

Comment 49 Micah Abbott 2020-09-28 18:53:55 UTC
I ran the script mentioned in comment #47 against 46.82.202009222340-0 and it was successful.  Moving to VERIFIED.

(Also the fact that we have a passing automated test per comment #48 would have been sufficient)

Comment 51 errata-xmlrpc 2020-10-27 15:55:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196

Comment 52 Red Hat Bugzilla 2023-09-14 05:52:10 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.