Bug 1276063 - rhel-osp-director: The time isn't synced on all overcloud nodes, deployed with the --ntp-server argument.
rhel-osp-director: The time isn't synced on all overcloud nodes, deployed wit...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
x86_64 Linux
high Severity high
: ga
: 8.0 (Liberty)
Assigned To: Hugh Brock
Alexander Chuzhoy
: Reopened, TestOnly
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-28 10:45 EDT by Alexander Chuzhoy
Modified: 2016-04-07 17:41 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-04-07 17:41:59 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Alexander Chuzhoy 2015-10-28 10:45:46 EDT
rhel-osp-director: The time isn't synced on all overcloud nodes, deployed with the --ntp-server argument.


Environment:
instack-undercloud-2.1.2-29.el7ost.noarch
openstack-heat-common-2015.1.1-5.el7ost.noarch


Steps to reproduce:
Attempt to deploy overcloud on nodes with skewed time, using the --ntp-server arg.


Result:
The time isn't synced with the ntp server.


Expected result:

The time on all nodes should be synced with the ntp server specified with --ntp-server.



Notes:
Runing ntpdate <server> manually on the nodes adjusts the time properly, so it seems like the command isn't triggered.
Also,
Looking at /usr/libexec/os-refresh-config/configure.d/50-ntp that's mentioned in the os-collect-config logs:
NTP_SERVER="$(os-apply-config --key 'ntp.servers.0.server' --type raw --key-default '')"

if [ -n "${NTP_SERVER}" ]; then

Running "os-apply-config --key 'ntp.servers.0.server' --type raw --key-default ''" results in empty string, so the condition isn't true.
Comment 2 Mike Burns 2015-11-09 12:44:04 EST
Sasha, can you please provide puppet logs?

Thanks
Comment 4 Mike Burns 2015-11-11 11:05:03 EST
Closing for now, if you reproduce, please reopen.
Comment 5 Dimitri Savineau 2015-12-22 15:58:48 EST
I have the same issue on an OSP-d 7.1 deployment

Environment:
instack-undercloud-2.1.2-29.el7ost.noarch
openstack-puppet-modules-2015.1.8-21.el7ost.noarch

openstack overcloud deploy --templates ~/templates/ -e ~/env.yaml -e ~/templates/environments/storage-environment.yaml -e ~/templates/environments/network-isolation.yaml --control-scale 3 --control-flavor control --compute-scale 1 --compute-flavor compute --ceph-storage-scale 1 --ceph-storage-flavor ceph-storage --swift-storage-scale 0 --swift-storage-flavor swift-storage --block-storage-scale 0 --neutron-network-type vxlan --neutron-tunnel-types vxlan --neutron-vni-ranges 1:10000 --neutron-bridge-mappings datacentre:br-bond --libvirt-type kvm --ntp-server pool.ntp.org -t 120

On overcloud side the value in hiera is right :

$ sudo hiera ntp::servers
["pool.ntp.org"]

But the ntp configuration is not apply with this value.

$ sudo grep ^server /etc/ntp.conf 
server 127.127.1.0 # local clock address

In the puppet logs we can see the changes but there are not applied on each puppet run (or maybe something rollback the changes after)

$ heat deployment-output-show <physical_resource_id> deploy_stdout|grep -i ntp
Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/content: content changed '{md5}c07b9a377faea45b96b7d3bf8976004b' to '{md5}7044a259392968d8dbe468670fd9c98c'
Notice: /File[/etc/ntp.conf]/seltype: seltype changed 'etc_t' to 'net_conf_t'
Notice: /Stage[main]/Ntp::Service/Service[ntp]/ensure: ensure changed 'stopped' to 'running'

Running the puppet command by hand only for ntp :

$ sudo  FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/1ac2a3a5-0b2a-4ab7-aea2-13379c20b3d0"  FACTER_fqdn="ctrl00-prv.localdomain"  FACTER_deploy_config_name="overcloud-ControllerNodesPostDeployment-l2fejf76ersk-ControllerPuppetConfig-bcgn5k5qg3vt-ControllerPuppetConfigImpl-4p6ejshk2opq"  puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/1ac2a3a5-0b2a-4ab7-aea2-13379c20b3d0.pp --tags Ntp
Warning: Scope(Class[Keystone]): Execution of db_sync does not depend on $enabled anymore. Please use sync_db instead.
Warning: Scope(Class[Glance::Registry]): Execution of db_sync does not depend on $manage_service or $enabled anymore. Please use sync_db instead.
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Concat::Setup]): concat::setup is deprecated as a public API of the concat module and should no longer be directly included in the manifest.
Notice: Compiled catalog for ctrl00-prv.localdomain in environment production in 13.35 seconds
Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/content: content changed '{md5}c07b9a377faea45b96b7d3bf8976004b' to '{md5}7044a259392968d8dbe468670fd9c98c'
Notice: /File[/etc/ntp.conf]/seltype: seltype changed 'etc_t' to 'net_conf_t'
Notice: /Stage[main]/Ntp::Service/Service[ntp]: Triggered 'refresh' from 1 events
Notice: Finished catalog run in 7.80 seconds

$ sudo grep ^server /etc/ntp.conf 
server pool.ntp.org
Comment 6 chris alfonso 2016-01-04 12:02:21 EST
Please retest this on OSP 7.2 which is now released.
Comment 7 Dimitri Savineau 2016-01-04 16:28:59 EST
Exactly the same issue with OSP-d 7.2

Environment:
instack-undercloud-2.1.2-36.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-94.el7ost.noarch
openstack-puppet-modules-2015.1.8-32.el7ost.noarch
Comment 8 Alexander Chuzhoy 2016-01-25 11:47:23 EST
Verified:
Environment:
instack-undercloud-2.2.0-1.el7ost.noarch
openstack-heat-common-5.0.0-1.el7ost.noarch


The reported issue doesn't reproduce.
Comment 10 Dimitri Savineau 2016-03-04 14:41:43 EST
The bug doesn't seem to be present in 7.3 :

# grep ^server /etc/ntp.conf 
server pool.ntp.org

The big difference between 7.{0,1,2} and 7.3 is the file 50-ntp in /usr/libexec/os-refresh-config/configure.d/

# in 7.0/7.1/7.2
# ls -lh /usr/libexec/os-refresh-config/configure.d/
total 36K
-rwxr-xr-x. 1 root root  396 Dec 16 15:08 10-sysctl-apply-config
-rwxr-xr-x. 1 root root   42 Dec 16 15:08 20-os-apply-config
-rwxr-xr-x. 1 root root 1.6K Dec 16 15:08 20-os-net-config
-rwxr-xr-x. 1 root root  629 Dec 16 15:08 25-set-network-gateway
-rwxr-xr-x. 1 root root 2.8K Dec 16 15:08 40-hiera-datafiles
-rwxr-xr-x. 1 root root  969 Dec 16 15:08 50-ntp
-rwxr-xr-x. 1 root root 1.4K Dec 16 15:08 51-hosts
-rwxr-xr-x. 1 root root 6.1K Dec 16 15:08 55-heat-config

# in 7.3
# ls -lh /usr/libexec/os-refresh-config/configure.d/
total 32K
-rwxr-xr-x. 1 root root  396 Feb 16 22:20 10-sysctl-apply-config
-rwxr-xr-x. 1 root root   42 Feb 16 22:20 20-os-apply-config
-rwxr-xr-x. 1 root root 1.6K Feb 16 22:20 20-os-net-config
-rwxr-xr-x. 1 root root  629 Feb 16 22:20 25-set-network-gateway
-rwxr-xr-x. 1 root root 2.8K Feb 16 22:20 40-hiera-datafiles
-rwxr-xr-x. 1 root root 1.4K Feb 16 22:20 51-hosts
-rwxr-xr-x. 1 root root 6.1K Feb 16 22:20 55-heat-config

But the problem is still present if we upgrade from 7.x to 7.3 because the file 50-ntp is not deleted.

The ntp server is configured during the puppet part then erased by the resource 50-ntp

Mar 04 11:56:37 sc-ctrl00.localdomain os-collect-config[3696]: Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/content: content changed '{md5}c07b9a377faea45b96b7d3bf8976004b' to '{md5}7044a259392968d8dbe468670fd9c98c'
Mar 04 11:56:37 sc-ctrl00.localdomain os-collect-config[3696]: Notice: /File[/etc/ntp.conf]/seltype: seltype changed 'etc_t' to 'net_conf_t'
Mar 04 11:56:37 sc-ctrl00.localdomain os-collect-config[3696]: Notice: /Stage[main]/Ntp::Service/Service[ntp]: Triggered 'refresh' from 1 events

# grep ^server /etc/ntp.conf 
server pool.ntp.org

Mar 04 16:57:23 sc-ctrl00.localdomain os-collect-config[3696]: [2016/03/04 04:57:23 PM] [INFO] writing /etc/ntp.conf
Mar 04 16:57:47 sc-ctrl00.localdomain os-collect-config[3696]: dib-run-parts Fri Mar 4 16:57:47 UTC 2016 Running /usr/libexec/os-refresh-config/configure.d/50-ntp
Mar 04 16:57:47 sc-ctrl00.localdomain os-collect-config[3696]: dib-run-parts Fri Mar 4 16:57:47 UTC 2016 50-ntp completed

# grep ^server /etc/ntp.conf 
server 127.127.1.0 # local clock address
Comment 11 Nicolas Hicher 2016-03-14 10:23:21 EDT
Any update for a fix ? I have the same issue than Dimitri when I'm upgrading from 7.1 to 7.3.

Thanks.
Comment 12 Matt Wisch 2016-03-16 11:05:41 EDT
There is additional discusson on this issue in https://bugzilla.redhat.com/show_bug.cgi?id=1296365

Since the goal is to have Puppet configure and maintain ntp.conf, not os-config (and certainly not both as was the case), the configuration script was removed in the OSP 7.3 overcloud-full image.

The issue is that the images are only used in new deployments, or scale ups, so right now they are saying the solution may just be manually removing /usr/libexec/os-refresh-config/configure.d/50-ntp
Comment 14 errata-xmlrpc 2016-04-07 17:41:59 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0604.html

Note You need to log in before you can comment on or make changes to this bug.