Bug 1276063
| Summary: | rhel-osp-director: The time isn't synced on all overcloud nodes, deployed with the --ntp-server argument. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> |
| Component: | rhosp-director | Assignee: | Hugh Brock <hbrock> |
| Status: | CLOSED ERRATA | QA Contact: | Alexander Chuzhoy <sasha> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 7.0 (Kilo) | CC: | dsavinea, karatecletus323, kbasil, mburns, mcornea, nhicher, ohochman, rhel-osp-director-maint, sasha |
| Target Milestone: | ga | Keywords: | Reopened, TestOnly |
| Target Release: | 8.0 (Liberty) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-04-07 21:41:59 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Sasha, can you please provide puppet logs? Thanks Closing for now, if you reproduce, please reopen. I have the same issue on an OSP-d 7.1 deployment
Environment:
instack-undercloud-2.1.2-29.el7ost.noarch
openstack-puppet-modules-2015.1.8-21.el7ost.noarch
openstack overcloud deploy --templates ~/templates/ -e ~/env.yaml -e ~/templates/environments/storage-environment.yaml -e ~/templates/environments/network-isolation.yaml --control-scale 3 --control-flavor control --compute-scale 1 --compute-flavor compute --ceph-storage-scale 1 --ceph-storage-flavor ceph-storage --swift-storage-scale 0 --swift-storage-flavor swift-storage --block-storage-scale 0 --neutron-network-type vxlan --neutron-tunnel-types vxlan --neutron-vni-ranges 1:10000 --neutron-bridge-mappings datacentre:br-bond --libvirt-type kvm --ntp-server pool.ntp.org -t 120
On overcloud side the value in hiera is right :
$ sudo hiera ntp::servers
["pool.ntp.org"]
But the ntp configuration is not apply with this value.
$ sudo grep ^server /etc/ntp.conf
server 127.127.1.0 # local clock address
In the puppet logs we can see the changes but there are not applied on each puppet run (or maybe something rollback the changes after)
$ heat deployment-output-show <physical_resource_id> deploy_stdout|grep -i ntp
Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/content: content changed '{md5}c07b9a377faea45b96b7d3bf8976004b' to '{md5}7044a259392968d8dbe468670fd9c98c'
Notice: /File[/etc/ntp.conf]/seltype: seltype changed 'etc_t' to 'net_conf_t'
Notice: /Stage[main]/Ntp::Service/Service[ntp]/ensure: ensure changed 'stopped' to 'running'
Running the puppet command by hand only for ntp :
$ sudo FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/1ac2a3a5-0b2a-4ab7-aea2-13379c20b3d0" FACTER_fqdn="ctrl00-prv.localdomain" FACTER_deploy_config_name="overcloud-ControllerNodesPostDeployment-l2fejf76ersk-ControllerPuppetConfig-bcgn5k5qg3vt-ControllerPuppetConfigImpl-4p6ejshk2opq" puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/1ac2a3a5-0b2a-4ab7-aea2-13379c20b3d0.pp --tags Ntp
Warning: Scope(Class[Keystone]): Execution of db_sync does not depend on $enabled anymore. Please use sync_db instead.
Warning: Scope(Class[Glance::Registry]): Execution of db_sync does not depend on $manage_service or $enabled anymore. Please use sync_db instead.
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_host'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_protocol'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_port'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::compute::vncproxy_path'; class ::nova::compute has not been evaluated
Warning: Scope(Class[Concat::Setup]): concat::setup is deprecated as a public API of the concat module and should no longer be directly included in the manifest.
Notice: Compiled catalog for ctrl00-prv.localdomain in environment production in 13.35 seconds
Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/content: content changed '{md5}c07b9a377faea45b96b7d3bf8976004b' to '{md5}7044a259392968d8dbe468670fd9c98c'
Notice: /File[/etc/ntp.conf]/seltype: seltype changed 'etc_t' to 'net_conf_t'
Notice: /Stage[main]/Ntp::Service/Service[ntp]: Triggered 'refresh' from 1 events
Notice: Finished catalog run in 7.80 seconds
$ sudo grep ^server /etc/ntp.conf
server pool.ntp.org
Please retest this on OSP 7.2 which is now released. Exactly the same issue with OSP-d 7.2 Environment: instack-undercloud-2.1.2-36.el7ost.noarch openstack-tripleo-heat-templates-0.8.6-94.el7ost.noarch openstack-puppet-modules-2015.1.8-32.el7ost.noarch Verified: Environment: instack-undercloud-2.2.0-1.el7ost.noarch openstack-heat-common-5.0.0-1.el7ost.noarch The reported issue doesn't reproduce. The bug doesn't seem to be present in 7.3 :
# grep ^server /etc/ntp.conf
server pool.ntp.org
The big difference between 7.{0,1,2} and 7.3 is the file 50-ntp in /usr/libexec/os-refresh-config/configure.d/
# in 7.0/7.1/7.2
# ls -lh /usr/libexec/os-refresh-config/configure.d/
total 36K
-rwxr-xr-x. 1 root root 396 Dec 16 15:08 10-sysctl-apply-config
-rwxr-xr-x. 1 root root 42 Dec 16 15:08 20-os-apply-config
-rwxr-xr-x. 1 root root 1.6K Dec 16 15:08 20-os-net-config
-rwxr-xr-x. 1 root root 629 Dec 16 15:08 25-set-network-gateway
-rwxr-xr-x. 1 root root 2.8K Dec 16 15:08 40-hiera-datafiles
-rwxr-xr-x. 1 root root 969 Dec 16 15:08 50-ntp
-rwxr-xr-x. 1 root root 1.4K Dec 16 15:08 51-hosts
-rwxr-xr-x. 1 root root 6.1K Dec 16 15:08 55-heat-config
# in 7.3
# ls -lh /usr/libexec/os-refresh-config/configure.d/
total 32K
-rwxr-xr-x. 1 root root 396 Feb 16 22:20 10-sysctl-apply-config
-rwxr-xr-x. 1 root root 42 Feb 16 22:20 20-os-apply-config
-rwxr-xr-x. 1 root root 1.6K Feb 16 22:20 20-os-net-config
-rwxr-xr-x. 1 root root 629 Feb 16 22:20 25-set-network-gateway
-rwxr-xr-x. 1 root root 2.8K Feb 16 22:20 40-hiera-datafiles
-rwxr-xr-x. 1 root root 1.4K Feb 16 22:20 51-hosts
-rwxr-xr-x. 1 root root 6.1K Feb 16 22:20 55-heat-config
But the problem is still present if we upgrade from 7.x to 7.3 because the file 50-ntp is not deleted.
The ntp server is configured during the puppet part then erased by the resource 50-ntp
Mar 04 11:56:37 sc-ctrl00.localdomain os-collect-config[3696]: Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/content: content changed '{md5}c07b9a377faea45b96b7d3bf8976004b' to '{md5}7044a259392968d8dbe468670fd9c98c'
Mar 04 11:56:37 sc-ctrl00.localdomain os-collect-config[3696]: Notice: /File[/etc/ntp.conf]/seltype: seltype changed 'etc_t' to 'net_conf_t'
Mar 04 11:56:37 sc-ctrl00.localdomain os-collect-config[3696]: Notice: /Stage[main]/Ntp::Service/Service[ntp]: Triggered 'refresh' from 1 events
# grep ^server /etc/ntp.conf
server pool.ntp.org
Mar 04 16:57:23 sc-ctrl00.localdomain os-collect-config[3696]: [2016/03/04 04:57:23 PM] [INFO] writing /etc/ntp.conf
Mar 04 16:57:47 sc-ctrl00.localdomain os-collect-config[3696]: dib-run-parts Fri Mar 4 16:57:47 UTC 2016 Running /usr/libexec/os-refresh-config/configure.d/50-ntp
Mar 04 16:57:47 sc-ctrl00.localdomain os-collect-config[3696]: dib-run-parts Fri Mar 4 16:57:47 UTC 2016 50-ntp completed
# grep ^server /etc/ntp.conf
server 127.127.1.0 # local clock address
Any update for a fix ? I have the same issue than Dimitri when I'm upgrading from 7.1 to 7.3. Thanks. There is additional discusson on this issue in https://bugzilla.redhat.com/show_bug.cgi?id=1296365 Since the goal is to have Puppet configure and maintain ntp.conf, not os-config (and certainly not both as was the case), the configuration script was removed in the OSP 7.3 overcloud-full image. The issue is that the images are only used in new deployments, or scale ups, so right now they are saying the solution may just be manually removing /usr/libexec/os-refresh-config/configure.d/50-ntp Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-0604.html |
rhel-osp-director: The time isn't synced on all overcloud nodes, deployed with the --ntp-server argument. Environment: instack-undercloud-2.1.2-29.el7ost.noarch openstack-heat-common-2015.1.1-5.el7ost.noarch Steps to reproduce: Attempt to deploy overcloud on nodes with skewed time, using the --ntp-server arg. Result: The time isn't synced with the ntp server. Expected result: The time on all nodes should be synced with the ntp server specified with --ntp-server. Notes: Runing ntpdate <server> manually on the nodes adjusts the time properly, so it seems like the command isn't triggered. Also, Looking at /usr/libexec/os-refresh-config/configure.d/50-ntp that's mentioned in the os-collect-config logs: NTP_SERVER="$(os-apply-config --key 'ntp.servers.0.server' --type raw --key-default '')" if [ -n "${NTP_SERVER}" ]; then Running "os-apply-config --key 'ntp.servers.0.server' --type raw --key-default ''" results in empty string, so the condition isn't true.