Bug 1233916

Summary: configure an ntp server on all overcloud nodes
Product: Red Hat OpenStack Reporter: Mike Burns <mburns>
Component: openstack-tripleo-puppet-elementsAssignee: James Slagle <jslagle>
Status: CLOSED ERRATA QA Contact: Leonid Natapov <lnatapov>
Severity: high Docs Contact:
Priority: high    
Version: DirectorCC: akrivoka, calfonso, dmacpher, dsneddon, ggillies, jslagle, mandreou, mburns, oblaut, ohochman, rhel-osp-director-maint, rrosa, ykawada, yrabl
Target Milestone: gaKeywords: Triaged
Target Release: Director   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-puppet-elements-0.0.1-3.el7ost Doc Type: Known Issue
Doc Text:
Overcloud nodes contained incorrectly synchronized system times. This resulted in various errors across the HA Controller cluster. As a workaround, pass the --ntp-server command line argument when running the "openstack overcloud deploy" command. This argument configures the ntp server in /etc/ntp.conf on each Overcloud node and starts the ntpd service. This produces properly synchronized system times and a successful Overcloud deployment.
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-05 13:54:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mike Burns 2015-06-19 17:21:10 UTC
Description of problem:
For HA, we must have ntp configured on the overcloud nodes.  This ability exists  in the CLI but needs testing.

Comment 1 Mike Burns 2015-06-19 17:22:20 UTC
Ana, can you add the options for setting the ntp server on the hosts?

Comment 2 Ofer Blaut 2015-06-25 15:36:10 UTC
no ntp rpm is installed on over cloud
no ntpstat 


[root@overcloud-controller-2 heat-admin]# rpm -qa | grep ntp
fontpackages-filesystem-1.44-8.el7.noarch

Comment 3 chris alfonso 2015-06-25 22:53:55 UTC
Is this the right package name?

Comment 4 Mike Burns 2015-06-25 23:15:51 UTC
It depends on what puppet is configuring.  There was a question of whether the image had chrony installed since that is the rhel 7 default, but it appears neither is installed today.

openstack-puppet-modules has many references to ntpd but no references to chrony, so this should be installing ntpd and ntpdate packages.

Comment 5 Marios Andreou 2015-06-26 15:32:52 UTC
I poked at this this afternoon and here are my notes, in case useful for anyone else, otherwise I will revisit on Monday,


* there is no ntp/ntpd package available for my rhel overcloud box and indeed chrony is available and already installed. 
* we are wired up to configure ntp (unified cli, heat templates etc)... like 
--ntp-server "0.fedora.pool.ntp.org" will make it so that we include ::ntp see https://github.com/rdo-management/python-rdomanager-oscplugin/blob/master/rdomanager_oscplugin/v1/overcloud_deploy.py#L610  and https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/manifests/overcloud_controller.pp#L43 
BUT
* when you do enable --ntp-server deploy fails because it tries to install ntp and there is no such thing on my box (no ntp package).--
* I am trying to trick ntp into configuring chrony (there is no official looking puppet-chrony), latest attempt looks like:

if count(hiera('ntp::servers')) > 0 { 
  class { 'ntp' :
    config      => "/etc/chrony.conf",
    keys_file   => "/etc/chrony.keys",
    service_name => "chronyd",
  } 
}

in the controller/compute hieradata (right now we just have include ::ntp) but that is failing like 

(i'll keep poking next week, unless someone else picks up)

4425:Jun 26 11:27:45 overcloud-controller-0.localdomain chronyd[20165]: Fatal error : Invalid command at line 7 in file /etc/chrony.conf
4559:Jun 26 11:28:31 overcloud-controller-0.localdomain os-collect-config[4368]: he package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.\n   (at /usr/share/ruby/vendor_ruby/puppet/type.rb:816:in `set_default')\u001b[0m\n\u001b[1;31mError: /Stage[main]/Ntp::Service/Service[ntp]: Failed to call refresh: Could not restart Service[ntp]: Execution of '/usr/bin/systemctl restart chronyd' returned 1: Job for chronyd.service failed. See 'systemctl status chronyd.service' and 'journalctl -xn' for details.\u001b[0m\n\u001b[1;31mError: /Stage[main]/Ntp::Service/Service[ntp]: Could not restart Service[ntp]: Execution of '/usr/bin/systemctl restart chronyd' returned 1: Job for chronyd.service failed. See 'systemctl status chronyd.service' and 'journalctl -xn' for details.\nWrapped exception:\nExecution of '/usr/bin/systemctl restart chronyd' returned 1: Job for chronyd.service failed. See 'systemctl status chronyd.service' and 'journalctl -xn' for details.\u001b[0m\n", "deploy_status_code": 6}
4660:Jun 26 11:28:31 overcloud-controller-0.localdomain os-collect-config[4368]: Error: /Stage[main]/Ntp::Service/Service[ntp]: Failed to call refresh: Could not restart Service[ntp]: Execution of '/usr/bin/systemctl restart chronyd' returned 1: Job for chronyd.service failed. See 'systemctl status chronyd.service' and 'journalctl -xn' for details.
4661:Jun 26 11:28:31 overcloud-controller-0.localdomain os-collect-config[4368]: Error: /Stage[main]/Ntp::Service/Service[ntp]: Could not restart Service[ntp]: Execution of '/usr/bin/systemctl restart chronyd' returned 1: Job for chronyd.service failed. See 'systemctl status chronyd.service' and 'journalctl -xn' for details.

Comment 6 Mike Burns 2015-06-26 15:55:24 UTC
chrony may be the default, but we don't have to use it.  It's probably simplest to include ntpd and ntpdate in the image and not include chrony.

FWIW, I didn't see chrony on my image (though I could certainly have typoed)

Comment 7 James Slagle 2015-06-26 21:17:28 UTC
(In reply to marios from comment #5)
> I poked at this this afternoon and here are my notes, in case useful for
> anyone else, otherwise I will revisit on Monday,
> 
> 
> * there is no ntp/ntpd package available for my rhel overcloud box and
> indeed chrony is available and already installed. 
> * we are wired up to configure ntp (unified cli, heat templates etc)... like 
> --ntp-server "0.fedora.pool.ntp.org" will make it so that we include ::ntp
> see
> https://github.com/rdo-management/python-rdomanager-oscplugin/blob/master/
> rdomanager_oscplugin/v1/overcloud_deploy.py#L610  and
> https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/
> manifests/overcloud_controller.pp#L43 
> BUT
> * when you do enable --ntp-server deploy fails because it tries to install
> ntp and there is no such thing on my box (no ntp package).--
> * I am trying to trick ntp into configuring chrony (there is no official
> looking puppet-chrony), latest attempt looks like:
> 
> if count(hiera('ntp::servers')) > 0 { 
>   class { 'ntp' :
>     config      => "/etc/chrony.conf",
>     keys_file   => "/etc/chrony.keys",
>     service_name => "chronyd",
>   } 
> }
> 
> in the controller/compute hieradata (right now we just have include ::ntp)
> but that is failing like 
> 
> (i'll keep poking next week, unless someone else picks up)
> 
> 4425:Jun 26 11:27:45 overcloud-controller-0.localdomain chronyd[20165]:
> Fatal error : Invalid command at line 7 in file /etc/chrony.conf
> 4559:Jun 26 11:28:31 overcloud-controller-0.localdomain
> os-collect-config[4368]: he package type's allow_virtual parameter will be
> changing its default value from false to true in a future release. If you do
> not want to allow virtual packages, please explicitly set allow_virtual to
> false.\n   (at /usr/share/ruby/vendor_ruby/puppet/type.rb:816:in
> `set_default')\u001b[0m\n\u001b[1;31mError:
> /Stage[main]/Ntp::Service/Service[ntp]: Failed to call refresh: Could not
> restart Service[ntp]: Execution of '/usr/bin/systemctl restart chronyd'
> returned 1: Job for chronyd.service failed. See 'systemctl status
> chronyd.service' and 'journalctl -xn' for
> details.\u001b[0m\n\u001b[1;31mError:
> /Stage[main]/Ntp::Service/Service[ntp]: Could not restart Service[ntp]:
> Execution of '/usr/bin/systemctl restart chronyd' returned 1: Job for
> chronyd.service failed. See 'systemctl status chronyd.service' and
> 'journalctl -xn' for details.\nWrapped exception:\nExecution of
> '/usr/bin/systemctl restart chronyd' returned 1: Job for chronyd.service
> failed. See 'systemctl status chronyd.service' and 'journalctl -xn' for
> details.\u001b[0m\n", "deploy_status_code": 6}
> 4660:Jun 26 11:28:31 overcloud-controller-0.localdomain
> os-collect-config[4368]: Error: /Stage[main]/Ntp::Service/Service[ntp]:
> Failed to call refresh: Could not restart Service[ntp]: Execution of
> '/usr/bin/systemctl restart chronyd' returned 1: Job for chronyd.service
> failed. See 'systemctl status chronyd.service' and 'journalctl -xn' for
> details.
> 4661:Jun 26 11:28:31 overcloud-controller-0.localdomain
> os-collect-config[4368]: Error: /Stage[main]/Ntp::Service/Service[ntp]:
> Could not restart Service[ntp]: Execution of '/usr/bin/systemctl restart
> chronyd' returned 1: Job for chronyd.service failed. See 'systemctl status
> chronyd.service' and 'journalctl -xn' for details.

marios, i'd probably just forgo trying to make puppet-ntp work with chrony. I've pushed up a patch to just install ntp on all the images.

Comment 8 Mike Burns 2015-06-27 22:57:56 UTC
*** Bug 1236338 has been marked as a duplicate of this bug. ***

Comment 9 Marios Andreou 2015-06-29 10:05:09 UTC
awesome thanks for the update

Comment 10 James Slagle 2015-06-29 11:49:15 UTC
*** Bug 1194571 has been marked as a duplicate of this bug. ***

Comment 11 Mike Burns 2015-06-30 11:56:56 UTC
*** Bug 1237108 has been marked as a duplicate of this bug. ***

Comment 13 Mike Burns 2015-07-02 11:14:53 UTC
*** Bug 1238528 has been marked as a duplicate of this bug. ***

Comment 14 Omri Hochman 2015-07-02 21:40:46 UTC
In order to verify the deployment command must be sent with --ntp-server 0.au.pool.ntp.org

Comment 21 Ofer Blaut 2015-07-19 09:07:17 UTC
Tested on controllers and computes

openstack-tripleo-puppet-elements-0.0.1-4.el7ost.noarch

Comment 24 errata-xmlrpc 2015-08-05 13:54:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549