Created attachment 989995 [details] neutron log Description of problem: We found out that neutron being restarted every 30 minutes by Puppet in environments deployed by staypuft (HA and non-ha), seems relate to puppet run "neutron-db-manage upgrade" Feb 10 08:43:47 mac441ea1733d43 systemd: Stopping OpenStack Neutron Server... Feb 10 08:43:48 mac441ea1733d43 systemd: Starting OpenStack Neutron Server... Feb 10 08:43:49 mac441ea1733d43 systemd: Started OpenStack Neutron Server. Feb 10 08:43:49 mac441ea1733d43 puppet-agent[27770]: (/Stage[main]/Neutron::Server/Service[neutron-server]) Triggered 'refresh' from 1 events Feb 10 08:43:49 mac441ea1733d43 puppet-agent[27770]: (/Stage[main]/Quickstack::Pacemaker::Neutron/Exec[pcs-neutron-server-set-up]/returns) executed successfully Feb 10 08:43:50 mac441ea1733d43 puppet-agent[27770]: (/Stage[main]/Quickstack::Pacemaker::Neutron/Exec[pcs-neutron-server-set-up-on-this-node]/returns) executed successfully Feb 10 08:43:51 mac441ea1733d43 puppet-agent[27770]: (/Stage[main]/Quickstack::Pacemaker::Neutron/Exec[all-neutron-nodes-are-up]/returns) executed successfully Feb 10 08:43:51 mac441ea1733d43 attrd[3078]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-neutron-server (1423089825) Feb 10 08:43:55 mac441ea1733d43 attrd[3078]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-neutron-server (1423089825) Feb 10 08:44:00 mac441ea1733d43 attrd[3078]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-neutron-server (1423089825) Feb 10 08:44:03 mac441ea1733d43 attrd[3078]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-neutron-server (1423089825) Feb 10 08:44:06 mac441ea1733d43 attrd[3078]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-neutron-server (1423089825) Version-Release number of selected component (if applicable): rhel-osp-installer-0.5.5-2.el7ost.noarch rhel-osp-installer-client-0.5.5-2.el7ost.noarch puppet-3.6.2-2.el7.noarch puppet-server-3.6.2-2.el7.noarch openstack-puppet-modules-2014.2.8-2.el7ost.noarch How reproducible: 3/3
reproduce steps : On controllers: check grep -ir "systemd: Stopping OpenStack Neutron Server" /var/log/messages check systemctl status neutron-server" Active time.
Pull request: https://github.com/redhat-openstack/astapor/pull/476
Workaround is to stop puppet-agent on controllers after RHEL-OSP installation
Merged. This patch resolved the retrigger of db-manage on each run, but there appears to be an additional trigger: Info: /Stage[main]/Neutron::Server::Notifications/Nova_admin_tenant_id_setter[nova_admin_tenant_id]: Scheduling refresh of Service[neutron-server This currently looks like a puppet-neutron bug, so there may be an additional BZ against opm needed to cover that.
Verified: failedQA Environment: rhel-osp-installer-client-0.5.5-5.el7ost.noarch ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch openstack-foreman-installer-3.0.15-1.el7ost.noarch rhel-osp-installer-0.5.5-5.el7ost.noarch ruby193-rubygem-staypuft-0.5.19-1.el7ost.noarch openstack-puppet-modules-2014.2.8-2.el7ost.noarch [root@maca25400702875 ~]# grep -ir "systemd: Stopping OpenStack Neutron Server" /var/log/messages Feb 17 14:54:10 maca25400702875 systemd: Stopping OpenStack Neutron Server... Feb 17 15:03:23 maca25400702875 systemd: Stopping OpenStack Neutron Server... Feb 17 15:25:04 maca25400702875 systemd: Stopping OpenStack Neutron Server... Feb 17 15:56:39 maca25400702875 systemd: Stopping OpenStack Neutron Server... Feb 17 16:26:41 maca25400702875 systemd: Stopping OpenStack Neutron Server... Feb 17 16:56:36 maca25400702875 systemd: Stopping OpenStack Neutron Server... Feb 17 17:26:35 maca25400702875 systemd: Stopping OpenStack Neutron Server... Feb 17 17:56:35 maca25400702875 systemd: Stopping OpenStack Neutron Server... Feb 17 18:26:30 maca25400702875 systemd: Stopping OpenStack Neutron Server... Feb 17 18:56:31 maca25400702875 systemd: Stopping OpenStack Neutron Server... Feb 17 19:26:29 maca25400702875 systemd: Stopping OpenStack Neutron Server... Feb 17 19:56:35 maca25400702875 systemd: Stopping OpenStack Neutron Server... Feb 17 20:26:37 maca25400702875 systemd: Stopping OpenStack Neutron Server... [root@maca25400702877 ~]# systemctl status neutron-server neutron-server.service - OpenStack Neutron Server Loaded: loaded (/usr/lib/systemd/system/neutron-server.service; disabled) Active: active (running) since Tue 2015-02-17 19:23:38 EST; 12min ago Main PID: 13644 (neutron-server) CGroup: /system.slice/neutron-server.service ├─13644 /usr/bin/python /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --log-file /var/log/ne... ├─13796 /usr/bin/python /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --log-file /var/log/ne... └─13797 /usr/bin/python /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --log-file /var/log/ne... Feb 17 19:23:36 maca25400702877.example.com systemd[1]: Starting OpenStack Neutron Server... Feb 17 19:23:38 maca25400702877.example.com systemd[1]: Started OpenStack Neutron Server. [root@maca25400702877 ~]# date Tue Feb 17 19:36:28 EST 2015
Here is the fix to puppet-neutron, I don't know if a BZ is needed to get it into a release of opm: https://review.openstack.org/#/c/156777/
Bug 1193889 filed for OPM to get that fix in. Until it does, we won't see the restart every 30 second issue go away. I'm updating the summary for this bug to reflect what was actually fixed. To test/verify this bug, check the 30 minute runs and ensure that neutron db-manage does not execute every puppet re-run. Moving back to ON_QA to verify just this specific issue.
Verified: Environment: ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch rhel-osp-installer-client-0.5.5-5.el7ost.noarch openstack-foreman-installer-3.0.16-1.el7ost.noarch rhel-osp-installer-0.5.5-5.el7ost.noarch ruby193-rubygem-staypuft-0.5.19-1.el7ost.noarch openstack-puppet-modules-2014.2.8-2.el7ost.noarch Based on comment #13: Checked the last puppet report and verified there's no mentioning of neutron db-manage.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-0641.html