Bug 1190990
| Summary: | don't run neutron db-manage on every puppet run | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Asaf Hirshberg <ahirshbe> | ||||
| Component: | openstack-foreman-installer | Assignee: | Jason Guiditta <jguiditt> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Alexander Chuzhoy <sasha> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 6.0 (Juno) | CC: | aberezin, dmacpher, mburns, morazi, oblaut, ohochman, rhos-maint, sasha, yeylon | ||||
| Target Milestone: | z1 | Keywords: | ZStream | ||||
| Target Release: | Installer | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-foreman-installer-3.0.15-1.el7ost | Doc Type: | Bug Fix | ||||
| Doc Text: |
The neutron-db-manage command ran on each Puppet run, which triggered a service restart of Neutron services. This fix moves quickstack Puppet code to use sync_db flag from puppet-neutron, which executes neutron-db-manage only once.
|
Story Points: | --- | ||||
| Clone Of: | |||||||
| : | 1193889 (view as bug list) | Environment: | |||||
| Last Closed: | 2015-03-05 18:20:10 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1193889 | ||||||
| Attachments: |
|
||||||
reproduce steps : On controllers: check grep -ir "systemd: Stopping OpenStack Neutron Server" /var/log/messages check systemctl status neutron-server" Active time. Pull request: https://github.com/redhat-openstack/astapor/pull/476 Workaround is to stop puppet-agent on controllers after RHEL-OSP installation Merged. This patch resolved the retrigger of db-manage on each run, but there appears to be an additional trigger: Info: /Stage[main]/Neutron::Server::Notifications/Nova_admin_tenant_id_setter[nova_admin_tenant_id]: Scheduling refresh of Service[neutron-server This currently looks like a puppet-neutron bug, so there may be an additional BZ against opm needed to cover that. Verified: failedQA
Environment:
rhel-osp-installer-client-0.5.5-5.el7ost.noarch
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch
openstack-foreman-installer-3.0.15-1.el7ost.noarch
rhel-osp-installer-0.5.5-5.el7ost.noarch
ruby193-rubygem-staypuft-0.5.19-1.el7ost.noarch
openstack-puppet-modules-2014.2.8-2.el7ost.noarch
[root@maca25400702875 ~]# grep -ir "systemd: Stopping OpenStack Neutron Server" /var/log/messages
Feb 17 14:54:10 maca25400702875 systemd: Stopping OpenStack Neutron Server...
Feb 17 15:03:23 maca25400702875 systemd: Stopping OpenStack Neutron Server...
Feb 17 15:25:04 maca25400702875 systemd: Stopping OpenStack Neutron Server...
Feb 17 15:56:39 maca25400702875 systemd: Stopping OpenStack Neutron Server...
Feb 17 16:26:41 maca25400702875 systemd: Stopping OpenStack Neutron Server...
Feb 17 16:56:36 maca25400702875 systemd: Stopping OpenStack Neutron Server...
Feb 17 17:26:35 maca25400702875 systemd: Stopping OpenStack Neutron Server...
Feb 17 17:56:35 maca25400702875 systemd: Stopping OpenStack Neutron Server...
Feb 17 18:26:30 maca25400702875 systemd: Stopping OpenStack Neutron Server...
Feb 17 18:56:31 maca25400702875 systemd: Stopping OpenStack Neutron Server...
Feb 17 19:26:29 maca25400702875 systemd: Stopping OpenStack Neutron Server...
Feb 17 19:56:35 maca25400702875 systemd: Stopping OpenStack Neutron Server...
Feb 17 20:26:37 maca25400702875 systemd: Stopping OpenStack Neutron Server...
[root@maca25400702877 ~]# systemctl status neutron-server
neutron-server.service - OpenStack Neutron Server
Loaded: loaded (/usr/lib/systemd/system/neutron-server.service; disabled)
Active: active (running) since Tue 2015-02-17 19:23:38 EST; 12min ago
Main PID: 13644 (neutron-server)
CGroup: /system.slice/neutron-server.service
├─13644 /usr/bin/python /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --log-file /var/log/ne...
├─13796 /usr/bin/python /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --log-file /var/log/ne...
└─13797 /usr/bin/python /usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --log-file /var/log/ne...
Feb 17 19:23:36 maca25400702877.example.com systemd[1]: Starting OpenStack Neutron Server...
Feb 17 19:23:38 maca25400702877.example.com systemd[1]: Started OpenStack Neutron Server.
[root@maca25400702877 ~]# date
Tue Feb 17 19:36:28 EST 2015
Here is the fix to puppet-neutron, I don't know if a BZ is needed to get it into a release of opm: https://review.openstack.org/#/c/156777/ Bug 1193889 filed for OPM to get that fix in. Until it does, we won't see the restart every 30 second issue go away. I'm updating the summary for this bug to reflect what was actually fixed. To test/verify this bug, check the 30 minute runs and ensure that neutron db-manage does not execute every puppet re-run. Moving back to ON_QA to verify just this specific issue. Verified: Environment: ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch rhel-osp-installer-client-0.5.5-5.el7ost.noarch openstack-foreman-installer-3.0.16-1.el7ost.noarch rhel-osp-installer-0.5.5-5.el7ost.noarch ruby193-rubygem-staypuft-0.5.19-1.el7ost.noarch openstack-puppet-modules-2014.2.8-2.el7ost.noarch Based on comment #13: Checked the last puppet report and verified there's no mentioning of neutron db-manage. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-0641.html |
Created attachment 989995 [details] neutron log Description of problem: We found out that neutron being restarted every 30 minutes by Puppet in environments deployed by staypuft (HA and non-ha), seems relate to puppet run "neutron-db-manage upgrade" Feb 10 08:43:47 mac441ea1733d43 systemd: Stopping OpenStack Neutron Server... Feb 10 08:43:48 mac441ea1733d43 systemd: Starting OpenStack Neutron Server... Feb 10 08:43:49 mac441ea1733d43 systemd: Started OpenStack Neutron Server. Feb 10 08:43:49 mac441ea1733d43 puppet-agent[27770]: (/Stage[main]/Neutron::Server/Service[neutron-server]) Triggered 'refresh' from 1 events Feb 10 08:43:49 mac441ea1733d43 puppet-agent[27770]: (/Stage[main]/Quickstack::Pacemaker::Neutron/Exec[pcs-neutron-server-set-up]/returns) executed successfully Feb 10 08:43:50 mac441ea1733d43 puppet-agent[27770]: (/Stage[main]/Quickstack::Pacemaker::Neutron/Exec[pcs-neutron-server-set-up-on-this-node]/returns) executed successfully Feb 10 08:43:51 mac441ea1733d43 puppet-agent[27770]: (/Stage[main]/Quickstack::Pacemaker::Neutron/Exec[all-neutron-nodes-are-up]/returns) executed successfully Feb 10 08:43:51 mac441ea1733d43 attrd[3078]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-neutron-server (1423089825) Feb 10 08:43:55 mac441ea1733d43 attrd[3078]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-neutron-server (1423089825) Feb 10 08:44:00 mac441ea1733d43 attrd[3078]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-neutron-server (1423089825) Feb 10 08:44:03 mac441ea1733d43 attrd[3078]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-neutron-server (1423089825) Feb 10 08:44:06 mac441ea1733d43 attrd[3078]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-neutron-server (1423089825) Version-Release number of selected component (if applicable): rhel-osp-installer-0.5.5-2.el7ost.noarch rhel-osp-installer-client-0.5.5-2.el7ost.noarch puppet-3.6.2-2.el7.noarch puppet-server-3.6.2-2.el7.noarch openstack-puppet-modules-2014.2.8-2.el7ost.noarch How reproducible: 3/3