Bug 1060709

Summary:

neutron-server start should survive lack of db availability

Product:

Red Hat OpenStack

Reporter:

Fabio Massimo Di Nitto <fdinitto>

Component:

openstack-neutron

Assignee:

Jakub Libosvar <jlibosva>

Status:

CLOSED ERRATA

QA Contact:

Roey Dekel <rdekel>

Severity:

urgent

Docs Contact:

Priority:

urgent

Version:

4.0

CC:

chrisw, fdinitto, jlibosva, lpeer, majopela, nyechiel, oblaut, yeylon

Target Milestone:

Keywords:

ZStream

Target Release:

4.0

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

openstack-neutron-2013.2.2-9.el6ost

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2014-05-29 20:23:43 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

1080561

Attachments:

Description	Flags
extra init script	none
Proposed patch for adding spec file	none
DB check patch to spec file (that goes together with the neutron-watch-agent)	none
extra init script	none

Description Fabio Massimo Di Nitto 2014-02-03 12:40:47 UTC

In a scalable/high available environment, there is no guarantee of service startup order and availability of all services during startup (some services could start or be visible at a later stage in the boot process).

In some cases:

mysql starts on node X
neutron-server starts on node Y

but in other cases:
neutron-server starts on node A -> will fail because mysql is not started yet
mysql is scheduled to start on node B

All services in openstack are capable in one way or another to survive this scenario. neutron-server runtime should continue attempting to connect to mysql even if at startup is not immediately available, and at the first connection perform db-check.

Comment 1 Fabio Massimo Di Nitto 2014-02-05 10:16:52 UTC

Created attachment 859563 [details]
extra init script

I need the file in attachment to be shipped as part of a neutron package that is installed together with neutron-server and any of the agent.

From a spec file perspective it must be installed in %{initdir}, chkconfig --add and off by default.

I can take care of everything else from pacemaker.

Comment 2 Fabio Massimo Di Nitto 2014-02-05 10:19:40 UTC

Unit test for QE:

mysql is running on configured host:

[root@rhos4-neutron1 ~]# /etc/init.d/neutron-db-check start
[root@rhos4-neutron1 ~]# echo $?
0

[root@rhos4-neutron1 ~]# /etc/init.d/neutron-db-check start
neutron-db-check: ERROR: unable to find version information for the neutron database. Check your logs for additional information.
[root@rhos4-neutron1 ~]# echo $?
1

at first it appears to duplicate information and checks from neutron-server init script, but this shorter version allows pacemaker to decouple check and server.

pacemaker is then configured to loop specifically on db-check and perform the correct action on "start failure".

Comment 3 Fabio Massimo Di Nitto 2014-02-05 10:21:44 UTC

Spec file testing should:

[root@rhos4-neutron1 ~]# rpm -q -l openstack-neutron |grep neutron-db-check
/usr/bin/neutron-db-check
/etc/init.d/neutron-db-check

after installing the package:

[root@rhos4-neutron1 ~]# chkconfig --list|grep neutron-db
neutron-db-check    0:off   1:off   2:off   3:off   4:off   5:off   6:off

Comment 4 Jakub Libosvar 2014-02-05 12:24:39 UTC

Created attachment 859625 [details]
Proposed patch for adding spec file

Comment 6 Miguel Angel Ajo 2014-03-28 15:06:50 UTC

Created attachment 879894 [details]
DB check patch to spec file (that goes together with the neutron-watch-agent)

Comment 7 Miguel Angel Ajo 2014-03-28 15:19:24 UTC

Created attachment 879910 [details]
extra init script

Comment 13 errata-xmlrpc 2014-05-29 20:23:43 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-0516.html