Bug 1049519
| Summary: | Nova 500 error: Connection to neutron failed: Maximum attempts reached | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Community] RDO | Reporter: | Steven Hardy <shardy> | ||||||||
| Component: | openstack-neutron | Assignee: | Ihar Hrachyshka <ihrachys> | ||||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Ofer Blaut <oblaut> | ||||||||
| Severity: | unspecified | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | unspecified | CC: | chrisw, ihrachys, lars, mrunge, ndipanov, rbryant, shardy, thomasch, yeylon | ||||||||
| Target Milestone: | --- | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2014-07-17 12:59:36 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
|
Description
Steven Hardy
2014-01-07 16:49:16 UTC
Created attachment 846778 [details]
Example heat template
Some further investigation - it seems that the neutron-server service had crashed: 2014-01-07 17:20:04.219 2159 CRITICAL neutron [-] (OperationalError) (2003, "Can't connect to MySQL server on '192.168.0.11' (101)") None None Full trace attached (from /var/log/neutron/server.log). I'm not sure what caused that, but re-starting neutron-server allows neutron and nova to work. Created attachment 846815 [details]
neutron server log error
Reassigning to openstack-neutron since this seems to be an issue with neutron-server not nova. I'm not sure why neutron couldn't connect to the mysql DB when other services were working OK. Also I'm unsure why everything worked OK after re-starting neutron-server (with no changes to the configuration). Strangely this problem recurred a couple of times after reboot, then I upgraded python-wsme to the latest version from updates-testing (to work around a ceilometer-api issue), and now neutron seems to start OK after reboot (other than the qpid issue reported under bug #1049488) Steve, Was this on a --allinone install? Or was neutron-server on a separate system? When neutron-server was reporting errors connecting to the database, were you table to connect using the credentials from /etc/neutron/plugin.ini and the mysql command line client? It's hard to tell what went wrong there. We probably need a detailed description of the setup. How many nodes? Where are Neutron and MySQL located? iptables from both nodes? Does ping work? Do MySQL credentials from the config file work with mysql client? Steven, can you describe that one? Without all that information, the only thing we can do is asking you to retest on latest Havana packages. Reproduced this bug again with Openstack Juno RDO install on CentOS7/minimal Server/ All-in-one install. ( this install is documented on : https://ask.openstack.org/en/question/48329/openstack-juno-using-rdo-fails-installation-amqp-server-closed-the-connection/ ) When the machine is rebooted.. the Neutron Server starts however it cannot connect to the MariaDB MYSQL Server as indicated in the log : [root@xCentStack neutron(keystone_admin)]# tail -f server.log 2014-09-30 16:31:18.869 1428 TRACE neutron File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 90, in connect 2014-09-30 16:31:18.869 1428 TRACE neutron return dialect.connect(*cargs, **cparams) 2014-09-30 16:31:18.869 1428 TRACE neutron File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 377, in connect 2014-09-30 16:31:18.869 1428 TRACE neutron return self.dbapi.connect(*cargs, **cparams) 2014-09-30 16:31:18.869 1428 TRACE neutron File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in Connect 2014-09-30 16:31:18.869 1428 TRACE neutron return Connection(*args, **kwargs) 2014-09-30 16:31:18.869 1428 TRACE neutron File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in __init__ 2014-09-30 16:31:18.869 1428 TRACE neutron super(Connection, self).__init__(*args, **kwargs2) 2014-09-30 16:31:18.869 1428 TRACE neutron OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.168.0.20' (101)") None None 2014-09-30 16:31:18.869 1428 TRACE neutron ^C // Restart the Neutron Server manually [root@xCentStack neutron(keystone_admin)]# sudo service neutron-server restart Redirecting to /bin/systemctl restart neutron-server.service // Now it again connects, as indicated in the log [root@xCentStack neutron(keystone_admin)]# tail -f server.log 2014-09-30 16:43:11.157 7437 INFO neutron.api.extensions [-] Extension 'Allowed Address Pairs' provides no backward compatibility map for extended attributes 2014-09-30 16:43:11.158 7437 INFO neutron.api.extensions [-] Extension 'Distributed Virtual Router' provides no backward compatibility map for extended attributes 2014-09-30 16:43:11.158 7437 INFO neutron.api.extensions [-] Extension 'Neutron Extra Route' provides no backward compatibility map for extended attributes 2014-09-30 16:43:11.180 7437 INFO keystonemiddleware.auth_token [-] Starting keystone auth_token middleware 2014-09-30 16:43:11.180 7437 WARNING keystonemiddleware.auth_token [-] Configuring admin URI using auth fragments. This is deprecated, use 'identity_uri' instead. 2014-09-30 16:43:11.182 7437 INFO keystonemiddleware.auth_token [-] Using /tmp/keystone-signing-NHA9nW as cache directory for signing certificate 2014-09-30 16:43:11.235 7437 INFO neutron.service [-] Neutron service started, listening on 0.0.0.0:9696 2014-09-30 16:43:11.236 7437 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on 192.168.0.20:5672 2014-09-30 16:43:11.246 7437 INFO neutron.wsgi [-] (7437) wsgi starting up on http://0.0.0.0:9696/ 2014-09-30 16:43:11.250 7437 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 192.168.0.20:5672 This may be relevant to bug 1144181. Thomas, could you please provide details in bug 1144181? *** This bug has been marked as a duplicate of bug 1144181 *** |