Bug 1154592

Restarting qpidd didn't help , then restarted Cinder service - fixed issue.
FYI during Cinder restart these warnings came up on api.log:  

2014-10-20 13:53:21.619 8181 TRACE cinder.api.middleware.fault 
2014-10-20 13:53:31.680 122539 WARNING keystoneclient.middleware.auth_token [-] Configuring admin URI using auth fragments. This is deprecated, use 'identity_uri' instead.
2014-10-20 13:53:31.736 122539 WARNING keystoneclient.middleware.auth_token [-] Configuring admin URI using auth fragments. This is deprecated, use 'identity_uri' instead.
2014-10-20 13:53:31.738 122539 WARNING cinder.service [-] Value of config option osapi_volume_workers must be integer greater than 1.  Input value ignored.

Cinder list now works fine, Horizon volumes errors also gone.

It looks like the biggest problem is that Cinder API is having trouble talking to the SQL database:

"OperationalError: (OperationalError) (1040, 'Too many connections')" is repeatedly in the logs.

Cinder-volume also showed some problems talking to the AMQP server, but those are further in the past so I assume not relevant now.

How big is this deployment and how many database connections will the SQL server allow?  I assume this is a configuration/deployment issue in that area without more info to go on at this point.

Is this reproducable in that environment?

Deployment includes 6 hosts, fully distributed each node also runs nova compute services. At the moment running only 6 instances, it's also used for automation testing maybe that's what caused the spike. 

Can't reproduce issue, I have no idea what was done to cause this. 
I'll keep an eye open for it and try to figure out what caused this. 

How does one know\calculate a valid connection limit to set for a deployment? 

mysql> show status like 'Conn%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Connections   | 69684 |
+---------------+-------+
1 row in set (0.03 sec)

mysql> select @@max_connections;
+-------------------+
| @@max_connections |
+-------------------+
|               151 |
+-------------------+
1 row in set (0.01 sec)

So this just happened again on Blue, attaching latest cinder.api logs. 

[root@puma51 ~(keystone_admin)]# cinder list
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-bd80f6ed-7475-439e-9eab-8374678150af)

Checking connection on MYSQL

mysql> select @@max_connections;
+-------------------+           
| @@max_connections |           
+-------------------+           
|               151 |           
+-------------------+           
1 row in set (0.00 sec)         

mysql> show status like 'Conn%';
+---------------+--------+      
| Variable_name | Value  |      
+---------------+--------+      
| Connections   | 125571 |      
+---------------+--------+      
1 row in set (0.00 sec)         

List of process 
http://pastebin.test.redhat.com/243685


Now all of a sudden without doing anything #Cinder list works OK.

If there is anything else I should check/do/report if this happens again please let me know.

Created attachment 952904 [details]
Recent cinder api.log

So I had run #cinder --debug list 
It does work but I noticed at the end of the output 

+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 7b04d6a7-f26b-4ca8-bb75-a1ac4baecaa6 | available |              |  1   |     None    |  false   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
[root@puma51 ~(keystone_admin)]# [root@puma51 ~(keystone_admin)]# cinder list
-bash: syntax error near unexpected token `('
[root@puma51 ~(keystone_admin)]# ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-bd80f6ed-7475-439e-9eab-8374678150af)
-bash: syntax error near unexpected token `('
[root@puma51 ~(keystone_admin)]# [root@puma51 ~(keystone_admin)]# cinder list
-bash: syntax error near unexpected token `('
[root@puma51 ~(keystone_admin)]# ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-bd80f6ed-7475-439e-9eab-8374678150af)


So my problem isn't gone, cinder manages to return an output despite still having some sort of http500 error.  Attaching full --debug output.

Created attachment 952905 [details]
Cinder --debug list error

Tzahi, please, reopen if it still happens.