DescriptionPURANDHAR SAIRAM MANNIDI
2016-12-13 01:35:59 UTC
Description of problem:
While deploying stack from heat, following error was observed
016-11-14 14:00:09.917 30197 INFO heat.engine.service [req-2b65a0ad-fc9e-4bf8-bb0f-246849078233 None] Creating stack Etcd-1.0.24-eDQ1ipYM1Ag
2016-11-14 14:00:30.874 30197 ERROR heat.engine.resource [-] DB error QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30
2016-11-14 14:00:31.138 30197 ERROR heat.engine.resource [-] Error marking resource as failed
The Customer wants to monitor the DB connection pool usage during the deployment
of templates using heat.
Customer was suggested with following command:
lsof -i4:3306 -sTCP:ESTABLISHED -c ^haproxy -c ^mysqld +c0 | sed '1d' | awk '{ print $1 }' | uniq -c | sort -k 2
Customer would like to have better approach to monitor DB connection pool.
Version-Release number of selected component (if applicable):
RH OSP 6.0
How reproducible:
Always
Steps to Reproduce:
1. deploy a stack with large template file
Actual results:
Horizon was inaccessible because of heat error mentioned above
Expected results:
Successful deployment of instances using templates
Additional info:
The issue was fixed by increasing max_pool_size=20 and max_overflow=30.
Customer still wants to monitor the DB connection pool.
Note that this was fixed in Mitaka (OSP 9) so that the max overflow size is always at least the size of the thread pool: http://git.openstack.org/cgit/openstack/heat/commit/?id=d7c2de60cb9ed0d3ddb077d603b7028f8d91d238
Heat uses Oslo libraries both for managing DB connections (oslo_db) and for providing debugging information at runtime (oslo_service). It wouldn't make sense to implement anything directly in Heat to report on connection usage, since that is not specific to Heat, so I'm changing the component to python-oslo-db.
Since the customer rejected your solution, perhaps they could elaborate on what they _would_ consider an acceptable solution?
Description of problem: While deploying stack from heat, following error was observed 016-11-14 14:00:09.917 30197 INFO heat.engine.service [req-2b65a0ad-fc9e-4bf8-bb0f-246849078233 None] Creating stack Etcd-1.0.24-eDQ1ipYM1Ag 2016-11-14 14:00:30.874 30197 ERROR heat.engine.resource [-] DB error QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30 2016-11-14 14:00:31.138 30197 ERROR heat.engine.resource [-] Error marking resource as failed The Customer wants to monitor the DB connection pool usage during the deployment of templates using heat. Customer was suggested with following command: lsof -i4:3306 -sTCP:ESTABLISHED -c ^haproxy -c ^mysqld +c0 | sed '1d' | awk '{ print $1 }' | uniq -c | sort -k 2 Customer would like to have better approach to monitor DB connection pool. Version-Release number of selected component (if applicable): RH OSP 6.0 How reproducible: Always Steps to Reproduce: 1. deploy a stack with large template file Actual results: Horizon was inaccessible because of heat error mentioned above Expected results: Successful deployment of instances using templates Additional info: The issue was fixed by increasing max_pool_size=20 and max_overflow=30. Customer still wants to monitor the DB connection pool.