1542538 – octavia-housekeeping cannot connect to database

Bug 1542538 - octavia-housekeeping cannot connect to database

Summary: octavia-housekeeping cannot connect to database

Keywords:
Status:	CLOSED WORKSFORME
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-octavia
Sub Component:
Version:	13.0 (Queens)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	---
Assignee:	Carlos Goncalves
QA Contact:	Alexander Stafeyev
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-02-06 14:40 UTC by Carlos Goncalves
Modified:	2019-09-10 14:09 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-02-20 10:10:07 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Carlos Goncalves 2018-02-06 14:40:14 UTC

Description of problem: octavia-housekeeping cannot connect to DB.


Version-Release number of selected component (if applicable): docker.io/tripleoupstream/centos-binary-octavia-housekeeping:latest (as of January 30).


How reproducible: containerized RDO deployment


Steps to Reproduce:
1. deploy overcloud with oooq and latest centos-binary-* docker images from registry docker.io/tripleoupstream


openstack overcloud deploy \
    --templates /usr/share/openstack-tripleo-heat-templates \
    --libvirt-type qemu --control-flavor oooq_control --compute-flavor oooq_compute --ceph-storage-flavor oooq_ceph --block-storage-flavor oooq_blockstorage --swift-storage-flavor oooq_objectstorage --timeout 90  -e /home/stack/cloud-names.yaml   -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml  -e /home/stack/containers-default-parameters.yaml  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /home/stack/network-environment.yaml  -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml   -e /home/stack/inject-trust-anchor.yaml   -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml   --validation-warnings-fatal     --ntp-server pool.ntp.org -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \
    ${DEPLOY_ENV_YAML:+-e $DEPLOY_ENV_YAML} "$@" && status_code=0 || status_code=$?


2. [root@overcloud-controller-0 heat-admin]# id=$(docker ps | grep octavia-housekeeping | awk {'print $1'})
3. [root@overcloud-controller-0 heat-admin]# docker exec -i $id cat /var/log/octavia/housekeeping.log

Actual results:

$ cat /var/log/octavia/housekeeping.log  
2018-01-30 16:58:35.562 5 INFO octavia.common.config [-] Logging enabled!
2018-01-30 16:58:35.562 5 INFO octavia.common.config [-] /usr/bin/octavia-housekeeping version 2.0.0.0b2.dev70
2018-01-30 16:58:35.562 5 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2018-01-30 16:58:35.562579
2018-01-30 16:58:35.562 5 INFO octavia.cmd.house_keeping [-] Spare check interval is set to 30 sec
2018-01-30 16:58:35.655 5 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec
2018-01-30 16:58:35.655 5 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds
2018-01-30 16:58:35.655 5 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds
2018-01-30 16:58:35.659 5 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec
2018-01-30 17:03:29.558 5 INFO octavia.common.config [-] Logging enabled!
2018-01-30 17:03:29.559 5 INFO octavia.common.config [-] /usr/bin/octavia-housekeeping version 2.0.0.0b2.dev70
2018-01-30 17:03:29.559 5 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2018-01-30 17:03:29.559343
2018-01-30 17:03:29.560 5 INFO octavia.cmd.house_keeping [-] Spare check interval is set to 30 sec
2018-01-30 17:03:29.599 5 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec
2018-01-30 17:03:29.599 5 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds
2018-01-30 17:03:29.600 5 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds
2018-01-30 17:03:29.612 5 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec
2018-02-01 10:11:21.815 5 INFO octavia.common.config [-] Logging enabled!
2018-02-01 10:11:21.843 5 INFO octavia.common.config [-] /usr/bin/octavia-housekeeping version 2.0.0.0b2.dev70
2018-02-01 10:11:21.844 5 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2018-02-01 10:11:21.844259
2018-02-01 10:11:21.844 5 INFO octavia.cmd.house_keeping [-] Spare check interval is set to 30 sec
2018-02-01 10:11:22.065 5 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec
2018-02-01 10:11:22.065 5 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds
2018-02-01 10:11:22.065 5 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds
2018-02-01 10:11:22.110 5 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '172.16.2.12' ([Errno 111] Connection refused)")
2018-02-01 10:11:22.111 5 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '172.16.2.12' ([Errno 111] Connection refused)")
2018-02-01 10:11:22.112 5 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec
2018-02-01 10:11:22.170 5 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '172.16.2.12' ([Errno 111] Connection refused)")
2018-02-01 12:18:30.976 5 INFO octavia.common.config [-] Logging enabled!
2018-02-01 12:18:30.999 5 INFO octavia.common.config [-] /usr/bin/octavia-housekeeping version 2.0.0.0b2.dev70
2018-02-01 12:18:31.000 5 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2018-02-01 12:18:31.000201
2018-02-01 12:18:31.002 5 INFO octavia.cmd.house_keeping [-] Spare check interval is set to 30 sec
2018-02-01 12:18:31.288 5 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec
2018-02-01 12:18:31.288 5 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds
2018-02-01 12:18:31.289 5 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds
2018-02-01 12:18:31.294 5 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
2018-02-01 12:18:31.298 5 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec
2018-02-01 12:18:31.304 5 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
2018-02-01 12:18:31.304 5 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')

Expected results:

octavia-housekeeping


Additional info:

Bug may not be observed in OSP as this was tested under a master upstream deployment. Yet reporting so that we can follow-up and confirm.

Comment 1 Carlos Goncalves 2018-02-06 14:43:11 UTC

Alex, could you verify this whenever you have an octavia-enabled OSP 13 env set up?

Comment 2 Alexander Stafeyev 2018-02-14 12:49:11 UTC

Yes I will try to reproduce this one

Comment 3 Alexander Stafeyev 2018-02-14 12:53:04 UTC

So far so good 
[root@controller-0 ~]# docker exec -i $id cat /var/log/octavia/housekeeping.log
2018-02-14 11:51:02.337 1 INFO octavia.common.config [-] Logging enabled!
2018-02-14 11:51:02.338 1 INFO octavia.common.config [-] /usr/bin/octavia-housekeeping version 2.0.0.0b3.dev60
2018-02-14 11:51:02.338 1 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2018-02-14 11:51:02.338402
2018-02-14 11:51:02.338 1 INFO octavia.cmd.house_keeping [-] Spare check interval is set to 30 sec
2018-02-14 11:51:02.353 1 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec
2018-02-14 11:51:02.353 1 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds
2018-02-14 11:51:02.354 1 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds
2018-02-14 11:51:02.360 1 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec
2018-02-14 11:55:39.518 1 INFO octavia.common.config [-] Logging enabled!
2018-02-14 11:55:39.518 1 INFO octavia.common.config [-] /usr/bin/octavia-housekeeping version 2.0.0.0b3.dev60
2018-02-14 11:55:39.518 1 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2018-02-14 11:55:39.518944
2018-02-14 11:55:39.519 1 INFO octavia.cmd.house_keeping [-] Spare check interval is set to 30 sec
2018-02-14 11:55:39.519 1 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec
2018-02-14 11:55:39.522 1 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds
2018-02-14 11:55:39.523 1 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds
2018-02-14 11:55:39.520 1 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec
2018-02-14 12:51:24.181 1 INFO octavia.controller.housekeeping.house_keeping [-] Attempting to delete Amphora id : 49f56a42-1fa6-4313-a885-9db6c1d4b228
2018-02-14 12:51:24.191 1 INFO octavia.controller.housekeeping.house_keeping [-] Deleted Amphora id : 49f56a42-1fa6-4313-a885-9db6c1d4b228

Comment 4 Carlos Goncalves 2018-02-20 10:10:07 UTC

Thanks, Alex! Closing as WORKSFORME. We can reopen if needed.

Note You need to log in before you can comment on or make changes to this bug.