Bug 1542538
| Summary: | octavia-housekeeping cannot connect to database | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Carlos Goncalves <cgoncalves> |
| Component: | openstack-octavia | Assignee: | Carlos Goncalves <cgoncalves> |
| Status: | CLOSED WORKSFORME | QA Contact: | Alexander Stafeyev <astafeye> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 13.0 (Queens) | CC: | astafeye, ihrachys, lpeer, majopela, nyechiel |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-02-20 10:10:07 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Alex, could you verify this whenever you have an octavia-enabled OSP 13 env set up? Yes I will try to reproduce this one So far so good [root@controller-0 ~]# docker exec -i $id cat /var/log/octavia/housekeeping.log 2018-02-14 11:51:02.337 1 INFO octavia.common.config [-] Logging enabled! 2018-02-14 11:51:02.338 1 INFO octavia.common.config [-] /usr/bin/octavia-housekeeping version 2.0.0.0b3.dev60 2018-02-14 11:51:02.338 1 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2018-02-14 11:51:02.338402 2018-02-14 11:51:02.338 1 INFO octavia.cmd.house_keeping [-] Spare check interval is set to 30 sec 2018-02-14 11:51:02.353 1 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec 2018-02-14 11:51:02.353 1 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds 2018-02-14 11:51:02.354 1 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds 2018-02-14 11:51:02.360 1 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec 2018-02-14 11:55:39.518 1 INFO octavia.common.config [-] Logging enabled! 2018-02-14 11:55:39.518 1 INFO octavia.common.config [-] /usr/bin/octavia-housekeeping version 2.0.0.0b3.dev60 2018-02-14 11:55:39.518 1 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2018-02-14 11:55:39.518944 2018-02-14 11:55:39.519 1 INFO octavia.cmd.house_keeping [-] Spare check interval is set to 30 sec 2018-02-14 11:55:39.519 1 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec 2018-02-14 11:55:39.522 1 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds 2018-02-14 11:55:39.523 1 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds 2018-02-14 11:55:39.520 1 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec 2018-02-14 12:51:24.181 1 INFO octavia.controller.housekeeping.house_keeping [-] Attempting to delete Amphora id : 49f56a42-1fa6-4313-a885-9db6c1d4b228 2018-02-14 12:51:24.191 1 INFO octavia.controller.housekeeping.house_keeping [-] Deleted Amphora id : 49f56a42-1fa6-4313-a885-9db6c1d4b228 Thanks, Alex! Closing as WORKSFORME. We can reopen if needed. |
Description of problem: octavia-housekeeping cannot connect to DB. Version-Release number of selected component (if applicable): docker.io/tripleoupstream/centos-binary-octavia-housekeeping:latest (as of January 30). How reproducible: containerized RDO deployment Steps to Reproduce: 1. deploy overcloud with oooq and latest centos-binary-* docker images from registry docker.io/tripleoupstream openstack overcloud deploy \ --templates /usr/share/openstack-tripleo-heat-templates \ --libvirt-type qemu --control-flavor oooq_control --compute-flavor oooq_compute --ceph-storage-flavor oooq_ceph --block-storage-flavor oooq_blockstorage --swift-storage-flavor oooq_objectstorage --timeout 90 -e /home/stack/cloud-names.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml -e /home/stack/containers-default-parameters.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /home/stack/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml -e /home/stack/inject-trust-anchor.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml --validation-warnings-fatal --ntp-server pool.ntp.org -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/octavia.yaml \ ${DEPLOY_ENV_YAML:+-e $DEPLOY_ENV_YAML} "$@" && status_code=0 || status_code=$? 2. [root@overcloud-controller-0 heat-admin]# id=$(docker ps | grep octavia-housekeeping | awk {'print $1'}) 3. [root@overcloud-controller-0 heat-admin]# docker exec -i $id cat /var/log/octavia/housekeeping.log Actual results: $ cat /var/log/octavia/housekeeping.log 2018-01-30 16:58:35.562 5 INFO octavia.common.config [-] Logging enabled! 2018-01-30 16:58:35.562 5 INFO octavia.common.config [-] /usr/bin/octavia-housekeeping version 2.0.0.0b2.dev70 2018-01-30 16:58:35.562 5 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2018-01-30 16:58:35.562579 2018-01-30 16:58:35.562 5 INFO octavia.cmd.house_keeping [-] Spare check interval is set to 30 sec 2018-01-30 16:58:35.655 5 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec 2018-01-30 16:58:35.655 5 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds 2018-01-30 16:58:35.655 5 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds 2018-01-30 16:58:35.659 5 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec 2018-01-30 17:03:29.558 5 INFO octavia.common.config [-] Logging enabled! 2018-01-30 17:03:29.559 5 INFO octavia.common.config [-] /usr/bin/octavia-housekeeping version 2.0.0.0b2.dev70 2018-01-30 17:03:29.559 5 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2018-01-30 17:03:29.559343 2018-01-30 17:03:29.560 5 INFO octavia.cmd.house_keeping [-] Spare check interval is set to 30 sec 2018-01-30 17:03:29.599 5 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec 2018-01-30 17:03:29.599 5 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds 2018-01-30 17:03:29.600 5 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds 2018-01-30 17:03:29.612 5 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec 2018-02-01 10:11:21.815 5 INFO octavia.common.config [-] Logging enabled! 2018-02-01 10:11:21.843 5 INFO octavia.common.config [-] /usr/bin/octavia-housekeeping version 2.0.0.0b2.dev70 2018-02-01 10:11:21.844 5 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2018-02-01 10:11:21.844259 2018-02-01 10:11:21.844 5 INFO octavia.cmd.house_keeping [-] Spare check interval is set to 30 sec 2018-02-01 10:11:22.065 5 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec 2018-02-01 10:11:22.065 5 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds 2018-02-01 10:11:22.065 5 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds 2018-02-01 10:11:22.110 5 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '172.16.2.12' ([Errno 111] Connection refused)") 2018-02-01 10:11:22.111 5 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '172.16.2.12' ([Errno 111] Connection refused)") 2018-02-01 10:11:22.112 5 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec 2018-02-01 10:11:22.170 5 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '172.16.2.12' ([Errno 111] Connection refused)") 2018-02-01 12:18:30.976 5 INFO octavia.common.config [-] Logging enabled! 2018-02-01 12:18:30.999 5 INFO octavia.common.config [-] /usr/bin/octavia-housekeeping version 2.0.0.0b2.dev70 2018-02-01 12:18:31.000 5 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2018-02-01 12:18:31.000201 2018-02-01 12:18:31.002 5 INFO octavia.cmd.house_keeping [-] Spare check interval is set to 30 sec 2018-02-01 12:18:31.288 5 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec 2018-02-01 12:18:31.288 5 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds 2018-02-01 12:18:31.289 5 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds 2018-02-01 12:18:31.294 5 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') 2018-02-01 12:18:31.298 5 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec 2018-02-01 12:18:31.304 5 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') 2018-02-01 12:18:31.304 5 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') Expected results: octavia-housekeeping Additional info: Bug may not be observed in OSP as this was tested under a master upstream deployment. Yet reporting so that we can follow-up and confirm.