Bug 1966429
| Summary: | Stale amphora's listed even after deleting loadbalancers with --cascade | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Asma Syed Hameed <asyedham> |
| Component: | openstack-octavia | Assignee: | Gregory Thiemonge <gthiemon> |
| Status: | CLOSED DUPLICATE | QA Contact: | Bruna Bonguardo <bbonguar> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 16.1 (Train) | CC: | ihrachys, jraju, lpeer, majopela, oschwart, scohen |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-09-07 15:22:20 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Asma Syed Hameed
2021-06-01 07:39:38 UTC
This is weird, there are amphorae that share the same management IP address, one is in ERROR, the second is ALLOCATED
| 99a7197c-b590-4a51-9bad-740fda9b392d | 34a9c851-2e32-484e-86f5-4079f778c171 | ERROR | NULL | 172.24.0.156 | 10.2.11.86 | 10.2.11.213 | 04a4061b-36ee-4d86-8ef4-381faa861f28 | 1b0e4b91-73c7-42d7-a030-735d769ff940 | STANDALONE | 2021-06-10 14:57:33 | 1 | NULL | 1 | NULL | nova | 2021-05-11 14:57:33 | 2021-05-27 15:53:31 | 0d8164aa-65f3-42b4-9fbf-289e522533fb | 65 |
| db011293-c33e-467b-a3b7-5bf3b34532fb | 68d7ed4a-3e53-4f18-a245-0cfaef66f264 | ALLOCATED | dc371928-24c5-4320-8b7f-eb3baf8bde6a | 172.24.0.156 | 10.2.5.48 | 10.2.5.41 | 9fcaf80f-ef20-4ecf-96db-06f984451ad3 | 3aee0bf2-247a-4109-b93a-3cbaae9e4530 | STANDALONE | 2021-06-25 15:38:28 | 0 | NULL | 1 | NULL | nova | 2021-05-26 15:38:28 | 2021-05-26 15:41:21 | 0d8164aa-65f3-42b4-9fbf-289e522533fb | 65 |
update_at looks good for both, but it doesn't mean the compute of the amphora in ERROR still exists.
I found logs about the creation of the amphorae that are now in ERROR (they were created without any issue and they were associated with a load balancer), but some logs are missing, I cannot see what happened after.
Some findings:
- the amphorae are no longer associated with a load balancer (load_balancer_id is NULL)
- the load_balancer_id no longer exists in the amphora list dump (it has probably been deleted)
There are also a lot of errors about failed deletions of LB and failed failovers, it looks like the VIP network/ports were deleted:
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks [-] Unable to unplug vip from load balancer 1f9a6a4c-d454-463e-8497-1a971c1993a8: octavia.network.base.PluggedVIPNotFound: Can't unplug vip because vip subnet 8e8df4ce-a293-4078-a724-d28ee1a04338 was not found
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks Traceback (most recent call last):
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks File "/usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py", line 193, in _get_resource
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks resource_type)(resource_id)
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 844, in show_subnet
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks return self.get(self.subnet_path % (subnet), params=_params)
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 354, in get
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks headers=headers, params=params)
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks headers=headers, params=params)
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks self._handle_fault_response(status_code, replybody, resp)
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks exception_handler_v20(status_code, error_body)
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks request_ids=request_ids)
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks neutronclient.common.exceptions.NotFound: Subnet 8e8df4ce-a293-4078-a724-d28ee1a04338 could not be found.
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks Neutron server returns request_ids: ['req-1c02a76f-38f1-4086-815b-7a1ebac60d68']
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks During handling of the above exception, another exception occurred:
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks Traceback (most recent call last):
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks File "/usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 571, in unplug_vip
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks subnet = self.get_subnet(vip.subnet_id)
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks File "/usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py", line 246, in get_subnet
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks return self._get_resource('subnet', subnet_id, context=context)
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks File "/usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/base.py", line 201, in _get_resource
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks [w.capitalize() for w in resource_type.split('_')]))(message)
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks octavia.network.base.SubnetNotFound: subnet not found (subnet id: 8e8df4ce-a293-4078-a724-d28ee1a04338).
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks During handling of the above exception, another exception occurred:
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks Traceback (most recent call last):
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks File "/usr/lib/python3.6/site-packages/octavia/controller/worker/v1/tasks/network_tasks.py", line 430, in execute
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks self.network_driver.unplug_vip(loadbalancer, loadbalancer.vip)
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks File "/usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 576, in unplug_vip
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks raise base.PluggedVIPNotFound(msg)
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks octavia.network.base.PluggedVIPNotFound: Can't unplug vip because vip subnet 8e8df4ce-a293-4078-a724-d28ee1a04338 was not found
controller-0/octavia/worker.log.6:2021-05-06 13:04:53.558 78 ERROR octavia.controller.worker.v1.tasks.network_tasks
2021-05-06 13:04:53.663 78 DEBUG neutronclient.v2_0.client [-] Error message: {"NeutronError": {"type": "PortNotFound", "message": "Port be94b98b-7905-4458-b1ce-8d500eb3e860 could not be found.", "detail": ""}} _handle_fault_response /usr/
lib/python3.6/site-packages/neutronclient/v2_0/client.py:259
2021-05-06 13:04:53.663 78 DEBUG octavia.network.drivers.neutron.allowed_address_pairs [-] VIP instance port be94b98b-7905-4458-b1ce-8d500eb3e860 already deleted. Skipping. deallocate_vip /usr/lib/python3.6/site-packages/octavia/network/dr
ivers/neutron/allowed_address_pairs.py:345
2021-05-06 13:04:53.738 78 DEBUG neutronclient.v2_0.client [-] Error message: {"NeutronError": {"type": "PortNotFound", "message": "Port 445dffc3-1cf8-4ee7-94f7-362f3b6c58a0 could not be found.", "detail": ""}} _handle_fault_response /usr/
lib/python3.6/site-packages/neutronclient/v2_0/client.py:259
2021-05-06 13:04:53.738 78 WARNING octavia.network.drivers.neutron.allowed_address_pairs [-] Can't deallocate VIP because the vip port 445dffc3-1cf8-4ee7-94f7-362f3b6c58a0 cannot be found in neutron. Continuing cleanup.: octavia.network.base.PortNotFound: port not found (port id: 445dffc3-1cf8-4ee7-94f7-362f3b6c58a0).
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs [-] Error creating a port on network 7ffb336b-0450-4d43-a950-ae5eca853f5c due to Network 7ffb336b-0450-4d43-a950-ae5eca853f5c could not be found.
Neutron server returns request_ids: ['req-a1fcb84e-ff8d-4f95-9e53-be1bab8df675'].: neutronclient.common.exceptions.NetworkNotFoundClient: Network 7ffb336b-0450-4d43-a950-ae5eca853f5c could not be found.
Neutron server returns request_ids: ['req-a1fcb84e-ff8d-4f95-9e53-be1bab8df675']
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs Traceback (most recent call last):
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python3.6/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py", line 839, in create_port
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs new_port = self.neutron_client.create_port({constants.PORT: port})
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 803, in create_port
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs return self.post(self.ports_path, body=body)
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 359, in post
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs headers=headers, params=params)
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs self._handle_fault_response(status_code, replybody, resp)
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs exception_handler_v20(status_code, error_body)
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs request_ids=request_ids)
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs neutronclient.common.exceptions.NetworkNotFoundClient: Network 7ffb336b-0450-4d43-a950-ae5eca853f5c could not be found.
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs Neutron server returns request_ids: ['req-a1fcb84e-ff8d-4f95-9e53-be1bab8df675']
2021-05-10 07:26:53.933 78 ERROR octavia.network.drivers.neutron.allowed_address_pairs
I think I figured out what the issue was: TLDR: this is probably a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1992691 which is targeted for OSP16.1z8 There's a bug in Octavia, the housekeeping service may try to rotate the certificates of amphorae in DELETE state. Basically, the housekeeping service is trying to reach some amphorae that have been deleted, with deleted compute servers, in order to perform some maintenance tasks. It triggers some issues present in the logs: housekeeping tries to connect to an amphora but it reports that the amphora has an incorrect certificate (1560 occurrences in the logs): 2021-05-23 06:10:08.711 7 ERROR urllib3.connection [-] Certificate did not match expected hostname: 489c62e1-53cb-4a9a-a6b1-ba4f7a48d43c. Certificate: {'subject': ((('commonName', 'a403e8bd-d00b-4d95-94aa-bc81275aa907'),),), 'issuer': ((('countryName', 'US'),), (('stateOrProvinceName', 'Denial'),), (('localityName', 'Springfield'),), (('organizationName', 'Dis'),), (('commonName', 'www.example.com'),)), 'version': 3, 'serialNumber': 'EA63F8F42CA74B62BDDBF9C89DB55A50', 'notBefore': 'May 21 16:43:01 2021 GMT', 'notAfter': 'Jun 20 16:43:01 2021 GMT', 'subjectAltName': (('DNS', 'a403e8bd-d00b-4d95-94aa-bc81275aa907'),)}: ssl.CertificateError: hostname '489c62e1-53cb-4a9a-a6b1-ba4f7a48d43c' doesn't match 'a403e8bd-d00b-4d95-94aa-bc81275aa907' -> The IP address of the amphora in the database has been reused by another amphora, this is confirmed by previous findings in the database dump, 2 amphora entries shared the same IP address. When the housekeeping fails to update the certificates in the deleted amphora, it updates its status to ERROR. So DELETED amphorae appear in the list of the amphorae in ERROR status. Hi, If you don't have any objections, I will close this BZ as a duplicate of BZ 1992691 Marked as a duplicate of BZ 1992691 *** This bug has been marked as a duplicate of bug 1992691 *** |