Bug 1961597

Summary: octavia_tempest_plugin.tests.api.v2.test_pool.PoolAPITest.test_pool_delete fails in ACTIVE_STANDBY jobs
Product: Red Hat OpenStack Reporter: Omer Schwartz <oschwart>
Component: openstack-octaviaAssignee: Gregory Thiemonge <gthiemon>
Status: CLOSED DUPLICATE QA Contact: Bruna Bonguardo <bbonguar>
Severity: high Docs Contact:
Priority: high    
Version: 16.1 (Train)CC: amaron, bbonguar, gthiemon, ihrachys, lpeer, majopela, scohen
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-09 11:54:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Omer Schwartz 2021-05-18 11:06:32 UTC 
Description of problem:
After a different issue was verified on https://bugzilla.redhat.com/show_bug.cgi?id=1845406 (using 4 threads led to memory error with haproxy, which led to timeouts), we tried using 2 threads and this time we face timeouts because of the load each backend server receives.

octavia_tempest_plugin.tests.api.v2.test_pool.PoolAPITest.test_pool_delete test fails when ACTIVE_STANDBY topology is enabled.


Ex: 
https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/octavia/job/DFG-network-octavia-16.1_director-rhel-virthost-3cont_3comp-ipv4-geneve-actstby/11/testReport/octavia_tempest_plugin.tests.api.v2.test_pool/PoolAPITest/test_PROXY_SI_pool_delete_id_277fba8f_d72b_47f0_9723_5e013f53fb7a_/

https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/octavia/job/DFG-network-octavia-16.1_director-rhel-virthost-3cont_3comp-ipv4-geneve-actstby/11/testReport/octavia_tempest_plugin.tests.api.v2.test_pool/PoolAPITest/test_TCP_RR_pool_delete_id_a455dea2_19ce_435c_90ae_e143fe84245e_/

https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/octavia/job/DFG-network-octavia-16.1_director-rhel-virthost-3cont_3comp-ipv4-geneve-actstby/11/testReport/octavia_tempest_plugin.tests.api.v2.test_pool/PoolAPITest/test_TCP_SI_pool_delete_id_8411295f_aec0_40ab_a25d_a4677c711d98_/


Version-Release number of selected component (if applicable):
16.1

How reproducible:
100%

Steps to Reproduce:
1. Configure Octavia for ACTIVE_STANDBY
2. Launch Octavia tempest plugin (can use 'PoolAPITest' regex)
3.

Actual results:
Traceback (most recent call last):
  File "/home/stack/plugins/octavia/octavia_tempest_plugin/tests/api/v2/test_pool.py", line 1541, in test_PROXY_SI_pool_delete
    self._test_pool_delete(const.PROXY, const.LB_ALGORITHM_SOURCE_IP)
  File "/home/stack/plugins/octavia/octavia_tempest_plugin/tests/api/v2/test_pool.py", line 1679, in _test_pool_delete
    CONF.load_balancer.check_timeout)
  File "/home/stack/plugins/octavia/octavia_tempest_plugin/tests/waiters.py", line 182, in wait_for_deleted_status_or_not_found
    raise exceptions.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: (PoolAPITest:test_PROXY_SI_pool_delete) show_pool provisioning_status failed to update to DELETED or become not found (404) within the required time 120. Current status of show_pool: PENDING_DELETE


Expected results:
Test should pass
Comment 1 Gregory Thiemonge 2021-05-19 11:47:50 UTC 
Pool deletion timed out because of a lost of connectivity between the octavia services and the amphora:

2021-05-09 13:01:25.590 31 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-b97ab21c-dd6e-4484-8b2d-e502b9073c43 - 62a072e951cb40c78122401fe79b9668 - - -] Could not connect to instance. Retrying.: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='172.24.1.139', port=9443): Read timed out. (read timeout=60.0)
2021-05-09 13:01:40.615 31 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-b97ab21c-dd6e-4484-8b2d-e502b9073c43 - 62a072e951cb40c78122401fe79b9668 - - -] Could not connect to instance. Retrying.: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='172.24.1.139', port=9443): Read timed out. (read timeout=10.0)
2021-05-09 13:01:55.634 31 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [req-b97ab21c-dd6e-4484-8b2d-e502b9073c43 - 62a072e951cb40c78122401fe79b9668 - - -] Could not connect to instance. Retrying.: requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='172.24.1.139', port=9443): Read timed out. (read timeout=10.0)
Comment 2 Omer Schwartz 2021-06-02 14:56:57 UTC 
*** Bug 1964451 has been marked as a duplicate of this bug. ***
Comment 3 Bruna Bonguardo 2021-06-09 11:54:27 UTC 

*** This bug has been marked as a duplicate of bug 1907965 ***