Bug 1787077

Summary: OVN load balancer not passing initial traffic
Product: Red Hat OpenStack Reporter: Carlos Goncalves <cgoncalves>
Component: python-networking-ovnAssignee: Maciej Józefczyk <mjozefcz>
Status: CLOSED DUPLICATE QA Contact: Eran Kuris <ekuris>
Severity: high Docs Contact:
Priority: high    
Version: 16.0 (Train)CC: amaron, apevec, lhh, majopela, mjozefcz, scohen
Target Milestone: z1Keywords: AutomationBlocker, Triaged
Target Release: 16.0 (Train on RHEL 8.1)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-31 10:54:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1788456    
Bug Blocks: 1874874, 1676631, 1792667    

Description Carlos Goncalves 2019-12-30 18:23:52 UTC 
Description of problem:
Tempest test octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest.test_basic_traffic fails with "Server did not begin passing traffic within the timeout period. Failing test."


Version-Release number of selected component (if applicable):
OSP 16 beta

How reproducible:
100%


Steps to Reproduce:
1. Deploy OSP 16 with ML2/OVN and Octavia enabled
2. Run Octavia tempest test test_basic_traffic


Actual results:

2019-12-27 21:55:38,608 296278 INFO     [tempest.lib.common.rest_client] Request (TrafficOperationsScenarioTest:test_basic_traffic): 200 GET http://10.0.0.107:9876/v2.0/lbaas/loadbalancers/41e01f6d-963b-4777-95f1-e1187c151a8c 0.081s
2019-12-27 21:55:38,609 296278 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Fri, 27 Dec 2019 21:55:38 GMT', 'server': 'Apache', 'content-length': '804', 'x-openstack-request-id': 'req-17acda0a-fa54-46c7-bf07-308ae182d0c0', 'vary': 'Accept-Encoding', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.107:9876/v2.0/lbaas/loadbalancers/41e01f6d-963b-4777-95f1-e1187c151a8c'}
        Body: b'{"loadbalancer": {"id": "41e01f6d-963b-4777-95f1-e1187c151a8c", "name": "tempest-lb_member_lb1_operations-1272789538", "description": "", "provisioning_status": "PENDING_UPDATE", "operating_status": "ONLINE", "admin_state_up": true, "project_id": "eba9c7ea76164d4d96294d19cf6264d7", "created_at": "2019-12-27T21:55:04", "updated_at": "2019-12-27T21:55:38", "vip_address": "10.1.1.233", "vip_port_id": "4d7702de-73e7-4918-b412-331ea7d6fcb9", "vip_subnet_id": "7dd9ed45-1708-41ec-826d-2ecc673e4483", "vip_network_id": "0913ddc2-a4e7-46e2-9221-89e7749c4dd2", "listeners": [{"id": "b41effed-6f60-476f-b243-fc2a2131d609"}], "pools": [{"id": "78143a52-741b-46f7-8fbc-9fe191b07410"}], "provider": "ovn", "flavor_id": null, "vip_qos_policy_id": null, "tags": [], "tenant_id": "eba9c7ea76164d4d96294d19cf6264d7"}}'
2019-12-27 21:55:43,746 296278 INFO     [tempest.lib.common.rest_client] Request (TrafficOperationsScenarioTest:test_basic_traffic): 200 GET http://10.0.0.107:9876/v2.0/lbaas/loadbalancers/41e01f6d-963b-4777-95f1-e1187c151a8c 0.132s
2019-12-27 21:55:43,748 296278 DEBUG    [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'date': 'Fri, 27 Dec 2019 21:55:43 GMT', 'server': 'Apache', 'content-length': '796', 'x-openstack-request-id': 'req-846c626d-3f41-47c6-b4e6-04e9bd2fe3ea', 'vary': 'Accept-Encoding', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.107:9876/v2.0/lbaas/loadbalancers/41e01f6d-963b-4777-95f1-e1187c151a8c'}
        Body: b'{"loadbalancer": {"id": "41e01f6d-963b-4777-95f1-e1187c151a8c", "name": "tempest-lb_member_lb1_operations-1272789538", "description": "", "provisioning_status": "ACTIVE", "operating_status": "ONLINE", "admin_state_up": true, "project_id": "eba9c7ea76164d4d96294d19cf6264d7", "created_at": "2019-12-27T21:55:04", "updated_at": "2019-12-27T21:55:38", "vip_address": "10.1.1.233", "vip_port_id": "4d7702de-73e7-4918-b412-331ea7d6fcb9", "vip_subnet_id": "7dd9ed45-1708-41ec-826d-2ecc673e4483", "vip_network_id": "0913ddc2-a4e7-46e2-9221-89e7749c4dd2", "listeners": [{"id": "b41effed-6f60-476f-b243-fc2a2131d609"}], "pools": [{"id": "78143a52-741b-46f7-8fbc-9fe191b07410"}], "provider": "ovn", "flavor_id": null, "vip_qos_policy_id": null, "tags": [], "tenant_id": "eba9c7ea76164d4d96294d19cf6264d7"}}'
2019-12-27 21:55:43,748 296278 INFO     [octavia_tempest_plugin.tests.waiters] show_loadbalancer's status updated to ACTIVE.
2019-12-27 21:55:45,755 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:55:48,772 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:55:51,782 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:55:54,791 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:55:57,798 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:00,806 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:03,816 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:06,825 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:09,832 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:12,842 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:15,858 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:18,868 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:21,876 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:24,883 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:27,890 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:30,897 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:33,908 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:36,917 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:39,925 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:42,938 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:45,948 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:48,962 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:51,974 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:54,983 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:56:57,994 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:01,002 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:04,010 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:07,019 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:10,027 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:13,034 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:16,046 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:19,054 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:22,063 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:25,077 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:28,086 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:31,098 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:34,106 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:37,115 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:40,125 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:43,138 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:46,144 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:49,151 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:52,162 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:55,172 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:57:58,178 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:01,188 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:04,198 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:07,205 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:10,214 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:13,225 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:16,231 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:19,241 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:22,250 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:25,257 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:28,265 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:31,275 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:34,282 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:37,290 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:40,300 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:43,308 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:46,316 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:49,325 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:52,336 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:55,346 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:58:58,355 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:01,364 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:04,375 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:07,393 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:10,404 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:13,412 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:16,425 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:19,434 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:22,442 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:25,448 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:28,455 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:31,462 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:34,470 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:37,478 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:40,485 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:43,494 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:46,512 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:49,522 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:52,532 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:55,546 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 21:59:58,555 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:01,564 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:04,571 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:07,579 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:10,585 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:13,591 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:16,599 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:19,611 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:22,620 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:25,628 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:28,637 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:31,645 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:34,655 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:37,665 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:40,673 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:43,683 296278 WARNING  [octavia_tempest_plugin.tests.test_base] Server is not passing initial traffic. Waiting.
2019-12-27 22:00:44,686 296278 ERROR    [octavia_tempest_plugin.tests.test_base] Server did not begin passing traffic within the timeout period. Failing test.
Comment 7 Arieh Maron 2020-09-17 07:13:41 UTC 
This bug is now (17-Sep-2020) being observed in OSP13 tempest runs for instance:

https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/DFG-network-octavia-13_director-rhel-virthost-3cont_2comp-ipv4-geneve-ovn/7//artifact/tempest-results/tempest-results-octavia.1.html

https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/DFG-network-octavia-13_director-rhel-virthost-3cont_2comp-ipv4-geneve-ovn-dvr/6//artifact/tempest-results/tempest-results-octavia.1.html

https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/DFG-network-octavia-13_director-rhel-virthost-3cont_2comp-ipv4-vxlan/15//artifact/tempest-results/tempest-results-octavia.1.html

https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/DFG-network-octavia-13_director-rhel-virthost-3cont_2comp_3net-ipv4-vxlan-composable/10//artifact/tempest-results/tempest-results-octavia.1.html