Description of problem: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest.test_http_l7policies_and_l7rules randomly fails Version-Release number of selected component (if applicable): 17.1 How reproducible: randomly ~10% Steps to Reproduce: 1. deploy OSP17.1 with Octavia 2. run the tempest suite at least 10 times 3. Backtrace: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest.test_http_l7policies_and_l7rules[id-3558186d-6dcd-4d9d-b7f7-adc190b66149] ----------------------------------------------------------------------------------------------------------------------------------------------------------------- Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request six.raise_from(e, None) File "<string>", line 3, in raise_from File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request httplib_response = conn.getresponse() File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse response.begin() File "/usr/lib64/python3.9/http/client.py", line 320, in begin version, status, reason = self._read_status() File "/usr/lib64/python3.9/http/client.py", line 281, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/lib64/python3.9/socket.py", line 704, in readinto return self._sock.recv_into(b) socket.timeout: timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen retries = retries.increment( File "/usr/lib/python3.9/site-packages/urllib3/util/retry.py", line 532, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/lib/python3.9/site-packages/urllib3/packages/six.py", line 709, in reraise raise value File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 447, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/usr/lib/python3.9/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='10.0.0.227', port=84): Read timed out. (read timeout=2) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 712, in test_http_l7policies_and_l7rules self.assertConsistentResponse((200, self.webserver1_response), File "/usr/lib/python3.9/site-packages/octavia_tempest_plugin/tests/validators.py", line 357, in assertConsistentResponse req = session.request(method, url, File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python3.9/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/usr/lib/python3.9/site-packages/requests/adapters.py", line 529, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPConnectionPool(host='10.0.0.227', port=84): Read timed out. (read timeout=2) tpcdump outputs from another run (not the same VIP address) undercloud: 12:53:48.039779 IP 10.0.0.54.52864 > 10.0.0.159.84: Flags [S], seq 692056855, win 64240, options [mss 1460,sackOK,TS val 176487310 ecr 0,nop,wscale 7], length 0 12:53:48.041801 IP 10.0.0.159.84 > 10.0.0.54.52864: Flags [S.], seq 79995390, ack 692056856, win 43090, options [mss 1402,sackOK,TS val 3299079005 ecr 176487310,nop,wscale 11], length 0 12:53:48.041852 IP 10.0.0.54.52864 > 10.0.0.159.84: Flags [.], ack 1, win 502, options [nop,nop,TS val 176487312 ecr 3299079005], length 0 12:53:48.041909 IP 10.0.0.54.52864 > 10.0.0.159.84: Flags [P.], seq 1:145, ack 1, win 502, options [nop,nop,TS val 176487312 ecr 3299079005], length 144 12:53:48.083807 IP 10.0.0.159.84 > 10.0.0.54.52864: Flags [.], ack 145, win 21, options [nop,nop,TS val 3299079048 ecr 176487312], length 0 12:53:50.044449 IP 10.0.0.54.52864 > 10.0.0.159.84: Flags [F.], seq 145, ack 1, win 502, options [nop,nop,TS val 176489315 ecr 3299079048], length 0 12:53:50.085643 IP 10.0.0.159.84 > 10.0.0.54.52864: Flags [.], ack 146, win 21, options [nop,nop,TS val 3299081050 ecr 176489315], length 0 12:53:56.107623 IP 10.0.0.159.84 > 10.0.0.54.52864: Flags [P.], seq 1:163, ack 146, win 21, options [nop,nop,TS val 3299087072 ecr 176489315], length 162 12:53:56.107668 IP 10.0.0.54.52864 > 10.0.0.159.84: Flags [R], seq 692057001, win 0, length 0 12:53:56.107741 IP 10.0.0.159.84 > 10.0.0.54.52864: Flags [F.], seq 163, ack 146, win 21, options [nop,nop,TS val 3299087072 ecr 176489315], length 0 12:53:56.107758 IP 10.0.0.54.52864 > 10.0.0.159.84: Flags [R], seq 692057001, win 0, length 0 The FIN packet sent from the client at 12:53:50.044449 is the termination of the tcp connection after a 2 sec timeout (default value in octavia-tempest-plugin) compute-0 (LB, vip port): tap7f5d8569-04.pcap 16:53:48.041495 IP 10.0.0.54.52864 > 10.1.1.178.84: Flags [S], seq 692056855, win 64240, options [mss 1460,sackOK,TS val 176487310 ecr 0,nop,wscale 7], length 0 16:53:48.041844 IP 10.1.1.178.84 > 10.0.0.54.52864: Flags [S.], seq 79995390, ack 692056856, win 43090, options [mss 1402,sackOK,TS val 3299079005 ecr 176487310,nop,wscale 11], length 0 16:53:48.042940 IP 10.0.0.54.52864 > 10.1.1.178.84: Flags [.], ack 1, win 502, options [nop,nop,TS val 176487312 ecr 3299079005], length 0 16:53:48.042971 IP 10.0.0.54.52864 > 10.1.1.178.84: Flags [P.], seq 1:145, ack 1, win 502, options [nop,nop,TS val 176487312 ecr 3299079005], length 144 16:53:48.084218 IP 10.1.1.178.84 > 10.0.0.54.52864: Flags [.], ack 145, win 21, options [nop,nop,TS val 3299079048 ecr 176487312], length 0 16:53:50.045287 IP 10.0.0.54.52864 > 10.1.1.178.84: Flags [F.], seq 145, ack 1, win 502, options [nop,nop,TS val 176489315 ecr 3299079048], length 0 16:53:50.086054 IP 10.1.1.178.84 > 10.0.0.54.52864: Flags [.], ack 146, win 21, options [nop,nop,TS val 3299081050 ecr 176489315], length 0 16:53:56.108080 IP 10.1.1.178.84 > 10.0.0.54.52864: Flags [P.], seq 1:163, ack 146, win 21, options [nop,nop,TS val 3299087072 ecr 176489315], length 162 16:53:56.108279 IP 10.1.1.178.84 > 10.0.0.54.52864: Flags [F.], seq 163, ack 146, win 21, options [nop,nop,TS val 3299087072 ecr 176489315], length 0 16:53:56.108412 IP 10.0.0.54.52864 > 10.1.1.178.84: Flags [R], seq 692057001, win 0, length 0 16:53:56.108457 IP 10.0.0.54.52864 > 10.1.1.178.84: Flags [R], seq 692057001, win 0, length 0 compute-0 (LB, member port): tapd9ee04bd-6c.pcap 16:53:48.043569 fa:16:3e:4a:e3:cd > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 10.2.1.201 tell 10.2.1.220, length 28 16:53:48.043774 fa:16:3e:e5:06:68 > fa:16:3e:4a:e3:cd, ethertype ARP (0x0806), length 42: Reply 10.2.1.201 is-at fa:16:3e:e5:06:68, length 28 16:53:48.043902 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 74: 10.2.1.220.45847 > 10.2.1.201.80: Flags [S], seq 479897052, win 43462, options [mss 1402,sackOK,TS val 4071114320 ecr 0,nop,wscale 11], length 0 16:53:49.045169 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 74: 10.2.1.220.45847 > 10.2.1.201.80: Flags [S], seq 479897052, win 43462, options [mss 1402,sackOK,TS val 4071115322 ecr 0,nop,wscale 11], length 0 16:53:51.093070 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 74: 10.2.1.220.45847 > 10.2.1.201.80: Flags [S], seq 479897052, win 43462, options [mss 1402,sackOK,TS val 4071117370 ecr 0,nop,wscale 11], length 0 16:53:53.050223 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 74: 10.2.1.220.45851 > 10.2.1.201.80: Flags [S], seq 1703562389, win 43462, options [mss 1402,sackOK,TS val 4071119327 ecr 0,nop,wscale 11], length 0 16:53:54.101034 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 74: 10.2.1.220.45851 > 10.2.1.201.80: Flags [S], seq 1703562389, win 43462, options [mss 1402,sackOK,TS val 4071120378 ecr 0,nop,wscale 11], length 0 16:53:56.105191 fa:16:3e:e5:06:68 > fa:16:3e:4a:e3:cd, ethertype IPv4 (0x0800), length 74: 10.2.1.201.80 > 10.2.1.220.45851: Flags [S.], seq 2546563750, ack 1703562390, win 65330, options [mss 1402,sackOK,TS val 593222582 ecr 4071119327,nop,wscale 4], length 0 16:53:56.105778 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 66: 10.2.1.220.45851 > 10.2.1.201.80: Flags [.], ack 1, win 22, options [nop,nop,TS val 4071122382 ecr 593222582], length 0 16:53:56.106345 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 205: 10.2.1.220.45851 > 10.2.1.201.80: Flags [P.], seq 1:140, ack 1, win 22, options [nop,nop,TS val 4071122383 ecr 593222582], length 139: HTTP: GET / HTTP/1.1 16:53:56.106704 fa:16:3e:e5:06:68 > fa:16:3e:4a:e3:cd, ethertype IPv4 (0x0800), length 66: 10.2.1.201.80 > 10.2.1.220.45851: Flags [.], ack 140, win 4075, options [nop,nop,TS val 593222585 ecr 4071122383], length 0 16:53:56.107530 fa:16:3e:e5:06:68 > fa:16:3e:4a:e3:cd, ethertype IPv4 (0x0800), length 228: 10.2.1.201.80 > 10.2.1.220.45851: Flags [P.], seq 1:163, ack 140, win 4075, options [nop,nop,TS val 593222586 ecr 4071122383], length 162: HTTP: HTTP/1.1 200 OK 16:53:56.107649 fa:16:3e:e5:06:68 > fa:16:3e:4a:e3:cd, ethertype IPv4 (0x0800), length 66: 10.2.1.201.80 > 10.2.1.220.45851: Flags [F.], seq 163, ack 140, win 4075, options [nop,nop,TS val 593222586 ecr 4071122383], length 0 16:53:56.107688 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 66: 10.2.1.220.45851 > 10.2.1.201.80: Flags [.], ack 163, win 22, options [nop,nop,TS val 4071122384 ecr 593222586], length 0 16:53:56.108630 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 66: 10.2.1.220.45851 > 10.2.1.201.80: Flags [F.], seq 140, ack 164, win 22, options [nop,nop,TS val 4071122385 ecr 593222586], length 0 16:53:56.109034 fa:16:3e:e5:06:68 > fa:16:3e:4a:e3:cd, ethertype IPv4 (0x0800), length 66: 10.2.1.201.80 > 10.2.1.220.45851: Flags [.], ack 141, win 4075, options [nop,nop,TS val 593222587 ecr 4071122385], length 0 16:53:57.256115 fa:16:3e:e5:06:68 > fa:16:3e:4a:e3:cd, ethertype IPv4 (0x0800), length 74: 10.2.1.201.80 > 10.2.1.220.45847: Flags [S.], seq 1063921752, ack 479897053, win 65330, options [mss 1402,sackOK,TS Many unreplied-SYN packets sent by the load balancer compute-1 (backend server): tap71ade383-18.pcap 16:53:48.044206 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 74: 10.2.1.220.45847 > 10.2.1.201.80: Flags [S], seq 479897052, win 43462, options [mss 1402,sackOK,TS val 4071114320 ecr 0,nop,wscale 11], length 0 16:53:48.044545 fa:16:3e:e5:06:68 > fa:16:3e:88:cd:6d, ethertype IPv4 (0x0800), length 74: 10.2.1.201.80 > 10.2.1.220.45847: Flags [S.], seq 1063921752, ack 479897053, win 65330, options [mss 1402,sackOK,TS val 593214523 ecr 4071114320,nop,wscale 4], length 0 16:53:49.045318 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 74: 10.2.1.220.45847 > 10.2.1.201.80: Flags [S], seq 479897052, win 43462, options [mss 1402,sackOK,TS val 4071115322 ecr 0,nop,wscale 11], length 0 16:53:49.045541 fa:16:3e:e5:06:68 > fa:16:3e:88:cd:6d, ethertype IPv4 (0x0800), length 74: 10.2.1.201.80 > 10.2.1.220.45847: Flags [S.], seq 1063921752, ack 479897053, win 65330, options [mss 1402,sackOK,TS val 593215524 ecr 4071114320,nop,wscale 4], length 0 16:53:50.055051 fa:16:3e:e5:06:68 > fa:16:3e:88:cd:6d, ethertype IPv4 (0x0800), length 74: 10.2.1.201.80 > 10.2.1.220.45847: Flags [S.], seq 1063921752, ack 479897053, win 65330, options [mss 1402,sackOK,TS val 593216534 ecr 4071114320,nop,wscale 4], length 0 16:53:51.092783 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 74: 10.2.1.220.45847 > 10.2.1.201.80: Flags [S], seq 479897052, win 43462, options [mss 1402,sackOK,TS val 4071117370 ecr 0,nop,wscale 11], length 0 16:53:51.093071 fa:16:3e:e5:06:68 > fa:16:3e:88:cd:6d, ethertype IPv4 (0x0800), length 74: 10.2.1.201.80 > 10.2.1.220.45847: Flags [S.], seq 1063921752, ack 479897053, win 65330, options [mss 1402,sackOK,TS val 593217572 ecr 4071114320,nop,wscale 4], length 0 16:53:53.050092 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 74: 10.2.1.220.45851 > 10.2.1.201.80: Flags [S], seq 1703562389, win 43462, options [mss 1402,sackOK,TS val 4071119327 ecr 0,nop,wscale 11], length 0 16:53:53.050590 fa:16:3e:e5:06:68 > fa:16:3e:88:cd:6d, ethertype IPv4 (0x0800), length 74: 10.2.1.201.80 > 10.2.1.220.45851: Flags [S.], seq 2546563750, ack 1703562390, win 65330, options [mss 1402,sackOK,TS val 593219529 ecr 4071119327,nop,wscale 4], length 0 16:53:53.094983 fa:16:3e:e5:06:68 > fa:16:3e:88:cd:6d, ethertype IPv4 (0x0800), length 74: 10.2.1.201.80 > 10.2.1.220.45847: Flags [S.], seq 1063921752, ack 479897053, win 65330, options [mss 1402,sackOK,TS val 593219574 ecr 4071114320,nop,wscale 4], length 0 16:53:53.159033 fa:16:3e:e5:06:68 > fa:16:3e:88:cd:6d, ethertype ARP (0x0806), length 42: Request who-has 10.2.1.220 tell 10.2.1.201, length 28 16:53:53.159674 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype ARP (0x0806), length 42: Reply 10.2.1.220 is-at fa:16:3e:4a:e3:cd, length 28 16:53:54.055053 fa:16:3e:e5:06:68 > fa:16:3e:88:cd:6d, ethertype IPv4 (0x0800), length 74: 10.2.1.201.80 > 10.2.1.220.45851: Flags [S.], seq 2546563750, ack 1703562390, win 65330, options [mss 1402,sackOK,TS val 593220534 ecr 4071119327,nop,wscale 4], length 0 16:53:54.100741 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 74: 10.2.1.220.45851 > 10.2.1.201.80: Flags [S], seq 1703562389, win 43462, options [mss 1402,sackOK,TS val 4071120378 ecr 0,nop,wscale 11], length 0 16:53:54.101008 fa:16:3e:e5:06:68 > fa:16:3e:88:cd:6d, ethertype IPv4 (0x0800), length 74: 10.2.1.201.80 > 10.2.1.220.45851: Flags [S.], seq 2546563750, ack 1703562390, win 65330, options [mss 1402,sackOK,TS val 593220580 ecr 4071119327,nop,wscale 4], length 0 16:53:54.183218 fa:16:3e:e5:06:68 > fa:16:3e:88:cd:6d, ethertype ARP (0x0806), length 42: Request who-has 10.2.1.220 tell 10.2.1.201, length 28 16:53:54.183645 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype ARP (0x0806), length 42: Reply 10.2.1.220 is-at fa:16:3e:4a:e3:cd, length 28 16:53:56.103047 fa:16:3e:e5:06:68 > fa:16:3e:4a:e3:cd, ethertype IPv4 (0x0800), length 74: 10.2.1.201.80 > 10.2.1.220.45851: Flags [S.], seq 2546563750, ack 1703562390, win 65330, options [mss 1402,sackOK,TS val 593222582 ecr 4071119327,nop,wscale 4], length 0 16:53:56.105709 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 66: 10.2.1.220.45851 > 10.2.1.201.80: Flags [.], ack 1, win 22, options [nop,nop,TS val 4071122382 ecr 593222582], length 0 16:53:56.105911 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 205: 10.2.1.220.45851 > 10.2.1.201.80: Flags [P.], seq 1:140, ack 1, win 22, options [nop,nop,TS val 4071122383 ecr 593222582], length 139: HTTP: GET / HTTP/1.1 16:53:56.106018 fa:16:3e:e5:06:68 > fa:16:3e:4a:e3:cd, ethertype IPv4 (0x0800), length 66: 10.2.1.201.80 > 10.2.1.220.45851: Flags [.], ack 140, win 4075, options [nop,nop,TS val 593222585 ecr 4071122383], length 0 16:53:56.106828 fa:16:3e:e5:06:68 > fa:16:3e:4a:e3:cd, ethertype IPv4 (0x0800), length 228: 10.2.1.201.80 > 10.2.1.220.45851: Flags [P.], seq 1:163, ack 140, win 4075, options [nop,nop,TS val 593222586 ecr 4071122383], length 162: HTTP: HTTP/1.1 200 OK 16:53:56.107020 fa:16:3e:e5:06:68 > fa:16:3e:4a:e3:cd, ethertype IPv4 (0x0800), length 66: 10.2.1.201.80 > 10.2.1.220.45851: Flags [F.], seq 163, ack 140, win 4075, options [nop,nop,TS val 593222586 ecr 4071122383], length 0 16:53:56.107256 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 66: 10.2.1.220.45851 > 10.2.1.201.80: Flags [.], ack 163, win 22, options [nop,nop,TS val 4071122384 ecr 593222586], length 0 16:53:56.108160 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 66: 10.2.1.220.45851 > 10.2.1.201.80: Flags [F.], seq 140, ack 164, win 22, options [nop,nop,TS val 4071122385 ecr 593222586], length 0 16:53:56.108381 fa:16:3e:e5:06:68 > fa:16:3e:4a:e3:cd, ethertype IPv4 (0x0800), length 66: 10.2.1.201.80 > 10.2.1.220.45851: Flags [.], ack 141, win 4075, options [nop,nop,TS val 593222587 ecr 4071122385], length 0 16:53:57.255104 fa:16:3e:e5:06:68 > fa:16:3e:4a:e3:cd, ethertype IPv4 (0x0800), length 74: 10.2.1.201.80 > 10.2.1.220.45847: Flags [S.], seq 1063921752, ack 479897053, win 65330, options [mss 1402,sackOK,TS val 593223734 ecr 4071114320,nop,wscale 4], length 0 16:53:57.256286 fa:16:3e:4a:e3:cd > fa:16:3e:e5:06:68, ethertype IPv4 (0x0800), length 54: 10.2.1.220.45847 > 10.2.1.201.80: Flags [R], seq 479897053, win 0, length 0 The backend server sent the SYN-ACK packets to an incorrect mac address. This was fixed at 16:53:56.103047 after it sent an ARP request The actual response of the load balancer was sent 8 sec after the first SYN (while the timeout in octavia-tempest-plugin is 2 sec) Searching for the ip address of the member port in the logs: controller-0 /var/log/containers/octavia/worker.log.2.gz:2023-04-06 16:52:38.849 20 DEBUG novaclient.v2.client [req-1158c1eb-1729-4265-b770-aea13fbea62d - 805c65b4a9e54d469f2fcc3ea75dc27c - - -] RESP BODY: {"interfaceAttachment": {"net_id": "6f714bea-bfd8-480d-978b-176910fe426a", "port_id": "d9ee04bd-6cbb-4112-b979-08f83f774b01", "mac_addr": "fa:16:3e:4a:e3:cd", "port_state": "ACTIVE", "fixed_ips": [{"subnet_id": "042d5e5f-e26c-4887-8119-65fdcfa32e0e", "ip_address": "10.2.1.220"}, {"subnet_id": "a50291c7-c2f8-4c57-8832-98f52b203c04", "ip_address": "fd7b:f9f7:fff:4eca::252"}]}} _http_log_response /usr/lib/python3.9/site-packages/keystoneauth1/session.py:582 controller-1 /var/log/containers/octavia/worker.log.2.gz:2023-04-06 16:39:24.600 14 DEBUG novaclient.v2.client [req-0bc9021c-2d2f-4d42-aa93-d4191316aaa8 - 227487505cc545bca3f3a297caf1f4b2 - - -] RESP BODY: {"interfaceAttachment": {"net_id": "6f714bea-bfd8-480d-978b-176910fe426a", "port_id": "3cca7c41-5943-4f7a-9ea2-44e51a00dfac", "mac_addr": "fa:16:3e:88:cd:6d", "port_state": "ACTIVE", "fixed_ips": [{"subnet_id": "042d5e5f-e26c-4887-8119-65fdcfa32e0e", "ip_address": "10.2.1.220"}, {"subnet_id": "a50291c7-c2f8-4c57-8832-98f52b203c04", "ip_address": "fd7b:f9f7:fff:4eca::135"}]}} _http_log_response /usr/lib/python3.9/site-packages/keystoneauth1/session.py:582 controller-2 /var/log/containers/octavia/worker.log.2.gz:2023-04-06 16:40:08.493 14 DEBUG novaclient.v2.client [req-74362e60-ca00-49e3-b70c-2bf3d601a82e - 227487505cc545bca3f3a297caf1f4b2 - - -] REQ: curl -g -i -X DELETE http://172.17.1.141:8774/v2.1/servers/94874609-d041-43a2-a392-c69121b577ab/os-interface/3cca7c41-5943-4f7a-9ea2-44e51a00dfac -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA256}517e872183a204bf09d94fb9e06363c4f9f023b2a6ded4fd0cd6b846553203b7" -H "X-OpenStack-Nova-API-Version: 2.15" _http_log_request /usr/lib/python3.9/site-packages/keystoneauth1/session.py:519 /var/log/containers/octavia/worker.log.2.gz:2023-04-06 16:40:08.587 14 DEBUG novaclient.v2.client [req-74362e60-ca00-49e3-b70c-2bf3d601a82e - 227487505cc545bca3f3a297caf1f4b2 - - -] DELETE call to compute for http://172.17.1.141:8774/v2.1/servers/94874609-d041-43a2-a392-c69121b577ab/os-interface/3cca7c41-5943-4f7a-9ea2-44e51a00dfac used request id req-9d4ae163-298e-4bc8-8dc0-9bf6949716ec request /usr/lib/python3.9/site-packages/keystoneauth1/session.py:954 /var/log/containers/octavia/worker.log.2.gz:2023-04-06 16:40:09.833 14 DEBUG octavia.network.drivers.neutron.allowed_address_pairs [req-74362e60-ca00-49e3-b70c-2bf3d601a82e - 227487505cc545bca3f3a297caf1f4b2 - - -] VIP instance port 3cca7c41-5943-4f7a-9ea2-44e51a00dfac already deleted. Skipping. delete_port /usr/lib/python3.9/site-packages/octavia/network/drivers/neutron/allowed_address_pairs.py:813 At 16:39:24, a port 3cca7c41-5943-4f7a-9ea2-44e51a00dfac was created with the ip address 10.2.1.220 and mac address fa:16:3e:88:cd:6d At 16:40:08, this port was deleted At 16:52:38, a port d9ee04bd-6cbb-4112-b979-08f83f774b01 was created with the same ip addr (10.2.1.220) but a different mac address fa:16:3e:4a:e3:cd It seems that the cirros VM didn't update the ARP table when it received a packet from the newly created port and replied using the mac address of the old port. This issue was mitigated automatically after 8 sec but it triggered an error in the test. Another way to mitigate it would be to send a GARP from the load balancer when a new port is plugged into the amphora.
Bulk moving target milestone to GA after the release of Beta on 14th June '23.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:4577