Description of problem: After restarting of ovn-metadata container, metadata server could not be reached from the nova server: Traceback (most recent call last): File "/home/stack/src/x/tobiko/tobiko/tests/faults/neutron/test_agents.py", line 854, in assert_metadata_is_reachable result = sh.execute(f"curl '{metadata_url}' I", File "/home/stack/src/x/tobiko/tobiko/shell/sh/_execute.py", line 157, in execute return execute_process(process=process, File "/home/stack/src/x/tobiko/tobiko/shell/sh/_execute.py", line 206, in execute_process error.reraise() File "/home/stack/src/x/tobiko/tobiko/common/_exception.py", line 109, in reraise reraise(*self) File "/home/stack/src/x/tobiko/tobiko/common/_exception.py", line 118, in reraise raise value File "/home/stack/src/x/tobiko/tobiko/shell/sh/_execute.py", line 185, in execute_process process.check_exit_status(expect_exit_status) File "/home/stack/src/x/tobiko/tobiko/shell/sh/_process.py", line 424, in check_exit_status raise ex tobiko.shell.sh._exception.ShellCommandFailed: command 'curl http://169.254.169.254/latest/meta-data/ -I' failed (exit status is 7); stdin: None stdout: stderr: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --::- -::- -::- 0 0 0 0 0 0 0 0 0 -::- -::- -::- 0 curl: (7) Failed to connect to 169.254.169.254 port 80: Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/stack/src/x/tobiko/tobiko/tests/faults/neutron/test_agents.py", line 903, in test_restart_metadata_containers self.wait_for_metadata_status(is_reachable=True) File "/home/stack/src/x/tobiko/tobiko/tests/faults/neutron/test_agents.py", line 834, in wait_for_metadata_status attempt.check_limits() File "/home/stack/src/x/tobiko/tobiko/common/_retry.py", line 103, in check_limits self.check_time_left() File "/home/stack/src/x/tobiko/tobiko/common/_retry.py", line 97, in check_time_left _exception.exc_info().reraise() File "/home/stack/src/x/tobiko/tobiko/common/_exception.py", line 109, in reraise reraise(*self) File "/home/stack/src/x/tobiko/tobiko/common/_exception.py", line 118, in reraise raise value File "/home/stack/src/x/tobiko/tobiko/tests/faults/neutron/test_agents.py", line 831, in wait_for_metadata_status self.assert_metadata_is_reachable(is_reachable) File "/home/stack/src/x/tobiko/tobiko/tests/faults/neutron/test_agents.py", line 858, in assert_metadata_is_reachable self.assertFalse(is_reachable, File "/usr/lib64/python3.9/unittest/case.py", line 682, in assertFalse raise self.failureException(msg) AssertionError: True is not false : Metadata server not reached from Nova server: exit_status=7 In the test [2] ovn_metadata container was restarted: 2024-08-21 14:10:33.650 201892 DEBUG tobiko.shell.sh._process - Waiting for process data 1.0 seconds... command: sudo podman restart ovn_metadata_agent attempt: number=1, elapsed_time=0.0 streams: {<tobiko.shell.sh._io.ShellStderr object at 0x7f7af4c7eb20>, <tobiko.shell.sh._io.ShellStdout object at 0x7f7b42b0adf0>} 2024-08-21 14:10:33.650 201892 DEBUG tobiko.common._retry - retrying without waiting... (number=1, elapsed_time=0.0) 2024-08-21 14:10:34.652 201892 DEBUG tobiko.shell.sh._process - Waiting for process data 1.0 seconds... And probably at the same time in neutron log[2] error was indicated: ++ grep -E ':(Exited|Created)' 08b9f5afc99c:neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19:Initialized' + ORPHANTS= + '[' -n '' ']' + printf '%s\n' '037fe288747f:neutron-haproxy-ovnmeta-b592da64-4cc6-44fe-aaf0-4d4707f75c3a:Up 3eb2f97ef09c:neutron-haproxy-ovnmeta-784a2da9-9102-44da-a026-a8a69328a2d4:Up b2aa01479419:neutron-haproxy-ovnmeta-c3d5f8af-a2ca-467f-be0b-daa0498828d3:Up 08b9f5afc99c:neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19:Initialized' + grep -q 'neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19$' + echo 'Starting a new child container neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19' + nsenter --net=/run/netns/ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19 --preserve-credentials -m -t 1 podman run --detach --log-driver k8s-file --log-opt path=/var/log/containers/stdouts/neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19.log -v /var/lib/config-data/puppet-generated/neutron/etc/neutron:/etc/neutron:ro -v /run/netns:/run/netns:shared -v /var/lib/neutron:/var/lib/neutron:shared -v /dev/log:/dev/log --net host --pid host --cgroupns host --privileged -u root --name neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-neutron-metadata-agent-ovn:17.1_20240725.1 /bin/bash -c 'HAPROXY="$(if [ -f /usr/sbin/haproxy-systemd-wrapper ]; then echo "/usr/sbin/haproxy -Ds"; else echo "/usr/sbin/haproxy -Ws"; fi)"; exec $HAPROXY -f /var/lib/neutron/ovn-metadata-proxy/5217afd7-b0ee-463f-8663-3bdb47ed2f19.conf' Error: creating container storage: the container name "neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19" is already in use by 08b9f5afc99cffab5b2f9f1d79f1742ab36337b8d9943ed36cba9b6aeaaca654. You have to remove that container to be able to reuse that name: that name is already in use Version-Release number of selected component (if applicable): How reproducible: Several times Steps to Reproduce: 1.Deploy OSP17.1 with composable roles 2.Run tobiko test 14:11:44.421956 tobiko/tests/faults/neutron/test_agents.py::OvnMetadataAgentTest::test_restart_metadata_containers 3. Actual results: Attempts to access metadata server failed with the above error Expected results: All health checks pass Additional info:
Hi Miro. Thank you so much for your informative reply. We'll consider adjusting these tests.