Bug 2310111 - [OSP17.1] Metadata server is not reachable from the Nova server
Summary: [OSP17.1] Metadata server is not reachable from the Nova server
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 17.1 (Wallaby)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: zstream
: ---
Assignee: Miro Tomaska
QA Contact: Eran Kuris
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-09-05 00:56 UTC by Julia Marciano
Modified: 2024-11-07 15:49 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-11-07 15:49:26 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-32763 0 None None None 2024-09-05 00:57:01 UTC

Description Julia Marciano 2024-09-05 00:56:13 UTC
Description of problem:
After restarting of ovn-metadata container, metadata server could not be reached from the nova server:
Traceback (most recent call last): File "/home/stack/src/x/tobiko/tobiko/tests/faults/neutron/test_agents.py", line 854, in assert_metadata_is_reachable result = sh.execute(f"curl '{metadata_url}' I", File "/home/stack/src/x/tobiko/tobiko/shell/sh/_execute.py", line 157, in execute return execute_process(process=process, File "/home/stack/src/x/tobiko/tobiko/shell/sh/_execute.py", line 206, in execute_process error.reraise() File "/home/stack/src/x/tobiko/tobiko/common/_exception.py", line 109, in reraise reraise(*self) File "/home/stack/src/x/tobiko/tobiko/common/_exception.py", line 118, in reraise raise value File "/home/stack/src/x/tobiko/tobiko/shell/sh/_execute.py", line 185, in execute_process process.check_exit_status(expect_exit_status) File "/home/stack/src/x/tobiko/tobiko/shell/sh/_process.py", line 424, in check_exit_status raise ex tobiko.shell.sh._exception.ShellCommandFailed: command 'curl http://169.254.169.254/latest/meta-data/ -I' failed (exit status is 7); stdin: None stdout: stderr: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --::- -::- -::- 0 0 0 0 0 0 0 0 0 -::- -::- -::- 0 curl: (7) Failed to connect to 169.254.169.254 port 80: Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/stack/src/x/tobiko/tobiko/tests/faults/neutron/test_agents.py", line 903, in test_restart_metadata_containers self.wait_for_metadata_status(is_reachable=True) File "/home/stack/src/x/tobiko/tobiko/tests/faults/neutron/test_agents.py", line 834, in wait_for_metadata_status attempt.check_limits() File "/home/stack/src/x/tobiko/tobiko/common/_retry.py", line 103, in check_limits self.check_time_left() File "/home/stack/src/x/tobiko/tobiko/common/_retry.py", line 97, in check_time_left _exception.exc_info().reraise() File "/home/stack/src/x/tobiko/tobiko/common/_exception.py", line 109, in reraise reraise(*self) File "/home/stack/src/x/tobiko/tobiko/common/_exception.py", line 118, in reraise raise value File "/home/stack/src/x/tobiko/tobiko/tests/faults/neutron/test_agents.py", line 831, in wait_for_metadata_status self.assert_metadata_is_reachable(is_reachable) File "/home/stack/src/x/tobiko/tobiko/tests/faults/neutron/test_agents.py", line 858, in assert_metadata_is_reachable self.assertFalse(is_reachable, File "/usr/lib64/python3.9/unittest/case.py", line 682, in assertFalse raise self.failureException(msg) AssertionError: True is not false : Metadata server not reached from Nova server: exit_status=7

In the test [2] ovn_metadata container was restarted:
2024-08-21 14:10:33.650 201892 DEBUG tobiko.shell.sh._process - Waiting for process data 1.0 seconds... 
  command: sudo podman restart ovn_metadata_agent
  attempt: number=1, elapsed_time=0.0
  streams: {<tobiko.shell.sh._io.ShellStderr object at 0x7f7af4c7eb20>, <tobiko.shell.sh._io.ShellStdout object at 0x7f7b42b0adf0>}
2024-08-21 14:10:33.650 201892 DEBUG tobiko.common._retry - retrying without waiting... (number=1, elapsed_time=0.0)
2024-08-21 14:10:34.652 201892 DEBUG tobiko.shell.sh._process - Waiting for process data 1.0 seconds... 

And probably at the same time in neutron log[2] error was indicated:
++ grep -E ':(Exited|Created)'
08b9f5afc99c:neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19:Initialized'
+ ORPHANTS=
+ '[' -n '' ']'
+ printf '%s\n' '037fe288747f:neutron-haproxy-ovnmeta-b592da64-4cc6-44fe-aaf0-4d4707f75c3a:Up
3eb2f97ef09c:neutron-haproxy-ovnmeta-784a2da9-9102-44da-a026-a8a69328a2d4:Up
b2aa01479419:neutron-haproxy-ovnmeta-c3d5f8af-a2ca-467f-be0b-daa0498828d3:Up
08b9f5afc99c:neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19:Initialized'
+ grep -q 'neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19$'
+ echo 'Starting a new child container neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19'
+ nsenter --net=/run/netns/ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19 --preserve-credentials -m -t 1 podman run --detach --log-driver k8s-file --log-opt path=/var/log/containers/stdouts/neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19.log -v /var/lib/config-data/puppet-generated/neutron/etc/neutron:/etc/neutron:ro -v /run/netns:/run/netns:shared -v /var/lib/neutron:/var/lib/neutron:shared -v /dev/log:/dev/log --net host --pid host --cgroupns host --privileged -u root --name neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp17-openstack-neutron-metadata-agent-ovn:17.1_20240725.1 /bin/bash -c 'HAPROXY="$(if [ -f /usr/sbin/haproxy-systemd-wrapper ]; then echo "/usr/sbin/haproxy -Ds"; else echo "/usr/sbin/haproxy -Ws"; fi)"; exec $HAPROXY -f /var/lib/neutron/ovn-metadata-proxy/5217afd7-b0ee-463f-8663-3bdb47ed2f19.conf'
Error: creating container storage: the container name "neutron-haproxy-ovnmeta-5217afd7-b0ee-463f-8663-3bdb47ed2f19" is already in use by 08b9f5afc99cffab5b2f9f1d79f1742ab36337b8d9943ed36cba9b6aeaaca654. You have to remove that container to be able to reuse that name: that name is already in use


Version-Release number of selected component (if applicable):


How reproducible:
Several times

Steps to Reproduce:
1.Deploy OSP17.1 with composable roles
2.Run tobiko test 14:11:44.421956	tobiko/tests/faults/neutron/test_agents.py::OvnMetadataAgentTest::test_restart_metadata_containers
3.

Actual results:
Attempts to access metadata server failed with the above error

Expected results:
All health checks pass

Additional info:

Comment 5 Julia Marciano 2024-09-10 11:41:30 UTC
Hi Miro.

Thank you so much for your informative reply. We'll consider adjusting these tests.


Note You need to log in before you can comment on or make changes to this bug.