Bug 2032295
| Summary: | [RHOSP 16.1] [ipv6 undercloud] undercloud update fails when you have ipv6 undercloud with enable_routed_networks=false in undercloud.conf; during stack update router interface port is tried to be re-created with the same ip | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Punit Kundal <pkundal> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Harald Jensås <hjensas> |
| Status: | CLOSED ERRATA | QA Contact: | David Rosenfeld <drosenfe> |
| Severity: | high | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 16.1 (Train) | CC: | abhijadh, hjensas, jhardee, jparoly, kurathod, mburns, morazi, sbaker, skaplons, sputhenp |
| Target Milestone: | z9 | Keywords: | Triaged |
| Target Release: | 16.1 (Train on RHEL 8.2) | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-11.3.2-1.20220125223751.29a02c1.el8ost | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-12-07 20:29:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
File \"/var/lib/heat-config/heat-config-script/713d1890-8f6c-4a76-ba57-b91194a3ce85\", line 129, in _neutron_subnet_update
File \"/usr/lib/python3.6/site-packages/openstack/network/v2/_proxy.py\", line 3469, in update_subnet
return self._update(_subnet.Subnet, subnet, **attrs)
File \"/usr/lib/python3.6/site-packages/openstack/proxy.py\", line 46, in check
return method(self, expected, actual, *args, **kwargs)
File \"/usr/lib/python3.6/site-packages/openstack/proxy.py\", line 393, in _update
return res.commit(self, base_path=base_path)
File \"/usr/lib/python3.6/site-packages/openstack/resource.py\", line 1402, in commit
retry_on_conflict=retry_on_conflict)
File \"/usr/lib/python3.6/site-packages/openstack/resource.py\", line 1428, in _commit
self._translate_response(response, has_body=has_body)
File \"/usr/lib/python3.6/site-packages/openstack/resource.py\", line 1107, in _translate_response
exceptions.raise_from_response(response, error_message=error_message)
File \"/usr/lib/python3.6/site-packages/openstack/exceptions.py\", line 229, in raise_from_response
http_status=http_status, request_id=request_id
openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: http://[2001:db8:88ec:9fb3::10]:9696/v2.0/subnets/6dfcdafe-042d-4014-99ba-d4dcdf740e02, Current gateway ip 2001:db8:88ec:9fb3::5 already in use by port e3259023-707e-4d2f-82f0-63e24fa7af5f. Unable to update.
The code that fails is[1], this method:
def _neutron_subnet_update(sdk, subnet_id, cidr, gateway, host_routes,
allocation_pools, name, dns_nameservers):
try:
if netaddr.IPNetwork(cidr).version == 6:
subnet = sdk.network.update_subnet(
subnet_id,
name=name,
gateway_ip=gateway,
allocation_pools=allocation_pools,
dns_nameservers=_get_nameservers_for_version(dns_nameservers,
6))
else:
subnet = sdk.network.update_subnet(
subnet_id,
name=name,
gateway_ip=gateway,
host_routes=host_routes,
allocation_pools=allocation_pools,
dns_nameservers=_get_nameservers_for_version(dns_nameservers,
4))
print('INFO: Subnet updated %s' % subnet)
except Exception:
print('ERROR: Update of subnet %s failed.' % name)
raise
I guess we need to only set the gateway_ip if it was changed in the update request to avoid neutron attempting to create a router port.
@skaplons, wdyt?
[1] https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/extraconfig/post_deploy/undercloud_ctlplane_network.py#L124-L147
We will backport this to 16.1, but the fix is actually in 16.2 so if you've switched to deploying with 16.2 then we can change the target of this bug and change its state. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenStack 16.1.9 (openstack-tripleo-heat-templates) security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:8796 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |
Description of problem: If we deploy an ipv6 undercloud with ipv6_address_mode=dhcpv6-stateful then in initial deploy a router is constructed on undercloud with the same ip for the tap port which is gateway ip in [ctlplane-subnet] section in undercloud.conf: +++ (undercloud) [stack@undercloud16 ~]$ neutron router-list neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+-----------------+----------------------------------+-----------------------+-------+ | id | name | tenant_id | external_gateway_info | ha | +--------------------------------------+-----------------+----------------------------------+-----------------------+-------+ | a3f95f05-7fd6-4e2d-8046-ed26f4a2ac5f | ctlplane-subnet | 31b153ae6ced41e1963e6da9ee84b379 | null | False | +--------------------------------------+-----------------+----------------------------------+-----------------------+-------+ (undercloud) [stack@undercloud16 ~]$ sudo ip netns qrouter-a3f95f05-7fd6-4e2d-8046-ed26f4a2ac5f qdhcp-44c0fe75-608a-4b8e-995a-84619acf5711 (id: 1) (undercloud) [stack@undercloud16 ~]$ sudo ip netns exec qdhcp-44c0fe75-608a-4b8e-995a-84619acf5711 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 10: tap45943298-8e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:03:cc:f3 brd ff:ff:ff:ff:ff:ff inet6 2001:db8:88ec:9fb3::5/64 scope global valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe03:ccf3/64 scope link valid_lft forever preferred_lft forever +++ here's the undercloud.conf from my lab: +++ (undercloud) [stack@undercloud16 ~]$ egrep -iv '^$|^#' undercloud.conf [DEFAULT] container_images_file = /home/stack/containers-prepare-parameter.yaml enable_telemetry = false enable_tempest = false enable_validations = false generate_service_certificate = false ipv6_address_mode = dhcpv6-stateful local_ip = 2001:db8:88ec:9fb3::10/64 overcloud_domain_name = site2a.rhlab.local undercloud_admin_host = 2001:db8:88ec:9fb3::15 undercloud_ntp_servers = XXX.XX.XX.XXX undercloud_public_host = 2001:db8:88ec:9fb3::12 [ctlplane-subnet] cidr = 2001:db8:88ec:9fb3::/64 dhcp_end = 2001:db8:88ec:9fb3::38 dhcp_start = 2001:db8:88ec:9fb3::28 gateway = 2001:db8:88ec:9fb3::5 inspection_iprange = 2001:db8:88ec:9fb3::49,2001:db8:88ec:9fb3::68 masquerade = false (undercloud) [stack@undercloud16 ~]$ +++ though say for example if you want to modify something on the undercloud node and re-run undercloud install; it fails with: +++ "[2021-12-14 05:23:08,046] (heat-config) [DEBUG] b'[2021-12-14 05:23:05,003] (heat-config) [INFO] config={\"cloud_name\": \"undercloud\", \"enable_routed_networks\": false, \"home_dir\": \"/home/stack\", \"ipv6_address_mod e\": \"dhcpv6-stateful\", \"local_ip\": \"2001:db8:88ec:9fb3::10\", \"local_subnet\": \"ctlplane-subnet\", \"mtu\": 1500, \"physical_network\": \"ctlplane\", \"subnets\": {\"ctlplane-subnet\": {\"AllocationPools\": [{\"end\": \"2001:db8:8 8ec:9fb3::38\", \"start\": \"2001:db8:88ec:9fb3::28\"}], \"DnsNameServers\": [], \"HostRoutes\": [], \"NetworkCidr\": \"2001:db8:88ec:9fb3::/64\", \"NetworkGateway\": \"2001:db8:88ec:9fb3::5\"}}}\\n[2021-12-14 05:23:05,004] (heat-config) [INFO] deploy_server_id=65db1ddd-9158-48cc-99bb-2b94f35f52c1\\n[2021-12-14 05:23:05,004] (heat-config) [INFO] deploy_action=CREATE\\n[2021-12-14 05:23:05,004] (heat-config) [INFO] deploy_stack_id=undercloud-AllNodesDeploySteps-cnjjucgzzvr 4-UndercloudExtraConfigPost-q5mkaa45phkp-UndercloudCtlplaneNetworkDeployment-po54lvmed3ms-0-27qendy7lqxz/1513ea34-bc34-425a-af59-948abb1e3637\\n[2021-12-14 05:23:05,004] (heat-config) [INFO] deploy_resource_name=TripleOSoftwareDeployment\ \n[2021-12-14 05:23:05,004] (heat-config) [INFO] deploy_signal_transport=NO_SIGNAL\\n[2021-12-14 05:23:05,004] (heat-config) [DEBUG] Running /var/lib/heat-config/heat-config-script/713d1890-8f6c-4a76-ba57-b91194a3ce85\\n[2021-12-14 05:23: 08,033] (heat-config) [INFO] b\"INFO: Network updated openstack.network.v2.network.Network(id=44c0fe75-608a-4b8e-995a-84619acf5711, name=ctlplane, mtu=1500, admin_state_up=True, status=ACTIVE, subnets=[\\'6dfcdafe-042d-4014-99ba-d4dcdf740 e02\\'], shared=False, availability_zone_hints=[], availability_zones=[\\'nova\\'], ipv4_address_scope=None, ipv6_address_scope=None, router:external=False, description=, port_security_enabled=True, tags=[\\'2001:db8:88ec:9fb3::/64\\'], c reated_at=2021-12-10T12:39:35Z, updated_at=2021-12-14T10:23:07Z, revision_number=5, project_id=31b153ae6ced41e1963e6da9ee84b379, provider:network_type=flat, provider:physical_network=ctlplane, provider:segmentation_id=None, location=Munch ({\\'cloud\\': \\'undercloud\\', \\'region_name\\': \\'regionOne\\', \\'zone\\': None, \\'project\\': Munch({\\'id\\': \\'31b153ae6ced41e1963e6da9ee84b379\\', \\'name\\': \\'admin\\', \\'domain_id\\': None, \\'domain_name\\': \\'Default\\ '})}))\\\\nERROR: Update of subnet ctlplane-subnet failed.\\\\n\"\\n[2021-12-14 05:23:08,033] (heat-config) [DEBUG] b\\'Traceback (most recent call last):\\\\n File \"/var/lib/heat-config/heat-config-script/713d1890-8f6c-4a76-ba57-b91194 a3ce85\", line 313, in <module>\\\\n net_cidrs = _local_neutron_segments_and_subnets(sdk, network.id, net_cidrs)\\\\n File \"/var/lib/heat-config/heat-config-script/713d1890-8f6c-4a76-ba57-b91194a3ce85\", line 243, in _local_neutron_s egments_and_subnets\\\\n s[\\\\\\'DnsNameServers\\\\\\'])\\\\n File \"/var/lib/heat-config/heat-config-script/713d1890-8f6c-4a76-ba57-b91194a3ce85\", line 129, in _neutron_subnet_update\\\\n 6))\\\\n File \"/usr/lib/python3.6/site -packages/openstack/network/v2/_proxy.py\", line 3469, in update_subnet\\\\n return self._update(_subnet.Subnet, subnet, **attrs)\\\\n File \"/usr/lib/python3.6/site-packages/openstack/proxy.py\", line 46, in check\\\\n return meth od(self, expected, actual, *args, **kwargs)\\\\n File \"/usr/lib/python3.6/site-packages/openstack/proxy.py\", line 393, in _update\\\\n return res.commit(self, base_path=base_path)\\\\n File \"/usr/lib/python3.6/site-packages/openst ack/resource.py\", line 1402, in commit\\\\n retry_on_conflict=retry_on_conflict)\\\\n File \"/usr/lib/python3.6/site-packages/openstack/resource.py\", line 1428, in _commit\\\\n self._translate_response(response, has_body=has_body )\\\\n File \"/usr/lib/python3.6/site-packages/openstack/resource.py\", line 1107, in _translate_response\\\\n exceptions.raise_from_response(response, error_message=error_message)\\\\n File \"/usr/lib/python3.6/site-packages/opensta ck/exceptions.py\", line 229, in raise_from_response\\\\n http_status=http_status, request_id=request_id\\\\nopenstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: http://[2001:db8:88ec:9fb3::10]:9696/v2. 0/subnets/6dfcdafe-042d-4014-99ba-d4dcdf740e02, Current gateway ip 2001:db8:88ec:9fb3::5 already in use by port e3259023-707e-4d2f-82f0-63e24fa7af5f. Unable to update.\\\\n\\'\\n[2021-12-14 05:23:08,033] (heat-config) [ERROR] Error runnin g /var/lib/heat-config/heat-config-script/713d1890-8f6c-4a76-ba57-b91194a3ce85. [1]\\n\\n'", +++ we can clearly see that heat is trying to re-create the gateway port with the same ip 2001:db8:88ec:9fb3::5; this port is already existing if we then try to do something like: +++ (undercloud) [stack@undercloud16 ~]$ neutron router-list neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+-----------------+----------------------------------+-----------------------+-------+ | id | name | tenant_id | external_gateway_info | ha | +--------------------------------------+-----------------+----------------------------------+-----------------------+-------+ | a3f95f05-7fd6-4e2d-8046-ed26f4a2ac5f | ctlplane-subnet | 31b153ae6ced41e1963e6da9ee84b379 | null | False | +--------------------------------------+-----------------+----------------------------------+-----------------------+-------+ (undercloud) [stack@undercloud16 ~]$ neutron port-list | grep -e 2001:db8:88ec:9fb3::5 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. | e3259023-707e-4d2f-82f0-63e24fa7af5f | | 31b153ae6ced41e1963e6da9ee84b379 | fa:16:3e:93:7a:1b | {"subnet_id": "6dfcdafe-042d-4014-99ba-d4dcdf740e02", "ip_address": "2001:db8:88ec:9fb3::5"} | (undercloud) [stack@undercloud16 ~]$ neutron l3-agent-list-hosting-router a3f95f05-7fd6-4e2d-8046-ed26f4a2ac5f neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +--------------------------------------+---------------------------------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+---------------------------------+----------------+-------+----------+ | a9f9c7c4-9719-4fdc-ab45-dd5fbdb7e5ac | undercloud16.site2a.rhlab.local | True | :-) | | +--------------------------------------+---------------------------------+----------------+-------+----------+ (undercloud) [stack@undercloud16 ~]$ neutron l3-agent-router-remove a9f9c7c4-9719-4fdc-ab45-dd5fbdb7e5ac a3f95f05-7fd6-4e2d-8046-ed26f4a2ac5f neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. Removed router a3f95f05-7fd6-4e2d-8046-ed26f4a2ac5f from L3 agent (undercloud) [stack@undercloud16 ~]$ sudo ip netns qdhcp-44c0fe75-608a-4b8e-995a-84619acf5711 (id: 1) (undercloud) [stack@undercloud16 ~]$ sudo podman ps | grep -e qdhcp ef74a8b3b4c1 undercloud16.ctlplane.site2a.rhlab.local:8787/rhosp-rhel8/openstack-neutron-dhcp-agent:16.1.6 /usr/sbin/dnsmasq... 7 minutes ago Up 7 minutes ago neutron-dnsmasq-qdhcp-44c0fe75-608a-4b8e-995a-84619acf5711 (undercloud) [stack@undercloud16 ~]$ (undercloud) [stack@undercloud16 ~]$ neutron router-interface-delete a3f95f05-7fd6-4e2d-8046-ed26f4a2ac5f ctlplane-subnet neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. Removed interface from router a3f95f05-7fd6-4e2d-8046-ed26f4a2ac5f. +++ if we then re-run undercloud install; the interface to the router is not attached: +++ (undercloud) [stack@undercloud16 ~]$ neutron router-port-list a3f95f05-7fd6-4e2d-8046-ed26f4a2ac5f neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +++ though the undercloud install goes fine one then needs to manually add the interface to the router: +++ (undercloud) [stack@undercloud16 ~]$ neutron router-interface-add a3f95f05-7fd6-4e2d-8046-ed26f4a2ac5f 6dfcdafe-042d-4014-99ba-d4dcdf740e02 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. Added interface eef890d8-443c-4444-89b0-b4df85251877 to router a3f95f05-7fd6-4e2d-8046-ed26f4a2ac5f. (undercloud) [stack@undercloud16 ~]$ sudo ip netns exec qrouter-a3f95f05-7fd6-4e2d-8046-ed26f4a2ac5f ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 11: qr-eef890d8-44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:de:63:e9 brd ff:ff:ff:ff:ff:ff inet6 2001:db8:88ec:9fb3::5/64 scope global valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fede:63e9/64 scope link valid_lft forever preferred_lft forever (undercloud) [stack@undercloud16 ~]$ +++ Version-Release number of selected component (if applicable): +++ (undercloud) [stack@undercloud16 ~]$ sudo rpm -qa | grep -i python3-tripleoclient python3-tripleoclient-12.3.2-1.20210608013304.ae58329.el8ost.noarch python3-tripleoclient-heat-installer-12.3.2-1.20210608013304.ae58329.el8ost.noarch +++ How reproducible: Always Steps to Reproduce: 1. 2. 3. Actual results: undercloud update tries to re-create the router port Expected results: undercloud update shouldn't have try to re-create the router port Additional info: