Description of problem: I've set up OSP11 with dvr (working fine) and I am following: https://docs.openstack.org/ocata/networking-guide/config-service-subnets.html Goal is to configure multiple subnets to split the ip assignment to the various ports (network:router_gateway, network:floatingip_agent_gateway, network:floatingip). I've started by defining two subnets, one for network:router_gateway and the other for network:floatingip_agent_gateway as I am only focusing to get the routers right for now: +-------------------+---------------------------------------+ | Field | Value | +-------------------+---------------------------------------+ | allocation_pools | 192.168.10.2-192.168.10.254 | | cidr | 192.168.10.0/24 | | created_at | 2018-02-08T07:54:43Z | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | 192.168.10.1 | | host_routes | | | id | 5be89e1a-b126-4419-9ce8-bb2683fbb243 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | demo-floating-ip-agent-gateway-subnet | | network_id | 833a76a9-4dfd-4bf0-9441-a86c17975125 | | project_id | 08493cd2fc7c46d5b1aeaa7a3dce43b8 | | revision_number | 3 | | segment_id | None | | service_types | network:floatingip_agent_gateway | | subnetpool_id | None | | updated_at | 2018-02-08T07:54:43Z | +-------------------+---------------------------------------+ +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 192.168.30.2-192.168.30.254 | | cidr | 192.168.30.0/24 | | created_at | 2018-02-08T07:54:47Z | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | 192.168.30.1 | | host_routes | | | id | b8e8a7e6-0960-4242-a18c-cffeb5ac8fa1 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | demo-snat-subnet | | network_id | 833a76a9-4dfd-4bf0-9441-a86c17975125 | | project_id | 08493cd2fc7c46d5b1aeaa7a3dce43b8 | | revision_number | 3 | | segment_id | None | | service_types | network:router_gateway | | subnetpool_id | None | | updated_at | 2018-02-08T07:54:47Z | +-------------------+--------------------------------------+ At this point I created a router and tried to set the gateway. Router got the IP from the right subnet, but port binding failed with: 2018-02-08 07:56:27.726 2662 DEBUG neutron.agent.l3.agent [-] Starting router update for 86a41cde-caa1-42a8-887b-88931d7ee6fc, action None, priority 0 _process_router_update /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:485 2018-02-08 07:56:27.727 2662 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 193bca14f873421ea17be8867a593252 exchange 'neutron' topic 'q-l3-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:562 2018-02-08 07:56:29.855 2662 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 15 __log_wakeup /usr/lib/python2.7/site-packages/ovs/poller.py:202 2018-02-08 07:56:30.386 2662 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 193bca14f873421ea17be8867a593252 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_dri vers/amqpdriver.py:419 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent [-] Failed to fetch router information for '86a41cde-caa1-42a8-887b-88931d7ee6fc' 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent Traceback (most recent call last): 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 495, in _process_router_update 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent [update.id]) 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 104, in get_routers 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent router_ids=router_ids) 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 151, in call 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent return self._original_context.call(ctxt, method, **kwargs) 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent retry=self.retry) 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent timeout=timeout, retry=retry) 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent retry=retry) 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent raise result 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent RemoteError: Remote error: IpAddressGenerationFailureNoMatchingSubnet No valid service subnet for the given device owner. 2018-02-08 07:56:30.388 2662 ERROR neutron.agent.l3.agent [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _pro cess_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch\n return self._do_dispatc h(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File " /usr/lib/python2.7/site-packages/neutron/db/api.py", line 95, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", lin e 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper\n ectxt.value = e.inner_exc\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 131, in wrapped\n traceback.format_exc())\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped\n return f(*dup_args, **dup_kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 126, in sync_routers\n self._ensure_host_set_on_ports(context, host, routers)\n', u' File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 140, in _ensure_host_set_on_ports\n router[\'id\'])\n', u' File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 183, in _ensure_host_set_on_port\n {\'port\': {portbindings.HOST_ID: host}})\n', u' File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 775, in inner\n return f(self, context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 166, in wrapped\n return method(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 95, in wrapped\n setattr(e, \'_RETRY_EXCEEDED\', True)\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 91, in wrapped\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 151, in wrapper\n ectxt.value = e.inner_exc\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 139, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 131, in wrapped\n traceback.format_exc())\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 126, in wrapped\n return f(*dup_args, **dup_kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 1316, in update_port\n port)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/api.py", line 166, in wrapped\n return method(*args, **kwargs)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1273, in update_port\n new_port=new_port)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/ipam_backend_mixin.py", line 773, in update_port\n context, old_port[\'network_id\'], host)\n', u' File "/usr/lib/python2.7/site-packages/neutron/db/ipam_backend_mixin.py", line 700, in _ipam_get_subnets\n raise ipam_exceptions.IpAddressGenerationFailureNoMatchingSubnet()\n', u'IpAddressGenerationFailureNoMatchingSubnet: No valid service subnet for the given device owner.\n']. [stack@undercloud-11 ~]$ openstack router list +--------------------------------------+-------------+--------+-------+-------------+-------+----------------------------------+ | ID | Name | Status | State | Distributed | HA | Project | +--------------------------------------+-------------+--------+-------+-------------+-------+----------------------------------+ | 86a41cde-caa1-42a8-887b-88931d7ee6fc | demo-router | ACTIVE | UP | True | False | 08493cd2fc7c46d5b1aeaa7a3dce43b8 | +--------------------------------------+-------------+--------+-------+-------------+-------+----------------------------------+ [stack@undercloud-11 ~]$ openstack port list +--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+ | 137b452f-d9ab-4cdc-bc83-b6bd2cd7dc57 | | fa:16:3e:bd:66:c9 | ip_address='172.16.0.6', subnet_id='126d91a6-58b0-415b-9ac4-965c960708f2' | DOWN | | 6d796319-6376-427f-bfd8-a9448b8d3e45 | | fa:16:3e:a1:a5:c8 | ip_address='172.16.0.2', subnet_id='126d91a6-58b0-415b-9ac4-965c960708f2' | ACTIVE | | 9399805c-79ff-49d8-9a42-be844a92e576 | | fa:16:3e:4d:d8:a5 | ip_address='172.16.0.7', subnet_id='126d91a6-58b0-415b-9ac4-965c960708f2' | ACTIVE | | b0ed23c9-f9d2-4d05-a969-73a88be07023 | | fa:16:3e:a1:52:27 | ip_address='172.16.0.1', subnet_id='126d91a6-58b0-415b-9ac4-965c960708f2' | ACTIVE | | eab5c60f-ca2f-4487-a384-1232187dd2be | | fa:16:3e:91:33:ed | ip_address='192.168.30.10', subnet_id='b8e8a7e6-0960-4242-a18c-cffeb5ac8fa1' | DOWN | +--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+ [stack@undercloud-11 ~]$ openstack port show eab5c60f-ca2f-4487-a384-1232187dd2be +-----------------------+------------------------------------------------------------------------------+ | Field | Value | +-----------------------+------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | normal | | created_at | 2018-02-08T07:56:24Z | | description | | | device_id | 86a41cde-caa1-42a8-887b-88931d7ee6fc | | device_owner | network:router_gateway | | dns_assignment | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='192.168.30.10', subnet_id='b8e8a7e6-0960-4242-a18c-cffeb5ac8fa1' | | id | eab5c60f-ca2f-4487-a384-1232187dd2be | | ip_address | None | | mac_address | fa:16:3e:91:33:ed | | name | | | network_id | 833a76a9-4dfd-4bf0-9441-a86c17975125 | | option_name | None | | option_value | None | | port_security_enabled | False | | project_id | | | qos_policy_id | None | | revision_number | 4 | | security_groups | | | status | DOWN | | subnet_id | None | | updated_at | 2018-02-08T07:56:24Z | +-----------------------+------------------------------------------------------------------------------+ This looks similar to what reported here: https://bugs.launchpad.net/neutron/+bug/1637366 - if I add one more subnet for 'network:floatingip' and try setting the gateway for that router I get the very same error as in the upstream bug (HostNotCompatibleWithFixedIps). It is reproducible at will but please let me know if/how I can help here. Thanks!
So I think I see the problem. On the original port creation, _ipam_get_subnets() was called with the port's device_owner as the service_type argument. But this code path in ipam_backend_mixin.py that is calling _ipam_get_subnets() is not passing the old port's device_owner. In your case, there is no "fallback" subnet without a service_type set, so the allocation fails. I think this one-line change would fix it in neutron/db/ipam_backend_mixin.py:update_port(): valid_subnets = self._ipam_get_subnets( context, old_port['network_id'], host, service_type=old_port.get('device_owner')) If you have a setup you can test on where you could modify the code and restart neutron-server to verify it helps I'd appreciate the feedback, else I'll try and do this locally soon. I also think it would be good to have a more descriptive message that also tells the network_id and the service_type that is failing, that might have helped me narrow this down a little quicker. I will make that change as well.
(In reply to Brian Haley from comment #2) > So I think I see the problem. > > On the original port creation, _ipam_get_subnets() was called with the > port's device_owner as the service_type argument. But this code path in > ipam_backend_mixin.py that is calling _ipam_get_subnets() is not passing the > old port's device_owner. In your case, there is no "fallback" subnet > without a service_type set, so the allocation fails. > > I think this one-line change would fix it in > neutron/db/ipam_backend_mixin.py:update_port(): > > valid_subnets = self._ipam_get_subnets( > > context, old_port['network_id'], host, > > service_type=old_port.get('device_owner')) > > If you have a setup you can test on where you could modify the code and > restart neutron-server to verify it helps I'd appreciate the feedback, else > I'll try and do this locally soon. > > I also think it would be good to have a more descriptive message that also > tells the network_id and the service_type that is failing, that might have > helped me narrow this down a little quicker. I will make that change as > well. Thanks Brian, Lab is still available, I'll test and report ASAP. Cheers Luca
Hi Brian, working like a charm :) [stack@undercloud-11 ~]$ openstack port list +--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+ | 040736af-1130-4aae-8378-ba09dee4e782 | | fa:16:3e:de:71:f1 | ip_address='172.16.0.8', subnet_id='126d91a6-58b0-415b-9ac4-965c960708f2' | ACTIVE | | 0dee6381-9637-466b-9694-426a474e16d1 | | fa:16:3e:65:14:e9 | ip_address='172.16.0.3', subnet_id='126d91a6-58b0-415b-9ac4-965c960708f2' | ACTIVE | | 4b271798-e03a-455b-af10-ee963f5f904a | | fa:16:3e:ea:6c:5f | ip_address='192.168.30.2', subnet_id='a37cbc9e-7667-465b-8de8-191d76c739be' | ACTIVE | | 6d796319-6376-427f-bfd8-a9448b8d3e45 | | fa:16:3e:a1:a5:c8 | ip_address='172.16.0.2', subnet_id='126d91a6-58b0-415b-9ac4-965c960708f2' | ACTIVE | | 8ccf2b8b-ee37-4e43-9fe7-4d899fd0ba81 | | fa:16:3e:13:10:9e | ip_address='192.168.10.8', subnet_id='78e06d59-a3ba-4afd-83d6-da70d605e944' | ACTIVE | | 9399805c-79ff-49d8-9a42-be844a92e576 | | fa:16:3e:4d:d8:a5 | ip_address='172.16.0.7', subnet_id='126d91a6-58b0-415b-9ac4-965c960708f2' | ACTIVE | | b0ed23c9-f9d2-4d05-a969-73a88be07023 | | fa:16:3e:a1:52:27 | ip_address='172.16.0.1', subnet_id='126d91a6-58b0-415b-9ac4-965c960708f2' | ACTIVE | | cff5f3fd-bf39-4438-8ae7-ff746550b869 | | fa:16:3e:14:68:50 | ip_address='192.168.20.2', subnet_id='8bf44b3f-afcd-4e12-b4de-0903ea34cfda' | N/A | +--------------------------------------+------+-------------------+-----------------------------------------------------------------------------+--------+ [stack@undercloud-11 ~]$ openstack subnet show a37cbc9e-7667-465b-8de8-191d76c739be +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 192.168.30.2-192.168.30.254 | | cidr | 192.168.30.0/24 | | created_at | 2018-02-12T17:55:35Z | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | 192.168.30.1 | | host_routes | | | id | a37cbc9e-7667-465b-8de8-191d76c739be | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | demo-snat-subnet | | network_id | 99b466e9-0108-4176-aa0b-dc43f9e583e9 | | project_id | 08493cd2fc7c46d5b1aeaa7a3dce43b8 | | revision_number | 3 | | segment_id | None | | service_types | network:router_gateway | | subnetpool_id | None | | updated_at | 2018-02-12T17:55:35Z | +-------------------+--------------------------------------+ [stack@undercloud-11 ~]$ openstack subnet show 8bf44b3f-afcd-4e12-b4de-0903ea34cfda +-------------------+--------------------------------------+ | Field | Value | +-------------------+--------------------------------------+ | allocation_pools | 192.168.20.2-192.168.20.254 | | cidr | 192.168.20.0/24 | | created_at | 2018-02-12T17:55:40Z | | description | | | dns_nameservers | | | enable_dhcp | False | | gateway_ip | 192.168.20.1 | | host_routes | | | id | 8bf44b3f-afcd-4e12-b4de-0903ea34cfda | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | demo-floating-ip-subnet | | network_id | 99b466e9-0108-4176-aa0b-dc43f9e583e9 | | project_id | 08493cd2fc7c46d5b1aeaa7a3dce43b8 | | revision_number | 3 | | segment_id | None | | service_types | network:floatingip | | subnetpool_id | None | | updated_at | 2018-02-12T17:55:40Z | +-------------------+--------------------------------------+ connectivity looks good as well: [root@garbd ~]# ping 192.168.30.2 PING 192.168.30.2 (192.168.30.2) 56(84) bytes of data. 64 bytes from 192.168.30.2: icmp_seq=1 ttl=64 time=1.11 ms 64 bytes from 192.168.30.2: icmp_seq=2 ttl=64 time=0.481 ms ^C --- 192.168.30.2 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 0.481/0.797/1.114/0.317 ms [root@garbd ~]# ping 192.168.20.2 PING 192.168.20.2 (192.168.20.2) 56(84) bytes of data. ^C [root@garbd ~]# ping 192.168.20.2 PING 192.168.20.2 (192.168.20.2) 56(84) bytes of data. 64 bytes from 192.168.20.2: icmp_seq=11 ttl=62 time=1.20 ms 64 bytes from 192.168.20.2: icmp_seq=12 ttl=62 time=0.834 ms 64 bytes from 192.168.20.2: icmp_seq=13 ttl=62 time=0.777 ms thank you very much for the fast turnaround!
Thanks for the quick testing, I'll get an upstream review up with the fix and work on a backport.
https://review.openstack.org/#/c/545615/ is now passing, was just a random upstream failure, I will start the backports to OSP.
(In reply to Brian Haley from comment #7) > https://review.openstack.org/#/c/545615/ is now passing, was just a random > upstream failure, I will start the backports to OSP. Hi Brian, I've seen the patch merged. Do you think it would be feasible to have this patch backported before the next OSP11 maint release? Thanks Luca
OSP11 change: https://code.engineering.redhat.com/gerrit/#/c/133138/ I'll clone this bug since it's also needed in 12 and 13.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1614