Description of problem: Once in a while , a network:dhcp reverts its IP to an older one and it creates a conflict with an IP that shouldn't be used [stack@undercloud ~]$ neutron port-list --network-id=8c1f75c9-bb82-4bcf-87bf-aae406f77714 --device_owner=network:dhcp +--------------------------------------+-----------------------------+-------------------+------------------------------------------------------------------------------------+ | id | name | mac_address | fixed_ips | +--------------------------------------+-----------------------------+-------------------+------------------------------------------------------------------------------------+ | 4153adcb-7ca2-4ccd-8243-fdfe06c85cf3 | dhcp3 | fa:16:3e:00:00:03 | {"subnet_id": "227578de-baef-45ca-9e7c-71c03042f20f", "ip_address": "10.10.10.10"} | | 574f9dfc-fd24-42e3-a804-1b9d848132db | dhcp2 | fa:16:3e:00:00:02 | {"subnet_id": "227578de-baef-45ca-9e7c-71c03042f20f", "ip_address": "10.10.10.11"} | | bbd72851-7373-4b75-861d-d01c021aff6a | dchp1 | fa:16:3e:00:00:01 | {"subnet_id": "227578de-baef-45ca-9e7c-71c03042f20f", "ip_address": "10.10.10.12"} | +--------------------------------------+-----------------------------+-------------------+------------------------------------------------------------------------------------+ In the above example, the port 4153adcb-7ca2-4ccd-8243-fdfe06c85cf3 ip was updated from 10.10.10.10 to 10.10.10.12 and somehow it got reverted to 10.10.10.10. Here is one of the logs where we see IPAM reserving 10.10.10.10 : server.log-20190715.gz:2019-07-15 01:31:39.862 5223 DEBUG neutron.api.rpc.handlers.dhcp_rpc [req-6c361634-ef0e-4831-affc-b7e81b79fe83 - - - - -] Update dhcp port {u'port': {u'network_id': u'8c1f75c9-bb82-4bcf-87bf-aae406f77714', 'binding:host_id': u'controller-0', u'fixed_ips': [{u'subnet_id': u'227578de-baef-45ca-9e7c-71c03042f20f'}]}, 'id': u'4153adcb-7ca2-4ccd-8243-fdfe06c85cf3'} from controller-0. update_dhcp_port /usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py:279 server.log-20190715.gz:2019-07-15 01:31:39.951 5223 DEBUG neutron.ipam.driver [req-6c361634-ef0e-4831-affc-b7e81b79fe83 - - - - -] Loading ipam driver: internal get_instance /usr/lib/python2.7/site-packages/neutron/ipam/driver.py:51 server.log-20190715.gz:2019-07-15 01:31:40.157 5223 DEBUG neutron.db.db_base_plugin_common [req-6c361634-ef0e-4831-affc-b7e81b79fe83 - - - - -] Allocated IP 10.10.10.10 (8c1f75c9-bb82-4bcf-87bf-aae406f77714/227578de-baef-45ca-9e7c-71c03042f20f/4153adcb-7ca2-4ccd-8243-fdfe06c85cf3) _store_ip_allocation /usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_common.py:115 server.log-20190715.gz:2019-07-15 01:31:40.223 5223 DEBUG neutron.plugins.ml2.db [req-6c361634-ef0e-4831-affc-b7e81b79fe83 - - - - -] For port 4153adcb-7ca2-4ccd-8243-fdfe06c85cf3, host controller-0, got binding levels [<neutron.plugins.ml2.models.PortBindingLevel[object at 7f579e200e10] {port_id=u'4153adcb-7ca2-4ccd-8243-fdfe06c85cf3', host=u'controller-0', level=0, driver=u'openvswitch', segment_id=u'193596f3-f6ba-427d-8dc2-1c18a070ad5f'}>] get_binding_levels /usr/lib/python2.7/site-packages/neutron/plugins/ml2/db.py:123 server.log-20190715.gz:2019-07-15 01:31:40.253 5223 DEBUG neutron.callbacks.manager [req-6c361634-ef0e-4831-affc-b7e81b79fe83 - - - - -] Notify callbacks [('neutron.db.l3_dvrscheduler_db._notify_l3_agent_port_update-8750894636925', <function _notify_l3_agent_port_update at 0x7f579f75f7d0>), ('neutron.services.trunk.plugin.TrunkPlugin._trigger_trunk_status_change-253764', <bound method TrunkPlugin._trigger_trunk_status_change of <neutron.services.trunk.plugin.TrunkPlugin object at 0x7f579fa01b10>>), ('neutron.db.l3_hascheduler_db._notify_l3_agent_ha_port_update--9223363285960138801', <function _notify_l3_agent_ha_port_update at 0x7f579f75fcf8>), ('neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api.DhcpAgentNotifyAPI._native_event_send_dhcp_notification--9223372036789493157', <bound method DhcpAgentNotifyAPI._native_event_send_dhcp_notification of <neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api.DhcpAgentNotifyAPI object at 0x7f579fd80850>>)] for port, after_update _notify_loop /usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:142 server.log-20190715.gz:2019-07-15 01:31:40.427 5223 DEBUG oslo_messaging._drivers.amqpdriver [req-6c361634-ef0e-4831-affc-b7e81b79fe83 - - - - -] CAST unique_id: 803a42a836824f55a11d48c918557e7e exchange 'neutron' topic 'dhcp_agent.lgpospiuctld1.gso.aexp.com' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568 server.log-20190715.gz:2019-07-15 01:31:40.429 5223 DEBUG oslo_messaging._drivers.amqpdriver [req-6c361634-ef0e-4831-affc-b7e81b79fe83 - - - - -] CAST unique_id: f9445507e918440c972ee2e9cb193a34 exchange 'neutron' topic 'dhcp_agent.lgpospiuctld0.gso.aexp.com' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568 server.log-20190715.gz:2019-07-15 01:31:40.431 5223 DEBUG oslo_messaging._drivers.amqpdriver [req-6c361634-ef0e-4831-affc-b7e81b79fe83 - - - - -] CAST unique_id: ce66b47685644443a4251605f043b431 exchange 'neutron' topic 'dhcp_agent.controller-0' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568 server.log-20190715.gz:2019-07-15 01:31:40.441 5223 DEBUG oslo_messaging._drivers.amqpdriver [req-6c361634-ef0e-4831-affc-b7e81b79fe83 - - - - -] CAST unique_id: 85c943e744bc49f5bb51ed9575bf8cd1 FANOUT topic 'q-agent-notifier-security_group-update' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:557 server.log-20190715.gz:2019-07-15 01:31:40.462 5223 DEBUG oslo_messaging._drivers.amqpdriver [req-6c361634-ef0e-4831-affc-b7e81b79fe83 - - - - -] CAST unique_id: eef6fdc9354b432794d1c426658435f5 FANOUT topic 'q-agent-notifier-port-update' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:557 server.log-20190715.gz:2019-07-15 01:31:40.481 5223 DEBUG oslo_messaging._drivers.amqpdriver [req-6c361634-ef0e-4831-affc-b7e81b79fe83 - - - - -] sending reply msg_id: ad5a0a7781e34a0987eabce42484ca27 reply queue: reply_27f06902ded04698a65bcfce878bbc34 time elapsed: 0.752347345s _send_reply /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:142 $ neutron port-show 4153adcb-7ca2-4ccd-8243-fdfe06c85cf3 +-----------------------+------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+------------------------------------------------------------------------------------+ | admin_state_up | True | | allowed_address_pairs | | | binding:host_id | controller-0 | | binding:profile | {} | | binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} | | binding:vif_type | ovs | | binding:vnic_type | normal | | created_at | 2018-10-02T21:20:38Z | | description | | | device_id | dhcpf8e27c09-cb0b-5a1e-8c1e-34b1de046f68-8c1f75c9-bb82-4bcf-87bf-aae406f77714 | | device_owner | network:dhcp | | extra_dhcp_opts | | | fixed_ips | {"subnet_id": "227578de-baef-45ca-9e7c-71c03042f20f", "ip_address": "10.10.10.10"} | | id | 4153adcb-7ca2-4ccd-8243-fdfe06c85cf3 | | mac_address | 00:00:00:00:00:00:0 | | name | dhcp:port | | network_id | 8c1f75c9-bb82-4bcf-87bf-aae406f77714 | | port_security_enabled | False | | project_id | 38fd5a11820b4dc9a5260218a5033279 | | qos_policy_id | | | revision_number | 1294 | | security_groups | | | status | ACTIVE | | tenant_id | 38fd5a11820b4dc9a5260218a5033279 | | updated_at | 2019-07-24T07:02:17Z | +-----------------------+------------------------------------------------------------------------------------+ Version-Release number of selected component (if applicable): neutron 9.4.1-32 How reproducible: random Steps to Reproduce: 1.unknown 2. 3. Actual results: ip of dhcp port is reverted to an old value Expected results: that shouldn't happen Additional info: