Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2245159

Summary: Error: Failed to create neutron ports for node - Ironic nodes stop mapping neutron segments and fail to clean or deploy with RPN
Product: Red Hat OpenStack Reporter: Chris Janiszewski <cjanisze>
Component: openstack-neutronAssignee: Harald Jensås <hjensas>
Status: CLOSED NEXTRELEASE QA Contact: Eran Kuris <ekuris>
Severity: high Docs Contact:
Priority: high    
Version: 16.2 (Train)CC: chrisw, hjensas, jlibosva, mburns, rhos-maint, sbaker, scohen
Target Milestone: async   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2263310 (view as bug list) Environment:
Last Closed: 2024-06-24 20:01:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2263310    
Bug Blocks:    

Description Chris Janiszewski 2023-10-19 19:33:38 UTC 
Description of problem:
We are seeing an issue where the ironic managed nodes association to network segments disappear resulting in the error - "Failed to prepare node <id> for cleaning: Failed to create neutron ports for node's". 

We have a customer has opened support case and attached all the logs to it -> https://access.redhat.com/support/cases/#/case/03642883

The customer problem has most likely been triggered by the control plane instability and that part of the issue is being taken care of ... but once this problem occurs there is no way to recover back the network segment mapping state back with those manages Ironic nodes. The only way to "fix" it is by removing and re-adding managed baremetal nodes back to the system.

We can see similar behavior in OSP16.2.5 as well as OSP17.0.1.
The neutron plugin is OVN


Version-Release number of selected component (if applicable):
OSP16.2.5


How reproducible:
With Ironic and RPN.
The exact steps to induce the failing behavior are not discovered, although it seems problem might be triggered with some components getting restarted on the control plane

Steps to Reproduce:
1. Create baremetal nodes with RPN network segment mapping
2. Wait long enough (or possibly restart different OSP components on OSP)
3. Attempt to clean or re-deploy the BM node

Actual results:
Error:
Failed to prepare node <id> for cleaning: Failed to create neutron ports for node's


Expected results:
Cleaning complete

Additional info:
Some internal troubleshooting with Red Hat engineering has been performed.

Quoting:
"there is a method, in baremetal_mech, that should be called when an agent is started" "try_to_bind_segment_for_agent"
"and there is an event, that should work by default when "segments" is loaded"
"that is _update_segment_host_mapping_for_agent, that is called in a agent event create and update"

https://github.com/openstack/networking-baremetal/blob/master/networking_baremetal/agent/ironic_neutron_agent.py
https://github.com/openstack/networking-baremetal/blob/8f3fcc073f415913b5db3b6fee51788578578e16/networking_baremetal/plugins/ml2/baremetal_mech.py
Comment 1 Harald Jensås 2023-10-23 16:07:55 UTC 
So the root cause here is that OvnSbSynchronizer is assuming all nodes are running OVN-Controller.

Additionally, the configuration option that is supposed to turn the sync to 'off' or 'log' mode does not honor the configuration option see [3]. 

neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync.OvnSbSynchronizer

When the `sync_hostname_and_physical_networks` method run's it will compare host physical_networks to mappings in neutrons `segmenthostmappings` table. Any mappings in neutron that is not seen on OVN chassis is then treated as "stale" and cleaned up.

This is problematic when there are other plug-ins/agents involved, for example ML2 networking-baremetal[2]. Any segment-host mapping created for baremetal nodes are deleted from the database. And since there is de-duplication[1] on updates from agents - the segment-host mappings are not re-created unless services are re-started or the baremetal node is deleted and re-created in the ironic service.

The OvnSbSynchronizer should not remove mappings unless they belong to OVN hosts.

[1] https://opendev.org/openstack/neutron/commit/176503e610aee16cb5799a77466579bc55129450
[2] https://opendev.org/openstack/networking-baremetal/src/branch/master/networking_baremetal/agent/ironic_neutron_agent.py



[3] https://bugs.launchpad.net/neutron/+bug/2040166
Comment 6 Steve Baker 2024-06-24 20:01:56 UTC 
This is now fixed in the current release of 17.1
Comment 7 Red Hat Bugzilla 2024-10-23 04:25:04 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days