Description of problem:


RHEL 8.4 setup

BONDING
  Device    Mode               ifcfg-File BONDING_OPTS                                                  Partner MAC Addr    Slaves (*=active; [n]=AggID)
  ========  =================  ========================                                                 ==================  ===============================
  bond0     4 (802.3ad)        mode=802.3ad lacp_rate=0 updelay=0 miimon=100 ad_select=0 downdelay=200  00:23:04:ee:c0:67   * [1] ens1f0 (aa:bb:cc:dd:ee:72)
                                                                                                                            * [1] ens4f0 (aa:bb:cc:dd:ee:82)
                                                                                                                            - - - - - - - - - - - - - - - -
  bond1     4 (802.3ad)        mode=802.3ad lacp_rate=0 updelay=0 miimon=100 ad_select=0 downdelay=200  00:23:04:ee:c0:69   * [1] ens1f1 (aa:bb:cc:dd:ee:73)
                                                                                                                            * [1] ens4f1 (aa:bb:cc:dd:ee:83)


generated udev rules
/etc/udev/rules.d/70-rhosp-persistent-net.rules 

SUBSYSTEM=="net",ACTION=="add",DRIVERS=="?*",NAME="eno1" ,ATTR{address}=="ab:ba:ab:ba:ab:16"
SUBSYSTEM=="net",ACTION=="add",DRIVERS=="?*",NAME="eno2" ,ATTR{address}=="ab:ba:ab:ba:ab:17"
SUBSYSTEM=="net",ACTION=="add",DRIVERS=="?*",NAME="eno3" ,ATTR{address}=="ab:ba:ab:ba:ab:18"
SUBSYSTEM=="net",ACTION=="add",DRIVERS=="?*",NAME="eno4" ,ATTR{address}=="ab:ba:ab:ba:ab:19"
SUBSYSTEM=="net",ACTION=="add",DRIVERS=="?*",NAME="ens1f0" ,ATTR{address}=="88:e9:a4:5b:02:72"
SUBSYSTEM=="net",ACTION=="add",DRIVERS=="?*",NAME="ens1f1" ,ATTR{address}=="88:e9:a4:5b:02:73"
SUBSYSTEM=="net",ACTION=="add",DRIVERS=="?*",NAME="ens4f0" ,ATTR{address}=="88:e9:a4:5b:02:72"
SUBSYSTEM=="net",ACTION=="add",DRIVERS=="?*",NAME="ens4f1" ,ATTR{address}=="88:e9:a4:5b:02:73"




6: ens1f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
    link/ether aa:bb:cc:dd:ee:72 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9978 
    bond_slave state ACTIVE mii_status UP link_failure_count 0 perm_hwaddr aa:bb:cc:dd:ee:72 queue_id 0 ad_aggregator_id 1 ad_actor_oper_port_state 61 ad_actor_oper_port_state_str <active,aggregating,in_sync,collecting,distributing> ad_partner_oper_port_state 61 ad_partner_oper_port_state_str <active,aggregating,in_sync,collecting,distributing> numtxqueues 504 numrxqueues 126 gso_max_size 65536 gso_max_segs 65535 
7: ens1f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond1 state UP group default qlen 1000
    link/ether aa:bb:cc:dd:ee:73 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9978 
    bond_slave state ACTIVE mii_status UP link_failure_count 0 perm_hwaddr aa:bb:cc:dd:ee:73 queue_id 0 ad_aggregator_id 1 ad_actor_oper_port_state 61 ad_actor_oper_port_state_str <active,aggregating,in_sync,collecting,distributing> ad_partner_oper_port_state 61 ad_partner_oper_port_state_str <active,aggregating,in_sync,collecting,distributing> numtxqueues 504 numrxqueues 126 gso_max_size 65536 gso_max_segs 65535 
8: ens4f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond0 state UP group default qlen 1000
    link/ether aa:bb:cc:dd:ee:72 brd ff:ff:ff:ff:ff:ff permaddr aa:bb:cc:dd:ee:82 promiscuity 1 minmtu 68 maxmtu 9978 
    bond_slave state ACTIVE mii_status UP link_failure_count 0 perm_hwaddr aa:bb:cc:dd:ee:82 queue_id 0 ad_aggregator_id 1 ad_actor_oper_port_state 61 ad_actor_oper_port_state_str <active,aggregating,in_sync,collecting,distributing> ad_partner_oper_port_state 61 ad_partner_oper_port_state_str <active,aggregating,in_sync,collecting,distributing> numtxqueues 504 numrxqueues 126 gso_max_size 65536 gso_max_segs 65535 
9: ens4f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond1 state UP group default qlen 1000
    link/ether aa:bb:cc:dd:ee:73 brd ff:ff:ff:ff:ff:ff permaddr aa:bb:cc:dd:ee:83 promiscuity 1 minmtu 68 maxmtu 9978 
    bond_slave state ACTIVE mii_status UP link_failure_count 0 perm_hwaddr aa:bb:cc:dd:ee:83 queue_id 0 ad_aggregator_id 1 ad_actor_oper_port_state 61 ad_actor_oper_port_state_str <active,aggregating,in_sync,collecting,distributing> ad_partner_oper_port_state 61 ad_partner_oper_port_state_str <active,aggregating,in_sync,collecting,distributing> numtxqueues 504 numrxqueues 126 gso_max_size 65536 gso_max_segs 65535 



Version-Release number of selected component (if applicable):
RHOSP 17.1
openstack-tripleo-heat-templates.noarch         14.3.1-17.1.20231103010826.el9ost

How reproducible:
- preserve nic naming (NICsPrefixesToUdev: ['en'] under system_upgrade.yaml) is enabled
- upgrade the os from RHEL 8.4 to RHEL 9.4 for hosts using bonds
$ openstack overcloud upgrade run --yes --tags system_upgrade --stack <stack> --limit compute-0,compute-1,compute-2


Actual results:
generated /etc/udev/rules.d/70-rhosp-persistent-net.rules is not consistent, it looks like perm_hwaddr is ignored
 

Additional info:
probably this patch is missing: https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-heat-templates/+/448594