Description of problem: Instance creation fails with the following error if the IP address of the associated port is referred by allowed-address of another port. ~~~ WARNING nova.virt.libvirt.driver [req-XXXX - - - default default] [instance: XXXXXXXXX] Timeout waiting for [('network-vif-plugged', 'XXXXXXXXXX')] for instance with vm_state building and task_state spawning.: eventlet.timeout.Timeout: 300 seconds ~~~ The issue here is that it is type: "virtual" with virtual-parents="d10f9306-21cc-47c0-9dd8-1e2bed6522da". This means the port is associated with "allowed-address" for those parent ports. Its not setup to be directly connected to a VM instance but rather unbound and available for those parent ports. Here is how I can reproduce in my lab: - create port # openstack port create --fixed-ip ip-address=10.1.1.99 --network net1 unbound1 - looking in OVN we see it is a normal port at the moment (note type: ""). This port is good to use directly on a VM. # ovn-nbctl list logical_switch_port e6f4f5e1-7bcb-4146-ac76-513230079a26 _uuid : e9cdcc36-cf8f-42b4-bfdf-ba947b1c0640 addresses : ["fa:16:3e:17:c0:a7 10.1.1.99"] dhcpv4_options : e70330e7-58eb-4403-8c5f-a6ea2bb7f994 dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids : {"neutron:cidrs"="10.1.1.99/24", "neutron:device_id"="", "neutron:device_owner"="", "neutron:network_name"=neutron-89eb5c23-cd96-4e37-b990-90a8ab770548, "neutron:port_name"=unbound1, "neutron:project_id"=a2d00d235246400daba696cc273def31, "neutron:revision_number"="1", "neutron:security_group_ids"="b215d0f5-1692-4fdd-bb5e-002a5d936c37"} ha_chassis_group : [] name : "e6f4f5e1-7bcb-4146-ac76-513230079a26" options : {mcast_flood_reports="true", requested-chassis=""} parent_name : [] port_security : ["fa:16:3e:17:c0:a7 10.1.1.99"] tag : [] tag_request : [] type : "" up : false - create a new port with 10.1.1.99 as an allowed-address-pair # openstack port create --network net1 --fixed-ip ip-address=10.1.1.10 --allowed-address ip-address=10.1.1.99 vm1-port1 - If we look again at the port for 10.1.1.99 in OVN it is now a virtual port with parent port. # ovn-nbctl list logical_switch_port e6f4f5e1-7bcb-4146-ac76-513230079a26 _uuid : e9cdcc36-cf8f-42b4-bfdf-ba947b1c0640 addresses : ["fa:16:3e:17:c0:a7 10.1.1.99"] dhcpv4_options : e70330e7-58eb-4403-8c5f-a6ea2bb7f994 dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids : {"neutron:cidrs"="10.1.1.99/24", "neutron:device_id"="", "neutron:device_owner"="", "neutron:network_name"=neutron-89eb5c23-cd96-4e37-b990-90a8ab770548, "neutron:port_name"=unbound1, "neutron:project_id"=a2d00d235246400daba696cc273def31, "neutron:revision_number"="1", "neutron:security_group_ids"="b215d0f5-1692-4fdd-bb5e-002a5d936c37"} ha_chassis_group : [] name : "e6f4f5e1-7bcb-4146-ac76-513230079a26" options : {mcast_flood_reports="true", requested-chassis="", virtual-ip="10.1.1.99", virtual-parents="d10f9306-21cc-47c0-9dd8-1e2bed6522da"} parent_name : [] port_security : ["fa:16:3e:17:c0:a7 10.1.1.99"] tag : [] tag_request : [] type : virtual up : false - create a VM with the 10.1.1.99 port now hangs and fails. # openstack server create --flavor m1.tiny --image cirros --key-name stack --port e6f4f5e1-7bcb-4146-ac76-513230079a26 test1 # openstack server list +--------------------------------------+-------+--------+----------+--------+---------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------+--------+----------+--------+---------+ | 3cacb7dc-805f-491b-859a-21a13e8c1c6e | test1 | ERROR | | cirros | m1.tiny | +--------------------------------------+-------+--------+----------+--------+---------+ - Once I remove the allowed-address-pair from the parent port I can successfully use the port (you can check the port in OVN as described above to see it no longer a "virtual" port). # openstack port set --no-allowed-address d10f9306-21cc-47c0-9dd8-1e2bed6522da # openstack server delete test1 # openstack server create --flavor m1.tiny --image cirros --key-name stack --port e6f4f5e1-7bcb-4146-ac76-513230079a26 test1 # openstack server list +--------------------------------------+-------+--------+----------------+--------+---------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-------+--------+----------------+--------+---------+ | 52b04c1b-8bc7-46ba-8907-6768dc1f3a8e | test1 | ACTIVE | net1=10.1.1.99 | cirros | m1.tiny | +--------------------------------------+-------+--------+----------------+--------+---------+ Version-Release number of selected component (if applicable): RHOSP 16.2.4. I've not verified yet, but CU says this issue didn't occur in RHOSP 16.2.3 and 16.2.1. So this issue might be a degration. How reproducible: Steps to Reproduce: 1. openstack port create --fixed-ip ip-address=10.1.1.99 --network net1 unbound1 2. openstack port create --network net1 --fixed-ip ip-address=10.1.1.10 --allowed-address ip-address=10.1.1.99 vm1-port1 3.openstack server create --flavor m1.tiny --image cirros --key-name stack --port unbound1 test1 Actual results: Instance creation fails Expected results: Instance creation succeeds
I've tested in my RHOSP 16.2.0 lab and confirmed that this issue doesn't occur on RHOPS 16.2.0. I think this issue only occurs on RHOSP 16.2.4 or later. ~~~ [root@overcloud-controller-1 ~]# cat /etc/rhosp-release Red Hat OpenStack Platform release 16.2.0 GA (Train) ~~~ I created a port unbound1 with 192.168.0.10. Then I created another port unbound2, which refers to the 192.168.0.10 as allowed-address. ~~~ (yatanaka) [stack@undercloud ~]$ openstack port create --fixed-ip ip-address=192.168.0.10 --network yatanaka_network0 unbound1 (yatanaka) [stack@undercloud ~]$ openstack port create --fixed-ip ip-address=192.168.0.11 --allowed-address ip-address=192.168.0.10 --network yatanaka_network0 unbound2 ~~~ At this moment, indeed, the port unbound1 is "virtual". This is the same behavior as 16.2.4. ~~~ [root@overcloud-controller-1 ~]# podman exec ovn-dbs-bundle-podman-1 ovn-nbctl --no-leader-only list Logical_Switch_Port|grep -A 6 192.168.0.10 -B 12 : _uuid : 7621b10f-8990-4c8a-97d6-503da8708344 addresses : ["fa:16:3e:96:e0:e1 192.168.0.10"] dhcpv4_options : e633ba42-c1b0-4853-a3bd-d28595331f54 dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids : {"neutron:cidrs"="192.168.0.10/24", "neutron:device_id"="", "neutron:device_owner"="", "neutron:network_name"=neutron-5de57161-7ba9-4061-8c51-9e0c13b3067a, "neutron:port_name"=unbound1, "neutron:project_id"=d8aa469d1b8d4a95a33e4fb344cc280c, "neutron:revision_number"="1", "neutron:security_group_ids"="6777bb29-532d-405a-8336-712346dd4944"} ha_chassis_group : [] name : "94e9fb6f-87bf-4b6b-ada1-61d09b50f9e1" options : {mcast_flood_reports="true", requested-chassis="", virtual-ip="192.168.0.10", virtual-parents="984adca0-d447-47cf-b124-923b253bb53d"} parent_name : [] port_security : ["fa:16:3e:96:e0:e1 192.168.0.10"] tag : [] tag_request : [] type : virtual <====================(*) up : false ~~~ However, instance creation succeeds in RHOSP 16.2.0 whereas it fails in RHOSP 16.2.4. ~~~ (yatanaka) [stack@undercloud ~]$ source yatanakarc; openstack server create --flavor yatanaka_flavor --image rhel-8.2-290 --port unbound1 --security-group yatanaka_sg --key-name yatanaka_keypair yatanaka1 (yatanaka) [stack@undercloud ~]$ openstack server list +--------------------------------------+-----------+--------+--------------------------------------------+--------------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------+--------+--------------------------------------------+--------------+--------+ | 3ddf0163-d3ca-4444-97bc-b734115aeade | yatanaka1 | BUILD | | rhel-8.2-290 | | +--------------------------------------+-----------+--------+--------------------------------------------+--------------+--------+ (yatanaka) [stack@undercloud ~]$ openstack server list +--------------------------------------+-----------+--------+--------------------------------------------+--------------+--------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------+--------+--------------------------------------------+--------------+--------+ | 3ddf0163-d3ca-4444-97bc-b734115aeade | yatanaka1 | ACTIVE | yatanaka_network0=192.168.0.10 | rhel-8.2-290 | | +--------------------------------------+-----------+--------+--------------------------------------------+--------------+--------+ ~~~ The difference between 16.2.0 and 16.2.4 is that the "virtual" flag is removed when instance creation is attempted in RHOSP 16.2.0. In RHOSP 16.2.4, the port retains "virtual" and instance creation fails. ~~~ [root@overcloud-controller-1 ~]# podman exec ovn-dbs-bundle-podman-1 ovn-nbctl --no-leader-only list Logical_Switch_Port|grep -A 6 192.168.0.10 -B 12 : _uuid : 7621b10f-8990-4c8a-97d6-503da8708344 addresses : ["fa:16:3e:96:e0:e1 192.168.0.10"] dhcpv4_options : e633ba42-c1b0-4853-a3bd-d28595331f54 dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids : {"neutron:cidrs"="192.168.0.10/24", "neutron:device_id"="3ddf0163-d3ca-4444-97bc-b734115aeade", "neutron:device_owner"="compute:AZ-overcloud-novacompute-2.yatanaka.example.com", "neutron:network_name"=neutron-5de57161-7ba9-4061-8c51-9e0c13b3067a, "neutron:port_name"=unbound1, "neutron:project_id"=d8aa469d1b8d4a95a33e4fb344cc280c, "neutron:revision_number"="4", "neutron:security_group_ids"="6777bb29-532d-405a-8336-712346dd4944"} ha_chassis_group : [] name : "94e9fb6f-87bf-4b6b-ada1-61d09b50f9e1" options : {mcast_flood_reports="true", requested-chassis=overcloud-novacompute-2.yatanaka.example.com} parent_name : [] port_security : ["fa:16:3e:96:e0:e1 192.168.0.10"] tag : [] tag_request : [] type : "" <====================(*) up : true ~~~
I believe the difference between RHOSP 16.2.0 and RHOSP 16.2.4 is due to the following change: https://review.opendev.org/c/openstack/neutron/+/842297/1 In RHOSP 16.2.0, if "device_owner" is not empty, "port_type" will be set to "". In RHOSP 16.2.4, even if "device_owner" is not empty, "port_type" will be set to "virtual". However, I'm not sure how we can modify this behavior. I think we should determine whether "virtual" ports should be able to be bound to instances or not. If "virtual" ports should be able to be bound to instances, we need to investigate why instance creation timed out and fix it. If "virtual" ports should not be able to be bound to instances, I believe instance creation should fail with an appropriate error message, rather than timing out.
Hello Yamato: This is the expected behaviour: the port "unbound1" (from your example), is used only to reserve the IP address in the network. Once this IP address (virtual IP address) is assigned to another port as "allowed address", it will be marked as "virtual". As you comment in c#2 (I'm setting this comment as visible), the patch than changes this behaviour is [1] (and the corresponding one in 16.2 attached in the link section). This patch is setting the OVN NB LSP.type to "virtual" for any port that has an IP address that is being used as a virtual IP. **These ports cannot be used as port VMs**. I'll check how to prevent this in ML2/OVN and how to raise an detailed exception in this case. Regards. [1]https://review.opendev.org/c/openstack/neutron/+/842297
Hi, The RHOSP 16.2 "Networking Guide" has been updated. Customers can see these changes here: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/networking_guide/config-allowed-address-pairs_rhosp-network#overview-allow-addr-pairs_config-allowed-address-pairs --Greg
Compose `RHOS-16.2-RHEL-8-20231005.n.3` includes the container with this RPM.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.2.6 (Train) bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:6307
*** Bug 2249793 has been marked as a duplicate of this bug. ***