Description of problem: I've got a running environment with overcloud ironic service working OK but when I try to re-deploy same templates but including ovn dvr ha suddenly baremetal nodes don't get ip (while "provide" step). Everything else works fine, even starting an regular nova instance (it gets the DHCP IP) in same baremetal flat network that I use for cleaning and provisioning. Since OVN uses it's own way to provide DHCP there should be something missed there. How reproducible: Steps to Reproduce: 1. Deploy overcloud ironic service + OVN (DVR HA in my case) 2. Try to enroll the baremetal node and move to "available" state 3. Check that when baremetal boots up it does not get any DHCP IP Actual results: Baremetal nodes are not getting IP or DHCP options Expected results: If that's something that still does not work, document it and possible workarounds ("NeutronEnableDHCPAgent: true" ?) Baremetal nodes get IP or DHCP options Additional info: Below I include some CLI outputs, I don see anything special, only a "binding_vif_type | binding_failed" that I think is "normal" when using flat networks with ironic [1].... or maybe I'm wrong and that's the key Taking a look to logs I see the above binding "error" /var/log/containers/neutron/server.log:2018-08-23 09:38:15.727 78 ERROR neutron.plugins.ml2.managers [req-f5fcf0d0-b9dd-4ff6-9504-63d7afbc0ec0 c61dd4f8130b47a092c1c9083ecbcfbb b5ceb706a1244277b97346110f8dbcb3 - default default] Failed to bind port d2aa6791-d369-41f7-9285-f2c0e0c64c14 on host 7c550978-59a8-4d89-886b-38947dfaac88 for vnic_type baremetal using segments [{'network_id': '5735672d-cf0f-4acc-988f-2223b0dde130', 'segmentation_id': None, 'physical_network': u'baremetal', 'id': '7eed297b-c207-4dcb-8abc-949e965ba66d', 'network_type': u'flat'}] [1] https://docs.openstack.org/ironic/queens/install/configure-networking.html --------------------------------- (rhosp) [stack@undercloud-vm ~]$ openstack baremetal node provide --wait 0 E-1 Waiting for provision state available on node E-1 +--------------------------------------+------+---------------+-------------+--------------------+-------------+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | +--------------------------------------+------+---------------+-------------+--------------------+-------------+ | 60eb3300-01ba-46a2-a58c-9efc453cf3bb | E-1 | None | power on | clean wait | False | | 7c550978-59a8-4d89-886b-38947dfaac88 | E-2 | None | power on | manageable | False | +--------------------------------------+------+---------------+-------------+--------------------+-------------+ [root@controller-0 ~]# tcpdump -nnei br-baremetal "not port 6385" tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on br-baremetal, link-type EN10MB (Ethernet), capture size 262144 bytes 09:24:51.142667 52:54:00:19:10:be > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 438: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:19:10:be, length 396 09:24:59.039742 52:54:00:19:10:be > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 438: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:19:10:be, length 396 09:25:14.859506 52:54:00:19:10:be > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 438: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:19:10:be, length 396 (rhosp) [stack@undercloud-vm ~]$ openstack baremetal port show aa024b42-73ce-42a4-bc09-d989fcd8babd +-----------------------+--------------------------------------------------------------------+ | Field | Value | +-----------------------+--------------------------------------------------------------------+ | address | 52:54:00:19:10:be | | created_at | 2018-08-23T09:11:26+00:00 | | extra | {} | | internal_info | {u'cleaning_vif_port_id': u'717cbd07-dca2-4e30-833a-6ca9695c3db3'} | | local_link_connection | {} | | node_uuid | 60eb3300-01ba-46a2-a58c-9efc453cf3bb | | physical_network | None | | portgroup_uuid | None | | pxe_enabled | True | | updated_at | 2018-08-23T09:24:09+00:00 | | uuid | aa024b42-73ce-42a4-bc09-d989fcd8babd | +-----------------------+--------------------------------------------------------------------+ (rhosp) [stack@undercloud-vm ~]$ openstack port list | grep 52:54:00:19:10:be | 717cbd07-dca2-4e30-833a-6ca9695c3db3 | | 52:54:00:19:10:be | ip_address='172.30.0.30', subnet_id='b856375a-b259-4c54-a717-f5a6c3a51a08' | DOWN | (rhosp) [stack@undercloud-vm ~]$ openstack port show 717cbd07-dca2-4e30-833a-6ca9695c3db3 +-----------------------+---------------------------------------------------------------------------------------+ | Field | Value | +-----------------------+---------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | 60eb3300-01ba-46a2-a58c-9efc453cf3bb | | binding_profile | local_link_information='[{}]' | | binding_vif_details | | | binding_vif_type | binding_failed | | binding_vnic_type | baremetal | | created_at | 2018-08-23T09:24:02Z | | data_plane_status | None | | description | | | device_id | 60eb3300-01ba-46a2-a58c-9efc453cf3bb | | device_owner | baremetal:none | | dns_assignment | None | | dns_name | None | | extra_dhcp_opts | ip_version='4', opt_name='server-ip-address', opt_value='172.30.0.13' | | | ip_version='4', opt_name='66', opt_value='172.30.0.13' | | | ip_version='4', opt_name='150', opt_value='172.30.0.13' | | | ip_version='4', opt_name='tag:ipxe,67', opt_value='http://172.30.0.13:8088/boot.ipxe' | | | ip_version='4', opt_name='tag:!ipxe,67', opt_value='undionly.kpxe' | | fixed_ips | ip_address='172.30.0.30', subnet_id='b856375a-b259-4c54-a717-f5a6c3a51a08' | | id | 717cbd07-dca2-4e30-833a-6ca9695c3db3 | | ip_address | None | | mac_address | 52:54:00:19:10:be | | name | | | network_id | 5735672d-cf0f-4acc-988f-2223b0dde130 | | option_name | None | | option_value | None | | port_security_enabled | True | | project_id | b5ceb706a1244277b97346110f8dbcb3 | | qos_policy_id | None | | revision_number | 14 | | security_group_ids | 5e3ff398-1edf-4ca8-aa26-c6becf4fa36d | | status | DOWN | | subnet_id | None | | tags | | | trunk_details | None | | updated_at | 2018-08-23T09:24:14Z | +-----------------------+---------------------------------------------------------------------------------------+ (rhosp) [stack@undercloud-vm ~]$
Hi Luis, Thanks for testing this. > I've got a running environment with overcloud ironic service working OK but > when I try to re-deploy same templates but including ovn dvr ha suddenly > baremetal nodes don't get ip (while "provide" step). Everything else works > fine, even starting an regular nova instance (it gets the DHCP IP) in same > baremetal flat network that I use for cleaning and provisioning. > > Since OVN uses it's own way to provide DHCP there should be something missed > there. > That's right, OVN's built-in DHCP can not be used for baremetal provisioning at the moment. I've tried to capture what is missing from te OVN side at: https://bugzilla.redhat.com/show_bug.cgi?id=1622154 > > Expected results: > > If that's something that still does not work, document it and possible > workarounds ("NeutronEnableDHCPAgent: true" ?) > > Baremetal nodes get IP or DHCP options Indeed, we should document that right now it's not possible to use OVN for the baremetal-to-tenant use case (or undercloud). I will keep this bug opened to track that. As to the workaround to use Neutron DHCP server, this might work (and has been suggested at 1622154 as well) but I'm not aware of anyone that has tried this yet. Cheers, Lucas
(In reply to Lucas Alvares Gomes from comment #1) > Hi Luis, > > Thanks for testing this. > > > I've got a running environment with overcloud ironic service working OK but > > when I try to re-deploy same templates but including ovn dvr ha suddenly > > baremetal nodes don't get ip (while "provide" step). Everything else works > > fine, even starting an regular nova instance (it gets the DHCP IP) in same > > baremetal flat network that I use for cleaning and provisioning. > > > > Since OVN uses it's own way to provide DHCP there should be something missed > > there. > > > > That's right, OVN's built-in DHCP can not be used for baremetal provisioning > at the moment. I've tried to capture what is missing from te OVN side at: > https://bugzilla.redhat.com/show_bug.cgi?id=1622154 > > > > > Expected results: > > > > If that's something that still does not work, document it and possible > > workarounds ("NeutronEnableDHCPAgent: true" ?) > > > > Baremetal nodes get IP or DHCP options > > Indeed, we should document that right now it's not possible to use OVN for > the baremetal-to-tenant use case (or undercloud). I will keep this bug > opened to track that. > > As to the workaround to use Neutron DHCP server, this might work (and has > been suggested at 1622154 as well) but I'm not aware of anyone that has > tried this yet. > > Cheers, > Lucas I deployed OVN but with Neutron DHCP + Baremetal and it's working for me. This is the environment that I've included to do it (changes from default is just commenting line that should disable neutron dhcp and change NeutronEnableDHCPAgent to "true"): # A Heat environment that can be used to deploy OVN services with non HA OVN DB servers. resource_registry: OS::TripleO::Docker::NeutronMl2PluginBase: /usr/share/openstack-tripleo-heat-templates/puppet/services/neutron-plugin-ml2-ovn.yaml OS::TripleO::Services::OVNController: /usr/share/openstack-tripleo-heat-templates/docker/services/ovn-controller.yaml OS::TripleO::Services::OVNDBs: /usr/share/openstack-tripleo-heat-templates/docker/services/pacemaker/ovn-dbs.yaml OS::TripleO::Services::OVNMetadataAgent: /usr/share/openstack-tripleo-heat-templates/docker/services/ovn-metadata.yaml # Disabling Neutron services that overlap with OVN OS::TripleO::Services::NeutronOvsAgent: OS::Heat::None OS::TripleO::Services::ComputeNeutronOvsAgent: OS::Heat::None OS::TripleO::Services::NeutronL3Agent: OS::Heat::None OS::TripleO::Services::NeutronMetadataAgent: OS::Heat::None #OS::TripleO::Services::NeutronDhcpAgent: OS::Heat::None OS::TripleO::Services::ComputeNeutronCorePlugin: OS::Heat::None parameter_defaults: NeutronMechanismDrivers: ovn OVNVifType: ovs OVNNeutronSyncMode: log OVNQosDriver: ovn-qos OVNTunnelEncapType: geneve NeutronEnableDHCPAgent: true NeutronTypeDrivers: 'geneve,vlan,flat' NeutronNetworkType: 'geneve' NeutronServicePlugins: 'qos,ovn-router,trunk' NeutronVniRanges: ['1:65536', ] NeutronEnableDVR: true
Further test shows that baremetal is getting the IP and enroll is complete, but once you deploy the user image metadata is not working with that config. That's on the OVN side since regular nova instances don't get metadata either.
Hi Luis, (In reply to Luis Arizmendi from comment #3) > Further test shows that baremetal is getting the IP and enroll is complete, > but once you deploy the user image metadata is not working with that config. > That's on the OVN side since regular nova instances don't get metadata > either. Thanks for trying this out! It's progress... May I suggest another test ? Currently the metadata agent relies on the DHCP ports created by Neutron but it expects only one port to exist (in the OVN's built-in DHCP model). See [0]. I believe that with Neutron DHCP Agent, it will spawn more than one DHCP agent instance per network by default resulting in multiple ports being created and that's possibly breaking the metadata model in networking-ovn. As a suggestion, can you please set the "NeutronDhcpAgentsPerNetwork" option in your environment file to 1 and see if that works? Appreciate that you're trying that stuff out! I will work on getting it documented and tested soon. [0] https://github.com/openstack/networking-ovn/blob/6de5e20ed6ed55e677a85ee6dd75341c394d446c/networking_ovn/common/ovn_client.py#L1754-L1762 Cheers, Lucas
(In reply to Lucas Alvares Gomes from comment #4) > Hi Luis, > > (In reply to Luis Arizmendi from comment #3) > > Further test shows that baremetal is getting the IP and enroll is complete, > > but once you deploy the user image metadata is not working with that config. > > That's on the OVN side since regular nova instances don't get metadata > > either. > > Thanks for trying this out! > > It's progress... May I suggest another test ? > > Currently the metadata agent relies on the DHCP ports created by Neutron but > it expects only one port to exist (in the OVN's built-in DHCP model). See > [0]. > > I believe that with Neutron DHCP Agent, it will spawn more than one DHCP > agent instance per network by default resulting in multiple ports being > created and that's possibly breaking the metadata model in networking-ovn. > > As a suggestion, can you please set the "NeutronDhcpAgentsPerNetwork" option > in your environment file to 1 and see if that works? > > Appreciate that you're trying that stuff out! I will work on getting it > documented and tested soon. > > [0] > https://github.com/openstack/networking-ovn/blob/ > 6de5e20ed6ed55e677a85ee6dd75341c394d446c/networking_ovn/common/ovn_client. > py#L1754-L1762 > > Cheers, > Lucas Thanks to you Lucas! I deployed with NeutronDhcpAgentsPerNetwork:1 and metadata is still not working. I think the problem is that I also deployed Octavia, and as part of Octavia you need to include NeutronEnableForceMetadata: true, and it seems that route in VM is not pointing to OVN metadata, it is pointing to regular neutron metadata, that I disabled as part of OVN configuration (OS::TripleO::Services::NeutronMetadataAgent: OS::Heat::None). I might need to disable it in OVN and enable on neutron, what do you think? Cheers <Check from VM> $ hostname cirros $ $ $ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc pfifo_fast qlen 1000 link/ether fa:16:3e:cd:c8:d3 brd ff:ff:ff:ff:ff:ff inet 172.19.1.7/24 brd 172.19.1.255 scope global eth0 inet6 fe80::f816:3eff:fecd:c8d3/64 scope link valid_lft forever preferred_lft forever $ curl http://169.254.169.254 curl: (7) couldn't connect to host $ ip r default via 172.19.1.1 dev eth0 169.254.169.254 via 172.19.1.2 dev eth0 172.19.1.0/24 dev eth0 src 172.19.1.7 <Check from compute node> [root@compute-2 ~]# ip netns ovnmeta-6d967fa2-0890-4d88-af76-2c49e08cbb69 (id: 0) [root@compute-2 ~]# ip netns exec ovnmeta-6d967fa2-0890-4d88-af76-2c49e08cbb69 ping 172.19.1.7 PING 172.19.1.7 (172.19.1.7) 56(84) bytes of data. 64 bytes from 172.19.1.7: icmp_seq=1 ttl=64 time=2.88 ms 64 bytes from 172.19.1.7: icmp_seq=2 ttl=64 time=1.37 ms 64 bytes from 172.19.1.7: icmp_seq=3 ttl=64 time=0.465 ms ^C --- 172.19.1.7 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 0.465/1.573/2.881/0.996 ms [root@compute-2 ~]# ip netns exec ovnmeta-6d967fa2-0890-4d88-af76-2c49e08cbb69 ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tap6d967fa2-01@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:21:b2:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.19.1.3/24 brd 172.19.1.255 scope global tap6d967fa2-01 valid_lft forever preferred_lft forever inet 169.254.169.254/16 brd 169.254.255.255 scope global tap6d967fa2-01 valid_lft forever preferred_lft forever <Review of ports> (rhosp) [stack@undercloud-vm whole]$ openstack port list | grep 172.19 | 58892b69-3369-499b-9cce-a49531fa16f7 | | fa:16:3e:69:04:c7 | ip_address='172.19.1.2', subnet_id='a5084ac6-7438-4b61-a9b6-0f6e4368ae6b' | DOWN | | bf79c902-25b2-4cc9-9192-a66a9d6fdbb2 | | fa:16:3e:d2:2d:fa | ip_address='172.19.1.1', subnet_id='a5084ac6-7438-4b61-a9b6-0f6e4368ae6b' | DOWN | | d66519de-4261-4f4a-a9d6-21bf7078269c | | fa:16:3e:21:b2:a5 | ip_address='172.19.1.3', subnet_id='a5084ac6-7438-4b61-a9b6-0f6e4368ae6b' | DOWN | | e0f01969-002e-472a-bb37-2f38e010b864 | | fa:16:3e:cd:c8:d3 | ip_address='172.19.1.7', subnet_id='a5084ac6-7438-4b61-a9b6-0f6e4368ae6b' | ACTIVE | (rhosp) [stack@undercloud-vm whole]$ openstack port show 58892b69-3369-499b-9cce-a49531fa16f7 +-----------------------+---------------------------------------------------------------------------+ | Field | Value | +-----------------------+---------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | normal | | created_at | 2018-08-28T18:39:31Z | | data_plane_status | None | | description | | | device_id | | | device_owner | network:dhcp | | dns_assignment | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='172.19.1.2', subnet_id='a5084ac6-7438-4b61-a9b6-0f6e4368ae6b' | | id | 58892b69-3369-499b-9cce-a49531fa16f7 | | ip_address | None | | mac_address | fa:16:3e:69:04:c7 | | name | | | network_id | fde7274b-1928-48cf-abb7-167719516be8 | | option_name | None | | option_value | None | | port_security_enabled | False | | project_id | 7a2d0ee020f9486c8c332f0c12f322f7 | | qos_policy_id | None | | revision_number | 7 | | security_group_ids | | | status | DOWN | | subnet_id | None | | tags | | | trunk_details | None | | updated_at | 2018-08-28T18:39:39Z | +-----------------------+---------------------------------------------------------------------------+ (rhosp) [stack@undercloud-vm whole]$ (rhosp) [stack@undercloud-vm whole]$ openstack port show d66519de-4261-4f4a-a9d6-21bf7078269c +-----------------------+-------------------------------------------------------------------------------+ | Field | Value | +-----------------------+-------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | controller-2.localdomain | | binding_profile | | | binding_vif_details | port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2018-08-28T18:39:39Z | | data_plane_status | None | | description | | | device_id | dhcpfa77521c-1630-5a1f-973c-c5c02c6d7e8d-fde7274b-1928-48cf-abb7-167719516be8 | | device_owner | network:dhcp | | dns_assignment | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='172.19.1.3', subnet_id='a5084ac6-7438-4b61-a9b6-0f6e4368ae6b' | | id | d66519de-4261-4f4a-a9d6-21bf7078269c | | ip_address | None | | mac_address | fa:16:3e:21:b2:a5 | | name | | | network_id | fde7274b-1928-48cf-abb7-167719516be8 | | option_name | None | | option_value | None | | port_security_enabled | False | | project_id | 7a2d0ee020f9486c8c332f0c12f322f7 | | qos_policy_id | None | | revision_number | 6 | | security_group_ids | | | status | DOWN | | subnet_id | None | | tags | | | trunk_details | None | | updated_at | 2018-08-28T18:39:42Z | +-----------------------+-------------------------------------------------------------------------------+
Problem is that there's two DHCP ports (one created by Neutron DHCP agent (with device id) and the other one created by OVN Metadata agent). We need to adapt networking-ovn to work with multiple DHCP ports. I think that it'll suffice with fetching DHCP owner ports without device_id populated for serving metadata in the OVN domain.
I have a question about ovn-metadata. ovn-metadata is deployed on compute nodes, and it can provide metadata service to VMs in the chassis. This function is ok. Can I use the ovn-metadata for ironic to get metadata? There is no compute host in ironic,so where should i deploy the ovn-metadata? Would it be possible to help me?
I was going through old issues and found this one. There haven't been any new comments or updates since December of 2019. Is this issue still relevant? Should it remain open? Thanks!
No answers to my question in nearly two months. Closing.