Bug 1844487
| Summary: | Fail to spawn Spawn Guest Instances With Macvtap(VF) Port Type | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | James Parker <jparker> |
| Component: | openstack-neutron | Assignee: | Rodolfo Alonso <ralonsoh> |
| Status: | CLOSED ERRATA | QA Contact: | Candido Campos <ccamposr> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 16.1 (Train) | CC: | amuller, atragler, bcafarel, beagles, chrisw, eolivare, njohnston, scohen |
| Target Milestone: | z2 | Keywords: | Triaged |
| Target Release: | 16.1 (Train on RHEL 8.2) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-neutron-15.1.1-0.20200623081322.4744f37.el8ost | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-10-28 15:37:32 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1666684 | ||
Verified in FFU 13-16.1 tests. CI test executed and pass: test_create_instance_with_macvtap_port_prov_net HOS-16.1-RHEL-8-20200925.n.1 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:4284 |
Description of problem: Instance fails to spawn when deploying with an SR-IOV instance vnic-type macvtap, problem not seen with direct type. Version-Release number of selected component (if applicable): RHOS-16.1-RHEL-8-20200603.n.0 How reproducible: 100% Steps to Reproduce: 1. Create a network, subnet, and port of vnic-type macvtap 2. Spawn an instance with the associated port 3. Actual results: Instance fails to deploy Expected results: Instance deploys successfully with the port attached Additional info: This is being executed on a standalone deployment. As mentioned earlier direct mode is not having an issue. [stack@sriov01 ~]$ sudo crudini --get /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers sriovnicswitch,ovn [stack@sriov01 ~]$ sudo crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf pci passthrough_whitelist {"devname": "enp6s0f1", "physical_network": "physnet2"} [stack@sriov01 ~]$ sudo crudini --get /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/ml2_conf.ini ml2 mechanism_drivers sriovnicswitch,ovn [stack@sriov01 ~]$ sudo crudini --get /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/ml2_conf.ini ml2_type_vlan network_vlan_ranges datacentre:1:1000,physnet2:2000:2005 [stack@sriov01 ~]$ sudo crudini --get /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf filter_scheduler enabled_filters RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,PciPassthroughFilter [stack@sriov01 ~]$ sudo crudini --get /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/sriov_agent.ini sriov_nic physical_device_mappings physnet2:enp6s0f1 [stack@sriov01 ~]$ sudo crudini --get /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/sriov_agent.ini securitygroup firewall_driver neutron.agent.firewall.NoopFirewallDriver [stack@sriov01 ~]$ openstack network show sriov-net +---------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +---------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2020-06-05T01:13:34Z | | description | | | dns_domain | | | id | 8872eab1-9712-43d8-a50a-a7e56d29f752 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | None | | is_vlan_transparent | None | | location | cloud='standalone', project.domain_id=, project.domain_name='Default', project.id='1c2b8a1af3f747edaa999a0b20e9ef09', project.name='admin', region_name='regionOne', zone= | | mtu | 1500 | | name | sriov-net | | port_security_enabled | True | | project_id | 1c2b8a1af3f747edaa999a0b20e9ef09 | | provider:network_type | vlan | | provider:physical_network | physnet2 | | provider:segmentation_id | 2000 | | qos_policy_id | None | | revision_number | 2 | | router:external | Internal | | segments | None | | shared | False | | status | ACTIVE | | subnets | e28aa732-0c1e-4aec-9105-57a0afa87548 | | tags | | | updated_at | 2020-06-05T01:14:10Z | +---------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [stack@sriov01 ~]$ openstack subnet show subnet1 +-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | allocation_pools | 40.0.0.100-40.0.0.200 | | cidr | 40.0.0.0/24 | | created_at | 2020-06-05T01:14:10Z | | description | | | dns_nameservers | | | enable_dhcp | True | | gateway_ip | 40.0.0.254 | | host_routes | | | id | e28aa732-0c1e-4aec-9105-57a0afa87548 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | location | cloud='standalone', project.domain_id=, project.domain_name='Default', project.id='1c2b8a1af3f747edaa999a0b20e9ef09', project.name='admin', region_name='regionOne', zone= | | name | subnet1 | | network_id | 8872eab1-9712-43d8-a50a-a7e56d29f752 | | prefix_length | None | | project_id | 1c2b8a1af3f747edaa999a0b20e9ef09 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2020-06-05T01:14:10Z | +-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ [stack@sriov01 ~]$ port_id=$(openstack port create --network $net_id --vnic-type direct sriov-port -c id -f value) [stack@sriov01 ~]$ openstack server create --flavor m1.small --image cirros --nic port-id=$port_id vm-sriov --wait +-------------------------------------+----------------------------------------------------------+ | Field | Value | +-------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | sriov01.localdomain | | OS-EXT-SRV-ATTR:hypervisor_hostname | sriov01.localdomain | | OS-EXT-SRV-ATTR:instance_name | instance-00000003 | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2020-06-05T01:41:38.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | sriov-net=40.0.0.180 | | adminPass | iHhtvn9bQYVn | | config_drive | | | created | 2020-06-05T01:41:30Z | | flavor | m1.small (94687f84-a1ce-4813-9c95-8ab604f0326e) | | hostId | e4dceaf294eaddf5deb2612d8243e1bc375bdb7e429f5ff8102e5bd3 | | id | b41317a6-fd8b-4180-9fe3-f05ae4021ecb | | image | cirros (8f78d0ab-6fa9-4970-b199-764c1ac56ab9) | | key_name | None | | name | vm-sriov | | progress | 0 | | project_id | 1c2b8a1af3f747edaa999a0b20e9ef09 | | properties | | | security_groups | name='default' | | status | ACTIVE | | updated | 2020-06-05T01:41:39Z | | user_id | bde7702828684aedb434dcbe4007e539 | | volumes_attached | | +-------------------------------------+----------------------------------------------------------+ [stack@sriov01 ~]$ sudo podman exec -it -u root nova_libvirt /bin/bash ()[root@sriov01 /]# virsh list Id Name State ----------------------------------- 1 instance-00000003 running ()[root@sriov01 /]# virsh dumpxml 1 ... <interface type='hostdev' managed='yes'> <mac address='fa:16:3e:01:ab:d9'/> <driver name='vfio'/> <source> <address type='pci' domain='0x0000' bus='0x06' slot='0x06' function='0x3'/> </source> <vlan> <tag id='2000'/> </vlan> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </interface> ... ()[root@sriov01 /]# exit exit [stack@sriov01 ~]$ openstack server delete vm-sriov [stack@sriov01 ~]$ openstack port delete $port_id # Reattempt with macvtap [stack@sriov01 ~]$ port_id=$(openstack port create --network $net_id --vnic-type macvtap sriov-port -c id -f value) [stack@sriov01 ~]$ openstack server create --flavor m1.small --image cirros --nic port-id=$port_id vm-sriov-macvtap --wait Error creating server: vm-sriov-macvtap Error creating server [stack@sriov01 ~]$ # Neutron logs while instance is attempting to deploy [root@sriov01 containers]# tail -f nova/nova-compute.log nova/nova-scheduler.log neutron/privsep-helper.log neutron/server.log neutron/sriov-nic-agent.log | grep --line-buffered -Eo '(ERROR|WARNING).*$' WARNING oslo.privsep.daemon [-] privsep log: WARNING oslo.privsep.daemon [-] privsep log: We trust you have received the usual lecture from the local System WARNING oslo.privsep.daemon [-] privsep log: Administrator. It usually boils down to these three things: WARNING oslo.privsep.daemon [-] privsep log: WARNING oslo.privsep.daemon [-] privsep log: #1) Respect the privacy of others. WARNING oslo.privsep.daemon [-] privsep log: #2) Think before you type. WARNING oslo.privsep.daemon [-] privsep log: #3) With great power comes great responsibility. WARNING oslo.privsep.daemon [-] privsep log: WARNING oslo.privsep.daemon [-] privsep log: sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent [req-43e680bd-275f-4a99-81b8-520b5028c152 - - - - -] Error in agent loop. Devices info: {}: oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1) ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent Traceback (most recent call last): ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 471, in daemon_loop ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent device_info = self.scan_devices(devices, updated_devices_copy) ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent result = f(*args, **kwargs) ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/sriov_nic_agent.py", line 240, in scan_devices ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent curr_devices = self.eswitch_mgr.get_assigned_devices_info() ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py", line 337, in get_assigned_devices_info ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent for device in embedded_switch.get_assigned_devices_info(): ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py", line 179, in get_assigned_devices_info ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent mac = self.get_pci_device(pci_slot) ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py", line 294, in get_pci_device ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent mac = self._get_macvtap_mac(vf_index) ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/mech_sriov/agent/eswitch_manager.py", line 267, in _get_macvtap_mac ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent upperdev).device(upperdev).link.address ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 484, in address ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent return self.attributes.get('link/ether') ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 517, in attributes ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent self._parent.namespace) ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python3.6/site-packages/neutron/privileged/agent/linux/ip_lib.py", line 73, in sync_inner ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent return input_func(*args, **kwargs) ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 244, in _wrap ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent self.start() ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 255, in start ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent channel = daemon.RootwrapClientChannel(context=self) ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 331, in __init__ ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent raise FailedToDropPrivileges(msg) ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent oslo_privsep.daemon.FailedToDropPrivileges: privsep helper command exited non-zero (1) ERROR neutron.plugins.ml2.drivers.mech_sriov.agent.sriov_nic_agent