Created attachment 1788132 [details] ovs vswitchd log Description of problem: Create ovs dpdk bonding failed on rhel9 when nic driver is ice and ovs versio is 2.15 [root@dell-per740-24 ~]# rpm -qa | grep openv kernel-kernel-networking-openvswitch-common-2.0-127.noarch openvswitch-selinux-extra-policy-1.0-28.el8fdp.noarch openvswitch2.15-2.15.0-16.el9fdp.x86_64 python3-openvswitch2.15-2.15.0-16.el9fdp.x86_64 openvswitch2.15-test-2.15.0-16.el9fdp.noarch kernel-kernel-networking-openvswitch-mem_leak_check-1.0-7.noarch [root@dell-per740-24 ~]# uname -a Linux dell-per740-24.rhts.eng.pek2.redhat.com 5.13.0-0.rc2.19.el9.x86_64 #1 SMP Tue May 18 10:44:29 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux [root@dell-per740-24 ~]# uname -r 5.13.0-0.rc2.19.el9.x86_64 [root@dell-per740-24 ~]# Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: Job link https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2021/05/54142/5414237/10052892/126497948/taskout.log :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: ovs_dpdk_bond_throughput_test single balance-tcp bind dpdk port to openvswitch :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ 03:34:33 ] :: [ BEGIN ] :: Running 'bonding_nic 40:a6:b7:18:ff:28 40:a6:b7:18:ff:29 balance-tcp 1500 4' CURRENT PMD CPU LIST IS 1 3 5 7 25 27 29 31 CURRENT PMD MASK IS 0xaa0000aa ovs-vswitchd PID is: 6801 Setting up watches. Watches established. ovs-vswitchd PID is: 6991 Setting up watches. Watches established. _uuid : 34cc9776-f814-4bc2-9408-0dc5409ed439 bridges : [] cur_cfg : 1 datapath_types : [netdev, system] datapaths : {} db_version : "8.2.0" dpdk_initialized : true dpdk_version : "DPDK 20.11.0" external_ids : {hostname=dell-per740-24.rhts.eng.pek2.redhat.com, rundir="/var/run/openvswitch", system-id="473be08e-7170-4ac0-a84b-1a51de4077c6"} iface_types : [bareudp, dpdk, dpdkvhostuser, dpdkvhostuserclient, erspan, geneve, gre, gtpu, internal, ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan] manager_options : [] next_cfg : 1 other_config : {dpdk-init="true", dpdk-socket-mem="4096,4096", pmd-cpu-mask=" 0xaa0000aa", vhost-iommu-support="true"} ovs_version : "2.15.1" ssl : [] statistics : {} system_type : rhel system_version : "9.0" ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device '0000:af:00.0' to DPDK. See ovs-vswitchd log for details. ovs-vsctl: Error detected while setting up 'dpdk1': Error attaching device '0000:af:00.1' to DPDK. See ovs-vswitchd log for details. ovs-vsctl: The default log directory is "/var/log/openvswitch". Actual results: 2021-05-27T07:34:37.976Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log 2021-05-27T07:34:38.008Z|00002|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 1 2021-05-27T07:34:38.008Z|00003|ovs_numa|INFO|Discovered 24 CPU cores on NUMA node 0 2021-05-27T07:34:38.008Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 48 CPU cores 2021-05-27T07:34:38.009Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2021-05-27T07:34:38.009Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2021-05-27T07:34:38.012Z|00007|dpdk|INFO|DPDK Disabled - Use other_config:dpdk-init to enable 2021-05-27T07:34:38.022Z|00008|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.15.1 2021-05-27T07:34:46.091Z|00009|dpdk|INFO|Using DPDK 20.11.0 2021-05-27T07:34:46.091Z|00010|dpdk|INFO|DPDK Enabled - initializing... 2021-05-27T07:34:46.091Z|00011|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2021-05-27T07:34:46.091Z|00012|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2021-05-27T07:34:46.091Z|00013|dpdk|INFO|POSTCOPY support for vhost-user-client disabled. 2021-05-27T07:34:46.091Z|00014|dpdk|INFO|Per port memory for DPDK devices disabled. 2021-05-27T07:34:46.091Z|00015|dpdk|INFO|EAL ARGS: ovs-vswitchd --socket-mem 1024,1024 --socket-limit 1024,1024 -l 0. 2021-05-27T07:34:46.097Z|00016|dpdk|INFO|EAL: Detected 48 lcore(s) 2021-05-27T07:34:46.097Z|00017|dpdk|INFO|EAL: Detected 2 NUMA nodes 2021-05-27T07:34:46.099Z|00018|dpdk|INFO|EAL: Multi-process socket /var/run/openvswitch/dpdk/rte/mp_socket 2021-05-27T07:34:46.146Z|00019|dpdk|INFO|EAL: rte_mem_virt2phy(): cannot open /proc/self/pagemap: Permission denied 2021-05-27T07:34:46.146Z|00020|dpdk|INFO|EAL: Selected IOVA mode 'VA' 2021-05-27T07:34:46.146Z|00021|dpdk|WARN|EAL: No available hugepages reported in hugepages-2048kB 2021-05-27T07:34:46.147Z|00022|dpdk|INFO|EAL: Probing VFIO support... 2021-05-27T07:34:46.147Z|00023|dpdk|INFO|EAL: VFIO support initialized 2021-05-27T07:34:46.923Z|00024|dpdk|INFO|EAL: using IOMMU type 1 (Type 1) 2021-05-27T07:34:47.243Z|00025|dpdk|INFO|EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:af:00.0 (socket 1) 2021-05-27T07:34:47.319Z|00026|dpdk|ERR|ice_load_pkg(): failed to open file: /lib/firmware/intel/ice/ddp/ice.pkg 2021-05-27T07:34:47.319Z|00027|dpdk|ERR|ice_dev_init(): Failed to load the DDP package,Use safe-mode-support=1 to enter Safe Mode 2021-05-27T07:34:47.320Z|00028|dpdk|INFO|EAL: Releasing pci mapped resource for 0000:af:00.0 2021-05-27T07:34:47.320Z|00029|dpdk|INFO|EAL: Calling pci_unmap_resource for 0000:af:00.0 at 0x2200000000 2021-05-27T07:34:47.320Z|00030|dpdk|INFO|EAL: Calling pci_unmap_resource for 0000:af:00.0 at 0x2202000000 2021-05-27T07:34:47.499Z|00031|dpdk|ERR|EAL: Requested device 0000:af:00.0 cannot be used 2021-05-27T07:34:47.499Z|00032|dpdk|INFO|EAL: using IOMMU type 1 (Type 1) 2021-05-27T07:34:47.783Z|00033|dpdk|INFO|EAL: Probe PCI driver: net_ice (8086:1592) device: 0000:af:00.1 (socket 1) 2021-05-27T07:34:47.832Z|00034|dpdk|ERR|ice_load_pkg(): failed to open file: /lib/firmware/intel/ice/ddp/ice.pkg 2021-05-27T07:34:47.832Z|00035|dpdk|ERR|ice_dev_init(): Failed to load the DDP package,Use safe-mode-support=1 to enter Safe Mode 2021-05-27T07:34:47.832Z|00036|dpdk|INFO|EAL: Releasing pci mapped resource for 0000:af:00.1 2021-05-27T07:34:47.832Z|00037|dpdk|INFO|EAL: Calling pci_unmap_resource for 0000:af:00.1 at 0x2202010000 2021-05-27T07:34:47.832Z|00038|dpdk|INFO|EAL: Calling pci_unmap_resource for 0000:af:00.1 at 0x2204010000 2021-05-27T07:34:48.005Z|00039|dpdk|ERR|EAL: Requested device 0000:af:00.1 cannot be used 2021-05-27T07:34:48.006Z|00040|dpdk|INFO|EAL: No legacy callbacks, legacy socket not created 2021-05-27T07:34:48.007Z|00041|dpdk|INFO|DPDK Enabled - initialized 2021-05-27T07:34:48.008Z|00042|timeval|WARN|Unreasonably long 1917ms poll interval (108ms user, 969ms system) 2021-05-27T07:34:48.008Z|00043|timeval|WARN|faults: 5344 minor, 0 major 2021-05-27T07:34:48.008Z|00044|timeval|WARN|disk: 168 reads, 0 writes 2021-05-27T07:34:48.008Z|00045|timeval|WARN|context switches: 22 voluntary, 159 involuntary 2021-05-27T07:34:48.008Z|00046|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour, hash=2beeb17f: 2021-05-27T07:34:48.008Z|00047|coverage|INFO|netlink_sent 8.8/sec 0.733/sec 0.0122/sec total: 44 2021-05-27T07:34:48.008Z|00048|coverage|INFO|netlink_recv_jumbo 1.2/sec 0.100/sec 0.0017/sec total: 6 2021-05-27T07:34:48.008Z|00049|coverage|INFO|netlink_received 9.2/sec 0.767/sec 0.0128/sec total: 46 2021-05-27T07:34:48.008Z|00050|coverage|INFO|util_xalloc 1684.0/sec 140.333/sec 2.3389/sec total: 8420 2021-05-27T07:34:48.008Z|00051|coverage|INFO|stream_open 0.2/sec 0.017/sec 0.0003/sec total: 1 2021-05-27T07:34:48.008Z|00052|coverage|INFO|pstream_open 0.2/sec 0.017/sec 0.0003/sec total: 1 2021-05-27T07:34:48.008Z|00053|coverage|INFO|seq_change 12.0/sec 1.000/sec 0.0167/sec total: 60 2021-05-27T07:34:48.008Z|00054|coverage|INFO|poll_create_node 19.0/sec 1.583/sec 0.0264/sec total: 95 2021-05-27T07:34:48.008Z|00055|coverage|INFO|txn_success 0.4/sec 0.033/sec 0.0006/sec total: 2 2021-05-27T07:34:48.008Z|00056|coverage|INFO|txn_incomplete 0.8/sec 0.067/sec 0.0011/sec total: 4 2021-05-27T07:34:48.008Z|00057|coverage|INFO|txn_unchanged 0.4/sec 0.033/sec 0.0006/sec total: 2 2021-05-27T07:34:48.008Z|00058|coverage|INFO|hmap_expand 94.8/sec 7.900/sec 0.1317/sec total: 474 2021-05-27T07:34:48.008Z|00059|coverage|INFO|hmap_pathological 0.8/sec 0.067/sec 0.0011/sec total: 4 2021-05-27T07:34:48.008Z|00060|coverage|INFO|miniflow_malloc 2.8/sec 0.233/sec 0.0039/sec total: 14 2021-05-27T07:34:48.008Z|00061|coverage|INFO|cmap_expand 2.4/sec 0.200/sec 0.0033/sec total: 12 2021-05-27T07:34:48.008Z|00062|coverage|INFO|bridge_reconfigure 0.4/sec 0.033/sec 0.0006/sec total: 2 2021-05-27T07:34:48.008Z|00063|coverage|INFO|126 events never hit 2021-05-27T07:34:48.009Z|00064|memory|INFO|49868 kB peak resident set size after 10.0 seconds Expected results: It works fine . Additional info:
This system is probably missing the linux-firmware package. Could you double check?
It means something has changed in the linux-firmware package. Looking at linux-firmware-20210315-119.el9.1.noarch in brew, I can see only xz'd firmware files. https://brewweb.engineering.redhat.com/brew/rpminfo?fileStart=1050&rpmID=9579816&fileOrder=name&buildrootOrder=-id&buildrootStart=0#filelist Can you confirm this is the case? # rpm -ql linux-firmware |grep ice.pkg
I confirmed the linux-firmware pkg content with a RHEL 9 vm I installed. This change in linux-firmware (all firmwares are compressed) is a problem for DPDK as it has no support for loading compressed firmwares. As a temporary workaround, you can do: [root@rhel9 ~]# rm -f /lib/firmware/intel/ice/ddp/ice.pkg.xz [root@rhel9 ~]# xz -d /lib/firmware/intel/ice/ddp/ice-1.3.16.0.pkg.xz [root@rhel9 ~]# ln -s ice-1.3.16.0.pkg /lib/firmware/intel/ice/ddp/ice.pkg [root@rhel9 ~]# ls -lrt /lib/firmware/intel/ice/ddp/ total 648 -rw-r--r-- 1 root root 659716 Apr 16 11:18 ice-1.3.16.0.pkg lrwxrwxrwx 1 root root 16 May 31 14:07 ice.pkg -> ice-1.3.16.0.pkg With this the dpdk ice driver should initialise successfully, and you should be able to run the rest of your checks for this nic with dpdk.
Posted a series upstream: http://patchwork.dpdk.org/project/dpdk/list/?series=17208&state=%2A&archive=both
v2 posted: http://patchwork.dpdk.org/project/dpdk/list/?series=17228&state=%2A&archive=both
*** Bug 1975595 has been marked as a duplicate of this bug. ***
*** Bug 2025392 has been marked as a duplicate of this bug. ***
Handling compressed firmware is now fixed in DPDK 21.11 and later. This version of DPDK is used with OVS 2.17. For now, we won't support RHEL 9 in FDP. This feature won't be backported to previous versions of DPDK (like 20.11 used for ovs 2.15). Closing this bz fixed in NEXTRELEASE (which will be ovs 2.17).
@
@dmarchan we used DPDK 22.11.2 (LTS) version on RHEL 9.1 and still facing the error with ice.pkg while executing test command with dpdk-testpmd and dpdk-l3fwd. Here are the logs. Issue is resolved after replacing ice.pkg.xz file with ice.pkg dpdk-testpmd -c 0xfffe -n 4 -- -i --portmask=0x1 --txq=1 --total-num-mbufs=2048 -cmdline-file=/root/testpmd_cmds.txt > /run_result.txt EAL: Detected CPU lcores: 48 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: VFIO support initialized EAL: Using IOMMU type 1 (Type 1) EAL: Ignore mapping IO port bar(1) EAL: Ignore mapping IO port bar(4) EAL: Probe PCI driver: net_ice (8086:1593) device: 0000:2a:00.2 (socket 0) ice_load_pkg(): ice_copy_and_init_hw failed: -6 ice_dev_init(): Failed to load the DDP package,Use safe-mode-support=1 to enter Safe Mode EAL: Releasing PCI mapped resource for 0000:2a:00.2 EAL: Calling pci_unmap_resource for 0000:2a:00.2 at 0x2101000000 EAL: Calling pci_unmap_resource for 0000:2a:00.2 at 0x2103000000 EAL: Requested device 0000:2a:00.2 cannot be used TELEMETRY: No legacy callbacks, legacy socket not created testpmd: No probed ethernet devices Fail: input txq (1) can't be greater than max_tx_queues (0) of port 0 EAL: Error - exiting with code: 1 Cause: txq 1 invalid - must be >= 0 && <= 0 dpdk-l3fwd -- -p 0x1 --config="(0,0,1)" EAL: Detected CPU lcores: 48 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: VFIO support initialized EAL: Using IOMMU type 1 (Type 1) EAL: Ignore mapping IO port bar(1) EAL: Ignore mapping IO port bar(4) EAL: Probe PCI driver: net_ice (8086:1593) device: 0000:2a:00.2 (socket 0) ice_load_pkg(): ice_copy_and_init_hw failed: -6 ice_dev_init(): Failed to load the DDP package,Use safe-mode-support=1 to enter Safe Mode EAL: Releasing PCI mapped resource for 0000:2a:00.2 EAL: Calling pci_unmap_resource for 0000:2a:00.2 at 0x2101000000 EAL: Calling pci_unmap_resource for 0000:2a:00.2 at 0x2103000000 EAL: Requested device 0000:2a:00.2 cannot be used TELEMETRY: No legacy callbacks, legacy socket not created Neither ACL, LPM, EM, or FIB selected, defaulting to LPM L3FWD: Missing 1 or more rule files, using default instead port 0 is not present on the board EAL: Error - exiting with code: 1 Cause: check_port_config failed
Quickly looking at the code, -6 means that an invalid ddp file was found. I don't think it is related to file compression. Please check which version of linux-firmware you have, and if files have been modified. [root@rhel9 ~]# rpm -Vv linux-firmware | grep intel/ice ......... /usr/lib/firmware/intel/ice ......... /usr/lib/firmware/intel/ice/ddp ......... /usr/lib/firmware/intel/ice/ddp-comms ......... /usr/lib/firmware/intel/ice/ddp-comms/ice_comms-1.3.31.0.pkg.xz ......... /usr/lib/firmware/intel/ice/ddp-wireless_edge ......... /usr/lib/firmware/intel/ice/ddp-wireless_edge/ice_wireless_edge-1.3.7.0.pkg.xz ......... /usr/lib/firmware/intel/ice/ddp/ice-1.3.28.0.pkg.xz ......... /usr/lib/firmware/intel/ice/ddp/ice.pkg.xz
we have ice-1.3.28 package under ddp. We downloaded ice-1.3.28 pkg from different source and copied to ddp folder in uncompressed format. Issue is not seen with that. Can you please try dpdk-testpmd on your setup? error code -6 with below files /usr/lib/firmware/intel/ice/ddp/ice-1.3.28.0.pkg.xz /usr/lib/firmware/intel/ice/ddp/ice.pkg.xz No issue seen with below files /usr/lib/firmware/intel/ice/ddp/ice-1.3.28.0.pkg /usr/lib/firmware/intel/ice/ddp/ice.pkg
I did not mention, but yes, DPDK runs fine on my system. This seems an issue on your system and not related to this bugzilla. Please open a new bz with requested info.